Imaging device, processing method for imaging device, and program

Abstract

This system enables color determination of objects in an image using methods tailored to the objects displayed within the image. [Solution] The imaging device comprises: a first image sensor having a photoelectric conversion unit capable of receiving non-visible light; a second image sensor having a photoelectric conversion unit capable of receiving visible light; a recognition processing unit that performs object recognition processing using a first image generated by the first image sensor; a color determination region selection unit that selects a region for performing color determination processing based on an object detected by the recognition processing unit; a color determination selection unit that selects a color determination process to be performed based on the object detected by the recognition processing unit; and a color determination unit that performs the color determination process selected by the color determination selection unit on the region selected by the color determination region selection unit, using a second image generated by the second image sensor.

Description

Technical Field

[0001] The present invention relates to an imaging device, a method for processing an imaging device, and a program.

Background Art

[0002] Conventionally, there is an RGB camera that performs color determination of an object displayed in a captured image. The result of color determination by the RGB camera is used for control of an operable device, such as control of automatic driving of a vehicle. In addition, in order to improve the accuracy of color determination of an object in an image captured in a state of poor visibility due to night or bad weather, an RGB-IR camera that simultaneously captures both RGB, which is visible light, and IR, which is non-visible light, is also known. Patent Document 1 discloses that a color image and a monochrome (IR) image are simultaneously acquired using a color imaging element and a monochrome imaging element, a vehicle body region is calculated from a license plate position recognized using the IR image, and the color of the vehicle body is determined using the color image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, according to the technique described in Patent Document 1, since color determination of an object is performed by a single color determination method regardless of the object to be determined, color determination of an object by a method according to the object displayed in the image cannot be performed. An object of the present invention is to realize color determination of an object by a method according to the object displayed in the image.

Means for Solving the Problems

[0005] To solve the above problems, the present invention provides an imaging device comprising: a first image sensor having a photoelectric conversion unit capable of receiving non-visible light; a second image sensor having a photoelectric conversion unit capable of receiving visible light; a recognition processing unit that performs object recognition processing using a first image generated by the first image sensor; a color determination region selection unit that selects a region for performing color determination processing based on an object detected by the recognition processing unit; a color determination selection unit that selects a color determination process to be performed based on the object detected by the recognition processing unit; and a color determination unit that performs the color determination process selected by the color determination selection unit on the region selected by the color determination region selection unit, using a second image generated by the second image sensor. [Effects of the Invention]

[0006] According to the present invention, it is possible to determine the color of an object using a method that corresponds to the object displayed in the image. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the configuration of a photoelectric conversion element. [Figure 2] This figure shows an example of the configuration of a sensor board. [Figure 3] This figure shows an example of the configuration of a color filter in a pixel. [Figure 4] This is a diagram showing an example of a circuit board configuration. [Figure 5] This diagram shows the equivalent circuit between a pixel and a signal processing circuit. [Figure 6] This diagram shows the relationship between APD operation and output signals. [Figure 7] This diagram shows the functional configuration of the IR emitter, camera, and mobile unit. [Figure 8] This figure shows an object management table. [Figure 9] (A) through (D) are diagrams showing specific examples of the object. [Figure 10]This is a flowchart illustrating the flow of the result output processing. [Figure 11] This is a diagram showing a relationship management table. [Figure 12] This is a flowchart illustrating the flow of motion processing for a moving object. [Figure 13] Figures (A) through (D) show examples of the movement of the moving object. [Figure 14] Figures (A) through (D) show examples of the movement of the moving object. [Modes for carrying out the invention]

[0008] In this embodiment, the imaging device captures images of objects and determines the color of the objects displayed in the captured images. More specifically, the imaging device in this embodiment detects objects displayed in the image and performs color determination using a method appropriate to the detected objects. In addition, in this embodiment, a mobile body on which the imaging device is mounted operates according to the result of the color determination. Examples of mobile bodies include automobiles, aircraft, trains, ships, drones, AGVs, robots, and other movable objects. Embodiments of the present invention will be described below with reference to the drawings. Figure 1 shows an example of the configuration of a photoelectric conversion element 100 in a camera. The photoelectric conversion element 100 has a sensor substrate 11 and a circuit board 21 stacked on top of each other. The sensor substrate 11 and the circuit board 21 are electrically connected.

[0009] The sensor substrate 11 has a pixel region 12. The circuit board 21 has a circuit region 22 that processes signals detected in the pixel region 12. In the example shown in Figure 1, the case where the photoelectric conversion element 100 is a so-called stacked structure photoelectric conversion device was described, but the invention is not limited to this. The photoelectric conversion element 100 may also have a so-called non-stacked structure in which the structural parts of the sensor substrate 11 and the structural parts of the circuit board 21 are included in a common semiconductor layer.

[0010] FIG. 2 is a diagram showing a configuration example of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 has pixels 101. In the pixel region 12, a plurality of pixels 101 are arranged two-dimensionally. The pixel 101 includes a photoelectric conversion unit 102 including an avalanche photodiode (hereinafter, APD). The photoelectric conversion unit 102 functions as a sensor unit that emits pulses at a frequency corresponding to the light reception frequency of photons. Note that the number of rows and columns in which the pixels 101 are arranged in the pixel region 12 may be any number.

[0011] FIG. 3 is a diagram showing a configuration example of the color filter 30 included in the pixel 101. The color filter 30 includes an RGB filter 31 that transmits visible light and an IR filter 32 that transmits non-visible light. The RGB filter 31 includes an R filter 31R that transmits light having a red wavelength, a B filter 31B that transmits light having a blue wavelength, and a G filter 31G that transmits light having a green wavelength. The IR filter 32 is a filter in the infrared region that transmits light having a wavelength of infrared light (IR).

[0012] In the pixel region 12, the color filter 30 included in each pixel 101 is constituted by any one of an R filter 31R, a B filter 31B, a G filter 31G, and an IR filter 32. In the present embodiment, as shown in FIG. 3, in the pixel region 12, a column in which the B filter 31B and the G filter 31G are alternately arranged and a column in which the IR filter 32 and the R filter 31R are alternately arranged are alternately arranged. However, the arrangement of the RGB filter 31 and the IR filter 32 in the pixel region 12 may be different from the illustrated example. As described above, the photoelectric conversion unit 102 of the present embodiment receives non-visible light and also receives visible light. Therefore, the photoelectric conversion element 100 can be regarded as a first imaging element having a photoelectric conversion unit 102 capable of receiving non-visible light. Further, the photoelectric conversion element 100 can also be regarded as a second imaging element having a photoelectric conversion unit 102 capable of receiving visible light.

[0013] Figure 4 shows an example of the configuration of the circuit board 21. The circuit board 21 includes a signal processing circuit 103 for processing the charge photoelectrically converted by the photoelectric conversion unit 102, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114. The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and sequentially supplies control pulses to a plurality of pixels 101 arranged in the left-right direction in Figure 2. Logic circuits such as a shift register and an address decoder are used in the vertical scanning circuit 110. A signal processing circuit 103 is provided for each photoelectric conversion unit 102 and processes the signal output from the photoelectric conversion unit 102. The signal processing circuit 103 is equipped with a counter and memory, and digital values ​​are stored in the memory. The horizontal scanning circuit 111 inputs a control pulse to the signal processing circuit 103 in order to read a signal from the memory of the pixel 101 in which the digital signal is held. The vertical signal line 113 is the signal line through which the signal output from the pixel 101 selected by the vertical scanning circuit 110 passes. The signal passing through the vertical signal line 113 is output to the outside of the photoelectric conversion element 100 via the readout circuit 112 and the output circuit 114. The readout circuit 112 has multiple buffers built in that are connected to the vertical signal line 113.

[0014] As shown in Figures 2 and 4, multiple signal processing circuits 103 are provided in the region that overlaps with the pixel region 12 in the front-to-back direction of the paper in the figures. Furthermore, the vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, and control pulse generation unit 115 are provided so as to overlap with the sensor substrate 11 in the front-to-back direction of the paper in the figures, but not overlap with the pixel region 12. In other words, the sensor substrate 11 has a pixel region 12 and a non-pixel region different from the pixel region 12. The vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, and control pulse generation unit 115 are provided so as to overlap with the non-pixel region of the sensor substrate 11 in the front-to-back direction of the paper in the figures.

[0015] Furthermore, the arrangement of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to the example shown in Figure 4. For example, the vertical signal line 113 may extend in the left-right direction in the figure, and the readout circuit 112 may be connected to the end of the vertical signal line 113. Also, the signal processing circuit 103 does not have to be provided for each photoelectric conversion unit 102. A single signal processing circuit 103 may sequentially perform signal processing on multiple photoelectric conversion units 102. In this embodiment, the circuit board 21 is provided with a signal processing circuit 103 for each of the R filter 31R, G filter 31G, B filter 31B, and IR filter 32. Each signal processing circuit 103 handles the light transmitted through the target color filter 30. The light transmitted through the R filter 31R may be referred to as the R signal below. The light transmitted through the G filter 31G may be referred to as the G signal below. The light transmitted through the B filter 31B may be referred to as the B signal below. The light transmitted through the IR filter 32 may be referred to as the IR signal below. The R signal, G signal, B signal, and IR signal are all signals after photoelectric conversion. The memory of the signal processing circuit 103 stores the digital value after photoelectric conversion corresponding to the target signal among the R signal, G signal, B signal, and IR signal.

[0016] Figure 5 shows the equivalent circuit of pixel 101 and signal processing circuit 103. The APD201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to incident light through photoelectric conversion. One of the two nodes of the APD201 is connected to a power line to which a drive voltage VL (first voltage) is supplied. The other of the two nodes of the APD201 is connected to a power line to which a drive voltage VH (second voltage), which is higher than the drive voltage VL, is supplied. In the example shown in Figure 5, one node of the APD201 is the anode, and the other node of the APD is the cathode. A reverse bias voltage is supplied to the anode and cathode of the APD201 so that the APD201 performs avalanche multiplication. When this voltage is supplied, the charge generated by the incident light undergoes avalanche multiplication, and an avalanche current is generated.

[0017] The APD201 has two modes: a Geiger mode, which operates with an anode-cathode voltage difference greater than the breakdown voltage when a reverse bias voltage is supplied, and a linear mode, which operates with an anode-cathode voltage difference near or below the breakdown voltage. Hereafter, the APD201 operating in Geiger mode may be referred to as SPAD. In SPAD, for example, the drive voltage VL (first voltage) is -30V and the drive voltage VH (second voltage) is 1V.

[0018] The signal processing circuit 103 includes a quench element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quench element 202 is connected to a power line to which a drive voltage VH is supplied, and to one of the nodes, either the anode or the cathode, of the APD 201. The quench element 202 functions as a load circuit (quench circuit) during signal multiplication by avalanche multiplication, suppressing the voltage supplied to the APD201 and thereby suppressing avalanche multiplication (quench operation). In addition, the quench element 202 returns the voltage supplied to the APD201 to the drive voltage VH by flowing the current that compensates for the voltage drop caused by the quench operation (recharge operation). The waveform shaping unit 210 shapes the cathode voltage change of the APD201 obtained during photon detection and outputs a pulse signal. An inverter circuit can be used as the waveform shaping unit 210. In the example shown in Figure 5, one inverter is used as the waveform shaping unit 210, but a circuit in which multiple inverters are connected in series may be used as the waveform shaping unit 210, or other circuits that have a waveform shaping effect may be used as the waveform shaping unit 210.

[0019] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When the control pulse RES is supplied via the drive line 213, the signal held by the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference in count values ​​between the start and end of the accumulation period. The memory circuit 212 receives control pulses SEL from the vertical scanning circuit 110 (see Figure 4) via the drive line 214, and switches the electrical connection and disconnection between the counter circuit 211 and the vertical signal line 113. The memory circuit 212 functions as a memory that temporarily stores the counter's count value and outputs the output signal from the counter circuit 211 of pixel 101 to the vertical signal line 113. Furthermore, the electrical connection may be switched by providing a switch such as a transistor between the quench element 202 and the APD201, or between the photoelectric conversion unit 102 and the signal processing circuit 103. In addition, the supply of the drive voltage VH and drive voltage VL supplied to the photoelectric conversion unit 102 may be electrically switched by a switch such as a transistor.

[0020] Figure 6 shows the relationship between the operation of the APD201 and the output signal. In Figure 6, the input side of the waveform shaping unit 210 is shown as nodeA, and the output side of the waveform shaping unit 210 is shown as nodeB. Between time t0 and time t1, a potential difference of VH-VL is applied to APD201. When a photon is incident on APD201 at time t1, avalanche multiplication occurs in APD201, an avalanche multiplication current flows through the quench element 202, and the voltage at nodeA drops. Subsequently, as the voltage drop becomes larger and the potential difference applied to APD201 decreases, the avalanche multiplication of APD201 stops at time t2, and the voltage level at nodeA no longer drops below a certain value. Then, between time t2 and time t3, a current flows through nodeA to compensate for the voltage drop from the drive voltage VL, and at time t3, nodeA settles to its original potential level. At this time, any portion of the output waveform at nodeA that exceeds a certain threshold is reshaped by the waveform shaping unit 210 and output as a pulse signal at nodeB.

[0021] Figure 7 shows the functional configuration of the IR emitter 500, camera 600, and mobile unit 700. The IR emitter 500, camera 600, and mobile unit 700 all realize various functions described later by having a computer (not shown) execute a computer program stored in memory (not shown) as a storage medium. However, some or all of the functions of the IR emitter 500, camera 600, and mobile unit 700 may be realized by hardware. Examples of hardware include dedicated circuits (ASICs), processors (reconfigurable processors, DSPs), etc.

[0022] The camera 600 has the photoelectric conversion element 100 described above. The camera 600 also has an imaging optical system 601, an image processing unit 603, a recognition unit 604, a color determination unit 605, a camera control unit 606, a storage unit 607, and a communication unit 608. The image processing unit 603 processes the image signal acquired by the photoelectric conversion element 100 to generate a final image signal. Examples of image processing performed by the image processing unit 603 include black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression. The image signal output by the photoelectric conversion element 100 passes through the color filter 30 to obtain R, G, B, and IR signals. The image processing unit 603 generates a color image by performing demosaicing on the R, G, and B signals. The image processing unit 603 may also perform white balance correction and color conversion. Furthermore, the image processing unit 603 generates an IR image using the IR signal. The IR image does not contain color information and is generated as a monochrome image. The image processing unit 603 may perform different image processing for color image generation and IR image generation. The IR image is an example of the first image, and the color image is an example of the second image. Furthermore, the image processing unit 603 outputs the generated image signal to the recognition unit 604, the color determination unit 605, the camera control unit 606, and the ECU (Electric Control Unit) 701 of the mobile body 700, which will be described later. At this time, the image signal output from the image processing unit 603 includes a color image and an IR image.

[0023] The recognition unit 604, as an example of a recognition processing unit, recognizes objects displayed in an image by performing image recognition on the image signal. Examples of objects recognized by the recognition unit 604 include animals such as humans, moving objects such as vehicles, traffic lights, road signs, and other signs. The recognition unit 604 performs image recognition by using, for example, deep learning. Examples of deep learning methods include YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Faster R-CNN (Regional Convolution Neural Network), Fast R-CNN, and R-CNN. In this embodiment, the recognition unit 604 performs image recognition using IR images. In this case, compared to a configuration using color images, even when images are taken in poor visibility conditions such as at night or in bad weather, the object's outline is easier to capture, resulting in higher accuracy in object recognition. This is because IR images are generated by imaging with infrared light, which has a longer wavelength than visible light, and are less likely to be scattered by water droplets in the air, allowing distant objects to be clearly imaged without being obstructed by fog. The recognition unit 604 also calculates the distance from the camera 600 to the recognized object. As a method of distance measurement by the recognition unit 604, for example, a method of estimating the distance using deep learning can be used. For example, the recognition unit 604 may calculate the distance by analyzing information such as blur in the image of the detected object using deep learning. Another example is that the recognition unit 604 may measure the distance using the principle of triangulation with the camera 600 as a stereo camera. The recognition unit 604 outputs information indicating the type of recognized object, the coordinates indicating the object's position in the image, and the distance from the camera 600 to the object as recognition results to the color determination unit 605 and the ECU 701 of the moving body 700. The process by which the recognition unit 604 recognizes the type of recognized object, the coordinates indicating the object's position in the image, and the distance from the camera 600 to the object can be considered as a recognition process. Furthermore, the processing steps performed by the recognition unit 604 can also be considered as a recognition process that performs object recognition processing using an IR image.

[0024] As an example of a color determination selection unit, the color determination unit 605 uses the color image received from the image processing unit 603 to determine the color of an object recognized by the recognition unit 604. The color determination unit 605 performs color determination on an object if the object recognized by the recognition unit 604 is one of the objects predetermined to be subject to color determination. In this case, the color determination unit 605 determines the area to be color determined in the color image based on the coordinates indicating the position of the object output as a recognition result by the recognition unit 604. Then, the color determination unit 605 performs color determination on the target area in the color image using a color determination method appropriate to the object recognized by the recognition unit 604. The target of color determination and the color determination method used by the color determination unit 605 will be described in detail later. The color determination unit 605 outputs the color determination result to the ECU 701 of the mobile unit 700. Furthermore, the processing step performed by the color determination unit 605 can also be understood as a color determination step in which the color determination process selected in the color determination selection step is performed on the region selected in the color determination region selection step, using the color image generated by the photoelectric conversion element 100.

[0025] The camera control unit 606 has a CPU and a memory that stores a computer program, and controls the camera 600 by having the CPU execute the computer program stored in the memory. Examples of control by the camera control unit 606 include control of the exposure period length for each frame of the photoelectric conversion element 100 via the control pulse generation unit 115, and control of the timing of the control signal. The camera control unit 606 also instructs the IR light emitter 500 to emit light via the communication unit 608. The memory unit 607 has a recording medium such as a memory card or a hard disk and stores image signals. The communication unit 608 has a wireless or wired interface and communicates with external devices. Through communication, the communication unit 608 outputs image signals to the outside of the camera 600 and receives various signals from the outside. The communication unit 608 also transmits instructions from the camera control unit 606 to the IR emitter 500 via the communication unit 503 of the IR emitter 500.

[0026] The IR emitter 500 includes an IR light-emitting unit 501, a light-emitting control unit 502, and a communication unit 503. The IR light-emitting unit 501 is, for example, a near-infrared LED. The IR light-emitting unit 501 has a lens (not shown) and a light-emitting unit (not shown), and outputs IR light at a timing and duration determined according to the control signal output from the light emission control unit 502. The light emission control unit 502 receives a light emission instruction from the camera control unit 606 via the communication unit 503, generates a control signal to emit light at the instructed timing, and outputs the generated control signal to the IR light emission unit 501. The control signal generated by the light emission control unit 502 includes information such as the duration of light emission, the period of light emission, and the number of times light is emitted. The communication unit 503 communicates with the camera 600 via the camera 600's communication unit 608, receives a light emission instruction from the camera control unit 606, and transmits the received light emission instruction to the light emission control unit 502.

[0027] The camera 600 and the IR emitter 500 are considered as an imaging device. In this embodiment, the camera 600 and the IR emitter 500 are mounted on the mobile body 700. Therefore, in a broad sense, the camera 600, the IR emitter 500, and the mobile body 700 can also be considered as an imaging device. In this embodiment, the camera unit, consisting of the imaging optical system 601 and the photoelectric conversion element 100, is configured to capture images in at least one direction: the front, rear, or side of the mobile body. Multiple camera units may be provided on the mobile body. The IR emitter 500 is positioned, for example, at the front of the mobile body 700.

[0028] In the following explanation, we will assume that the mobile device 700 is an autonomous vehicle. The mobile unit 700 includes an ECU 701, a vehicle control unit 702, and a display unit 703. As an example of an operation determination unit, the ECU 701 has a CPU and a memory that stores a computer program, and controls the mobile body 700 by having the CPU execute the computer program stored in the memory. When the ECU 701 receives the recognition result from the recognition unit 604 and the color determination result from the color determination unit 605, it determines the control content for the mobile body 700 according to the received recognition result and color determination result. The control content for the mobile body 700 may include, for example, stopping control of the mobile body 700 by automatic braking. The ECU 701 outputs information indicating the determined control content to the vehicle control unit 702 and the display unit 703. In addition, when the ECU 701 receives a color image and an IR image from the image processing unit 603, it transmits the received image to the display unit 703.

[0029] The vehicle control unit 702 controls the operation of the mobile body 700 based on instructions from the ECU 701. Examples of control of the mobile body 700 by the vehicle control unit 702 include driving, stopping, and direction control. The display unit 703 is a display means for displaying images. Examples of the display unit 703 include liquid crystal devices and organic EL displays. In response to instructions from the ECU 701, the display unit 703 displays various information such as images acquired by the photoelectric conversion element 100, recognition results by the recognition unit 604, color determination results by the color determination unit 605, and the driving status of the mobile body 700.

[0030] The image processing unit 603 and recognition unit 604 of the camera 600 may not be mounted on the mobile unit 700, but may be provided, for example, on an external terminal. Examples of external terminals include a terminal for remotely controlling the mobile unit 700 and a terminal for monitoring the movement of the mobile unit 700. Furthermore, the functional units of the camera 600, the IR light emitter 500, and the mobile unit 700 shown in Figure 7 may not only be provided in a single device, but may also be provided separately in multiple devices.

[0031] Figure 8 shows an object management table. The object management table is a table for managing objects that are subject to color determination by the color determination unit 605 of the camera 600. The object management table is stored, for example, in the storage unit 607. The object management table shows the "object," "target area," "color determination process," and "determination result" in relation to each other.

[0032] The contents of the object management table will be explained in detail. The "Object" column indicates the type of object that is the target of color determination by the color determination unit 605. When the color determination unit 605 receives a recognition result from the recognition unit 604, it identifies the type of object recognized by the recognition unit 604 from the received recognition result. The color determination unit 605 then performs color determination on the object if the identified object type is shown in the "Object" column of the object management table, and does not perform color determination on the object if it is not shown in the "Object" column. The "target area" indicates the area of ​​the "objects" displayed in the color image that is subject to color determination processing by the color determination unit 605. Color determination processing is the process by which the color determination unit 605 determines the color of the objects displayed in the color image. Hereafter, the area of ​​the "objects" displayed in the color image that is subject to color determination processing by the color determination unit 605 may be referred to as the target area. The "color determination process" section describes the color determination process that the color determination unit 605 performs on the "target area" of the "object". The "Judgment Result" section displays candidate color judgment results from the color judgment unit 605. The color judgment unit 605 determines the result of the color judgment process for the "Object" from the candidates shown in the "Judgment Result" associated with this "Object".

[0033] The contents of each item in the object management table will be explained along with specific examples of objects shown in Figure 9. In this embodiment, when it is necessary to control the moving body 700 according to the color of a specific part of an object recognized by the recognition unit 604, this object is designated as the target of color determination processing by the color determination unit 605. More specifically, this object is shown in the "Object" column of the object management table, and the specific part of the object mentioned above is shown in the "Target Area" column of the object management table.

[0034] The "Object" in the object management table shows "Vehicle" as shown in Figure 9(A). When camera 600 is photographing the area in front of the mobile body 700, knowing whether the vehicle in front is moving, stopped, accelerating, decelerating, etc., is necessary information for realizing automatic driving of the mobile body 700 in response to the state of the vehicle in front. An example of automatic driving of the mobile body 700 in response to the state of the vehicle in front is that, if the vehicle in front is stopped, the mobile body 700 stops to prevent a collision with this vehicle in front. Automatic driving of the mobile body 700 in response to the state of the vehicle in front is a control required even in conditions of poor visibility such as nighttime or bad weather, where it is more difficult to detect the vehicle displayed in the image compared to conditions with good visibility such as clear weather. Furthermore, while the recognition unit 604 recognizes the vehicle from the IR image in which the vehicle is displayed, it does not recognize whether the vehicle is moving, stopped, accelerating, decelerating, etc. Therefore, in this embodiment, the vehicle is shown in the "Object" in the object management table as the target of color determination processing. Furthermore, the brake lights, which require information about their color to determine if the vehicle is decelerating, are shown in the "Target Area" of the object management table. The "Brake Lights" shown in the "Target Area" refer to the left brake light 800a and the right brake light 800b shown in Figure 9(A). Furthermore, whether or not a vehicle is decelerating is determined by whether or not the brake lights are illuminated in red. Therefore, the details of the processing related to whether or not the brake lights are illuminated in red are shown in the "Color Judgment Processing" of the object management table. "Comparison of the red luminance of the target area with the threshold" shown in "Color Judgment Processing" means that the color judgment unit 605 calculates the average value of the red luminance of the "target area" in the color image as a color judgment process and determines whether or not the calculated average value exceeds the threshold. Also, "Comparison of contrast with the threshold" shown in "Color Judgment Processing" means that the color judgment unit 605 calculates the average value of the contrast of the "objects" in the color image as a color judgment process and determines whether or not the calculated average value exceeds the threshold. Furthermore, the "Judgment Result" associated with "Vehicle" shows "Undeterminable," "Brake lights off," and "Brake lights on." The color determination unit 605 determines the judgment result to "Brake lights off" if the average value of the red luminance calculated for the "Target area" as a result of the "Color determination process" is below a threshold. The color determination unit 605 also determines the judgment result to "Brake lights on" if the average value of the red luminance calculated for the "Target area" as a result of the "Color determination process" exceeds a threshold. In addition, the color determination unit 605 determines the judgment result to "Undeterminable" if the average value of the contrast calculated for the "Object" as a result of the "Color determination process" is below a threshold, regardless of whether the average value of the red luminance calculated for the "Target area" exceeds a threshold. This means that if the average contrast of the object is low, it is difficult for the color determination unit 605 to determine whether or not the brake lights are on.

[0035] Furthermore, the "Object" in the object management table shows the "traffic light" as shown in Figure 9(B). When the camera 600 is photographing the area in front of the mobile body 700, knowing whether the traffic light ahead is red, yellow, or green is necessary information for realizing automatic driving of the mobile body 700 according to the traffic light status. Examples of automatic driving of the mobile body 700 according to the traffic light status include the mobile body 700 stopping when the traffic light ahead is red or yellow, or the mobile body 700 continuing to move when the traffic light ahead is green. In addition, automatic driving of the mobile body 700 according to the traffic light status is required even in conditions with poor visibility, such as at night or in bad weather, where it is more difficult to detect the traffic light displayed in the image compared to conditions with good visibility, such as in clear weather. Furthermore, while the recognition unit 604 recognizes the traffic light from the IR image in which the traffic light is displayed, it does not recognize which color the traffic light is illuminated in. Therefore, in this embodiment, the traffic light is shown as an object in the object management table as the target of the color determination process. In addition, the lamps, which are the parts necessary for determining which color the traffic light is illuminated in, are shown in the object management table as the target area. The lamps shown in the target area refer to the blue lamp 810a, yellow lamp 810b, and red lamp 810c shown in Figure 9(B). Furthermore, the state in which a traffic light is illuminated is determined by whether the light is illuminated in blue, yellow, or red. Therefore, the details of the processing related to the illumination status of the blue light 810a, yellow light 810b, and red light 810c are shown in the "Color Determination Processing" of the object management table. "Comparison of the brightness of blue in the target area with the threshold" in "Color Determination Processing" means that the color determination unit 605 calculates the average brightness of blue in the "target area" corresponding to the blue light 810a in the color image as a color determination process, and determines whether the calculated average value exceeds the threshold. "Comparison of the brightness of yellow in the target area with the threshold" in "Color Determination Processing" means that the color determination unit 605 calculates the average brightness of yellow in the "target area" corresponding to the yellow light 810b in the color image as a color determination process, and determines whether the calculated average value exceeds the threshold. The "comparison of the red luminance of the target area with the threshold" in the "color determination process" means that the color determination unit 605 calculates the average value of the red luminance of the "target area" corresponding to the red light 810c in the color image as part of the color determination process, and determines whether the calculated average value exceeds the threshold. Furthermore, the "judgment results" associated with the "traffic light" include "undeterminable," "green light on," "yellow light on," and "red light on." The color determination unit 605 determines the judgment result to "green light on" if, as a result of the "color determination processing," the average value of the blue luminance calculated for the "target area" corresponding to the blue light 810a exceeds the threshold. The color determination unit 605 also determines the judgment result to "yellow light on" if, as a result of the "color determination processing," the average value of the yellow luminance calculated for the "target area" corresponding to the yellow light 810b exceeds the threshold. The color determination unit 605 also determines the judgment result to "red light on" if, as a result of the "color determination processing," the average value of the red luminance calculated for the "target area" corresponding to the red light 810c exceeds the threshold. Furthermore, the average value of the blue luminance in the "target area" of the blue light 810a, the average value of the yellow luminance in the "target area" of the yellow light 810b, and the average value of the red luminance in the "target area" of the red light 810c may all be below the threshold. In this case, the color determination unit 605 determines the determination result to be "undeterminable." This "undeterminable" result may occur in conditions of poor visibility such as nighttime or bad weather, or when the distance from the mobile body 700 to the traffic light is large. Note that the threshold used for comparison with the blue luminance, the threshold used for comparison with the yellow luminance, and the threshold used for comparison with the red luminance may be the same value or may be different values. Furthermore, the color determination processing method by the color determination unit 605 is not limited to the above example. The color determination unit 605 compares the average value of blue luminance calculated for the "target area" of the blue light 810a, the average value of yellow luminance calculated for the "target area" of the yellow light 810b, and the average value of red luminance calculated for the "target area" of the red light 810c. The color determination unit 605 may then determine that the traffic light is lit in the color with the highest average luminance value. In addition, there are cases where two or more of the average values ​​of blue luminance in the "target area" of the blue light 810a, the average value of yellow luminance in the "target area" of the yellow light 810b, and the average value of red luminance in the "target area" of the red light 810c exceed a threshold. In this case, the color determination unit 605 may determine the determination result to be "undeterminable".

[0036] Furthermore, the "Object" in the object management table shows "Sign A". "Sign A" refers to the road sign with a leftward-pointing arrow shown in Figure 9(C). Hereafter, the road sign with a leftward-pointing arrow shown in Figure 9(C) may simply be referred to as Sign A. In Japan, Sign A with a white arrow means that vehicles are only allowed to travel in the direction indicated by the arrow, i.e., a one-way street. Also in Japan, Sign A with a blue arrow means that vehicles are allowed to turn in the direction indicated by the arrow, i.e., a left turn permitted. Therefore, when camera 600 is photographing the area in front of the mobile vehicle 700, whether the arrow on the sign A ahead is white or blue is information necessary for realizing automated driving of the mobile vehicle 700 according to the type of sign A. Examples of automated driving of the mobile vehicle 700 according to the type of sign A include restricting the mobile vehicle 700 from moving in a direction different from the direction indicated by the arrow on sign A, depending on whether the arrow on the sign A ahead is white. Furthermore, the automatic driving of the mobile body 700 according to the type of sign A is required even in conditions of poor visibility such as nighttime or bad weather, where it is more difficult to detect the traffic signals displayed in the image compared to conditions of good visibility such as clear weather. In addition, the recognition unit 604 recognizes sign A from the IR image in which sign A is displayed, but does not recognize whether sign A is a road sign indicating one-way or left turn permitted. Therefore, in this embodiment, sign A is shown as an object in the object management table as the target of color determination processing. In addition, the inside of the arrow, which is the part necessary for determining the color of the arrow shown on sign A, is shown in the object management table as the "target area". The "inside of the arrow" shown in the "target area" means the inside of the arrow 820 of sign A shown in Figure 9(C). Furthermore, the type of marker A is determined by whether the arrow of marker A is indicated in blue or white. The details of the processing regarding the presence or absence of blue indication inside the arrow 820 of marker A are shown in the "Color Judgment Processing" of the object management table. "Comparison of blue luminance of the target area with threshold" in "Color Judgment Processing" means that the color judgment unit 605 calculates the average value of the blue luminance of the "target area" in the color image as a color judgment process and determines whether the calculated average value exceeds the threshold. Also, "Comparison of contrast with threshold" shown in "Color Judgment Processing" means that the color judgment unit 605 calculates the average value of the contrast of the "object" in the color image as a color judgment process and determines whether the calculated average value exceeds the threshold. Furthermore, the "Judgment Result" associated with "Sign A" shows "Undeterminable," "One-Way Street," and "Left Turn Permitted." The color determination unit 605 determines the judgment result to "One-Way Street" if the average value of the blue luminance calculated for the "Target Area" as a result of the "Color Judgment Processing" exceeds a threshold. The color determination unit 605 also determines the judgment result to "Left Turn Permitted" if the average value of the blue luminance calculated for the "Target Area" as a result of the "Color Judgment Processing" is below the threshold. In addition, the color determination unit 605 determines the judgment result to "Undeterminable" if the average value of the contrast calculated for the "Object" as a result of the "Color Judgment Processing" is below the threshold, regardless of whether the average value of the blue luminance calculated for the "Target Area" exceeds the threshold. This means that if the average contrast of the object is low, it is difficult for the color determination unit 605 to determine the display color of the arrow on Sign A. Thus, in this embodiment, taking into account that there are objects with the same shape but different display colors, these objects are predetermined in the object management table as targets for color determination processing.

[0037] Furthermore, the "Object" in the object management table shows "Sign B". "Sign B" refers to the road sign shown in Figure 9(D), which has a diagonal line inside a circle. Hereafter, the road sign shown in Figure 9(D), which has a diagonal line inside a circle, may simply be referred to as Sign B. In Japan, Sign B with a blue color inside the circle means no parking for vehicles. Also in Japan, Sign B with a white color inside the circle means no vehicle traffic. Therefore, when camera 600 is photographing the area in front of the mobile vehicle 700, whether the inside of the circle shown on the sign B ahead is blue or white is information necessary to realize automated driving of the mobile vehicle 700 according to the type of sign B. An example of automated driving of the mobile vehicle 700 according to the type of sign B is to restrict parking of the mobile vehicle 700 depending on whether the inside of the circle shown on the sign B ahead is blue. Furthermore, an example of automatic driving of the mobile body 700 according to the type of sign B is to restrict the passage of the mobile body 700 according to whether the inside of the circle shown on the sign B ahead is white. In addition, automatic driving of the mobile body 700 according to the type of sign B is a control required even in conditions of poor visibility such as nighttime or bad weather, where it is more difficult to detect the traffic signals displayed in the image compared to conditions of good visibility such as clear weather. Furthermore, while the recognition unit 604 recognizes sign B from the IR image on which sign B is displayed, it does not recognize whether this sign B is a road sign indicating no parking or no vehicle traffic.Therefore, in this embodiment, sign B is shown as an object in the object management table as the target of color determination processing.In addition, the inside of the circle, which is the part necessary for determining the type of sign B, is shown in the object management table as the target area.The "inside of the circle" shown in the "target area" means the inside of the circle 830 of sign B shown in Figure 9(D). Furthermore, the type of marker B is determined by whether the inside 830 of the circle of marker B is indicated in blue or white. The details of the processing regarding the presence or absence of blue indication in the inside 830 of the circle of marker B are shown in the "Color Determination Processing" of the object management table. "Comparison of blue luminance of the target area with threshold" in "Color Determination Processing" means that the color determination unit 605 calculates the average value of the blue luminance of the "target area" in the color image as a color determination process and determines whether the calculated average value exceeds the threshold. Also, "Comparison of contrast with threshold" shown in "Color Determination Processing" means that the color determination unit 605 calculates the average value of the contrast of the "object" in the color image as a color determination process and determines whether the calculated average value exceeds the threshold. Furthermore, the "Judgment Result" associated with "Sign B" shows "Undetermined," "No Parking," and "No Entry." The color determination unit 605 determines the judgment result to "No Parking" if the average value of the blue luminance calculated for the "Target Area" as a result of the "Color Judgment Processing" exceeds a threshold. The color determination unit 605 also determines the judgment result to "No Entry" if the average value of the blue luminance calculated for the "Target Area" as a result of the "Color Judgment Processing" is below the threshold. In addition, the color determination unit 605 determines the judgment result to "Undetermined" if the average value of the contrast calculated for the "Object" as a result of the "Color Judgment Processing" is below the threshold, regardless of whether the average value of the blue luminance calculated for the "Target Area" exceeds the threshold. This means that if the average contrast of the object is low, it is difficult for the color determination unit 605 to determine the display color inside the circle 830 of Sign B.

[0038] In this embodiment, when an object to be subjected to color determination processing is recognized by the recognition unit 604, the target area is identified based on the size and shape of the recognized object and the "target area" information shown in the object management table associated with this object. The identification of the target area may be performed by the recognition unit 604 or by the color determination unit 605. Therefore, both the recognition unit 604 and the color determination unit 605 can be considered as color determination area selection units that select an area to perform color determination processing based on the object detected by the recognition unit 604. Furthermore, the processing steps performed by the recognition unit 604 and the color determination unit 605 can also be considered as color determination area selection steps that select an area to perform color determination processing based on the object detected in the recognition step. Furthermore, the target area may be specified according to the object recognized by the recognition unit 604. For example, when the recognition unit 604 recognizes "vehicle," which is an "object" shown in the object management table, different areas may be specified as the target area depending on whether this "vehicle" is a passenger car, a truck, or a motorcycle. Also, for example, when the recognition unit 604 recognizes "traffic light," which is an "object" shown in the object management table, different areas may be specified as the target area depending on the shape of the traffic light, such as whether it is vertical or horizontal.

[0039] Figure 10 is a flowchart showing the flow of the result output processing. The result output processing is the process in which the camera 600 outputs the result of the color determination processing. In this embodiment, for example, when the camera 600 is in a mode for performing color determination processing, the result output processing is started when it receives an imaging instruction in response to the user's operation of the camera 600. The camera control unit 606 sets the emission settings for the IR emitter 500 (step (hereinafter sometimes referred to as "S") 101). More specifically, the camera control unit 606 sets the timing of emission by the IR emitter 500, the duration of emission, the emission period, the number of emission cycles, etc.

[0040] The camera control unit 606 configures the photoelectric conversion element 100 (S102). More specifically, the camera control unit 606 configures various settings on the circuit board 21 of the photoelectric conversion element 100 for photoelectric conversion from the optical image from the imaging optical system 601 to generate an image signal. The camera control unit 606 may configure different settings for the pixels 101 of the RGB filter 31 and the pixels 101 of the IR filter 32 in this photoelectric conversion configuration. At this time, the camera control unit 606 also configures the parameters of the image processing unit 603, the recognition unit 604, and the color determination unit 605. The camera control unit 606 initiates imaging (S103). More specifically, the camera control unit 606 instructs the IR light emitter 500 to emit light according to the settings set in step 101, and instructs the photoelectric conversion element 100 to output a vertical synchronization signal, thereby initiating exposure and image signal generation.

[0041] The camera control unit 606 instructs the image processing unit 603 to perform various image processing operations on the image signal output from the photoelectric conversion element 100 as a result of the imaging initiated in step 103, thereby generating a final image signal. As a result, the image processing unit 603 acquires a color image using the R, G, and B signals output from the photoelectric conversion element 100, and also acquires a monochrome IR image using the IR signal (S104). The recognition unit 604 recognizes the object displayed in the IR image acquired in step 104 (S105). More specifically, the recognition unit 604 detects the object displayed in the IR image, identifies the coordinates indicating the position of the detected object in the IR image, and calculates the distance of the detected object from the camera 600.

[0042] The recognition unit 604 determines whether the object recognized in step 105 is subject to color determination processing by the color determination unit 605 (S106). The recognition unit 604 makes the determination in step 106 based on whether the recognized object is of the type shown in "Object" in the object management table (see Figure 8). If the object recognized by the recognition unit 604 is subject to color determination processing (Yes in S106), the color determination unit 605 selects the content of the color determination processing according to the recognized object (S107). The color determination unit 605 selects the content of the color determination processing to be executed from the content indicated in the "color determination processing" associated with the recognized "object" in the object management table. At this time, the target area is also identified according to the recognized object.

[0043] The color determination unit 605 performs color determination processing on the target area of ​​the object displayed in the color image, based on the content selected in step S107 (S108). In this embodiment, since the IR image and the color image are acquired by a common photoelectric conversion element 100, the IR image and the color image have the same field of view. Therefore, the coordinates corresponding to the target area identified from the IR image are also applied as the coordinates of the target area in the color image. Furthermore, different photoelectric conversion elements may be used for acquiring IR images and color images. In this case, the field of view may differ between the IR image and the color image depending on the arrangement of pixels in each photoelectric conversion element. If the field of view differs between the IR image and the color image, the coordinates of the target area in the color image may be determined by performing a coordinate transformation process (alignment process) corresponding to the target area identified from the IR image.

[0044] The camera control unit 606 transmits the color image and IR image acquired in step 104, the recognition result by the recognition unit 604 performed in step 105, and the result of the color determination process performed in step 108 to the mobile body 700 (S109). If the object recognized by the recognition unit 604 is not subject to color determination processing (No in S106), the camera control unit 606 transmits the color image and IR image, along with the recognition result by the recognition unit 604, to the mobile body 700. The camera control unit 606 determines whether the imaging instruction is continuing (S110). If the imaging instruction is continuing (Yes in S110), the process from step 103 is repeated. That is, if the imaging instruction is continuing, the process from imaging is repeated for the next frame. If the imaging instruction is not continuing (No in S110), the result output process ends.

[0045] Figure 11 shows a relationship management table. The relationship management table is a table for managing the relationship between the object that is the target of the color determination process, the determination result of the color determination process, and the operation of the mobile body 700. The relationship management table is stored, for example, in the ECU 701. The relationship management table shows the relationships between "objects," "judgment results," and "action details."

[0046] Let me explain the contents of the relationship management table in detail. The "Object" column indicates the type of object that has undergone color determination processing by the color determination unit 605. This "Object" column contains the same items as the "Object" column in the object management table (see Figure 8). The "Judgment Result" column shows the result of the color determination process performed by the color determination unit 605. This "Judgment Result" column contains the same items as the "Judgment Result" column in the object management table. The "Operation Details" section describes how the ECU 701 of the mobile unit 700 operates the mobile unit 700.

[0047] When the ECU 701 receives the recognition results of the IR image from the recognition unit 604 and the results of the color determination processing from the camera 600, it determines what actions to take the mobile body 700 according to the received information. More specifically, the ECU 701 determines what actions to take the mobile body 700 based on the "action content" associated with the "object" corresponding to the recognition result of the recognition unit 604 and the "determination result" corresponding to the color determination processing result in the relationship management table.

[0048] I will now explain in detail how the relationship management table works. In the relationship management table, the "Action Details" associated with the object "Vehicle" and the judgment result "Undetermined" shows "Deceleration / Slow Driving". In situations where it is not possible to determine whether the vehicle's brake lights are illuminated, if the vehicle's brake lights are illuminated, in other words, the vehicle may be slowing down or stopped. In this case, it is necessary to restrict the movement of the moving object 700 in order to prevent a collision between the moving object 700 and the vehicle. Therefore, in this embodiment, in order to prioritize safety, "deceleration / low-speed driving" is defined as the "operation content" so that the movement of the moving object 700 is restricted even in situations where it is not possible to determine whether the vehicle's brake lights are illuminated.

[0049] Furthermore, in the relationship management table, the "Action Details" associated with the object "Vehicle" and the judgment result "Brake lights not illuminated" show "Continue driving". If the vehicle's brake lights are not illuminated, even if the moving object 700 does not decelerate, the moving object 700 will not suddenly approach the vehicle, and a situation is identified in which a collision between the moving object 700 and the vehicle is unlikely. Therefore, in this embodiment, "continued driving" is defined as the "action" when the vehicle's brake lights are not illuminated.

[0050] Furthermore, in the relationship management table, the "action content" associated with the object "vehicle" and the judgment result "brake lights on" shows "deceleration / stop". If the vehicle's brake lights are illuminated, and the mobile unit 700 continues to move, there is a risk that the mobile unit 700 will rapidly approach the vehicle and collide with it. Therefore, in this embodiment, "deceleration and stopping" is defined as the "action to take" when the vehicle's brake lights are illuminated.

[0051] Furthermore, in the relationship management table, the "operation details" associated with the object "traffic light" and the judgment result "undeterminable" show "deceleration / slow driving". In situations where it is not possible to determine which color the traffic light is illuminated in, the traffic light may be illuminated in either yellow or red. In such cases, it is necessary to restrict the movement of the mobile unit 700 before it reaches the traffic light. Therefore, in this embodiment, in order to prioritize safety, "deceleration / low-speed driving" is defined as the "operation" so that the movement of the mobile unit 700 is restricted even in situations where it is not possible to determine which color the traffic light is illuminated in.

[0052] Furthermore, in the relationship management table, the "action content" associated with the object "traffic light" and the judgment result "green light on" shows "continue driving". When the traffic light is lit green, the moving vehicle 700 does not need to slow down. Therefore, in this embodiment, "continue driving" is defined as the "operation" when the traffic light is lit green.

[0053] Furthermore, in the relationship management table, the "action content" associated with the object "traffic light" and the judgment results "yellow light on" and "red light on" shows "slow down / stop". If the traffic light is illuminated yellow or red, the mobile unit 700 must be stopped before it reaches the traffic light. Therefore, in this embodiment, "deceleration and stopping" is defined as the "action to be performed" when the traffic light is illuminated yellow or red.

[0054] Furthermore, in the relationship management table, the "Action Details" associated with the object "Sign A" and the judgment result "Undetermined" shows "Continue Driving". Furthermore, in the relationship management table, the "action content" associated with the object "Sign A" and the judgment result "One-way street" indicates "No driving in directions other than the arrow direction." When Sign A is a road sign indicating a one-way street, it is necessary to restrict the mobile body 700 from traveling in directions other than the one indicated by the arrow in order to comply with traffic regulations. Therefore, in this embodiment, "No driving in directions other than the arrow direction" is defined as the "action content" when Sign A is a road sign indicating a one-way street. Furthermore, in the relationship management table, the "action content" associated with the object "sign A" and the judgment result "left turn permitted" indicates "no restriction on driving to the left." If sign A is a road sign indicating that a left turn is permitted, there is no need to restrict the left turn of the mobile body 700. Therefore, in this embodiment, "no restriction on driving to the left" is defined as the "action content" when sign A is a road sign indicating that a left turn is permitted.

[0055] Furthermore, in the relationship management table, the "Action Details" associated with the object "Sign B" and the judgment result "Undetermined" shows "Continue Driving". Furthermore, in the relationship management table, the "action content" associated with the object "Sign B" and the judgment result "No Parking" is set to "No Parking in the Sign Area." If Sign B is a road sign indicating no parking, it is necessary to restrict the parking of the mobile body 700 in the area corresponding to a predetermined range centered on the no parking road sign in order to comply with traffic regulations. Therefore, in this embodiment, "No Parking in the Sign Area" is set as the "action content" when Sign B is a road sign indicating no parking. Furthermore, in the relationship management table, the "action content" associated with the object "Sign B" and the judgment result "No Vehicle Entry" is set to "Slow Down / Stop". If Sign B is a road sign indicating no vehicle entry, it is necessary to restrict the passage of the mobile body 700 in order to comply with traffic regulations. Therefore, in this embodiment, "Slow Down / Stop" is defined as the "action content" when Sign B is a road sign indicating no vehicle entry.

[0056] Figure 12 is a flowchart showing the flow of the mobile body motion processing. The mobile body motion processing is a process in which the mobile body 700 operates according to the result of the color determination processing by the color determination unit 605. In this embodiment, for example, if the mobile body motion processing is not being executed, the mobile body motion processing is started at predetermined time intervals. The predetermined time interval can be any time, but for example, it is 0.1 seconds. In the result output processing, ECU 701 determines whether or not it has received the IR image and color image transmitted from camera control unit 606 (see S109 in Figure 10) (S201). If the negative result continues (No in S201), the process in step 201 is repeated.

[0057] Furthermore, if the image is received by the ECU 701 (Yes in S201), the process proceeds to the next step. The ECU 701 determines whether or not there is an object requiring the mobile body 700 to act within a predetermined range in front of the camera 600 (S202). The predetermined range is, for example, a range shorter than the range in which the mobile body 700 can stop without colliding with an object if it applies emergency brakes. An object requiring the mobile body 700 to act is an object that needs to be detected in order to realize the automatic operation of the mobile body 700, such as a human or other animal. In the result output processing, the ECU 701 makes the determination in step 202 based on the recognition result from the recognition unit 604 transmitted from the camera control unit 606.

[0058] If an object exists within a predetermined range (Yes in S202), proceed to the next step. ECU 701 determines whether the object within the predetermined range is an object subject to color determination processing (S203). In the result output processing, ECU 701 makes the determination in step 203 based on the recognition result from the recognition unit 604 transmitted from the camera control unit 606 and the relationship management table.

[0059] If an object within a predetermined range is the target of the color determination process (Yes in S203), the process proceeds to the next step. The ECU 701 determines whether the color determination unit 605 was able to determine the color (S204). In the result output process, the ECU 701 compares the result of the color determination process transmitted from the camera control unit 606 with the "determination result" in the relationship management table. The ECU 701 then determines that the color determination unit 605 was able to determine the color if the referenced result is different from the "determination result" in the relationship management table. The ECU 701 also determines that the color determination unit 605 was not able to determine the color if the referenced result is "determination result" in the relationship management table.

[0060] If the color determination unit 605 is able to make a determination in the color determination process (Yes in S204), the ECU 701 selects the operation content of the mobile body 700 according to the determination result in the color determination process (S205). More specifically, in the result output process, the ECU 701 compares the result of the color determination process transmitted from the camera control unit 606 with the "determination result" in the relationship management table. Then, the ECU 701 selects the "operation content" associated with the "determination result" in the relationship management table that corresponds to the referenced result as the operation content of the mobile body 700. The "determination result" in the relationship management table that corresponds to the result referenced by the ECU 701 is the "determination result" associated with the "object" recognized by the recognition unit 604.

[0061] Furthermore, if the color determination unit 605 is unable to determine the color during the color determination process (No in S204), the ECU 701 selects an action for the mobile body 700 corresponding to the determination result of the color determination process (S206). More specifically, the ECU 701 selects an action for the mobile body 700 that is associated in the relationship management table with the "object" recognized by the recognition unit 604 and the determination result of "unable to determine" in the color determination process. The ECU 701 operates the mobile body 700 according to the selection made in step 205 or step 206 (S207). In some cases, objects within a predetermined range may not be subject to color determination processing (No in S203). In this case, the ECU 701 operates the mobile body 700 according to predetermined actions for each object. For example, this action could be an action to avoid collision between the mobile body 700 and the object. Furthermore, this action may differ depending on the distance calculated by the recognition unit 604 as the distance to the camera 600 object.

[0062] If no object exists within a predetermined range (No in S202), or after step 207, the process proceeds to the next step. The ECU 701 generates an image to be displayed on the display unit 703 (S208). The ECU 701 generates an image by superimposing the object detection result of the recognition unit 604 for the IR image, the result of the color determination processing, and information indicating the operating status of the moving body 700 onto the color image acquired in step 201. The ECU 701 displays the image generated in step 208 on the display unit 703 (S209). This allows the occupant of the mobile unit 700 to recognize the object detection results, the color determination processing results, and the operating status of the mobile unit 700. The ECU701 determines whether or not the next frame exists (S210). If the next frame exists (Yes in S210), the process from step 201 is repeated. If the next frame does not exist (No in S210), the mobile body operation process ends.

[0063] Figures 13 and 14 show examples of the operation of the mobile body 700. In the following example, a camera 600 is provided on the front of the mobile body 700, and the camera 600 is assumed to be imaging the area in front of the mobile body 700 in the direction of travel. First, we will describe an example of the operation of the mobile body 700 when a car is in front of it. When a car is in front of the mobile body 700, the color image generated by imaging in the result output processing will show the car 920, as shown in Figure 13(A). In addition, this color image will show fog 910 covering the car 920, and this fog 910 will make the car 920 difficult to see.

[0064] Furthermore, Figure 13(B) shows an IR image. The IR image shown in Figure 13(B) is an image generated by camera 600 through simultaneous imaging with the color image shown in Figure 13(A). This IR image also shows the recognition result of the recognition unit 604 in the result output processing, including a region image 921 indicating the area where the automobile 920 was detected, and information indicating the distance of the detected automobile 920 from camera 600. In the illustrated example, the text "40m" is shown as the distance from camera 600 to automobile 920.

[0065] Furthermore, in the result output processing, the color determination unit 605 performs color determination processing based on the recognition result of the recognition unit 604 shown in Figure 13(B). In this case, as shown in Figure 13(C), the color determination unit 605 displays the region image 921 superimposed on the color image, and identifies the display area of ​​the left brake lamp 922a and the display area of ​​the right brake lamp 922b as target areas from the region image 921. Then, the color determination unit 605 performs color determination processing on the identified target areas (see S108 in Figure 10). In the following, it is assumed that the color determination unit 605 has obtained a determination result of "brake lamp illuminated" through the color determination processing. In the color image, the coordinates for the region image 921, the left brake light 922a, and the right brake light 922b are the same as the coordinates for the positions of each image in the IR image.

[0066] In the mobile body operation processing, the ECU 701 controls the mobile body 700 using "Decelerate / Stop" as the operation content, which is associated with "Vehicle" corresponding to the automobile 920 in the relationship management table and "Brake light illumination," which is the result of the color determination processing. In addition, as shown in Figure 13(D), the ECU 701 displays a notification image 923 on the display unit 703, which includes the selected operation content and the recognition result of the recognition unit 604 (see S209 in Figure 12). The notification image 923 shows the recognition result of the recognition unit 604, the area image 921, and the operation content selected by the ECU 701. When the notification image 923 and the color image are displayed on the display unit 703, the occupant of the mobile body 700 recognizes that the vehicle 920 is located beyond the fog 910 and that the mobile body 700 will decelerate and stop as part of the automatic driving of the mobile body 700. In addition, the result of the color determination process for the color image shown in Figure 13(C) may be "brake lights off". In this case, the ECU 701 controls the mobile body 700 in the mobile body operation process, using "continued driving" as the operation content, which is associated with "vehicle" corresponding to the automobile 920 in the relationship management table and "brake lights off", which is the result of the color determination process.

[0067] In this way, the camera 600 can detect vehicles in front of the mobile unit 700 with high accuracy from IR images, even in conditions of poor visibility such as nighttime or bad weather. The camera 600 also determines the operating status of the vehicle, such as whether the brake lights are on, by performing color judgment processing on the color image. The mobile unit 700 then takes action according to the judgment result from the color judgment processing. This suppresses collisions with vehicles in front when the mobile unit 700 is driving autonomously.

[0068] Next, we will describe an example of the operation of the mobile body 700 when a traffic light is present in front of it. When a traffic light is present in front of the mobile body 700, the color image generated by imaging in the result output processing will show the traffic light 940, as shown in Figure 14(A). In addition, this color image will show fog 930 covering the traffic light 940, and this fog 930 will make the traffic light 940 difficult to see.

[0069] Furthermore, Figure 14(B) shows an IR image. The IR image shown in Figure 14(B) is an image generated by camera 600 through simultaneous imaging with the color image shown in Figure 14(A). This IR image also shows the recognition result of the recognition unit 604 in the result output processing, including a region image 941 indicating the area where the traffic light 940 was detected, and information indicating the distance of the detected traffic light 940 from camera 600. In the illustrated example, the text "50m" is shown as the distance from camera 600 to traffic light 940.

[0070] Furthermore, in the result output processing, the color determination unit 605 performs color determination processing based on the recognition result of the recognition unit 604 shown in Figure 14(B). In this case, as shown in Figure 14(C), the color determination unit 605 displays the region image 941 superimposed on the color image. The color determination unit 605 also identifies the display area of ​​the blue light 942a, the display area of ​​the yellow light 942b, and the display area of ​​the red light 942c as target areas from the region image 941. Then, the color determination unit 605 performs color determination processing on the identified target areas (see S108 in Figure 10). In the following, it is assumed that at the time the color image shown in Figure 14(C) was captured, the traffic light 940 was far from the camera 600, and in the color determination processing, the average value of the brightness of each color fell below the threshold, resulting in a "cannot determine" determination result being output to the color determination unit 605. In the color image, the coordinates for the region image 941, the blue light 942a, the yellow light 942b, and the red light 942c are the same as the coordinates for the positions of each image in the IR image.

[0071] In the mobile body operation processing, the ECU 701 controls the mobile body 700 using "Decelerate / Slow Speed ​​Driving" as the operation content, which is associated with "Traffic Light" corresponding to traffic light 940 in the relationship management table and "Unable to Determine" which is the result of the color determination process. The ECU 701 also overlays the notification image 943 shown in Figure 14(C) onto the color image. The notification image 943 is an image that notifies information indicating the recognition result of the recognition unit 604 and the operation content selected by the ECU 701. In the illustrated example, the notification image 943 shows the text "Traffic light ahead," which is the recognition result of the recognition unit 604, and the text "Decelerating," which is the operation content selected by the ECU 701.

[0072] Subsequently, as the mobile unit 700 continues to move, it approaches the traffic light 940. When the distance from the camera 600 to the traffic light 940 is 30m, the result output processing is started, and imaging, recognition of the traffic light 940 by the recognition unit 604, and color determination processing by the color determination unit 605 are performed. In the following, it is assumed that the color determination unit 605 has determined that the traffic light is "on" through the color determination processing.

[0073] In the mobile vehicle operation processing, the ECU 701 controls the mobile vehicle 700 using "Decelerate / Stop" as the operation content, which is associated with "Traffic Light" corresponding to traffic light 940 in the relationship management table and "Red light on," which is the result of the color determination processing. In addition, as shown in Figure 14(D), the ECU 701 displays a notification image 944, which includes the selected operation content and the recognition result of the recognition unit 604, on the display unit 703 (see S209 in Figure 12). The notification image 944 shows the recognition result of the recognition unit 604, the area image 941, and the operation content selected by the ECU 701. When the notification image 944 and the color image are displayed on the display unit 703, the occupant of the mobile vehicle 700 recognizes that there is a traffic light beyond the fog and that the mobile vehicle 700 will decelerate and stop as part of the automatic operation of the mobile vehicle 700. In addition, the result of the color determination process for the color image shown in Figure 14(D) may be "green light on". In this case, the ECU 701 controls the mobile body 700 in the mobile body operation process by setting the operation content to "continue driving" which is associated with "traffic light" corresponding to traffic light 940 in the relationship management table and "green light on", which is the result of the color determination process.

[0074] In this way, the camera 600 can detect traffic lights in front of the mobile vehicle 700 with high accuracy from IR images, even in conditions of poor visibility such as nighttime or bad weather. The camera 600 also determines the illumination status of the traffic lights by performing color judgment processing on the color image. The mobile vehicle 700 then operates according to the judgment result of the color judgment processing. This allows the mobile vehicle 700 to operate in response to the status of the traffic lights ahead when it is driving autonomously. Furthermore, as shown in Figures 14(C) and 14(D), if color determination is impossible because the object to be color-determined is far from the camera 600, the mobile unit 700 operates according to the actions appropriate to the inability to determine the color. Then, when the object to be color-determined approaches the camera 600 and color determination becomes possible, the mobile unit 700 operates according to the actions appropriate to the result of the color determination.

[0075] The actions of the mobile body 700 in response to the color determination process are not limited to the examples above. The actions of the mobile body 700 in response to the color determination process may include acceleration, left turns, right turns, lane changes, etc. Furthermore, the objects subject to color detection processing are not limited to vehicles, traffic lights, and certain road signs.

[0076] Furthermore, although this embodiment describes the photoelectric conversion element 100 of the camera 600 as a SPAD sensor, it is not limited to this. The photoelectric conversion element 100 may also be a CMOS image sensor equipped with an RGB filter 31 and an IR filter 32. In other words, the first image sensor and the second image sensor may be either a SPAD sensor or a CMOS image sensor. Furthermore, the image sensor equipped with the RGB filter 31 and the image sensor equipped with the IR filter 32 may be separate cameras. In other words, the camera that captures color images and the camera that captures IR images may be different cameras.

[0077] Alternatively, so-called range gate control may be performed, in which the IR light-emitting section 501 of the IR light emitter 500 emits pulsed light at a predetermined period, and the image sensor of the camera 600 is exposed at a predetermined timing corresponding to the distance from the camera 600 to the subject. In other words, the photoelectric conversion element 100 of the camera 600 may be exposed in synchronization with the light emission period of the IR light-emitting section 501.

[0078] Furthermore, although this embodiment describes the color determination unit 605 determining whether or not color determination processing is necessary for an object recognized by the recognition unit 604, it is not limited to this. The recognition unit 604 may also determine whether or not color determination processing by the color determination unit 605 is necessary for the recognized object. Therefore, the recognition processing by the recognition unit 604 can also be considered as processing to recognize the type of object or whether or not color determination processing is necessary. Furthermore, in this embodiment, the color determination unit 605 selects a color determination process to be executed according to the object recognized by the recognition unit 604, but this is not limited to this. The recognition unit 604 may also select a color determination process to be executed by the color determination unit 605 according to the object it has recognized. Therefore, the recognition unit 604 can also be considered as a color determination selection unit that selects a color determination process to be performed based on the object detected by the recognition unit 604. Moreover, the processing steps by the recognition unit 604 and the color determination unit 605 can also be considered as a color determination selection step that selects a color determination process to be performed based on the object detected in the recognition step.

[0079] As described above, in this embodiment, the color determination unit 605 performs the color determination process selected by the color determination area selection unit on the target area selected by the color determination area selection unit, using a color image. In this case, the color of the object is determined using a method appropriate to the object displayed in the image.

[0080] Furthermore, in this embodiment, the first image sensor and the second image sensor are CMOS image sensors. In this case, a CMOS image sensor enables color determination of objects displayed in the image using a method appropriate to the object.

[0081] Furthermore, in this embodiment, the first image sensor and the second image sensor are SPAD sensors. In this case, the SPAD sensor enables color determination of objects in the image using a method appropriate to the object being displayed.

[0082] Furthermore, in this embodiment, the recognition process is the process of recognizing the type of object or whether or not a color determination process is necessary. In this case, color determination of the object is performed using a method appropriate to the type of object displayed in the image, or the need for color determination processing is switched depending on the object displayed in the image.

[0083] Furthermore, in this embodiment, the imaging device further includes an IR light-emitting unit 501 that emits infrared light. In this case, it becomes unnecessary to use a separate infrared light-emitting unit in addition to the imaging device to acquire IR images.

[0084] Furthermore, in this embodiment, the first image sensor performs exposure in synchronization with the light emission period of the IR light emission unit 501. In this case, even in poor visibility conditions, the subject being photographed can be captured more clearly.

[0085] In this embodiment, the color determination unit 605 calculates the contrast value in the color image of the object detected by the recognition unit 604, and determines whether or not color determination is possible based on the calculated contrast value. Examples of contrast values ​​include the average contrast of the color image of the object. In this case, compared to a configuration where color determination is performed regardless of the contrast value in the object's color image, the output of color determination results with low accuracy is suppressed.

[0086] Furthermore, in this embodiment, the imaging device includes an ECU 701 that determines the operation of the mobile body 700 on which the imaging device is mounted, according to the recognition processing result by the recognition unit 604 and the color determination result by the color determination unit 605. In this case, the movement of the mobile body 700 is realized in accordance with the object displayed in the image.

[0087] Furthermore, in this embodiment, the ECU 701 determines the operation of the mobile body 700 in cases where the color determination result from the color determination unit 605 is color determination possible and cases where color determination is not possible. In this case, the possibility that the automatic operation of the mobile unit 700 will not be realized depending on the color determination result is prevented.

[0088] In this embodiment, the color determination unit 605 performs color determination processing on a region of the color image that is the same as the region of the IR image on which the object detected by the recognition unit 604 is displayed. In this case, it becomes unnecessary to convert the area of ​​the IR image in which the object detected by the recognition unit 604 is displayed into an area corresponding to the color image.

[0089] Furthermore, in this embodiment, when the recognition unit 604 recognizes that the object displayed in the IR image is a vehicle, the color determination area selection unit selects the left or right brake lamp area of ​​the vehicle as the target area (see Figure 8). The color determination area selection unit also selects a process to determine the color of the left or right brake lamp area of ​​the vehicle based on the intensity of red. The intensity of red refers to the luminance of red. The color determination unit 605 then determines that the vehicle's brake lamps are lit if the intensity of red exceeds a threshold, and determines that the vehicle's brake lamps are not lit if the intensity of red falls below the threshold. In this case, the accuracy of determining whether a vehicle's brake lights are on or off is improved compared to a configuration where colors other than red are included in the color determination.

[0090] Furthermore, in this embodiment, when the recognition unit 604 recognizes an object displayed in the IR image as a traffic light, the color determination area selection unit selects the illuminated area of ​​the traffic light as the target area (see Figure 8). The color determination area selection unit also selects a process to determine the color of the illuminated area of ​​the traffic light based on the intensity of blue, the intensity of yellow, or the intensity of red. The intensity of blue, the intensity of yellow, and the intensity of red refer to the brightness of blue, the brightness of yellow, and the brightness of red, respectively. The color determination unit 605 then determines that the traffic light is illuminated blue if the intensity of blue exceeds the threshold, the traffic light is illuminated yellow if the intensity of yellow exceeds the threshold, and the traffic light is illuminated red if the intensity of red exceeds the threshold. In this case, the accuracy of determining which color the traffic light is illuminated by is improved compared to a configuration where colors other than blue, yellow, and red are included in the color determination.

[0091] Furthermore, in this embodiment, when the recognition unit 604 recognizes that the object displayed in the IR image is sign A, the color determination area selection unit selects the arrow area of ​​sign A as the target area (see Figure 8). The color determination area selection unit also selects a process to determine the color of the arrow area based on the intensity of blue. The intensity of blue refers to the brightness of blue. The color determination unit 605 then determines that if the intensity of blue exceeds a threshold, it is a road sign indicating that vehicles can turn left, and if the intensity of blue falls below the threshold, it is a road sign indicating that vehicles are traveling in one direction. In this case, compared to a configuration where colors other than blue are included in the color determination, the accuracy of determining whether sign A is a road sign indicating that vehicles can turn left or a road sign indicating that vehicles are one-way is improved.

[0092] Furthermore, in this embodiment, when the recognition unit 604 recognizes that the object displayed in the IR image is sign B, the color determination area selection unit selects the area inside the circle of sign B as the target area (see Figure 8). The color determination area selection unit also selects a process to determine the color of the area inside the circle based on the intensity of blue. The intensity of blue refers to the brightness of blue. The color determination unit 605 then determines that it is a road sign indicating no parking if the intensity of blue exceeds a threshold, and determines that it is a road sign indicating no vehicle access if the intensity of blue falls below the threshold. In this case, compared to a configuration where colors other than blue are included in the color determination, the accuracy of determining whether sign B is a road sign indicating no parking or a road closure is improved.

[0093] In this embodiment, the color determination selection unit selects a color that is the subject of color determination by the color determination unit 605, based on the object detected by the recognition unit 604. In this case, compared to a configuration where the color of the object being color-determined does not change regardless of the object detected by the recognition unit 604, the accuracy of color determination of objects displayed in a color image is improved.

[0094] Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of this embodiment to an imaging device via a network or storage medium, and by having one or more processors in the imaging device's computer read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0095] This embodiment includes the following configuration. (Composition 1) A first image sensor having a photoelectric conversion unit capable of receiving non-visible light, A second image sensor having a photoelectric conversion unit capable of receiving visible light, A recognition processing unit that performs object recognition processing using a first image generated by the first image sensor, A color determination region selection unit selects a region in which to perform color determination processing based on an object detected by the recognition processing unit, A color determination selection unit that selects a color determination process to be performed based on the object detected by the recognition processing unit, A color determination unit that performs the color determination process selected by the color determination selection unit on the region selected by the color determination selection unit, using a second image generated by the second image sensor, An imaging device characterized by comprising: (Configuration 2) The imaging apparatus according to configuration 1, characterized in that the first image sensor and the second image sensor are CMOS image sensors. (Composition 3) The imaging apparatus according to configuration 1 or 2, characterized in that the first image sensor and the second image sensor are SPAD sensors. (Composition 4) The imaging apparatus according to any one of configurations 1 to 3, characterized in that the recognition process is a process for recognizing the type of object or whether the color determination process is necessary. (Composition 5) The aforementioned invisible light is infrared light. An imaging apparatus according to any one of configurations 1 to 4, further comprising a light-emitting unit that emits infrared light. (Composition 6) The imaging apparatus according to configuration 5, characterized in that the first image sensor performs exposure in synchronization with the light emission period of the light emission unit. (Composition 7) The imaging apparatus according to any one of configurations 1 to 6, characterized in that the color determination unit calculates the contrast value in the second image of the object detected by the recognition processing unit, and determines whether or not color determination is possible from the calculated contrast value. (Composition 8) The imaging device according to any one of configurations 1 to 7, further comprising an operation determination unit that determines the operation of a mobile body on which the imaging device is mounted, in accordance with the recognition processing result by the recognition processing unit and the color determination result by the color determination unit. (Composition 9) The imaging apparatus according to configuration 8, characterized in that the operation determination unit determines the operation of the moving body in cases where the color determination result by the color determination unit is color-determinable and cases where color determination is not possible. (Composition 10) The imaging apparatus according to any one of configurations 1 to 9, characterized in that the color determination unit performs the color determination process on a region of the second image that is the same as the region of the first image on which the object detected by the recognition processing unit is displayed. (Composition 11) The recognition processing unit recognizes that the object is a vehicle, The color determination area selection unit selects the left or right brake lamp area of ​​the vehicle. The color determination selection unit selects a process to determine the color of the left or right brake lamp area of ​​the vehicle based on the intensity of red. The imaging device according to any one of configurations 1 to 10, characterized in that the color determination unit determines that the vehicle's brake lights are illuminated when the intensity of the red exceeds a threshold, and determines that the vehicle's brake lights are not illuminated when the intensity of the red falls below the threshold. (Composition 12) The recognition processing unit recognizes the object as a traffic light, The color determination area selection unit selects the illuminated area of ​​the traffic light, The color determination selection unit selects a process to determine the color of the illuminated area of ​​the traffic light based on the intensity of blue, yellow, or red. The imaging device according to any one of configurations 1 to 11, characterized in that the color determination unit determines that the blue light of the traffic light is lit when the intensity of the blue light exceeds a threshold, determines that the yellow light of the traffic light is lit when the intensity of the yellow light exceeds a threshold, and determines that the red light of the traffic light is lit when the intensity of the red light exceeds a threshold. (Composition 13) The recognition processing unit recognizes that the object is a road sign with an arrow indicated, The color determination area selection unit selects the arrow area in the road sign, The color determination selection unit selects a process to determine the color of the arrow region based on the intensity of blue, The imaging device according to any one of configurations 1 to 12, characterized in that the color determination unit determines that the road sign indicates that a vehicle can turn left when the intensity of the blue exceeds a threshold, and determines that the road sign indicates that a vehicle is traveling in one direction when the intensity of the blue falls below the threshold. (Composition 14) The recognition processing unit recognizes that the object is a road sign with a diagonal line inside a circle, The color determination area selection unit selects the inner area of ​​the circle in the road sign, The color determination selection unit selects a process to determine the color of the inner region of the circle based on the intensity of blue, The imaging device according to any one of configurations 1 to 13, characterized in that the color determination unit determines that the road sign indicates no parking when the intensity of the blue exceeds a threshold, and determines that the road sign indicates no vehicle access when the intensity of the blue falls below the threshold. (Composition 15) The imaging apparatus according to any one of configurations 1 to 14, characterized in that the color determination selection unit selects a color that is the subject of color determination by the color determination unit based on the object detected by the recognition processing unit. (Composition 16) A processing method for an imaging device comprising a first image sensor having a photoelectric conversion unit capable of receiving non-visible light, and a second image sensor having a photoelectric conversion unit capable of receiving visible light, A recognition step of performing object recognition processing using a first image generated by the first image sensor, A color determination region selection step in which a region is selected to perform color determination processing based on the object detected in the recognition step, A color determination selection step, which selects a color determination process to be performed based on the object detected in the recognition step, A color determination step in which the color determination process selected in the color determination selection step is performed on the region selected in the color determination selection step using a second image generated by the second image sensor, A processing method for an imaging device, characterized by having the following features. (Composition 17) A program to cause a computer to function as an imaging device as described in any of configurations 1 to 15.

[0096] Although the present invention has been described in detail above based on its preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible in accordance with the spirit of the invention, and these modifications are not excluded from the scope of the present invention. Furthermore, some of the above embodiments may be combined as appropriate. [Explanation of Symbols]

[0097] 500...IR emitter, 600...camera, 700...mobile device

Claims

1. A first image sensor having a photoelectric conversion unit capable of receiving non-visible light, A second image sensor having a photoelectric conversion unit capable of receiving visible light, A recognition processing unit that performs object recognition processing using a first image generated by the first image sensor, A color determination region selection unit selects a region in which to perform color determination processing based on an object detected by the recognition processing unit, A color determination selection unit that selects a color determination process to be performed based on the object detected by the recognition processing unit, A color determination unit that performs the color determination process selected by the color determination selection unit on the region selected by the color determination selection unit, using a second image generated by the second image sensor, An imaging device characterized by comprising:

2. The imaging apparatus according to claim 1, characterized in that the first image sensor and the second image sensor are CMOS image sensors.

3. The imaging apparatus according to claim 1, characterized in that the first image sensor and the second image sensor are SPAD sensors.

4. The imaging apparatus according to claim 1, characterized in that the recognition process is a process for recognizing the type of object or whether the color determination process is necessary.

5. The aforementioned invisible light is infrared light. The imaging apparatus according to claim 1, further comprising a light-emitting unit that emits infrared light.

6. The imaging apparatus according to claim 5, characterized in that the first image sensor performs exposure in synchronization with the light emission period of the light emission unit.

7. The imaging apparatus according to claim 1, characterized in that the color determination unit calculates the contrast value in the second image of the object detected by the recognition processing unit, and determines whether or not color determination is possible from the calculated contrast value.

8. The imaging device according to claim 1, further comprising an operation determination unit that determines the operation of a mobile body on which the imaging device is mounted, according to the recognition processing result by the recognition processing unit and the color determination result by the color determination unit.

9. The imaging apparatus according to claim 8, characterized in that the operation determination unit determines the operation of the moving body in cases where the color determination result by the color determination unit is color-determinable and cases where color determination is not possible.

10. The imaging apparatus according to claim 1, characterized in that the color determination unit performs the color determination process on an area of ​​the second image which is the same as the area of ​​the first image on which the object detected by the recognition processing unit is displayed.

11. The recognition processing unit recognizes that the object is a vehicle, The color determination area selection unit selects the left or right brake lamp area of ​​the vehicle. The color determination selection unit selects a process to determine the color of the left or right brake lamp area of ​​the vehicle based on the intensity of red. The imaging device according to claim 1, characterized in that the color determination unit determines that the vehicle's brake lights are illuminated when the intensity of the red exceeds a threshold, and determines that the vehicle's brake lights are not illuminated when the intensity of the red falls below the threshold.

12. The recognition processing unit recognizes the object as a traffic light, The color determination area selection unit selects the illuminated area of ​​the traffic light, The color determination selection unit selects a process to determine the color of the illuminated area of ​​the traffic light based on the intensity of blue, yellow, or red. The imaging device according to claim 1, characterized in that the color determination unit determines that the blue light of the traffic light is lit when the intensity of the blue light exceeds a threshold, determines that the yellow light of the traffic light is lit when the intensity of the yellow light exceeds a threshold, and determines that the red light of the traffic light is lit when the intensity of the red light exceeds a threshold.

13. The recognition processing unit recognizes that the object is a road sign with an arrow indicated, The color determination area selection unit selects the arrow area in the road sign, The color determination selection unit selects a process to determine the color of the arrow region based on the intensity of blue, The imaging device according to claim 1, characterized in that the color determination unit determines that the road sign indicates that a vehicle can turn left when the intensity of the blue exceeds a threshold, and determines that the road sign indicates that a vehicle is one-way when the intensity of the blue falls below the threshold.

14. The recognition processing unit recognizes that the object is a road sign with a diagonal line inside a circle, The color determination area selection unit selects the inner area of ​​the circle in the road sign, The color determination selection unit selects a process to determine the color of the inner region of the circle based on the intensity of blue, The imaging device according to claim 1, characterized in that the color determination unit determines that the road sign indicates no parking when the intensity of the blue exceeds a threshold, and determines that the road sign indicates no vehicle access when the intensity of the blue falls below the threshold.

15. The imaging apparatus according to claim 1, characterized in that the color determination selection unit selects a color that is the subject of color determination by the color determination unit based on the object detected by the recognition processing unit.

16. A processing method for an imaging device comprising a first image sensor having a photoelectric conversion unit capable of receiving non-visible light, and a second image sensor having a photoelectric conversion unit capable of receiving visible light, A recognition step of performing object recognition processing using a first image generated by the first image sensor, A color determination region selection step in which a region is selected to perform color determination processing based on the object detected in the recognition step, A color determination selection step, which selects a color determination process to be performed based on the object detected in the recognition step, A color determination step in which the color determination process selected in the color determination selection step is performed on the region selected in the color determination selection step using a second image generated by the second image sensor, A processing method for an imaging device, characterized by having the following features.

17. A program for causing a computer to function as an imaging device according to any one of claims 1 to 15.

Images