Imaging apparatus, processing method of imaging apparatus, recording medium, and computer program product

Abstract

The present disclosure relates to an image pickup apparatus, a processing method of an image pickup apparatus, a recording medium, and a computer program product. The image pickup apparatus includes a first image pickup element configured to have a photoelectric conversion unit capable of receiving invisible light, a second image pickup element configured to have a photoelectric conversion unit capable of receiving visible light, at least one memory storing an instruction, and at least one processor executing the stored instruction, thereby causing the image pickup apparatus to perform object recognition processing using a first image generated by the first image pickup element, select a region to be subjected to color determination processing based on an object detected by the object recognition processing, select color determination processing to be performed based on the object detected by the object recognition processing, and execute the selected color determination processing on the selected region using a second image generated by the second image pickup element.

Description

Technical Field

[0001] This disclosure relates to camera equipment, and methods and procedures for processing camera equipment. Background Technology

[0002] Traditionally, RGB cameras exist, which determine the color of objects displayed in captured images. The results of this color determination by the RGB camera are used, for example, to control operable devices (such as autonomous driving of vehicles). Furthermore, to improve the accuracy of color determination of objects in images captured under poor visibility conditions such as nighttime or inclement weather, RGB-IR cameras that simultaneously capture both RGB (visible light) and IR (invisible light) images are also known. Japanese Patent Application Publication No. 2010-250710 discloses a method that simultaneously acquires color and monochrome (IR) images using a color imaging element and a monochrome imaging element, calculates the vehicle body area based on the license plate position identified using the IR image, and determines the color of the vehicle body using the color image.

[0003] However, according to the technology described in Japanese Patent Application Publication No. 2010-250710, since a single color determination method is used to determine the color of an object regardless of the object to be determined, it is not possible to determine the color of an object using a method corresponding to the object shown in the image. Summary of the Invention

[0004] The purpose of this disclosure is to enable the determination of the color of an object using a method corresponding to the object displayed in the image.

[0005] To address the aforementioned problems, the camera device of this disclosure includes: a first camera element configured to have a photoelectric conversion unit capable of receiving invisible light; a second camera element configured to have a photoelectric conversion unit capable of receiving visible light; at least one memory storing instructions; and at least one processor executing the stored instructions to cause the camera device to: perform object recognition processing using a first image generated by the first camera element; select an area to be color-determined based on an object detected by the object recognition processing; select a color-determined area to be performed based on an object detected by the object recognition processing; and perform the selected color-determined processing on the selected area using a second image generated by the second camera element.

[0006] The features of this disclosure will become apparent from the following description of embodiments with reference to the accompanying drawings. The following description of the embodiments is illustrated by way of example. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating an example of the configuration of a photoelectric conversion element.

[0008] Figure 2 This is a diagram illustrating an example of the configuration of a sensor substrate.

[0009] Figure 3 This is a diagram illustrating an example configuration of color filters in a pixel.

[0010] Figure 4 This is a diagram illustrating an example of the configuration of a circuit board.

[0011] Figure 5 This is a diagram illustrating the equivalent circuit of a pixel and signal processing circuit.

[0012] Figure 6 This is a diagram illustrating the relationship between the operation of the APD and its output signal.

[0013] Figure 7 This is a diagram illustrating the functional arrangement of an IR emitter, camera, and moving body.

[0014] Figure 8 This is a diagram illustrating the object management table.

[0015] Figures 9A to 9D It is a diagram that illustrates a specific example of the instance object.

[0016] Figure 10 This is a flowchart illustrating the process of outputting results.

[0017] Figure 11 This is a diagram illustrating a relational management table.

[0018] Figure 12 This is a flowchart illustrating the process of handling moving objects.

[0019] Figures 13A to 13D This is a diagram illustrating an example of the operation of a moving object.

[0020] Figures 14A to 14D This is a diagram illustrating an example of the operation of a moving object. Detailed Implementation

[0021] In this embodiment, the camera device captures an image of an object (such as an item) and determines the color of the object displayed in the captured image. More specifically, the camera device in this embodiment detects objects displayed in the image and uses a method corresponding to the detected objects to determine the color. Furthermore, in this embodiment, a moving body equipped with the camera device operates based on the color determination result. Examples of moving bodies include vehicles, aircraft, trains, ships, drones, AGVs, and robots.

[0022] In the following description, embodiments of the present disclosure will be illustrated with reference to the accompanying drawings.

[0023] Figure 1 This is a diagram illustrating an example configuration of a photoelectric conversion element 100 in a camera. The photoelectric conversion element 100 has a laminated sensor substrate 11 and a circuit substrate 21. The sensor substrate 11 and the circuit substrate 21 are electrically connected to each other.

[0024] The sensor substrate 11 has a pixel area 12.

[0025] The circuit board 21 has a circuit region 22 for processing the signal detected in the pixel region 12.

[0026] Note that, although in Figure 1 In the illustrated example, the photoelectric conversion element 100 has been described as a photoelectric conversion device having a laminated structure, but this disclosure is not limited thereto. The photoelectric conversion element 100 may also be referred to as a non-laminated structure, wherein the structural portions of the sensor substrate 11 and the structural portions of the circuit substrate 21 are included in a common semiconductor layer.

[0027] Figure 2 This is a diagram illustrating an example configuration of the sensor substrate 11. The pixel region 12 of the sensor substrate 11 has pixels 101. A plurality of pixels 101 are arranged two-dimensionally in the pixel region 12. Each pixel 101 includes a photoelectric conversion unit 102, which includes an avalanche photodiode (hereinafter referred to as an APD).

[0028] The photoelectric conversion unit 102 is used as a sensor unit to generate pulses at a frequency corresponding to the frequency of the received photons.

[0029] Note that the number of rows and columns in which pixels 101 are arranged in pixel region 12 can be any number.

[0030] Figure 3 This is a diagram illustrating an example configuration of the color filter 30 in pixel 101. The color filter 30 includes an RGB filter 31 that transmits visible light and an IR filter 32 that transmits invisible light.

[0031] The RGB filter 31 includes an R filter 31R that transmits red wavelength light, a B filter 31B that transmits blue wavelength light, and a G filter 31G that transmits green wavelength light.

[0032] IR filter 32 is a filter for transmitting infrared (IR) wavelength light in the infrared region.

[0033] In pixel region 12, the color filter 30 included in each pixel 101 consists of one of R filter 31R, B filter 31B, G filter 31G and IR filter 32.

[0034] In this embodiment, as Figure 3 As illustrated, in pixel region 12, there is an alternating arrangement of columns of B filters 31B and G filters 31G, and columns of columns of IR filters 32 and R filters 31R. However, the arrangement of RGB filters 31 and IR filters 32 in pixel region 12 may differ from the illustrated example.

[0035] Therefore, the photoelectric conversion unit 102 of this embodiment receives both invisible light and visible light. Thus, the photoelectric conversion element 100 can also be considered as a first imaging element having a photoelectric conversion unit 102 capable of receiving invisible light. Furthermore, the photoelectric conversion element 100 can also be considered as a second imaging element having a photoelectric conversion unit 102 capable of receiving visible light.

[0036] Figure 4 This is a diagram illustrating an example of the configuration of the circuit board 21. The circuit board 21 includes a signal processing circuit 103, a readout circuit 112, a control pulse generation unit 115, a horizontal scanning circuit 111, a vertical signal line 113, a vertical scanning circuit 110, and an output circuit 114. The signal processing circuit 103 processes the charge that has been photoelectrically converted in the photoelectric conversion unit 102.

[0037] The vertical scanning circuit 110 receives control pulses supplied from the control pulse generation unit 115 and directs them to the vertical scanning circuit. Figure 2 Multiple pixels 101 arranged in the left and right directions are sequentially supplied with control pulses. Logic circuits such as shift registers and address decoders are used in the vertical scanning circuit 110.

[0038] The signal processing circuit 103 is provided for each photoelectric conversion unit 102 and processes the signals output from the photoelectric conversion unit 102. The signal processing circuit 103 is provided with counters and memory, etc., and the digital values ​​are stored in the memory.

[0039] The horizontal scanning circuit 111 inputs control pulses to the signal processing circuit 103 to read signals from the memory of the pixel 101 that holds digital signals.

[0040] Vertical signal line 113 is a signal line through which the signal output from pixel 101 selected by vertical scanning circuit 110 passes. The signal passing through vertical signal line 113 is output to the outside of photoelectric conversion element 100 via readout circuit 112 and output circuit 114. Readout circuit 112 incorporates multiple buffers connected to vertical signal line 113.

[0041] like Figure 2 and Figure 4As illustrated, multiple signal processing circuits 103 are disposed in the region overlapping with pixel region 12 in the front-back direction of the sheet surface. Furthermore, the vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, and control pulse generation unit 115 are configured to overlap with sensor substrate 11 in the front-back direction of the sheet surface, but not with pixel region 12. That is, sensor substrate 11 has pixel region 12 and non-pixel region different from pixel region 12. The vertical scanning circuit 110, horizontal scanning circuit 111, readout circuit 112, output circuit 114, and control pulse generation unit 115 are configured to overlap with the non-pixel region of sensor substrate 11 in the front-back direction of the sheet surface.

[0042] It should be noted that the arrangement of the vertical signal line 113, the readout circuit 112, and the output circuit 114 is not limited to... Figure 4 The illustrated example is shown below. For instance, the vertical signal line 113 can be configured to extend in the left-right direction of the figure, and the readout circuit 112 can be connected to the front end of the vertical signal line 113. Furthermore, it is not necessary to provide signal processing circuits 103 for each photoelectric conversion unit 102. A single signal processing circuit 103 can sequentially process signals from multiple photoelectric conversion units 102.

[0043] In the circuit board 21 of this embodiment, signal processing circuits 103 are provided for each of the R filter 31R, G filter 31G, B filter 31B, and IR filter 32, and each signal processing circuit 103 processes the light that has been transmitted through the corresponding color filter 30. It should be noted that the light that has been transmitted through the R filter 31R is referred to as the R signal below. Similarly, the light that has been transmitted through the G filter 31G is referred to as the G signal below. The light that has been transmitted through the B filter 31B is referred to as the B signal below. Finally, the light that has been transmitted through the IR filter 32 is referred to as the IR signal below. The R signal, G signal, B signal, and IR signal are all signals obtained after photoelectric conversion. The digital values ​​obtained after photoelectric conversion, corresponding to the target signals from the R signal, G signal, B signal, and IR signal, are stored in the memory of the signal processing circuit 103.

[0044] Figure 5 This is a diagram illustrating the equivalent circuit of pixel 101 and signal processing circuit 103.

[0045] The APD 201 included in the photoelectric conversion unit 102 generates charge pairs corresponding to the incident light through photoelectric conversion. One of the two nodes of the APD 201 is connected to a power supply line supplied with a driving voltage VL (first voltage). The other node of the APD 201 is connected to a power supply line supplied with a driving voltage VH (second voltage) that is higher than the driving voltage VL.

[0046] exist Figure 5 In the illustrated example, one node of APD 201 is the anode, and the other node is the cathode. A reverse bias voltage is supplied to both the anode and cathode of APD 201 to enable avalanche multiplication operation. The charge generated by the incident light undergoes avalanche multiplication, and the avalanche current is generated by the state in which this voltage is supplied.

[0047] It should be noted that the APD 201 has Geiger mode and linear mode. In Geiger mode, with a reverse bias voltage supplied, the APD 201 operates with a voltage difference between the anode and cathode greater than the breakdown voltage. In linear mode, the APD 201 operates with a voltage difference between the anode and cathode close to or below the breakdown voltage. In the following text, the APD 201 operating in Geiger mode may be referred to as a SPAD. In a SPAD, for example, the drive voltage VL (first voltage) is -30V, and the drive voltage VH (second voltage) is 1V.

[0048] The signal processing circuit 103 includes a quenching element 202, a waveform shaping unit 210, a counter circuit 211, and a memory circuit 212. The quenching element 202 is connected to a power supply line supplied with a drive voltage VH and is connected to one of the nodes of the anode and cathode of the APD 201.

[0049] Quenching element 202 acts as a load circuit (quenching circuit) during signal multiplication caused by avalanche multiplication, suppressing the voltage supplied to APD 201 and thereby suppressing avalanche multiplication (quenching operation). Quenching element 202 also restores the voltage supplied to APD 201 to the drive voltage VH (recharge operation) by allowing current flow corresponding to the amount of voltage drop during quenching operation.

[0050] Waveform shaping unit 210 shapes the voltage change at the cathode of APD 201 obtained during photon detection and outputs a pulse signal. An example of waveform shaping unit 210 includes an inverter circuit. Although in Figure 5 In the illustrated example, an inverter is used as waveform shaping unit 210, but a circuit with multiple inverters connected in series can be used as waveform shaping unit 210, or any other circuit with waveform shaping effect can be used as waveform shaping unit 210.

[0051] The counter circuit 211 counts the number of pulses output from the waveform shaping unit 210 and holds the count value. When a control pulse RES is supplied via the drive line 213, the signal held in the counter circuit 211 is reset. Here, the counter circuit 211 generates a signal based on the difference between the count value at the beginning of the accumulation period and the count value at the end of the accumulation period.

[0052] The memory circuit 212 is supplied with power from the vertical scan circuit 110 via drive line 214 (see [link]). Figure 4 The control pulse SEL is applied to the counter circuit 211, and the memory circuit 212 switches between the connected and disconnected states of the counter circuit 211 and the vertical signal line 113. The memory circuit 212 serves as a memory for temporarily storing the count value of the counter, and outputs the output signal from the counter circuit 211 of the pixel 101 to the vertical signal line 113.

[0053] Note that the electrical connection can be switched on and off by providing a switch, such as a transistor, between the quenching element 202 and the APD 201 and / or between the photoelectric conversion unit 102 and the signal processing circuit 103. Additionally, the supply of the drive voltage VH and drive voltage VL to the photoelectric conversion unit 102 can be electrically switched by a switch, such as a transistor.

[0054] Figure 6 This is a diagram illustrating the relationship between the operation and output signals of the APD 201.

[0055] exist Figure 6 In this diagram, the input side of the waveform shaping unit 210 is represented as node A, and the output side of the waveform shaping unit 210 is represented as node B. Between time t0 and time t1, a potential difference of VH-VL is applied to APD 201. When a photon is incident on APD 201 at time t1, avalanche multiplication occurs in APD 201, the avalanche multiplication current flows through the quenching element 202, and the voltage at node A drops. Subsequently, as the voltage drop further increases and the potential difference applied to APD 201 decreases, the avalanche multiplication of APD 201 stops at time t2, and the voltage level at node A does not drop to a value equal to or below a certain value. Subsequently, between time t2 and time t3, a current compensating for the voltage drop from the driving voltage VL flows to node A, and at time t3, node A stabilizes at its original potential level. At this time, the portion of the output waveform at node A that exceeds a certain threshold is shaped by the waveform shaping unit 210 and output as a pulse signal at node B.

[0056] Figure 7This is a diagram illustrating the functional arrangement of the IR emitter 500, camera 600, and moving body 700.

[0057] The IR emitter 500, camera 600, and moving body 700 each implement the various functions described below by causing a computer (not shown) to execute a computer program stored in a memory (not shown) used as a storage medium. However, some or all of the functions of the IR emitter 500, camera 600, and moving body 700 can be implemented in hardware. Examples of hardware include application-specific integrated circuits (ASICs), processors (reconfigurable processors, DSPs), etc.

[0058] The camera 600 includes the aforementioned photoelectric conversion element 100. The camera 600 also includes an imaging optical system 601, an image processing unit 603, a recognition unit 604, a color determination unit 605, a camera control unit 606, a storage unit 607, and a communication unit 608.

[0059] The image processing unit 603 processes the image signal acquired by the photoelectric conversion element 100 and generates a final image signal. Examples of processing performed by the image processing unit 603 include image processing such as black level correction, gamma curve adjustment, noise reduction, digital gain adjustment, demosaicing, and data compression. The image signal output from the photoelectric conversion element 100 passes through the color filter 30 and is converted into, for example, R, G, B, and IR signals. The image processing unit 603 generates a color image by performing demosaicing or similar processing on the R, G, and B signals. The image processing unit 603 can also perform processing such as white balance correction and color conversion. Additionally, the image processing unit 603 generates an IR image using the IR signal. The IR image has no color information and is generated as a monochrome image. It should be noted that the image processing unit 603 can perform different image processing for color image generation and IR image generation. Note that the IR image is an example of a first image. The color image is an example of a second image.

[0060] In addition, the image processing unit 603 outputs the generated image signal to the recognition unit 604, the color determination unit 605, the camera control unit 606, and the ECU (electronic control unit) 701 of the moving body 700, which will be described below. At this time, the image signal output from the image processing unit 603 includes a color image and an IR image.

[0061] The recognition unit 604, used as an example of a recognition processing unit, performs image recognition on an image signal to identify objects displayed in the image. Examples of objects recognized by the recognition unit 604 include animals such as people, moving objects such as vehicles, traffic lights, and signs such as road signs. The recognition unit 604 performs image recognition using, for example, deep learning. Examples of deep learning include YOLO (You Only Look Once), SSD (Single Shot Multiple Box Detector), Faster R-CNN (Region Convolutional Neural Network), Fast R-CNN, and R-CNN.

[0062] In this embodiment, the recognition unit 604 performs image recognition using IR images. In this case, compared to a configuration using color images, the object is recognized with higher accuracy, even when shooting in poor visibility conditions such as at night or in inclement weather, as the object's outline becomes easier to capture. This is because IR images are generated by photographing with infrared light, which has a longer wavelength than visible light, and is less likely to be scattered by water droplets in the air, allowing distant objects to be clearly photographed without interference from fog. Additionally, the recognition unit 604 calculates the distance from the camera 600 to the recognized object. Examples of distance measurement methods performed by the recognition unit 604 include methods that estimate distance using deep learning. As an example, the recognition unit 604 can calculate the distance by using deep learning to analyze information such as blur in the image of the detected object. Alternatively, as another example, the recognition unit 604 can perform distance measurement by using the camera 600 as a stereo camera and applying the principles of triangulation. The recognition unit 604 outputs information indicating the type of the recognized object, coordinates indicating the object's position in the image, and distance from the camera 600 to the ECU 701 of the moving body 700 as recognition results to the color determination unit 605 and the ECU 701 of the moving body 700. Note that the process by which the recognition unit 604 identifies the type of the recognized object, the coordinates indicating the object's position in the image, and the distance from the camera 600 to the object can be considered as recognition processing. Furthermore, the processing steps performed by the recognition unit 604 can also be considered as recognition steps involving object recognition using IR images.

[0063] As an example of a color determination unit, the color determination unit 605 determines the color of an object identified by the recognition unit 604 using a color image received from the image processing unit 603. When the object identified by the recognition unit 604 is an object that has already been pre-determined as a target for color determination, the color determination unit 605 performs color determination on that object. In this case, the color determination unit 605 determines a region in the color image as a target for color determination based on coordinates used to indicate the position of the object, wherein these coordinates are output as the recognition result of the recognition unit 604. Then, the color determination unit 605 performs color determination on the target region in the color image using a color determination method corresponding to the object identified by the recognition unit 604. Note that the target and the method of color determination performed by the color determination unit 605 will be described in detail below. The color determination unit 605 outputs the color determination result to the ECU 701 of the moving body 700. Note that the processing steps performed by the color determination unit 605 can also be considered as color determination steps. In the color determination steps, the color determination processing selected in the color determination selection step is performed on the area selected in the color determination area selection step by using the color image generated by the photoelectric conversion element 100.

[0064] The camera control unit 606 has a CPU and a memory storing a computer program, and controls the camera 600 by causing the CPU to execute the computer program stored in the memory. Examples of control performed by the camera control unit 606 include controlling the length of the exposure time period for each frame of the photoelectric conversion element 100 via the control pulse generation unit 115, and controlling the timing of control signals. Additionally, the camera control unit 606 instructs the IR emitter 500 to emit light via the communication unit 608.

[0065] The storage unit 607 has a recording medium such as a memory card and a hard disk, and stores image signals.

[0066] The communication unit 608 has a wireless or wired interface and communicates with external devices. The communication unit 608 outputs image signals to the outside of the camera 600 and receives various signals from the outside of the camera 600 via communication. The communication unit 608 also transmits commands from the camera control unit 606 to the IR emitter 500 via the communication unit 503 of the IR emitter 500.

[0067] The IR emitter 500 has an IR emitting unit 501, an emitting control unit 502, and a communication unit 503.

[0068] The IR light-emitting unit 501 is, for example, a near-infrared LED. The IR light-emitting unit 501 has a lens (not shown) and a light-emitting unit (not shown), and emits IR light at a timing and time period determined according to a control signal output from the light-emitting control unit 502.

[0069] The light emission control unit 502 receives a light emission command from the camera control unit 606 via the communication unit 503, generates a control signal for emitting light at an indicated timing, and outputs the generated control signal to the IR light emission unit 501. The control signal generated in the light emission control unit 502 includes information for indicating the light emission time period, light emission cycle, and number of light emission cycles.

[0070] The communication unit 503 communicates with the camera 600 via the communication unit 608 of the camera 600, receives light emission commands from the camera control unit 606, and sends the received light emission commands to the light emission control unit 502.

[0071] The camera 600 and the IR emitter 500 can be considered as imaging devices. Furthermore, in this embodiment, the camera 600 and the IR emitter 500 are mounted on the moving body 700. Therefore, the camera 600, the IR emitter 500, and the moving body 700 can also be broadly considered as imaging devices. Additionally, in this embodiment, a camera unit including an imaging optical system 601 and a photoelectric conversion element 100 is provided to capture images from at least one direction among the front, rear, and side of the moving body. Furthermore, multiple camera units can be provided on the moving body. Additionally, the IR emitter 500 is positioned, for example, at the front of the moving body 700.

[0072] The following description will assume that the mobile body 700 is an autonomous vehicle. The mobile body 700 has an ECU 701, a vehicle control unit 702, and a display unit 703.

[0073] ECU 701 serves as an example of an operation determination unit and includes a CPU and a memory storing a computer program. It controls the moving body 700 by having the CPU execute the computer program stored in the memory. When ECU 701 receives an identification result from identification unit 604 and a color determination result from color determination unit 605, ECU 701 determines control of the moving body 700 based on the received identification and color determination results. Examples of control measures for the moving body 700 include stopping the moving body 700 via an automatic brake. ECU 701 outputs information indicating the determined control measures to vehicle control unit 702 and display unit 703. When receiving color and IR images from image processing unit 603, ECU 701 also sends the received images to display unit 703.

[0074] The vehicle control unit 702 controls the operation of the moving body 700 based on instructions from the ECU 701. Examples of the vehicle control unit 702's control over the moving body 700 include driving, stopping, and controlling the direction of the moving body 700.

[0075] Display unit 703 is a display unit for displaying images. Examples of display units 703 include liquid crystal devices, organic EL devices, etc. According to instructions from ECU 701, display unit 703 displays various information, such as images acquired by photoelectric conversion element 100, recognition results from recognition unit 604, color determination results from color determination unit 605, and the movement status of moving body 700.

[0076] It should be noted that the image processing unit 603 and the recognition unit 604 of the camera 600 may be located in, for example, an external terminal, rather than being installed on the mobile body 700. Examples of external terminals include terminals for remotely controlling the mobile body 700 and terminals for monitoring the movement status of the mobile body 700.

[0077] In addition, such as Figure 7 As illustrated, the functional units of camera 600, IR emitter 500, and moving body 700 can be configured not only by setting them in a single device, but also by setting them individually in multiple devices.

[0078] Figure 8 This is a diagram illustrating an object management table. The object management table is used to manage objects that are targets for color determination by the color determination unit 605 of the camera 600. The object management table is stored, for example, in storage unit 607.

[0079] In the object management table, "Object", "Target Area", "Color Determination Processing" and "Determination Result" are shown in relation to each other.

[0080] A detailed explanation of the contents of the object management table will be provided.

[0081] The "Object" field indicates the type of object that is the target of color determination by the color determination unit 605. When the color determination unit 605 receives a recognition result from the recognition unit 604, the color determination unit 605 specifies the type of object recognized by the recognition unit 604 based on the received recognition result. Then, if the specified object type is indicated in the "Object" field of the object management table, the color determination unit 605 performs color determination on the object; otherwise, if the specified object type is not indicated in the "Object" field, the color determination unit 605 does not perform color determination on the object.

[0082] In the "target area," the region within the "object" displayed in the color image is designated as the target area for color determination by the color determination unit 605. The color determination process is the process by which the color determination unit 605 determines the color of the object displayed in the color image. Hereinafter, the region within the "object" displayed in the color image that is to be the target area for color determination by the color determination unit 605 may be referred to as the target area.

[0083] The “color determination process” indicates the content of the color determination process performed by the color determination unit 605 on the “target area” of the “object”.

[0084] The "Determination Result" indicates the candidates for the color determination result of the color determination unit 605. As a result of the color determination process for the "object", the color determination unit 605 determines the result from the candidates shown in the "Determination Result" associated with the "object".

[0085] The contents shown for each item in the object management table will be provided, as well as... Figures 9A to 9D Explanation of specific examples of the objects shown in the text.

[0086] In this embodiment, when it becomes necessary to control the moving body 700 based on the color of a specific portion of an object already identified by the recognition unit 604, the object is defined as the target of the color determination process of the color determination unit 605. More specifically, the object is indicated in the "Object" field of the object management table, and the specific portion of the object is indicated in the "Target Area" field of the object management table.

[0087] The "Object" field in the object management table indicates... Figure 9AThe "vehicle" exemplified in the image. When camera 600 is capturing an image of the area in front of the mobile body 700, in order to achieve autonomous driving of the mobile body 700 based on the state of the vehicle in front of the mobile body 700, information is needed to indicate which state the vehicle in front of the mobile body 700 is in: moving, stopped, accelerating, or decelerating. As an example of autonomous driving of the mobile body 700 based on the state of the vehicle in front of the mobile body 700, an example could be given such as the mobile body 700 stopping to prevent a collision with the vehicle in front of the mobile body 700 if the vehicle in front of the mobile body 700 is stopped. Autonomous driving of the mobile body 700 based on the state of the vehicle in front is necessary even when the visibility is poor compared to good visibility (such as in clear weather), making it difficult to detect the state of the vehicle shown in the image (such as in poor visibility conditions such as at night or in inclement weather). Furthermore, the recognition unit 604 recognizes the vehicle from the IR image displaying the vehicle, but does not identify which state the vehicle is in: moving, stopped, accelerating, or decelerating. Therefore, in this embodiment, the vehicle is indicated in the "Object" field of the object management table as the target of the color determination process. Additionally, the brake light is indicated in the "Target Area" field of the object management table, where the brake light is the part of the information needed to indicate what color is being displayed in order to determine whether a vehicle is decelerating. The "brake light" indicated in the "Target Area" means... Figure 9A The left brake light 800a and the right brake light 800b are illustrated in the figure.

[0088] Furthermore, whether the vehicle is decelerating is indicated by whether the brake lights are illuminated in red. Therefore, the processing related to the presence or absence of red brake lights is indicated in the "Color Determination Process" of the object management table. The "Comparison between the red brightness of the target area and a threshold" indicated in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average red brightness of the "target area" in the color image and determines whether the calculated average exceeds a threshold. Similarly, the "Comparison between contrast and a threshold" indicated in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average contrast of the "object" in the color image and determines whether the calculated average exceeds a threshold.

[0089] Furthermore, "Unable to determine," "Brake lights off," and "Brake lights on" are indicated in the "Determination Result" associated with the "vehicle." As a result of the "color determination process," if the average red brightness calculated for the "target area" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "Brake lights off." Conversely, as a result of the "color determination process," if the average red brightness calculated for the "target area" exceeds a threshold, the color determination unit 605 determines the determination result as "Brake lights on." Furthermore, as a result of the "color determination process," if the average contrast calculated for the "object" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "Unable to determine," regardless of whether the average red brightness calculated for the "target area" exceeds the threshold. This means that when the average contrast of the object is low, the color determination unit 605 has difficulty determining whether the brake lights are on or off.

[0090] in addition, Figure 9B The "traffic light" illustrated is indicated in the "Object" field of the object management table. When the camera 600 captures an image of the area in front of the mobile body 700, in order to achieve automatic driving of the mobile body 700 based on the state of the traffic light, the following information is needed to indicate the state of the traffic light in front of the mobile body 700 from the states of red, yellow, and green traffic lights. Examples of automatic driving of the mobile body 700 based on the state of the traffic light include: stopping the mobile body 700 when the traffic light in front of the mobile body 700 is in a red or yellow state; and moving the mobile body 700 when the traffic light in front of the mobile body 700 is in a green state. Furthermore, automatic driving of the mobile body 700 based on the state of the traffic light is necessary even when the traffic light shown in the image is difficult to detect compared to good visibility (such as in clear weather) (such as at night or in inclement weather with poor visibility). Furthermore, although the recognition unit 604 identifies traffic lights from the IR image displaying traffic lights, it does not identify the color of the illuminated traffic lights. Therefore, in this embodiment, traffic lights are indicated in the "Object" field of the object management table as the target of the color determination process. Additionally, lights that are necessary for determining the illuminated color of the traffic lights are indicated in the "Target Area" field of the object management table. The "lights" indicated in the "Target Area" mean... Figure 9B The green light 810a, yellow light 810b, and red light 810c are shown in the diagram.

[0091] Furthermore, the color of the traffic light is specified based on whether the light is illuminated as green, yellow, or red. Therefore, the processing related to the presence or absence of green light 810a, yellow light 810b, and red light 810c is indicated as "Color Determination Process" in the object management table. The "Comparison between the green brightness of the target area and a threshold" in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average green brightness of the "target area" corresponding to green light 810a in the color image and determines whether the calculated average value exceeds a threshold. The "Comparison between the yellow brightness of the target area and a threshold" in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average yellow brightness of the "target area" corresponding to yellow light 810b in the color image and determines whether the calculated average value exceeds a threshold. The “comparison between the red brightness of the target area and the threshold” in the “color determination process” means that, as a color determination process, the color determination unit 605 calculates the average value of the red brightness of the “target area” in the color image corresponding to the red light 810c and determines whether the calculated average value exceeds the threshold.

[0092] Furthermore, "Unable to determine," "Green signal light on," "Yellow signal light on," and "Red signal light on" are indicated in the "determination result" associated with the "traffic light." As a result of the "color determination process," if the average green brightness calculated for the "target area" corresponding to the green light 810a exceeds a threshold, the color determination unit 605 determines the result as "Green signal light on." Similarly, as a result of the "color determination process," if the average yellow brightness calculated for the "target area" corresponding to the yellow light 810b exceeds a threshold, the color determination unit 605 determines the result as "Yellow signal light on." Finally, as a result of the "color determination process," if the average red brightness calculated for the "target area" corresponding to the red light 810c exceeds a threshold, the color determination unit 605 determines the result as "Red signal light on." There are cases where the average green brightness in the "target area" of green light 810a, the average yellow brightness in the "target area" of yellow light 810b, and the average red brightness in the "target area" of red light 810c are all equal to or less than a threshold. In this case, the color determination unit 605 determines the determination result as "cannot be determined". This "cannot be determined" result may occur under conditions of poor visibility (such as at night or in inclement weather) or when the distance from the moving object 700 to the traffic light is long. Note that the threshold used for comparison with green brightness, the threshold used for comparison with yellow brightness, and the threshold used for comparison with red brightness can be the same or different.

[0093] Furthermore, the color determination process performed by the color determination unit 605 is not limited to the example described above. The color determination unit 605 compares the average green brightness calculated for the "target area" of the green light 810a, the average yellow brightness calculated for the "target area" of the yellow light 810b, and the average red brightness calculated for the "target area" of the red light 810c. Then, the color determination unit 605 can output a determination result indicating that the traffic light's illumination state is the color with the highest average brightness. Additionally, there may be cases where two or more of the average green brightness in the "target area" of the green light 810a, the average yellow brightness in the "target area" of the yellow light 810b, and the average red brightness in the "target area" of the red light 810c exceed a threshold. In this case, the color determination unit 605 can determine the result as "cannot be determined".

[0094] Additionally, "Flag A" is also indicated in the "Object" field of the object management table. For example... Figure 9C As illustrated, "sign A" means a road sign indicating a left-hand arrow. In the following text, as... Figure 9CThe road sign showing a left-pointing arrow can be simply referred to as sign A. In Japan, a sign A with a white arrow indicates so-called one-way traffic, meaning that vehicles can only travel in the direction indicated by the arrow. Additionally, in Japan, a sign A with a blue arrow indicates so-called left-turn permitted, meaning that vehicles can turn in the direction indicated by the arrow. Therefore, when camera 600 is photographing the area in front of the moving body 700, whether the arrow shown in sign A in front of the moving body 700 is white or blue is information needed to achieve autonomous driving of the moving body 700 according to the type of sign A. Examples of autonomous driving for the moving body 700 according to the type of sign A include: in response to the arrow shown in sign A in front of the moving body 700 being white, restricting the moving body 700 to travel in a direction different from the direction indicated by the arrow in sign A. Furthermore, autonomous driving for the moving body 700 according to the type of sign A is a control necessary even in situations where it is difficult to detect the sign A shown in the image compared to good visibility (such as in clear weather) (such as in poor visibility conditions such as at night or in inclement weather). Furthermore, although the recognition unit 604 identifies sign A from the IR image displaying sign A, the recognition unit 604 does not identify whether sign A is a road sign indicating one-way traffic or a road sign indicating that a left turn is permitted. Therefore, in this embodiment, sign A is shown in the "Object" field of the object management table as a target for color determination processing. Additionally, the interior of the arrow is indicated in the "Target Area" of the object management table; this interior of the arrow is the part needed to determine what color the arrow shown in sign A is. The "interior of the arrow" indicated in the "Target Area" means... Figure 9C The arrow inside the symbol A shown is 820.

[0095] Furthermore, the type of marker A is specified by whether the arrow of marker A is blue or white. Therefore, the processing related to the presence or absence of blue within the arrow 820 of marker A is indicated in the "Color Determination Processing" field of the object management table. The "Comparison between blue brightness of the target area and a threshold" in the "Color Determination Processing" means that, as a color determination process, the color determination unit 605 calculates the average blue brightness value of the "target area" in the color image and determines whether the calculated average value exceeds a threshold. Additionally, the "Comparison between contrast and a threshold" indicated in the "Color Determination Processing" means that, as a color determination process, the color determination unit 605 calculates the average contrast of the "object" in the color image and determines whether the calculated average value exceeds a threshold.

[0096] Furthermore, "Unable to determine," "One-way traffic," and "Left turn permitted" are indicated in the "Determination Result" associated with "Sign A." As a result of the "color determination process," if the average blue brightness calculated for the "target area" exceeds a threshold, the color determination unit 605 determines the determination result as "One-way traffic." Additionally, as a result of the "color determination process," if the average blue brightness calculated for the "target area" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "Left turn permitted." Furthermore, as a result of the "color determination process," if the average contrast calculated for the "object" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "Unable to determine," regardless of whether the average blue brightness calculated for the "target area" exceeds the threshold. This means that when the average contrast value of the object is low, the color determination unit 605 has difficulty determining the display color of the arrow of Sign A.

[0097] Therefore, in this embodiment, taking into account the fact that there are objects with similar shapes and different display colors, these objects are predefined in the object management table as targets for the color determination process.

[0098] Additionally, "Flag B" is also indicated in the "Object" field of the object management table. For example... Figure 9D As illustrated, "sign B" means a road sign shown diagonally inside a circle. In the following text, as... Figure 9DThe road sign exemplified by showing a diagonal line inside a circle can be simply referred to as sign B. In Japan, if the color inside the circle in sign B is blue, it means that parking is prohibited at that location. Additionally, in Japan, if the color inside the circle in sign B is white, it indicates that vehicles are prohibited from passing. Therefore, when camera 600 is capturing images of the area in front of the moving vehicle 700, whether the inside of the circle in sign B in front of the moving vehicle 700 is blue or white is information needed to achieve autonomous driving of the moving vehicle 700 according to the type of sign B. Examples of autonomous driving for the moving vehicle 700 according to the type of sign B include: restricting the parking of the moving vehicle 700 in response to the inside of the circle in sign B in front of the moving vehicle 700 being blue. Additionally, examples of autonomous driving for the moving vehicle 700 according to the type of sign B include: restricting the movement of the moving vehicle 700 in response to the inside of the circle in sign B in front of the moving vehicle 700 being white. Furthermore, autonomous driving of the mobile body 700 based on the type of sign B is necessary even in situations where it is difficult to detect the sign B displayed in the image compared to good visibility (such as in clear weather) (such as in poor visibility conditions such as at night or in inclement weather). Additionally, although the recognition unit 604 identifies sign B from the IR image displaying sign B, the recognition unit 604 does not identify whether sign B indicates no parking or no vehicle passage. Therefore, in this embodiment, sign B is indicated in the "Object" field of the object management table, serving as the target for color determination processing. Furthermore, the color shown inside the circle is shown in the "Target Area" field of the object management table; the area inside the circle is the part needed to determine the type of sign B. The "inside the circle" shown in the "Target Area" means... Figure 9D The circle inside the symbol B shown in the image is 830.

[0099] Furthermore, the type of marker B is specified based on whether the interior 830 of the circle of marker B is shown in blue or white. Therefore, the processing related to the presence or absence of blue display within the interior 830 of the circle of marker B is indicated in the "Color Determination Process" field of the object management table. The "Comparison between blue brightness of the target area and threshold" in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average blue brightness value of the "target area" in the color image and determines whether the calculated average value exceeds the threshold. Additionally, the "Comparison between contrast and threshold" indicated in the "Color Determination Process" means that, as a color determination process, the color determination unit 605 calculates the average contrast of the "object" in the color image and determines whether the calculated average value exceeds the threshold.

[0100] Furthermore, "Unable to determine," "No parking," and "No vehicle passage" are indicated in the "Determination Result" associated with "Mark B." As a result of the "Color Determination Processing," if the average value of the blue brightness calculated for the "target area" exceeds a threshold, the color determination unit 605 determines the determination result as "No parking." Additionally, as a result of the "Color Determination Processing," if the average value of the blue brightness calculated for the "target area" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "No vehicle passage." Furthermore, as a result of the "Color Determination Processing," if the average value of the contrast calculated for the "object" is equal to or less than a threshold, the color determination unit 605 determines the determination result as "Unable to determine," regardless of whether the average value of the blue brightness calculated for the "target area" exceeds the threshold. This means that when the average contrast of the object is low, the color determination unit 605 has difficulty determining the display color within the circle 830 of Mark B.

[0101] In this embodiment, when an object intended for color determination processing is identified by the identification unit 604, a target area is designated based on the size, shape, etc., of the identified object and information about the "target area" indicated in the object management table associated with that object. It should be noted that the designation of the target area can be performed by either the identification unit 604 or the color determination unit 605. Therefore, both the identification unit 604 and the color determination unit 605 can be considered as color determination area selection units for selecting the area to be color determined based on objects already detected by the identification unit 604. Furthermore, the processing steps performed by the identification unit 604 and the color determination unit 605 can also be considered as color determination area selection steps for selecting the area to be color determined based on objects already detected during the identification step.

[0102] Furthermore, the area corresponding to the object identified by the identification unit 604 can be designated as the target area. For example, when the identification unit 604 identifies a "vehicle" as an "object" in the object management table, different areas can be designated as target areas depending on whether the "vehicle" is a regular car, a truck, or a motorcycle. Additionally, for example, when the identification unit 604 identifies a "traffic light" as an "object" indicated in the object management table, different areas can be designated as target areas based on the shape of the traffic light (such as whether the traffic light is vertical or horizontal).

[0103] Figure 10This is a flowchart illustrating the process for result output processing. Result output processing is the process by which the camera 600 outputs the result of color determination processing. In this embodiment, for example, when the camera 600 is in a color determination processing mode, result output processing begins when a recording instruction is received based on the user's operation of the camera 600.

[0104] The camera control unit 606 sets the IR emitter 500 to emit light (step (hereinafter sometimes referred to as "S") 101). More specifically, the camera control unit 606 sets the IR emitter 500's emission timing, emission time period, emission cycle, number of emission cycles, etc.

[0105] The camera control unit 606 sets the photoelectric conversion element 100 (S102). More specifically, the camera control unit 606 makes various settings for the circuit board 21 of the photoelectric conversion element 100 to perform photoelectric conversion on the optical image from the imaging optical system 601 and generate an image signal. It should be noted that in these photoelectric conversion settings, the camera control unit 606 can make different settings for the pixels 101 of the RGB filter 31 and the pixels 101 of the IR filter 32. In addition, at this time, the camera control unit 606 sets the parameters of the image processing unit 603, the recognition unit 604, and the color determination unit 605.

[0106] The camera control unit 606 causes the recording to be performed (S103). More specifically, the camera control unit 606 instructs the IR emitter 500 to emit light according to the settings already made in step S101, and instructs the photoelectric conversion element 100 to output a vertical synchronization signal, thereby enabling exposure and image signal generation.

[0107] The camera control unit 606 causes the image processing unit 603 to perform various image processing operations on the image signal output from the photoelectric conversion element 100 during the imaging process that began in step S103, thereby generating a final image signal. As a result, the image processing unit 603 acquires a color image by using the R, G, and B signals output from the photoelectric conversion element 100, and further acquires a monochrome IR image by using the IR signal (S104).

[0108] The recognition unit 604 identifies the objects displayed in the IR image acquired in step S104 (S105). More specifically, the recognition unit 604 detects the objects displayed in the IR image, specifies coordinates to indicate the position of the detected objects in the IR image, and calculates the distance between the camera 600 and the detected objects.

[0109] The identification unit 604 determines whether the object identified in step S105 is an object to be color-determined by the color determination unit 605 (S106). The identification unit 604 determines this based on whether the identified object is in the object management table (see...). Figure 8 The determination in step S106 is made using the type indicated in the "object" field of the object.

[0110] If the object identified by the identification unit 604 is the target of the color determination process ("Yes" in S106), the color determination unit 605 selects the content of the color determination process based on the identified object (S107). The color determination unit 605 selects the content indicated in the "Color Determination Process" field associated with the identified "Object" field in the object management table as the content of the color determination process to be performed. Additionally, at this time, the target area is specified based on the identified object.

[0111] The color determination unit 605 performs color determination processing (S108) based on the content selected in step S107 for the target area of ​​the object displayed in the color image. In this embodiment, since the IR image and the color image are acquired by a shared photoelectric conversion element 100, the IR image and the color image have the same viewing angle. Therefore, the coordinates corresponding to the target area specified from the IR image are also applied as the coordinates of the target area in the color image.

[0112] It should be noted that different photoelectric conversion elements can be used for both IR and color image acquisition. In this case, the viewing angles in the IR and color images can differ depending on the pixel arrangement of each photoelectric conversion element. When the viewing angles differ between the IR and color images, coordinate transformation (alignment processing) can be performed on the coordinates corresponding to the target area specified from the IR image, allowing the coordinates of the target area in the color image to be specified.

[0113] The camera control unit 606 sends the color image and IR image acquired in step S104, the recognition result obtained by the recognition unit 604 in step S105, and the result of the color determination processing performed in step S108 to the moving body 700 (S109). Additionally, if the object identified by the recognition unit 604 is not the target of the color determination processing ("No" in S106), the camera control unit 606 sends the color image and IR image, along with the recognition result obtained by the recognition unit 604, to the moving body 700.

[0114] The camera control unit 606 determines whether a recording command is continuing (S110). If the recording command is continuing ("Yes" in S110), the processing from step S103 is repeated. That is, if the recording command is continuing, the processing from the start of recording is repeated for the next frame. Otherwise, if the recording command is not continuing ("No" in S110), the result output processing ends.

[0115] Figure 11 This is a diagram illustrating a relationship management table. The relationship management table is used to manage the relationships between objects that are the targets of color determination processing, the results of color determination processing, and the operations of the moving body 700. The relationship management table is stored, for example, in ECU 701.

[0116] In the relationship management table, "object", "determine result" and "operation content" are shown in relation to each other.

[0117] A detailed explanation of the contents of the relational management table will be provided.

[0118] The "Object" field indicates the type of object that has undergone color determination processing by the color determination unit 605. Within this "Object" field, the object management table (see [link to table]) is also specified. Figure 8 The same item in the "object" field of the item.

[0119] The "Determined Result" field indicates the result of the color determination process performed by the color determination unit 605. This "Determined Result" field also indicates the same items as those in the "Determined Result" field of the object management table.

[0120] The “Operation Content” section indicates the operation by which the ECU 701 causes the moving body 700 to operate.

[0121] When the ECU 701 receives the recognition result and color determination processing result of the IR image from the recognition unit 604 from the camera 600, the ECU 701 determines the content for operating the moving body 700 based on the received information. More specifically, the ECU 701 determines the "operation content" associated with the "object" corresponding to the recognition result of the recognition unit 604 and the "determination result" corresponding to the result of the color determination processing in the relationship management table as the content for operating the moving body 700.

[0122] A detailed explanation of the operations related to relational management tables will be provided.

[0123] In the relationship management table, the "Operation Content" field, which is associated with "Vehicle" as the object and "Undetermined" as the result, indicates "Decelerate / Low Speed".

[0124] If the vehicle's brake lights are not specified as illuminated, they may be illuminated, indicating that the vehicle may be decelerating or stopped. In this case, it is necessary to restrict the movement of the moving body 700 to prevent a collision between the moving body 700 and the vehicle. Therefore, in this embodiment, for safety reasons, "deceleration / low-speed movement" is defined as "operation content," so that the movement of the moving body 700 is restricted even if the vehicle's brake lights are not specified as illuminated.

[0125] Additionally, in the relationship management table, the "Operation Content" field, which is associated with "Vehicle" as the object and "Brake Lights Off" as the result, indicates "Continue Driving".

[0126] When the vehicle's brake lights are not illuminated, the following scenario is specified: even if the moving body 700 does not decelerate, the moving body 700 will not rapidly approach the vehicle, and a collision between the moving body 700 and the vehicle is unlikely to occur. Therefore, in this embodiment, "continue moving" is defined as the "operation content" when the vehicle's brake lights are not illuminated.

[0127] Additionally, in the relationship management table, the "Operation Content" field, which is associated with "Vehicle" as the object and "Brake Light On" as the result, indicates "Decelerate / Stop".

[0128] When the vehicle's brake lights are illuminated, the moving body 700 may rapidly approach and collide with the vehicle as it continues to move. Therefore, in this embodiment, "deceleration / stopping" is defined as the "operation" when the vehicle's brake lights are illuminated.

[0129] Additionally, in the relationship management table, the "Operation Content" field, which is associated with "Traffic Lights" as the object and "Undetermined" as the result, indicates "Slow Down / Low Speed".

[0130] If the color of the traffic light is not specified, the traffic light may be illuminated in yellow or red. In this case, it is necessary to restrict the movement of the mobile body 700 before it reaches the traffic light. Therefore, in this embodiment, for the sake of safety, "deceleration / low-speed movement" is defined as "operation content," so that the movement of the mobile body 700 is restricted even if the color of the traffic light is not specified.

[0131] Additionally, in the relationship management table, in the "Operation Content" associated with the "Traffic Light" as the object and the "Green Light On" as the result, the instruction is "Continue."

[0132] When the traffic light is green, there is no need for the moving body 700 to decelerate. Therefore, in this embodiment, "continue moving" is defined as the "operation content" when the traffic light is green.

[0133] Additionally, in the relationship management table, the "Operation Content" associated with the "Traffic Light" as the object and the "Yellow Light On" and "Red Light On" as the results indicates "Slow Down / Stop".

[0134] When the traffic light is illuminated in yellow or red, the moving body 700 needs to stop before reaching the traffic light. Therefore, in this embodiment, "deceleration / stopping" is defined as the "operation content" when the traffic light is illuminated in yellow or red.

[0135] Additionally, in the relationship management table, in the "Operation Content" associated with "Flag A" as the object and "Undeterminable" as the result, the instruction is "Continue".

[0136] Furthermore, in the relationship management table, the "Operation Content" associated with "Sign A" as the object and "One-Way Traffic" as the result indicates "Prohibition of travel in directions other than the arrow direction". When Sign A is a road sign indicating one-way traffic, it is necessary to restrict the movement of the mobile body 700 in directions other than those indicated by the arrow to comply with traffic regulations. Therefore, in this embodiment, "Prohibition of travel in directions other than the arrow direction" is defined as the "Operation Content" when Sign A is a road sign indicating one-way traffic.

[0137] Furthermore, in the relationship management table, the "Operation Content" associated with "Mark A" as the object and "Left Turn Permitted" as the determination result indicates "No restriction on leftward movement". When Mark A is a road sign indicating that a left turn is permitted, there is no need to restrict the left turn of the moving body 700. Therefore, in this embodiment, "No restriction on leftward movement" is defined as the "Operation Content" when Mark A is a road sign indicating that a left turn is permitted.

[0138] Additionally, in the relationship management table, in the "Operation Content" associated with "Flag B" as the object and "Undeterminable" as the result, the instruction is "Continue".

[0139] Furthermore, in the relationship management table, the "Operation Content" associated with "Sign B" as the object and "No Parking" as the result indicates "No parking in the marked area". When Sign B is a road sign indicating no parking, it is necessary to restrict the parking of the mobile body 700 to an area corresponding to a predetermined range centered on the no parking road sign to comply with traffic regulations. Therefore, in this embodiment, "No parking in the marked area" is defined as the "Operation Content" when Sign B is a road sign indicating no parking.

[0140] Furthermore, in the relationship management table, the "Operation Content" associated with "Sign B" as the object and "Vehicle Prohibition" as the result indicates "Slow Down / Stop". When Sign B is a road sign indicating that vehicles are prohibited from passing, it is necessary to restrict the movement of the moving body 700 to comply with traffic regulations. Therefore, in this embodiment, "Slow Down / Stop" is defined as the "Operation Content" when Sign B is a road sign indicating that vehicles are prohibited from passing.

[0141] Figure 12 This is a flowchart illustrating the process of mobile body operation processing. Mobile body operation processing is the process by which mobile body 700 performs operations based on the result of color determination processing by color determination unit 605. In this embodiment, for example, if mobile body operation processing is not being performed, mobile body operation processing begins at a predetermined time interval. The predetermined time interval can be any time, such as 0.1 seconds.

[0142] ECU 701 determines whether it has received the IR image and color image that have already been sent from camera control unit 606 in the result output processing (see [link]). Figure 10 (Step S109)(S201). If the negative result continues ("No" in S201), repeat the processing of step S201.

[0143] Furthermore, if the ECU 701 has already received the image ("Yes" in S201), the processing proceeds to the next step. The ECU 701 determines whether there is an object requiring operation by the moving body 700 within a predetermined range in front of the camera 600 (S202). For example, the predetermined range is a range closer to the range within which the moving body 700 can stop without colliding with the object during an emergency braking operation. Furthermore, the object requiring operation by the moving body 700 is an object that needs to be detected to achieve autonomous driving of the moving body 700, such as an animal (e.g., a person). In the result output processing, the ECU 701 performs the determination in step S202 based on the recognition result sent from the recognition unit 604 to the camera control unit 606.

[0144] If an object exists within the predetermined range ("Yes" in S202), the process proceeds to the next step. ECU 701 determines whether the object existing within the predetermined range is the target object for the color determination process (S203). In the result output process, ECU 701 performs the determination in step S203 based on the recognition result of the recognition unit 604 and the relationship management table that has been sent from the camera control unit 606.

[0145] If the object existing within the predetermined range is the target of the color determination process ("Yes" in S203), the process proceeds to the next step. ECU 701 determines whether the color determination unit 605 can perform determination in the color determination process (S204). ECU 701 compares the result of the color determination process sent from the camera control unit 606 in the result output process with the "Determination Result" field in the relation management table. If the result referenced by ECU 701 is a "Determination Result" different from "Cannot be determined" in the relation management table, ECU 701 determines that the color determination unit 605 can perform determination in the color determination process. Otherwise, if the result referenced by ECU 701 is "Cannot be determined" in the "Determination Result" field of the relation management table, ECU 701 determines that the color determination unit 605 cannot perform determination in the color determination process.

[0146] If the color determination unit 605 can determine the color in the color determination process ("Yes" in step S204), the ECU 701 selects the operation content of the moving body 700 based on the determination result (S205). More specifically, the ECU 701 compares the result of the color determination process sent from the camera control unit 606 in the result output process with the "determination result" field in the relationship management table. The ECU 701 selects the "operation content" associated with the "determination result" field in the relationship management table corresponding to the result already referenced by the ECU 701 as the operation content of the moving body 700. It should be noted that the "determination result" field in the relationship management table corresponding to the result already referenced by the ECU 701 is the "determination result" associated with the "object" identified by the identification unit 604.

[0147] Furthermore, if the color determination unit 605 cannot make a determination in the color determination process ("No" in step S204), the ECU 701 selects the operation content of the moving body 700 based on the determination result "cannot be determined" in the color determination process (S206). More specifically, the ECU 701 selects the "operation content" associated with the "object" identified by the identification unit 604 and the determination result "cannot be determined" in the color determination process in the relationship management table as the operation content of the moving body 700.

[0148] ECU 701 operates the moving body 700 according to the selection made in step S205 or S206 (S207). Additionally, there may be a case where an object within a predetermined range is not the target of the color determination process ("No" in S203). In this case, ECU 701 operates the moving body 700 according to predetermined operations for each object. Examples of such operations include operations to avoid collisions between the moving body 700 and the object. Furthermore, this operation may vary depending on the distance calculated by the recognition unit 604, which measures the distance from the camera 600 to the object.

[0149] If no object exists within the predetermined range ("No" in S202), or after step S207, the process proceeds to the next step. ECU 701 generates an image to be displayed on display unit 703 (S208). ECU 701 generates the image by overlaying the object detection result of recognition unit 604 on the IR image, the result of color determination processing, and information indicating the operating state of moving body 700 onto the color image acquired in step S201.

[0150] The ECU 701 displays the image generated in step S208 on the display unit 703 (S209). As a result, the occupants of the mobile body 700 can identify the detection results of the object, the results of the color determination process, and the operating status of the mobile body 700.

[0151] ECU 701 determines whether a next frame exists (S210). If a next frame exists ("Yes" in S210), the processing from step S201 is repeated. If no next frame exists ("No" in S210), the moving body operation processing ends.

[0152] Figures 13 and 14 are diagrams illustrating an example of the operation of the moving body 700. In the following example, it is assumed that a camera 600 is positioned at the front of the moving body 700, and the camera 600 captures an image of the moving body 700 in the direction of travel.

[0153] First, an example of the operation of the mobile body 700 when a car is in front of it will be explained. When a car is in front of the mobile body 700, as follows... Figure 13A As shown, car 920 is displayed in a color image generated by imaging during the result output processing. Additionally, this color image shows fog 910 covering car 920, making it difficult to see car 920 due to the fog 910.

[0154] in addition, Figure 13B An IR image is shown as an example. Figure 13B The illustrated IR image was captured by camera 600 through a lens targeting... Figure 13A The image shown is generated simultaneously with the capture of the color image. Furthermore, in this IR image, as a result of the recognition unit 604's recognition of the car 920 in the result output processing, a region image 921 indicating the area where the car 920 was detected and information indicating the distance of the detected car 920 from the camera 600 are shown. In the illustrated example, the text "40m" is shown as the distance of the car 920 from the camera 600.

[0155] In addition, in the result output processing, the color determination unit 605 is based on Figure 13B The recognition result of the recognition unit 604 shown is used for color determination processing. In this case, such as Figure 13C As illustrated, the color determination unit 605 displays a region image 921 overlaid on a color image, and designates the display areas of the left brake light 922a and the right brake light 922b from the region image 921 as target regions. Then, the color determination unit 605 performs color determination processing on the designated target regions (see [link to example]). Figure 10 Step S108). In the following text, it is assumed that the color determination unit 605 obtains the determination result of "brake light on" through the color determination process.

[0156] It should be noted that in the color image, the same coordinates as those of the individual images in the IR image are applied as the coordinates of the area image 921, the left brake light 922a, and the right brake light 922b.

[0157] In the movement operation processing, ECU 701 controls movement 700 using "deceleration / stop" as the operation content. "Deceleration / stop" is associated with the "vehicle" corresponding to car 920 in the relationship management table, and is also associated with "brake light activation" as a result of color determination processing. Additionally, as... Figure 13D As illustrated, ECU 701 displays a notification image 923 on display unit 703, including the selected operation content and the recognition result of recognition unit 604 (see [link]). Figure 12 (Step S209). The notification image 923 shows the recognition result of the recognition unit 604, the area image 921, and the operation content selected by the ECU 701. By displaying the notification image 923 and the color image on the display unit 703, the occupants of the mobile body 700 recognize the presence of the vehicle 920 in front of the fog 910, and the mobile body 700 decelerates / stops during autonomous driving.

[0158] It should be noted that there are targets targeting Figure 13CThe result of the color determination process for the illustrated color image is the "brake lights off" condition. In this case, during the moving body operation process, the ECU 701 controls the moving body 700 with the operation content of "continue moving", where "continue moving" is associated with the "vehicle" corresponding to the car 920 in the relationship management table, and is also associated with "brake lights off" as a result of the color determination process.

[0159] Therefore, even in conditions of poor visibility due to nighttime conditions or inclement weather, camera 600 detects vehicles present in front of the moving body 700 from the IR image with high accuracy. Furthermore, camera 600 determines the vehicle's operational status (such as whether the brake lights are illuminated) through color determination processing of the color image. Then, the moving body 700 operates based on the determination results of the color determination processing. As a result, when the moving body 700 is autonomously driving, collisions between the moving body 700 and vehicles in front of it can be prevented.

[0160] Next, an example of the operation of the mobile body 700 when there is a traffic light in front of it will be described. When there is a traffic light in front of the mobile body 700, such as... Figure 14A As illustrated, the traffic light 940 is displayed in a color image generated by imaging during the result output processing. Additionally, this color image shows fog 930 covering the traffic light 940, making it difficult to see the traffic light 940 due to the fog 930.

[0161] In addition, IR images in Figure 14B Example in. Figure 14B The illustrated IR image was generated by camera 600 through a sensor connected to a camera. Figure 14A The illustrated color image is generated simultaneously with the captured image. Furthermore, in this IR image, as a result of the recognition unit 604's recognition of the traffic light 940 during the result output processing, a region image 941 indicating the area where the traffic light 940 was detected and information indicating the distance of the detected traffic light 940 from the camera 600 are displayed. In the illustrated example, the text "50m" is displayed to indicate the distance from the camera 600 to the traffic light 940.

[0162] In addition, in the result output processing, the color determination unit 605 is based on Figure 14B The recognition result of the recognition unit 604 shown is used for color determination processing. In this case, such as Figure 14CAs illustrated, the color determination unit 605 overlays the area image 941 onto the color image. Furthermore, the color determination unit 605 designates the display areas of the green light 942a, yellow light 942b, and red light 942c from the area image 941 as target areas. Then, the color determination unit 605 performs color determination processing on the designated target areas (see [link to example]). Figure 10 Step S108). In the following text, it is assumed that during the shooting... Figure 14C In the illustrated color image, the traffic light 940 is far from the camera 600, and because the average brightness value of each color is lower than the threshold in the color determination process, the determination result of "cannot be determined" is output to the color determination unit 605.

[0163] It should be noted that in the color image, the coordinates used to locate the area images 941, green light 942a, yellow light 942b, and red light 942c are the same coordinates used for the locations of the individual images in the IR image.

[0164] In the mobile body operation processing, ECU 701 controls the mobile body 700 with "deceleration / low-speed travel" as the operation content. "Deceleration / low-speed travel" is associated with the "traffic light" corresponding to traffic light 940 in the relationship management table and with "undeterminable" as the result of color determination processing. Additionally, ECU 701... Figure 14C The notification image 943, as illustrated, is overlaid on the color image. Notification image 943 is an image used to provide information indicating the recognition result of the recognition unit 604 and the operation selected by the ECU 701. In the illustrated example, notification image 943 shows the text "Traffic light ahead" as the recognition result of the recognition unit 604 and the text "Slowing down" as the operation selected by the ECU 701.

[0165] Subsequently, as the moving body 700 continues its movement, it approaches the traffic light 940. Then, when the distance from the camera 600 to the traffic light 940 is 30m, the result output processing begins. This result output processing includes image capture, recognition of the traffic light 940 by the recognition unit 604, and color determination processing by the color determination unit 605. In the following text, it is assumed that the color determination unit 605 obtains the determination result of "red traffic light illuminated" through the color determination processing.

[0166] In the movement operation processing, ECU 701 controls movement 700 using "deceleration / stop" as the operation content. "Deceleration / stop" is associated in the relationship management table with the "traffic light" corresponding to traffic light 940 and with the "red light illuminating" as a result of color determination processing. Additionally, as... Figure 14DAs illustrated, ECU 701 displays a notification image 944 on display unit 703, including the selected operation content and the recognition result of recognition unit 604 (see [link]). Figure 12 (Step S209). The notification image 944 indicates the recognition result of the recognition unit 604, the area image 941, and the operation selected by the ECU 701. By displaying the notification image 944 and the color image on the display unit 703, the occupants of the mobile body 700 recognize the presence of traffic lights ahead of the fog, and the mobile body 700 decelerates / stops during autonomous driving.

[0167] It should be noted that there are targets targeting Figure 14D The result of the color determination process for the illustrated color image is a case where "the green traffic light is on". In this case, during the movement operation process, ECU 701 controls the movement 700 based on "continue moving" as the operation content, where "continue moving" is associated in the relationship management table with "traffic light" corresponding to traffic light 940 and with "green traffic light on" as the result of the color determination process.

[0168] Therefore, even in conditions of poor visibility due to nighttime conditions and inclement weather, the camera 600 detects traffic lights in front of the mobile body 700 from the IR image with high accuracy. Furthermore, the camera 600 determines the illumination state of the traffic lights through color determination processing of the color image. Then, the mobile body 700 operates based on the determination result of the color determination processing. As a result, when the mobile body 700 is driving autonomously, it operates according to the state of the traffic lights in front of it.

[0169] In addition, such as Figure 14C and Figure 14D As illustrated, when color determination cannot be performed because the object being processed for color determination is far from the camera 600, the moving body 700 operates according to the operation content associated with "cannot be determined". Then, when the object being processed for color determination approaches the camera 600 and color determination becomes possible, the moving body 700 operates according to the operation content already determined by the result of color determination.

[0170] It should be noted that the operation of the moving body 700, whose processing result is determined based on color, is not limited to the examples above. The operation of the moving body 700, whose processing result is determined based on color, may include acceleration, left turn, right turn, lane change, etc.

[0171] Furthermore, the objects targeted by color determination processing are not limited to vehicles, traffic lights, and specific road signs.

[0172] Furthermore, although it has been described in this embodiment that the photoelectric conversion element 100 of the camera 600 is a SPAD sensor, this disclosure is not limited thereto. The photoelectric conversion element 100 may be a CMOS image sensor provided with an RGB filter 31 and an IR filter 32. That is, the first imaging element and the second imaging element may be a SPAD sensor or a CMOS image sensor.

[0173] Furthermore, the imaging element equipped with the RGB filter 31 and the imaging element equipped with the IR filter 32 can be housed in separate cameras. In other words, the camera for capturing color images and the camera for capturing IR images can be different cameras.

[0174] Additionally, control known as distance gating control can be performed, in which the IR emitting unit 501 of the IR emitting unit 500 emits pulsed light at a predetermined period, and the image sensor of the camera 600 exposes at a predetermined time according to the distance from the camera 600 to the subject. That is, the photoelectric conversion element 100 of the camera 600 can be exposed synchronously with the emission period of the IR emitting unit 501.

[0175] Furthermore, although the necessity for the color determination unit 605 to determine the color determination process for an object already identified by the recognition unit 604 has been described in this embodiment, this disclosure is not limited thereto. The recognition unit 604 can determine the necessity for the color determination process for the identified object by the color determination unit 605. Therefore, the recognition process performed by the recognition unit 604 can also be considered as a process for identifying the type of the object and the necessity of the color determination process.

[0176] Furthermore, although in this embodiment the color determination unit 605 selects the color determination process to be performed based on the object already identified by the recognition unit 604, this disclosure is not limited thereto. The recognition unit 604 can select the color determination process to be performed by the color determination unit 605 based on the identified object. Therefore, the recognition unit 604 can also be considered as a color determination selection unit for selecting the color determination process to be performed based on the object already detected by the recognition unit 604. In addition, the processing steps performed by the recognition unit 604 and the color determination unit 605 can also be considered as a color determination selection step for selecting the color determination process to be performed based on the object already detected in the recognition step.

[0177] As described above, in this embodiment, the color determination unit 605 uses a color image to perform color determination processing for the target area that has been selected by the color determination area selection unit.

[0178] In this case, the color of the object displayed in the image is determined by a method corresponding to that object.

[0179] In addition, in this embodiment, the first camera element and the second camera element are CMOS image sensors.

[0180] In this case, the color of the object is determined by a method corresponding to the object displayed in the image, which is achieved through a CMOS image sensor.

[0181] In addition, in this embodiment, the first camera element and the second camera element are SPAD sensors.

[0182] In this case, the color of the object is determined by a method corresponding to the object shown in the image, which is achieved through a SPAD sensor.

[0183] In addition, in this embodiment, the identification process is a necessary process for determining the type or color of the object.

[0184] In this case, the color of the object is determined by a method corresponding to the type of object displayed in the image, and the necessity of color determination processing is switched according to the object displayed in the image.

[0185] In addition, in this embodiment, the camera device also includes an IR light-emitting unit 501 that emits infrared light.

[0186] In this case, it is not necessary to use an infrared light-emitting unit separately from the camera equipment to obtain IR images.

[0187] In addition, in this embodiment, the first camera element and the IR light-emitting unit 501 perform exposure synchronously.

[0188] In this situation, even in poor visibility conditions, the subject can be photographed more clearly.

[0189] In addition, in this embodiment, the color determination unit 605 calculates the contrast value in the color image of the object already detected by the recognition unit 604, and determines the probability or impossibility of color determination based on the calculated contrast value. For example, the contrast value may include the average contrast value in the color image of the object.

[0190] In this case, compared to a configuration that performs color determination regardless of the contrast value in the object's color image, the output of color determination results with low accuracy is suppressed.

[0191] In addition, in this embodiment, the camera device includes an ECU 701, which determines the operation of the moving body 700 on which the camera device is installed based on the recognition processing result of the recognition unit 604 and the color determination result of the color determination unit 605.

[0192] In this case, the operation of the moving body 700 based on the object displayed in the image is implemented.

[0193] In addition, in this embodiment, the ECU 701 determines the operation of the moving body 700 when color determination is possible and when color determination is not possible.

[0194] In this case, the color determination result prevented the inability to achieve autonomous driving of the mobile body 700.

[0195] In addition, in this embodiment, the color determination unit 605 performs color determination processing on the region of the color image that is the same as the region displaying the IR image of the object detected by the recognition unit 604.

[0196] In this case, it is not necessary to convert the area of ​​the IR image that displays the object that has been detected by the recognition unit 604 into the corresponding area of ​​the color image.

[0197] In addition, in this embodiment, when the recognition unit 604 identifies the object displayed in the IR image as a vehicle, the color determination area selection unit selects the left brake light area or the right brake light area of ​​the vehicle as the target area (see [link]). Figure 8 Additionally, the color determination and selection unit selects the color for determining the left and right brake light areas of the vehicle based on the intensity of red. Examples of red intensity could include red brightness. Then, the color determination unit 605 determines that the vehicle's brake lights are on if the red intensity exceeds a threshold, and determines that the vehicle's brake lights are off if the red intensity drops below the threshold.

[0198] In this case, compared to a configuration where colors other than red are included as the target for color determination, the accuracy of determining whether the vehicle's brake lights are illuminated is improved.

[0199] Furthermore, in this embodiment, when the recognition unit 604 identifies the object displayed in the IR image as a traffic light, the color determination area selection unit selects the illuminated area of ​​the traffic light as the target area (see [link]). Figure 8Additionally, the color determination and selection unit selects the color used to determine the illuminated area of ​​the traffic light based on the intensity of green, yellow, or red. The values ​​for the intensities of green, yellow, and red can include green brightness, yellow brightness, and red brightness, respectively. Then, the color determination unit 605 determines that the traffic light is illuminated in green if the green intensity exceeds a threshold, in yellow if the yellow intensity exceeds a threshold, and in red if the red intensity exceeds a threshold.

[0200] In this case, compared to a configuration where colors other than green, yellow, and red are included as targets for color determination, the accuracy of color determination for the illuminated portion of the traffic light is improved.

[0201] Furthermore, in this embodiment, when the recognition unit 604 identifies the object displayed in the IR image as mark A, the color determination area selection unit selects the arrow area of ​​mark A as the target area (see...). Figure 8 Additionally, the color determination unit selects the color used to determine the arrow area based on the intensity of the blue. Examples of blue intensity could include blue luminance. Then, the color determination unit 605 determines that the road sign indicates permitted left turns if the blue intensity exceeds a threshold, and determines that the road sign indicates one-way traffic if the blue intensity drops below the threshold.

[0202] In this case, compared to a configuration where colors other than blue are included as targets in color determination, the accuracy of determining whether sign A is a road sign indicating permitted left turns or a road sign indicating one-way traffic is improved.

[0203] Furthermore, in this embodiment, when the recognition unit 604 identifies the object displayed in the IR image as mark B, the color determination area selection unit selects the inner area of ​​the circle of mark B as the target area (see [link]). Figure 8 Additionally, the color determination unit selects the color for determining the inner region of the circle based on the intensity of the blue. Examples of blue intensity could include blue luminance. Then, the color determination unit 605 determines that a road sign indicates no parking if the blue intensity exceeds a threshold, and determines that a road sign indicates no passage if the blue intensity drops below the threshold.

[0204] In this case, compared to a configuration where colors other than blue are included as the target for color determination, the accuracy of determining whether sign B is a road sign indicating no parking or a road sign indicating no passage for vehicles is improved.

[0205] In addition, in this embodiment, the color determination and selection unit selects the color as the target color for color determination by the color determination unit 605 based on the object already detected by the recognition unit 604.

[0206] In this case, compared to the configuration where the color of the target for color determination remains unchanged regardless of the object detected by the recognition unit 604, the accuracy of color determination of the object displayed in the color image is improved.

[0207] In addition, this disclosure also includes situations where a software program for implementing the functions of each of the foregoing embodiments is supplied directly from a recording medium or via wired / wireless communication to a system or apparatus having a computer capable of executing the program, and the program is executed.

[0208] Therefore, the program code supplied to and installed in a computer to implement the aforementioned functional processing of this disclosure also implements the present invention. That is, this disclosure also includes the computer program itself for implementing the functional processing of the present invention. In this case, as long as it is used as a program, the program can take any form, such as object code, a program executed by an interpreter, or script data supplied to an OS. For example, the recording medium used to supply the program can be a hard disk, a magnetic recording medium such as magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory. In addition, regarding the method for supplying the program, the following method can also be considered: storing the computer program forming the present invention in a server on a computer network, and connecting client computers downloading and programming the computer program.

[0209] The embodiments of this disclosure can also be implemented by reading and executing computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform one or more functions of the above-described embodiments and / or by a computer of a system or device including one or more circuits (e.g., application-specific integrated circuits (ASICs)) for performing one or more functions of the above-described embodiments and by the following method, wherein the computer of the system or device performs the above-described methods by, for example, reading and executing computer-executable instructions from the storage medium to perform one or more functions of the above-described embodiments and / or controlling the one or more circuits to perform one or more functions of the above-described embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessor unit (MPU)) and may include a network of separate computers or separate processors for reading and executing computer-executable instructions. For example, these computer-executable instructions can be provided to a computer from a network or storage medium. The storage medium may include one or more of the following: a hard disk, random access memory (RAM), read-only memory (ROM), the storage unit of a distributed computing system, optical discs (such as CDs, DVDs, or Blu-ray Discs™), flash memory devices, and memory cards.

[0210] Embodiments of the present invention can also be implemented by providing software (including computer program products of computer programs) that performs the functions of the above embodiments to a system or device via a network or various storage media, and the computer (central processing unit (CPU) or microprocessor unit (MPU) of the system or device) reads and executes the computer program.

[0211] While this disclosure has been described with reference to embodiments, it should be understood that this disclosure is not limited to the disclosed embodiments. The scope of the appended claims should be given the broadest interpretation to cover all such modifications and equivalent structures and functions.

[0212] According to this disclosure, the color of an object can be determined by a method corresponding to the object displayed in the image.

[0213] This application claims the benefit of Japanese Patent Application 2024-219783, filed on December 16, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. A camera device, comprising: The first camera element is configured to have a photoelectric conversion unit capable of receiving invisible light; The second camera element is configured to have a photoelectric conversion unit capable of receiving visible light; At least one memory that stores instructions; as well as At least one processor executes stored instructions to cause the camera device to: The first image generated by the first camera element is used for object recognition processing. Based on the objects detected by the object recognition process, the region to be processed for color determination is selected. Based on the objects detected by the object recognition process, the color determination process to be performed is selected, and The selected region is processed using the second image generated by the second camera element to determine the selected color.

2. The camera device according to claim 1, wherein, The first camera element and the second camera element are CMOS image sensors.

3. The camera device according to claim 1, wherein, The first camera element and the second camera element are SPAD sensors.

4. The camera device according to claim 1, in, The identification process is a process used to identify the type of the object or the necessity of the color determination process.

5. The camera device according to claim 1, in, The invisible light mentioned is infrared light, and The camera device also includes a light-emitting unit configured to emit infrared light.

6. The camera device according to claim 5, wherein, The first imaging element performs exposure synchronously with the light emission cycle of the light-emitting unit.

7. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to: calculate the contrast value of the object in the second image detected by the object recognition process, and determine the probability or impossibility of color determination based on the calculated contrast value.

8. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to determine the operation of the moving body on which the camera device is mounted, based on the results of the object recognition processing and the color determination processing.

9. The camera device according to claim 8, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to determine the operation of the moving body in various cases, including when the result of the color determination process is obtainable and when the result of the color determination process is not obtainable.

10. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further causing the camera device to perform the color determination process on a region of the second image that is the same region as the area of ​​the object detected in the display of the first image.

11. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to: The object is identified as a vehicle. Select the left brake light area and the right brake light area of ​​the vehicle. The process of selecting the color for determining the left and right brake light areas of the vehicle based on the intensity of red, and... If the intensity of the red light exceeds a threshold, it is determined that the vehicle's brake lights are illuminated; if the intensity of the red light drops below the threshold, it is determined that the vehicle's brake lights are not illuminated.

12. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to: The object is identified as a traffic light. Select the illuminated area of ​​the traffic light. The process of selecting the color for determining the illuminated area of ​​the traffic light based on the intensity of green, yellow, and red, and... If the intensity of green exceeds a threshold, the traffic light is determined to be lit in green; if the intensity of yellow exceeds the threshold, the traffic light is determined to be lit in yellow; and if the intensity of red exceeds the threshold, the traffic light is determined to be lit in red.

13. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to: The object is identified as a road sign with an arrow. Select the arrow area in the road sign. The process of selecting the color for determining the arrow region based on the intensity of blue, and If the intensity of blue exceeds a threshold, the road sign is determined to be a road sign indicating that vehicles are allowed to turn left; if the intensity of blue drops below the threshold, the road sign is determined to be a road sign indicating one-way traffic for vehicles.

14. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to: The object is identified as a road sign with a circle, wherein the circle has a diagonal inside. Select the inner area of ​​the circle in the road sign. The process of selecting the color for determining the inner region of the circle based on the intensity of blue, and If the intensity of blue exceeds a threshold, the road sign is determined to be a road sign indicating no parking; if the intensity of blue drops below the threshold, the road sign is determined to be a road sign indicating no passage for vehicles.

15. The camera device according to claim 1, wherein, The at least one processor executes the stored instructions, thereby further enabling the camera device to select a color as the target of the color determination process based on the object detected by the object recognition process.

16. A processing method for a camera device, the camera device comprising a first imaging element having a photoelectric conversion unit capable of receiving invisible light and a second imaging element having a photoelectric conversion unit capable of receiving visible light, the processing method comprising: The step is to perform object recognition processing using the first image generated by the first camera element; The region selection step is used to select the region to be processed for color determination based on the objects detected in the object recognition process. The processing selection step is used to select the color determination process to be performed based on the object detected in the object recognition process; as well as The execution step is used to perform a selected color determination process on the selected area using the second image generated by the second camera element.

17. A non-transient recording medium storing a control program for a camera device, the camera device comprising a first imaging element having a photoelectric conversion unit capable of receiving invisible light and a second imaging element having a photoelectric conversion unit capable of receiving visible light, the control program causing a computer to perform steps of a control method for the camera device, the control method comprising: The step is to perform object recognition processing using the first image generated by the first camera element; The region selection step is used to select the region to be processed for color determination based on the objects detected by the object recognition process. The processing selection step is used to select the color determination process to be performed based on the object detected by the object recognition process; as well as The execution step is used to perform a selected color determination process on the selected area using the second image generated by the second camera element.

18. A computer program product comprising a control program for a camera device, the camera device including a first imaging element having a photoelectric conversion unit capable of receiving invisible light and a second imaging element having a photoelectric conversion unit capable of receiving visible light, the control program causing a computer to perform steps of a control method for the camera device, the control method comprising: The step is to perform object recognition processing using the first image generated by the first camera element; The region selection step is used to select the region to be processed for color determination based on the objects detected by the object recognition process. The processing selection step is used to select the color determination process to be performed based on the object detected by the object recognition process; as well as The execution step is used to perform a selected color determination process on the selected area using the second image generated by the second camera element.

Images