Image processing device
The image processing device in vehicles switches between CNN and Transformer models based on environmental conditions to maintain high-precision image processing, addressing accuracy drops from disturbances like rain and brightness.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- DENSO CORP
- Filing Date
- 2023-03-28
- Publication Date
- 2026-07-07
AI Technical Summary
Advanced driver-assistance systems using neural networks face accuracy decreases due to disturbances such as rain and changes in brightness, necessitating high-precision image processing that burdens machine learning and requires selecting a neural network based on disturbance type.
An image processing device for vehicles equipped with multiple neural network models (CNN and Transformer) and sensors to detect environmental conditions, dynamically selecting the appropriate model for processing based on brightness or rain intensity to maintain high-precision image processing.
Enables high-precision image processing by adaptively switching between neural networks based on environmental conditions, enhancing accuracy in varying light conditions and reducing computational burden.
Smart Images
Figure 0007886293000001 
Figure 0007886293000002 
Figure 0007886293000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing apparatus.
Background Art
[0002] In Patent Document 1, an image recording apparatus is disclosed that automatically switches and records a captured image at a frame rate higher than the frame rate set when the weather is not stormy when the weather is stormy. The image recording apparatus is a drive recorder and is mounted on a vehicle. This image recording apparatus determines whether it is stormy based on a captured image captured by an imaging unit. Specifically, a rectangular area surrounding the wiper is extracted from the captured image by a wiper detection unit. The extraction of the rectangular area is realized using an object detection algorithm such as SSD or YOLO.
[0003] A wiper operation period estimation unit estimates the operation period of the wiper by referring to the data of the wiper area supplied from the wiper detection unit and the data of a plurality of past wiper areas. Then, when the wiper operation period is less than a threshold value, a stormy weather determination unit determines that it is stormy. And a frame rate switching unit changes the frame rate. Thereby, it is possible to prevent insufficient information in a dangerous scene while minimizing the recording capacity of the image recording apparatus.
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0005] Incidentally, in general, advanced driver-assistance systems sometimes use neural networks to detect objects around a vehicle. However, it is known that the accuracy of image processing using neural networks decreases due to disturbances such as rain and changes in brightness. To achieve high-precision image processing, it is necessary to accumulate image data when disturbances occur and to spend time on machine learning, which places a heavy burden on those performing the work. Here, it is known that the way in which the accuracy of a neural network decreases when subjected to various disturbances differs depending on the type of neural network. Therefore, the inventors of this application considered achieving high-precision image processing by selecting a neural network according to the type of disturbance that occurs in the image. [Means for solving the problem]
[0006] This disclosure can be implemented in the following forms:
[0007] (1) According to one embodiment of the present disclosure, an image processing device (1) to be mounted on a vehicle (VW) is provided. The image processing device includes: an imaging unit (10) that images an object and generates an image including the object; one or more element acquisition units (20) that acquire numerical values representing the intensity of one or more elements of the environment surrounding the vehicle; a first image processing unit (30) that can perform computational processing on the image using a first neural network model; a second image processing unit (40) that can perform computational processing on the image using a second neural network model; and a processing selection unit (50) that selects an image processing unit from the first image processing unit and the second image processing unit to perform computational processing on the image. A vehicle control unit (60) controls the vehicle based on the data obtained by the calculation process,The system is equipped with such that, for some of the elements, in a first range of intensity, the accuracy of the first image processing unit is higher than that of the second image processing unit, and in a second range lower than the first range, the accuracy of the second image processing unit is higher than that of the first image processing unit, and the processing selection unit selects an image processing unit to perform calculation processing based on a numerical value representing the intensity of the element acquired by one or more element acquisition units (10). The vehicle control unit includes a first mode (MD1) in which it performs control based on the image processing unit selected by the processing selection unit, and a second mode (MD2) in which it performs control of the vehicle based on calculation processing data output by the first image processing unit, without being based on the image processing unit selected by the processing selection unit. When the processing selection unit has not made a selection of the image processing unit, it performs control of the vehicle in the second mode. (2) In another embodiment of the present disclosure, an image processing device (1) to be mounted on a vehicle (VW) is provided. The image processing device includes: an imaging unit (10) that images an object and generates an image including the object; one or more element acquisition units (20) that acquire numerical values representing the intensity of one or more elements of the environment surrounding the vehicle; a first image processing unit (30) that can perform calculation processing on the image using a first neural network model; a second image processing unit (40) that can perform calculation processing on the image using a second neural network model; and a processing selection unit (50) that selects an image processing unit from the first image processing unit and the second image processing unit to perform calculation processing on the image, wherein, for some of the elements, in a first range of intensity, the accuracy of the first image processing unit is higher than the accuracy of the second image processing unit, and a second range is lower than the first range. Within the range, the accuracy of the second image processing unit is higher than that of the first image processing unit, and the processing selection unit selects an image processing unit to perform calculation processing based on a numerical value representing the intensity of the element acquired by the one or more element acquisition units (10). One of the one or more element acquisition units includes a network information unit (210B) that acquires a first numerical value (FN) as a numerical value representing the intensity of rain via a network, and a detection unit (220B) that acquires a second numerical value (SN) as a numerical value representing the intensity of rain by directly detecting rain around the vehicle, and outputs the first numerical value and the second numerical value. The processing selection unit uses the larger of the first numerical value and the second numerical value as the numerical value representing the intensity of rain acquired by the element acquisition unit to select the image processing unit.
[0008] In this type of image processing apparatus, the processing selection unit causes either the first image processing unit or the second image processing unit to perform calculations based on the numerical values of the elements acquired by the element acquisition unit. This enables high-precision image processing. [Brief explanation of the drawing]
[0009] [Figure 1] A diagram illustrating the configuration of the vehicle according to this embodiment. [Figure 2] A block diagram showing the configuration of the first image processing unit, the second image processing unit, the processing selection unit, and the vehicle control unit. [Figure 3] This diagram shows the detection accuracy of the first image processing unit and the second image processing unit in relation to brightness. [Figure 4] A process diagram showing an example of the selection of an image processing unit by an image processing device. [Figure 5] A diagram showing a vehicle according to the second embodiment. [Figure 6] A diagram showing the accuracy of the image processing unit in relation to rainfall. [Figure 7] A process diagram showing an example of the selection of an image processing unit by the image processing apparatus of the second embodiment. [Figure 8] A process diagram showing an example of the selection of an image processing unit by the image processing apparatus of the third embodiment. [Modes for carrying out the invention]
[0010] A. First Embodiment: A1. Configuration of the first embodiment: Figure 1 is a diagram illustrating the configuration of the vehicle VW in this embodiment. The vehicle VW is capable of driving using ADAS (Advanced Driving Assistant System). As shown in Figure 1, the vehicle VW is equipped with an image processing device 1. The image processing device 1 detects objects around the vehicle VW and controls the vehicle VW based on the detection results. In this embodiment, the image processing device 1 comprises an imaging unit 10, a first element acquisition unit 20, a first image processing unit 30, a second image processing unit 40, a processing selection unit 50, and a vehicle control unit 60.
[0011] The imaging unit 10 captures images of objects around the vehicle VW and generates an image that includes these objects. These objects include other vehicles, people, animals, traffic lights, buildings, etc. The imaging unit 10 is electrically connected to the first image processing unit 30 and the second image processing unit 40, and transmits the generated image data to the first image processing unit 30 and the second image processing unit 40. In this embodiment, the imaging unit 10 is a camera. However, any configuration other than a camera can be used as the imaging unit 10, as long as it captures images of objects around the vehicle VW and generates an image.
[0012] The first element acquisition unit 20 acquires a numerical value representing the intensity of elements in the environment surrounding the vehicle VW. In this embodiment, the elements in the environment surrounding the vehicle VW are brightness, and the first element acquisition unit 20 is an illuminance sensor. The illuminance sensor is positioned on the vehicle VW so as to receive ambient light from the front of the vehicle VW. The illuminance sensor acquires a numerical value representing the intensity of the received light. In this embodiment, the unit of the numerical value is lux (lx). The numerical value acquired by the element acquisition unit in this specification includes the element acquisition unit detecting the degree of intensity of the element and outputting a value corresponding to that degree. The first element acquisition unit 20 transmits a signal indicating the acquired numerical value to the processing selection unit 50.
[0013] Figure 2 is a block diagram showing the configuration of the first image processing unit 30, the second image processing unit 40, the processing selection unit 50, and the vehicle control unit 60. The first image processing unit 30 can perform computational processing on the image generated by the imaging unit 10 using a first neural network model. The first image processing unit 30 transmits the data obtained by the computational processing to the processing selection unit 50. In this embodiment, the first neural network model is a CNN (Convolutional neural network). The first image processing unit 30 includes an NPU (Neural Processing Unit) 31, a processor 32, and a memory 33. The NPU 31 performs computational processing using the CNN. The processor 32 controls the NPU 31 overall. The memory 33 stores the results obtained by the computational processing.
[0014] The second image processing unit 40 can perform calculations on the image generated by the imaging unit 10 using a second neural network model. The second image processing unit 40 transmits the data obtained by the calculations to the processing selection unit 50. In this embodiment, the second neural network model is a Transformer. The second image processing unit 40 includes an NPU 41, a processor 42, and a memory 43. The NPU 41 performs calculations using the Transformer. The processor 42 controls the NPU 41 overall. The memory 43 stores the results obtained by the calculations.
[0015] The processing selection unit 50 selects an image processing unit that performs arithmetic processing on an image from among the two image processing units, the first image processing unit 30 and the second image processing unit 40. Specifically, the processing selection unit 50 selects an image processing unit that performs arithmetic processing on an image based on a numerical value representing the intensity of the elements acquired by the first element acquisition unit 20. Further, the processing selection unit 50 transmits the data of the result of the arithmetic processing of the selected image processing unit to the vehicle control unit 60. In the present embodiment, the processing selection unit 50 selects an image processing unit based on a numerical value representing the intensity of the light received by the illuminance sensor. In the present embodiment, the processing selection unit 50 includes a processor 51 and a memory 52. The processor 51 selects an image processing unit. The memory 52 stores, in time series, the image processing unit selected by the processing selection unit 50 and the data of the result of the arithmetic processing by the selected image processing unit.
[0016] The vehicle control unit 60 controls the vehicle VW based on the data obtained by the arithmetic processing of the image processing unit selected by the processing selection unit 50. For example, when the data indicates that another vehicle is included in the image, the vehicle control unit 60 can operate a brake (not shown) of the vehicle VW to stop the vehicle VW. The vehicle control unit 60 can transmit and receive signals to and from the processing selection unit 50. The vehicle control unit 60 includes a processor 61, a ROM 62, and a RAM 63. The processor 61 executes various programs by expanding the various programs stored in the ROM 62 onto the RAM 63.
[0017] In the present embodiment, the vehicle control unit 60 includes a first mode MD1 and a second mode MD2. The first mode MD1 is a mode in which control is performed based on the image processing unit selected by the processing selection unit 50. The second mode MD2 is a mode in which the vehicle VW is controlled based on the data of the arithmetic processing output by the first image processing unit 30 without relying on the image processing unit selected by the processing selection unit 50. The vehicle control unit 60 executes control of the vehicle VW in the second mode MD2 while the image processing unit is not being selected by the processing selection unit 50.
[0018] Also, in the present embodiment, while the second mode MD2 is being executed by the vehicle control unit 60, arithmetic processing is being executed by both the first image processing unit 30 and the second image processing unit 40. Then, in the first mode MD1, the vehicle control unit 60 controls the vehicle VW based on the data of the result of the arithmetic processing output by the image processing unit selected by the processing selection unit 50 among the first image processing unit 30 and the second image processing unit 40.
[0019] A2. Features of each of the first image processing unit 30 and the second image processing unit 40: FIG. 3 is a diagram showing the detection accuracy of each of the first image processing unit 30 and the second image processing unit 40 with respect to brightness. In FIG. 3, the intensity of brightness is shown on the horizontal axis, and the detection accuracy of the image processing unit is shown on the vertical axis. The detection accuracy in this specification refers to the ratio of correct answers when the image processing unit performs arithmetic processing on a previously prepared image. The detection accuracy is pre-tested by an operator for each element. In the present embodiment, it shows the detection accuracy when the image processing unit performs arithmetic processing on a plurality of images with different brightnesses. Hereinafter, the detection accuracy of the image processing unit is referred to as "accuracy".
[0020] As shown in FIG. 3, in the first range FR1, the accuracy of the first image processing unit 30 using CNN is higher than the accuracy of the second image processing unit 40 using Transformer. Also, in the second range SR1 of intensity, the accuracy of the second image processing unit 40 is higher than the accuracy of the first image processing unit 30. In FIG. 3, the line representing the accuracy of the first image processing unit 30 is shown as a solid line, and the line representing the accuracy of the second image processing unit 40 is shown as a dashed line. The same applies to FIG. 6 described later. In the present embodiment, the smallest value in the first range FR1 is larger than the first threshold T1. Also, the largest value in the second range SR1 is smaller than the first threshold T1.
[0021] In this embodiment, the first threshold T1 is the brightness intensity of image data at which the accuracy of the first image processing unit 30 becomes equal to the accuracy of the second image processing unit 40, when the first image processing unit 30 and the second image processing unit 40 each perform calculation processing on multiple images with different brightness levels prepared by the operator.
[0022] Furthermore, the first threshold T1 can be determined by methods other than those described above. For example, the first threshold T1 may be the highest intensity among the brightness intensities measured by a worker at the entrances of any multiple tunnels. The brightness at the entrances of any multiple tunnels is measured by a worker 1m inside the tunnel from the entrance of each tunnel. Alternatively, brightness may be measured at multiple locations in any multiple tunnels, and the highest intensity among them may be used as the first threshold T1.
[0023] Furthermore, it is generally known that CNNs exhibit less accuracy change compared to Transformers in response to changes in brightness over time during sunny daytime conditions. The brightness during sunny daytime conditions falls within the first range, FR1, in Figure 3. On the other hand, CNNs are known to exhibit lower accuracy compared to Transformers in situations where the brightness intensity of the external environment of the vehicle VW is lower than during sunny daytime conditions, such as inside tunnels or in rainy weather. The brightness in situations where the intensity is lower than during sunny daytime conditions falls within the second range, SR1, in Figure 3.
[0024] A3. Selection of the first image processing unit 30 or the second image processing unit 40 by the processing selection unit 50: Figure 4 is a process diagram showing an example of the selection of an image processing unit by the image processing device 1. In step S10 of Figure 4, after the vehicle VW is powered on by the user, the vehicle control unit 60 starts controlling the vehicle VW in second mode MD2. After a predetermined time has elapsed since the start of second mode MD2, the process moves to step S20. The predetermined time can be arbitrarily set to 5 seconds, 10 seconds, 1 minute, etc.
[0025] In step S20, the image data generated by the imaging unit 10 is transmitted to the first image processing unit 30 and the second image processing unit 40. In step S30, the first image processing unit 30 and the second image processing unit 40 perform calculations on the image data. The first image processing unit 30 and the second image processing unit 40 then transmit the data of the calculation results to the processing selection unit 50. The processing selection unit 50 transmits the data of the calculation results from the first image processing unit 30 to the vehicle control unit 60. The vehicle control unit 60 then controls the vehicle VW based on the calculation results output by the first image processing unit 30. From this point until processing is complete, the generation of images by the imaging unit 10 and the calculations and output of results by the first image processing unit 30 and the second image processing unit 40 are performed every second.
[0026] In step S40, the illuminance sensor acquires a numerical value corresponding to the intensity of the received light and transmits the numerical data to the processing selection unit 50. Note that the processing in step S40 may be performed before the processing in step S20. In step S50, the processing selection unit 50 selects an image processing unit based on the numerical value representing the intensity of the light received by the illuminance sensor. Specifically, if the numerical value is greater than or equal to the first threshold T1, the processing selection unit 50 selects the first image processing unit 30. If the numerical value is less than the first threshold T1, the processing selection unit 50 selects the second image processing unit 40. In this embodiment, the processing selection unit 50 selects the first image processing unit 30 and transmits the data of the calculation result received from the first image processing unit 30 in step S30 to the vehicle control unit 60.
[0027] In step S60, the vehicle control unit 60 controls the vehicle VW using the first mode MD1. In this embodiment, the vehicle control unit 60 controls the vehicle VW based on the data from the calculation processing by the first image processing unit 30. In step S70, the vehicle control unit 60 determines whether or not to terminate the process. If the vehicle VW is stopped and the power to the vehicle VW is turned off by the user within a predetermined time from the execution of the process in step S60, the process is terminated. If the vehicle VW is still being driven by the user after a predetermined time has elapsed from the execution of the process in step S60, the vehicle control unit 60 determines not to terminate the process, and the process proceeds back to step S40 with the first mode MD1 still running.
[0028] In this embodiment, the processing selection unit 50 causes either the first image processing unit 30 or the second image processing unit 40 to perform calculation processing based on the numerical values of the elements acquired by the first element acquisition unit 20. This enables high-precision image processing.
[0029] In this embodiment, the image processing unit 50 selects an image processing unit based on a numerical value representing the intensity of light received by the illuminance sensor. By selecting the image processing unit based on brightness, which is considered to have a high frequency of influence on the accuracy of the calculation processing performed by the image processing unit, high-precision image processing can be achieved.
[0030] It is assumed that the vehicle VW is most often driven during the daytime on sunny days. In this embodiment, as shown in Figure 3, in the first brightness intensity range FR1, the accuracy of the first image processing unit 30 using a CNN is higher than the accuracy of the second image processing unit 40 using a Transformer. Furthermore, while the second mode MD2 is being executed, and the image processing unit 50 has not made a selection, the vehicle VW is controlled based on the data obtained by the calculations of the first image processing unit 30. This makes it possible to suppress the decrease in the accuracy of the calculations that occurs in response to brightness intensity when the image processing unit has not made a selection by the image processing unit 50, especially during the daytime on sunny days, which is a highly probable driving environment.
[0031] Furthermore, in the second mode MD2, calculation processing is performed by both the first image processing unit 30 and the second image processing unit 40. Therefore, when control is switched from the second mode MD2 to the first mode MD1, the data resulting from the calculation processing is prepared. As a result, the vehicle control unit 60 can quickly control the vehicle VW based on the data.
[0032] Furthermore, it is generally known that CNNs are affected by noise and their accuracy decreases when the noise during A / D conversion, which digitizes the input analog signal, becomes large. Noise during A / D conversion is likely to increase in situations where the surroundings of the vehicle VW become dark, such as when the vehicle VW enters a tunnel or enters a building. Therefore, when the numerical value acquired by the first element acquisition unit 20 falls within the second range SR1, which is smaller than the first threshold T1, the second image processing unit 40 using a Transformer is selected by the processing selection unit 50, thereby executing high-precision calculation processing.
[0033] B. Second Embodiment: The image processing apparatus 1B of the second embodiment differs from the first embodiment in that the first image processing unit 30B is an image processing unit using a Transformer, the second image processing unit 40B is an image processing unit using a CNN, the processing selection unit 50 selects an image processing unit based on the intensity of rain rather than brightness, and the element acquisition unit and the processing of step S30 in Figure 4. Other components are the same as in the first embodiment, so the same reference numerals are used and detailed descriptions are omitted.
[0034] Figure 5 shows a vehicle VW according to the second embodiment. First, in the second embodiment, as described above, a first image processing unit 30B using a Transformer and a second image processing unit 40B using a CNN perform calculations on the image.
[0035] The second element acquisition unit 20B of the second embodiment acquires a numerical value representing the intensity of rain in the environment surrounding the vehicle VW. The second element acquisition unit 20B comprises a network information unit 210B and a rain amount detection unit 220B. The network information unit 210B acquires the current location using signals from GPS satellites. Then, it acquires the amount of precipitation at the current location via the network. Specifically, the network information unit 210B acquires the amount of precipitation that has reached the horizontal surface of the ground at the present time within a certain period of time. The network information unit 210B converts the acquired amount of precipitation into the amount of rain that falls per square meter per second and acquires it as a numerical value representing the intensity of rain. The numerical value acquired by the network information unit 210B is denoted as the first numerical value FN. The first numerical value FN is output to the processing selection unit 50.
[0036] The rainfall detection unit 220B directly detects rain around the vehicle VW. The rainfall detection unit 220B is installed on the windshield FR of the vehicle VW. In this embodiment, the rainfall detection unit 220B includes a rain sensor. The rain sensor detects rain in contact with the windshield FR using a combination of an infrared LED and a Si photodiode. Specifically, infrared light is emitted from the infrared LED toward the windshield FR. If there are no water droplets on the windshield FR, the infrared LED is reflected by the windshield FR and incident on the Si photodiode. On the other hand, if there are water droplets on the windshield FR, the infrared LED does not reflect off the windshield FR but passes through the water droplets. The amount of rainfall is measured by the amount of reflected light incident on the Si photodiode. The rainfall detection unit 220B converts the amount of rainfall measured by the rain sensor into the amount of rain that falls per square meter per second and outputs it as a numerical value representing the intensity of the rain. The numerical value acquired by the rainfall detection unit 220B is denoted as the second numerical value SN. The second numerical value SN is output to the processing selection unit 50.
[0037] The processing selection unit 50 uses the larger of the first numerical value FN and the second numerical value SN as a numerical value representing the rain intensity acquired by the second element acquisition unit 20B to make a selection for the image processing unit.
[0038] Figure 6 shows the accuracy of the image processing unit with respect to rainfall. The horizontal axis represents the amount of rainfall per square meter per second, and the vertical axis represents the accuracy of the image processing unit. Generally, it is known that the accuracy of CNNs decreases due to rain, snow, dust, etc., transferred to the image data. As shown in Figure 6, in the first range FR2, the accuracy of the first image processing unit 30B using a Transformer is higher than the accuracy of the second image processing unit 40B using a CNN. Also, in the second range SR2, the accuracy of the second image processing unit 40B using a CNN is higher than the accuracy of the first image processing unit 30B using a Transformer. In the second embodiment, the smallest value in the first range FR2 is greater than the second threshold T2. Also, the largest value in the second range SR2 is less than the second threshold T2.
[0039] In the second embodiment, the second threshold T2 is the amount of rainfall at which the accuracy of the first image processing unit 30B and the second image processing unit 40B matches, as determined using multiple images taken during rainy weather, where the amount of rainfall per second per square meter differs. The images are arbitrary images prepared by the operator. The second threshold T2 may also be an arbitrary value determined by the operator. For example, the second threshold T2 can be defined as the amount of rainfall at which the area occupied by raindrops in a 1-square-meter image is greater than or equal to a predetermined area.
[0040] Figure 7 is a process diagram showing an example of the selection of an image processing unit by the image processing device 1B of the second embodiment. The processing in steps S10 and S20 is the same as in Figure 4, so the details are omitted. In step S30B of Figure 7, the vehicle control unit 60 controls the vehicle VW based on the data resulting from the calculation processing by the second image processing unit 40. In other words, the vehicle VW is controlled based on the data resulting from the calculation processing using CNN.
[0041] In step S40B, the rain sensor outputs a first numerical value FN, and the rain amount detection unit 220B outputs a second numerical value SN. In step S50B, the processing selection unit 50 uses the larger of the first numerical value FN and the second numerical value SN to select the image processing unit. Specifically, if the numerical value to be used is greater than or equal to the second threshold T2, the processing selection unit 50 selects the first image processing unit 30B. On the other hand, if the numerical value to be used is less than the second threshold T2, it selects the second image processing unit 40B. The processing from step S60 onward is the same as in Figure 4.
[0042] In the second embodiment, the processing selection unit 50 uses the larger of the first numerical value FN and the second numerical value SN to select the image processing unit. This enables highly accurate calculation processing according to the intensity of the rain.
[0043] C. Third Embodiment: The image processing apparatus 1C of the third embodiment differs from the above embodiment in that the image processing apparatus 1C comprises a first element acquisition unit 20 and a second element acquisition unit 20B, and the image processing unit is selected by the processing selection unit 50. Other components are the same as in the first embodiment, so the same reference numerals are used, and detailed descriptions are omitted.
[0044] In the third embodiment, the image processing device 1C includes a first element acquisition unit 20 and a second element acquisition unit 20B. The first element acquisition unit 20 acquires a numerical value representing the brightness intensity of the environment surrounding the vehicle VW. The second element acquisition unit 20B acquires a numerical value representing the intensity of rain. The first element acquisition unit 20 is an illuminance sensor and has the same configuration as in the first embodiment. The second element acquisition unit 20B includes a network information unit 210B and a rain amount detection unit 220B. The network information unit 210B and the rain amount detection unit 220B have the same configuration as in the second embodiment.
[0045] In the third embodiment, the first image processing unit 30 is an image processing unit using a CNN, and the second image processing unit 40 is an image processing unit using a Transformer. In the third embodiment, in the first range FR1 of brightness intensity shown in Figure 3, the accuracy of the first image processing unit 30 is higher than that of the second image processing unit 40. Also, in the second range SR1, the accuracy of the second image processing unit 40 is higher than that of the first image processing unit 30. On the other hand, in the first range FR2 of rain intensity shown in Figure 6, the accuracy of the second image processing unit 40 is higher than that of the first image processing unit 30. Also, in the second range SR2, the accuracy of the first image processing unit 30 is higher than that of the second image processing unit 40.
[0046] Figure 8 is a process diagram showing an example of the selection of an image processing unit by the image processing device 1C of the third embodiment. The processing from step S10 to step S30 is the same as in Figure 3, so the explanation is omitted. In step S40C, numerical values representing the intensity of the elements are output from the first element acquisition unit 20 and the second element acquisition unit 20B, respectively. In the third embodiment, the net information unit 210B and the rainfall detection unit 220B are electrically connected, and the larger of the first numerical value FN and the second numerical value SN is output from the second element acquisition unit 20B.
[0047] In step S50C, the processing selection unit 50 selects an image processing unit based on numerical values representing the intensity of the elements acquired by the first element acquisition unit 20 and the second element acquisition unit 20B, respectively. The selection of the image processing unit based on the numerical values acquired by the first element acquisition unit 20 is performed in the same manner as in Figure 4. The selection of the image processing unit based on the numerical values acquired by the second element acquisition unit 20B is performed in the same manner as in Figure 7.
[0048] In step 60C, the processing selection unit 50 determines whether the second image processing unit 40 has been selected a predetermined number of times or more. In step S50C, if the second image processing unit 40 is selected based on the numerical value acquired by the first element acquisition unit 20, it is indicated that the second image processing unit 40 has been selected once. In step S50C, if the second image processing unit 40 is selected based on the numerical value acquired by the second element acquisition unit 20B, it is indicated that the second image processing unit 40 has been selected once. In the third embodiment, the predetermined number of times is one. If the processing selection unit 50 selects the second image processing unit 40 one or more times, the process proceeds to step S70C. If the processing selection unit 50 does not select the second image processing unit 40 even once, and only the first image processing unit 30 is selected, the process proceeds to step S80C. The predetermined number of times is determined by the operator and input into the processing selection unit 50 in advance.
[0049] In step S70C, the vehicle control unit 60 executes the first mode MD1. In step S70C, the second image processing unit 40 performs calculation processing. In step S80C, the vehicle control unit 60 executes the first mode MD1. In step 80C, the first image processing unit 30 performs calculation processing. After that, the process moves to step S90C. The processing in step S90C is the same as the processing in step S70 in Figure 4.
[0050] In the third embodiment, the image processing device 1C can select an image processing unit based on numerical values representing the intensity of one or more elements.
[0051] D. Fourth Embodiment: The fourth embodiment differs from the first embodiment in that the image processing apparatus 1D includes a third element acquisition unit 20D (not shown), and the third element acquisition unit 20D acquires numerical values representing the intensity of elements in the environment surrounding the vehicle VW based on the image acquired by the imaging unit 10, and does not include an illuminance sensor. The other components are the same as in the first embodiment, so the same reference numerals are used and detailed descriptions are omitted.
[0052] In the fourth embodiment, the imaging unit 10 transmits the generated image data to the third element acquisition unit 20D. The third element acquisition unit 20D detects the brightness per unit pixel of the image data. It then calculates the average brightness of the image, obtains a numerical value representing the intensity, and outputs the numerical value to the processing selection unit 50. Based on the numerical value obtained by the third element acquisition unit 20D, the processing selection unit 50 selects an image processing unit in the same manner as in the first embodiment.
[0053] In the fourth embodiment, numerical values representing the intensity of elements in the environment surrounding the vehicle VW can be obtained from the image acquired by the imaging unit 10 without using a device to acquire numerical values representing the intensity of elements.
[0054] E. Other embodiments: E1: Other Embodiments 1: (1) In the first embodiment described above, the first element acquisition unit 20 acquires a numerical value representing the brightness intensity. In the second embodiment described above, the second element acquisition unit 20B acquires a numerical value representing the rain intensity. For example, the element acquisition unit may acquire a numerical value representing the amount of dust or snow adhering to the windshield.
[0055] (2) In the above embodiment, the smallest value in the first range FR1 is greater than the first threshold T1. Also, the largest value in the second range SR1 is less than the first threshold T1. Also, the smallest value in the first range FR2 is greater than the second threshold T2. Also, the largest value in the second range SR2 is less than the second threshold T2. Depending on the method for determining the first threshold, the first threshold may belong to either the first range or the second range. For example, if the first threshold is the greatest intensity among the brightness intensities of any multiple tunnels measured by an operator, the first threshold may belong to either the first range or the second range. Similarly, depending on the method for determining the second threshold, the second threshold may belong to either the first range or the second range.
[0056] (3) In the above embodiment, the processing selection unit 50 selects an image processing unit based on whether the numerical value acquired by the element acquisition unit is equal to or greater than the first threshold T1 or the second threshold. In this embodiment, the processing selection unit may select an image processing unit without basing it on the threshold, for example, in an embodiment where the threshold belongs to either the first range or the second range. In this embodiment, the processing selection unit may select the first image processing unit if the numerical value acquired by the element acquisition unit belongs to the first range, select the second image processing unit if the numerical value belongs to the second range, and select the first image processing unit or the second image processing unit, whichever is predetermined by the operator, if the numerical value does not belong to either the first range or the second range.
[0057] (4) In the first embodiment described above, after the vehicle VW is powered on by the user in step S10 of Figure 4, the vehicle control unit 60 controls the vehicle VW in second mode MD2. The second mode may be executed at times other than when the vehicle is powered on. For example, the vehicle control unit may execute vehicle control in second mode after automatic driving has started, or the vehicle control unit may execute vehicle control in second mode after a predetermined amount of time has elapsed since the vehicle started moving.
[0058] (5) For example, the image processing device may include a control unit that functions as a first image processing unit, a second image processing unit, and a processing selection unit. In this embodiment, the control unit includes an NPU, a processor, and memory, the NPU performs calculations using CNN and Transformer, and the processing selection unit causes either the first image processing unit or the second image processing unit to perform the calculations. The processor performs overall control of the NPU. In this form of image processing device, since the first image processing unit, the second image processing unit, and the processing selection unit are controlled by the same control unit, the processing selection unit can quickly switch between the first image processing unit and the second image processing unit.
[0059] (6) In the above embodiment, if the numerical value representing the intensity of light received by the illuminance sensor is greater than or equal to the first threshold, the processing selection unit 50 selects the first image processing unit 30. If the numerical value is greater than the first threshold, the processing selection unit may select the first image processing unit, and if it is less than or equal to the first threshold, it may select the second image processing unit.
[0060] (7) In the above embodiment, the vehicle VW is capable of driving using ADAS (Advanced Driving Assistant System). For example, the vehicle may be configured to be capable of driving using AD (Autonomous Driving).
[0061] (8) In the first embodiment described above, if the vehicle control unit 60 determines that the process should not be terminated, the process proceeds back to step S40 in Figure 4. The process may also be terminated if the vehicle control unit determines that the automatic driving is finished. Alternatively, the process may proceed to step S40 after the process in step S60 in Figure 4. In this embodiment, if the user terminates the driving, the process will be terminated regardless of which process has been executed.
[0062] (9) In the first embodiment described above, after a predetermined time has elapsed since the start of the second mode MD2, the process proceeds to step S20. Alternatively, for example, the process may proceed to step S20 immediately after the start of the second mode, and then proceed to step S60 after a predetermined time has elapsed.
[0063] E2: Other Embodiments 2: (1) In the third embodiment described above, the image processing apparatus 1B is equipped with two element acquisition units: a first element acquisition unit 20 and a second element acquisition unit 20B. The image processing apparatus may be equipped with more than two element acquisition units, such as three or five. In this embodiment, each element acquisition unit acquires a numerical value representing the intensity of one element from among one or more elements such as rain, brightness, and dust. The processing selection unit then selects an image processing unit based on the numerical values of the elements acquired by the multiple element acquisition units.
[0064] (2) In the third embodiment described above, the predetermined number of times is one. However, the operator can determine the predetermined number of times depending on the environment in which the vehicle is driven, and the type and number of elements. For example, in an embodiment in which the image processing device has two element acquisition units, the second image processing unit may be configured to perform calculation processing in the first mode only when the second image processing unit is selected twice by the processing selection unit. Alternatively, for example in an embodiment in which the image processing device has five element acquisition units, the second image processing unit may be configured to perform calculation processing in the first mode when the second image processing unit is selected once or more times, or when it is selected five times.
[0065] E3: Other Embodiments 3: In the third embodiment described above, if the second image processing unit 40 has been selected a predetermined number of times or more, the processing selection unit 50 selects the second image processing unit 40. For example, in an embodiment in which the image processing device includes a plurality of element acquisition units, one of which is an illuminance sensor, the processing selection unit may select the image processing unit based solely on the numerical value acquired by the illuminance sensor.
[0066] E4: Other Embodiments 4: (1) In the above embodiment, the image processing device 1 includes a vehicle control unit 60. However, in an embodiment in which, for example, the processing selection unit controls the vehicle, the image processing device does not need to include a vehicle control unit.
[0067] (2) In the above embodiment, the vehicle control unit 60 includes a first mode MD1 and a second mode MD2. However, in an embodiment in which the vehicle is driven by the user until numerical values are acquired by the element acquisition unit, the vehicle control unit does not include the second mode and only includes the first mode, and control by the vehicle control unit is not performed while the image processing unit has not made a selection. Also, in an embodiment in which the image processing unit is selected by the processing selection unit immediately after the vehicle is powered on, the second mode does not need to be executed.
[0068] E5: Other Embodiments 5: (1) In the above embodiment, while the second mode MD2 is being executed, calculation processing is performed by the first image processing unit 30 and the second image processing unit 40, and in the first mode MD1, the vehicle VW is controlled based on the calculation processing data output by either the first image processing unit 30 or the second image processing unit 40. In an embodiment in which the vehicle control unit does not have a second mode, for example, while the processing selection unit has not made a selection of the image processing unit, for example, only the calculation processing by the first image processing unit may be executed, and calculation processing may not be performed by either the first image processing unit or the second image processing unit. Also, in an embodiment in which the vehicle control unit does not have a second mode, for example, the processing selection unit that receives a signal indicating a numerical value acquired by the element acquisition unit may select an image processing unit, and then the selected image processing unit may receive image data from the imaging unit and execute calculation processing.
[0069] E6: Other Embodiments 6: (1) In the fourth embodiment described above, the third element acquisition unit 20D acquires a numerical value representing the intensity of an element based on the image generated by the imaging unit 10. For example, the element acquisition unit may include an illuminance sensor and an illuminance processing unit that acquires a numerical value representing the intensity of an element based on the image generated by the imaging unit, and the illuminance processing unit and the illuminance sensor may be used in combination to acquire a numerical value representing the intensity of an element. In this embodiment, the image processing unit may be selected by the processing selection unit based on the smaller of the numerical values acquired by the illuminance processing unit and the illuminance sensor, respectively. Also in this embodiment, if an abnormality occurs in the illuminance sensor, the processing selection unit may select the image processing unit using the numerical value acquired by the illuminance processing unit as a substitute for the illuminance sensor.
[0070] (2) The element acquisition unit may use the network information unit, the rainfall detection unit, and the imaging unit in combination to acquire a numerical value representing the intensity of rain.
[0071] E7: Other Embodiments 7: (1) In the second embodiment described above, the second element acquisition unit 20B comprises a network information unit 210B and a rainfall detection unit 220B. For example, the element acquisition unit may comprise only one of the network information unit and the rainfall detection unit.
[0072] (2) In the second embodiment described above, the network information unit 210B and the rainfall detection unit 220B output the amount of rain that fell per square meter in one second. Alternatively, the network information unit and the rainfall detection unit may output the amount of rain that fell per square meter in one minute.
[0073] (3) In the second embodiment described above, the processing selection unit 50 uses the larger of the first numerical value FN and the second numerical value SN as a numerical value representing the rain intensity acquired by the second element acquisition unit 20B to make a selection in the image processing unit. Alternatively, for example, the second element acquisition unit may transmit a signal to the processing selection unit indicating the larger of the first numerical value and the second numerical value.
[0074] Furthermore, for example, when the network information unit and the rainfall detection unit each acquire a value exceeding a predetermined threshold, they may transmit a signal indicating that rain has been detected and a signal indicating the value to the processing selection unit, and the processing selection unit may select an image processing unit based on the received value. In this embodiment, when either the network information unit or the rainfall detection unit detects rain, the processing selection unit can select an image processing unit. Because the image processing unit is selected quickly when it is raining in the environment in which the vehicle is driving, high-precision calculation processing becomes possible.
[0075] E8: Other Embodiments 8: In the first embodiment described above, the first threshold T1 is the brightness intensity of image data in which the accuracy of the first image processing unit 30 becomes equal to the accuracy of the second image processing unit 40 when the first image processing unit 30 and the second image processing unit 40 each perform calculation processing on multiple images with different brightness levels prepared by the operator, and the first range FR1 is a range of values greater than the first threshold T1. For example, the first range of brightness may refer to a range greater than the brightness intensity inside the tunnel. In this specification, the brightness intensity inside the tunnel is either the average road surface luminance of the basic lighting of the tunnel as defined in the Road Lighting Facility Installation Standards of the Ministry of Land, Infrastructure, Transport and Tourism, or a range greater than the road surface luminance of each part of the entrance lighting as defined in the Road Lighting Facility Installation Standards of the Ministry of Land, Infrastructure, Transport and Tourism.
[0076] This disclosure is not limited to the embodiments and modifications described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments and modifications corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate in order to solve some or all of the above-mentioned problems, or to achieve some or all of the above-mentioned effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate. [Explanation of Symbols]
[0077] 1, 1B, 1C, 1D...Image processing device, 10...Imaging unit, 20...First element acquisition unit, 20B...Second element acquisition unit, 20D...Third element acquisition unit, 30, 30B...First image processing unit, 31...NPU of the first image processing unit, 32...Processor of the first image processing unit, 33...Memory of the first image processing unit, 40, 40B...Second image processing unit, 41...NPU of the second image processing unit, 42...Processor of the second image processing unit, 43...Memory of the second image processing unit, 50...Processing Selection unit, 51...Processor of the processing selection unit, 52...Memory of the processing selection unit, 60...Vehicle control unit, 61...Processor of the vehicle control unit, 62...ROM, 63...RAM, 210B...Network information unit, 220B...Rain amount detection unit, FN...First numerical value, FR...Windshield, FR1, FR2...First range, MD1...First mode, MD2...Second mode, SN...Second numerical value, SR1, SR2...Second range, T1...First threshold, T2...Second threshold, VW...Vehicle
Claims
1. An image processing device (1) mounted on a vehicle (VW), An imaging unit (10) that images an object and generates an image including the object, One or more element acquisition units (20) that acquire numerical values representing the intensity of one or more elements of the environment surrounding the vehicle, A first image processing unit (30) that can perform computational processing on the image using a first neural network model, A second image processing unit (40) that can perform computational processing on the image using a second neural network model, A processing selection unit (50) selects an image processing unit from among the first image processing unit and the second image processing unit to perform calculation processing on the image, A vehicle control unit (60) controls the vehicle based on the data obtained through calculation processing, Equipped with, For some of the aforementioned elements, in a first range of intensity, the accuracy of the first image processing unit is higher than that of the second image processing unit, and in a second range lower than the first range, the accuracy of the second image processing unit is higher than that of the first image processing unit. The processing selection unit, Based on the numerical values representing the intensity of the elements acquired by the one or more element acquisition units (10), an image processing unit is selected to perform calculation processing. The vehicle control unit, A first mode (MD1) is performed based on the image processing unit selected by the processing selection unit, A second mode (MD2) controls the vehicle based on the calculation data output by the first image processing unit, rather than based on the image processing unit selected by the processing selection unit. Equipped with, An image processing device that controls the vehicle in the second mode while the image processing unit has not made a selection of the image processing device.
2. An image processing apparatus according to claim 1, An image processing apparatus wherein, with respect to other elements, the accuracy of the second image processing unit is higher than that of the first image processing unit in a first range of intensity, and the accuracy of the first image processing unit is higher than that of the second image processing unit in a second range lower than the first range.
3. An image processing apparatus according to claim 1, An image processing apparatus in which, with respect to brightness among the one or more elements, the accuracy of the calculation processing of the first image processing unit is higher than that of the calculation processing of the second image processing unit in the first range, and the accuracy of the calculation processing of the second image processing unit is higher than that of the calculation processing of the first image processing unit in the second range.
4. An image processing apparatus according to claim 1, While the second mode is being executed by the vehicle control unit, The first image processing unit and the second image processing unit perform calculation processing, The vehicle control unit, An image processing apparatus that, in the first mode, controls the vehicle based on the calculation processing data output by the image processing apparatus selected by the processing selection unit from among the first image processing apparatus and the second image processing apparatus.
5. An image processing apparatus according to claim 1, The element acquisition unit (20D) is an image processing device that acquires numerical values representing the intensity of elements in the environment surrounding the vehicle based on the image generated by the imaging unit.
6. An image processing apparatus according to claim 1, Of the one or more element acquisition units, one of the element acquisition units is: A network information unit (210B) obtains a first numerical value (FN) representing the intensity of rainfall via the network, The system includes a detection unit (220B) that directly detects rain around the vehicle and obtains a second numerical value (SN) representing the intensity of the rain, Output the first numerical value and the second numerical value. The processing selection unit selects the image processing unit by using the larger of the first and second numerical values as a numerical value representing the rain intensity acquired by the element acquisition unit.
7. An image processing apparatus according to claim 1, The first range of brightness among the one or more elements is: An image processing device whose range is greater than the numerical value representing the brightness intensity inside the tunnel.
8. An image processing device (1) mounted on a vehicle (VW), An imaging unit (10) that images an object and generates an image including the object, One or more element acquisition units (20) that acquire numerical values representing the intensity of one or more elements of the environment surrounding the vehicle, A first image processing unit (30) that can perform computational processing on the image using a first neural network model, A second image processing unit (40) that can perform computational processing on the image using a second neural network model, A processing selection unit (50) selects an image processing unit from among the first image processing unit and the second image processing unit to perform calculation processing on the image, Equipped with, For some of the aforementioned elements, in a first range of intensity, the accuracy of the first image processing unit is higher than that of the second image processing unit, and in a second range lower than the first range, the accuracy of the second image processing unit is higher than that of the first image processing unit. The processing selection unit, Based on the numerical values representing the intensity of the elements acquired by the one or more element acquisition units (10), an image processing unit is selected to perform calculation processing. Of the one or more element acquisition units, one of the element acquisition units is: A network information unit (210B) obtains a first numerical value (FN) representing the intensity of rainfall via the network, The system includes a detection unit (220B) that directly detects rain around the vehicle and obtains a second numerical value (SN) representing the intensity of the rain, Output the first numerical value and the second numerical value. The processing selection unit selects the image processing unit by using the larger of the first and second numerical values as a numerical value representing the rain intensity acquired by the element acquisition unit.