Image processing apparatus, imaging apparatus, and image processing method

Abstract

The present invention, in one aspect thereof, provides an image processing device for obtaining ambient-environment recognition information for controlling a vehicle by processing a series of pairs of captured images acquired by means of a pair of camera modules that capture images of the ambient environment of the vehicle, wherein the image processing device is configured such that it is possible to select captured-image regions to be used when generating parallax images from the pairs of captured images, and the selection of regions is switched depending on the number of pairs of captured images that are loaded for image processing per unit time.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method for generating a disparity image from images captured by a plurality of cameras.

Background Art

[0002] In order to improve the driving safety of a vehicle, a system has been studied that detects vehicles and pedestrians crossing in front with sensors mounted on the vehicle and warns the driver or activates an automatic brake when there is a possibility of colliding with the vehicle or pedestrian. As sensors for monitoring the front of the vehicle, there are millimeter-wave radars, lidars, cameras, etc. As types of cameras, there are monocular cameras and stereo cameras using a plurality of cameras.

[0003] A stereo camera can measure the distance to a photographed object by utilizing the disparity of an overlapping area photographed by two cameras arranged at a predetermined interval. Therefore, a stereo camera can accurately grasp the collision risk degree to an object in front. However, a stereo camera processes and uses captured images obtained from two or more cameras. Therefore, usually, in a stereo camera, when using the same resolution / angle of view for the image processing part, the amount of calculation is larger than that of an electronic circuit that handles one camera, and the scale of the electronic circuit increases.

[0004] For this reason, in order to suppress the angle of view of a stereo camera, a stereo camera device has been devised that reduces the overlapping area (stereo area) of a plurality of cameras, treats the non-overlapping area as a monocular area, and reduces the angle of view for image processing as a stereo camera for the entire captured image (see Patent Document 1). In this stereo camera device, the distance information of the monocular area detects the distance to a front object based on the fact that the stereo area and the road surface height are equal or the front object information reflected in the past. That is, in the monocular area, the distance to a front object is estimated with high accuracy based on the distance information detected in the stereo area.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2017-96777 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, when objects suddenly appear from the side or when turning right or left at intersections, the object to be detected first appears in the monocular field, and information about the object detected in the stereo field in the past cannot be used. At this time, the distance to the object must be estimated under the assumption that the stereo field and the road surface are at the same height. When the object finally approaches the vehicle, it will appear in the stereo field, and the distance to the object can be calculated more accurately in the stereo field. However, even if the distance to the object is calculated accurately at this time, if there is a large error between the distance obtained in the monocular field and the actual distance, the risk of collision and the possibility of sudden braking increase, affecting driving safety.

[0007] This invention has been made in view of the above circumstances, and provides a method for processing overlapping regions of images captured by multiple cameras that simultaneously suppresses an increase in the size of the electronic circuit and ensures driving safety. [Means for solving the problem]

[0008] To solve the above problems, an image processing device according to one aspect of the present invention is an image processing device for processing a series of captured images obtained by a pair of camera modules that capture the surrounding environment of a vehicle, and obtaining surrounding environment recognition information for controlling the vehicle, wherein the region of the captured images to be used when generating a disparity image from the pair of captured images can be selected, and the selection of the region is switched according to the number of captured images of the pair that are incorporated into the image processing per unit time. [Effects of the Invention]

[0009] According to at least one aspect of the present invention, in processing overlapping regions of images captured by multiple cameras, by switching regions of the captured images according to the number of captured images to be incorporated into image processing per unit time, it is possible to suppress an increase in the size of the electronic circuit and ensure driving safety at the same time. Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the functional configuration of an imaging device equipped with an image processing device according to the first embodiment, and an example of retained data during high-speed travel. [Figure 2] This is an enlarged view of an example of retained data (image) during high-speed travel in the imaging device according to the first embodiment. [Figure 3] This figure shows an example of an image captured and input to an image processing device while driving on a highway, according to the first embodiment. [Figure 4] This is a timing chart showing an example of the image processing operation of the image processing device according to the first embodiment while driving on a highway. [Figure 5] This figure shows an example of the functional configuration of an imaging device equipped with an image processing device according to the first embodiment, and an example of retained data during low-speed driving. [Figure 6] This is an enlarged view of an example of retained data (image) during low-speed driving in the image processing apparatus according to the first embodiment. [Figure 7] This figure shows an example of an image captured and input to an image processing device while driving on a public road, according to the first embodiment. [Figure 8] This is a timing chart showing an example of the image processing operation of the image processing device according to the first embodiment when driving on a public road. [Figure 9] This figure shows an example of an image captured and input to an image processing device in the second embodiment. [Figure 10] This figure shows an example of an captured image input to an image processing device in the third embodiment. [Figure 11]It is a diagram showing an example of the functional configuration of an imaging device including an image processing device according to the fourth embodiment. [Figure 12] It is a diagram enlarging an example of held data (image) during vehicle travel in an image processing device according to the fourth embodiment. [Figure 13] It is a timing chart showing an example of an image processing operation during congested travel on a highway in an image processing device according to the fourth embodiment. [Figure 14] It is a block diagram showing an example of the hardware configuration of an image processing device included in an imaging device according to each embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, examples of embodiments for carrying out the present invention (hereinafter referred to as "embodiments") will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, the same reference numerals are given to the same components or components having substantially the same functions, and redundant explanations are omitted.

[0012] <First Embodiment> [Configuration of Imaging Device Including Image Processing Device] First, the configuration of an imaging device including an image processing device according to the first embodiment of the present invention will be described with reference to FIGS. 1 and 2. FIG. 1 is a diagram showing an example of the functional configuration of an imaging device including an image processing device according to the first embodiment and an example of held data during high-speed travel. FIG. 2 is a diagram enlarging an example of held data (image) during high-speed travel in an image processing device according to the first embodiment.

[0013] In FIG. 1, the imaging device 10 is a device that images the surrounding environment of the vehicle and recognizes the surrounding environment of the vehicle based on the acquired imaging image. The vehicle is assumed to be a vehicle performing automatic driving (AD) or a vehicle equipped with an advanced driver assistance system (ADAS).

[0014] The imaging device 10 includes a pair of camera modules 1 and 2 attached to a vehicle to image the surrounding environment of the vehicle such as the front of the vehicle, and an image processing device 3 that generates a disparity image from a plurality of pairs of captured image groups 11 and 12 acquired by the pair of camera modules 1 and 2 and finally outputs surrounding environment recognition information. The image processing device 3 processes a series of a pair of captured images acquired by the pair of camera modules 1 and 2 to obtain an identification information group 17 as surrounding environment recognition information for controlling the vehicle.

[0015] The pair of camera modules 1 and 2 is composed of a right camera module 1 and a left camera module 2. Hereinafter, the right camera module 1 will be referred to as the "right camera module 1", and the left camera module 2 will be referred to as the "left camera module 2". Each of the right camera module 1 and the left camera module 2 includes an imaging element and a circuit element that processes an electrical signal output from the imaging element. As the imaging element, a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like is used.

[0016] Each of the right camera module 1 and the left camera module 2 converts the light transmitted through the lens into an electrical signal by the imaging element, and performs AD conversion on the electrical signal by the circuit element and outputs it as a digital signal. The imaging device 10 monitors the front in a normal vehicle running state, and outputs a captured image group 11 (hereinafter also referred to as the "right captured image group 11") acquired by the right camera module 1 and a captured image group 12 (hereinafter also referred to as the "left captured image group 12") acquired by the left camera module 2 to the image processing device 3.

[0017] The image processing device 3 is configured to be able to select an area of the captured image used when generating a disparity image from a pair of captured images, and the selection of the area is switched according to the number of pairs of captured images taken into image processing per unit time. Hereinafter, the configuration and operation of the image processing device 3 will be described in detail.

[0018]

[0018] The image processing device 3 comprises blocks comprising an image preprocessing unit 4, a data storage area 5, a disparity image generation unit 6, a data storage area 7, and a recognized object processing unit 8. The disparity image generation unit 6, the data storage area 7, and the recognized object processing unit 8 are connected to each other via a system bus 9 so as to be able to communicate data with each other.

[0019] The image preprocessing unit 4 selects which images to process from the right image group 11 and the left image group 12. For this selection, information about the driving scene is provided to the image preprocessing unit 4 from the control core 84 of the recognition object processing unit 8. After selection, the image preprocessing unit 4 processes the images to be processed, aiming to remove noise and sharpen the images. Specific examples of processing include adjusting the overall brightness of the image, adjusting the brightness of each pixel, adjusting the gain for each brightness level, geometric correction, and shading correction. The processed right image (referred to as "right camera image" in the figure) 13 from the right camera module 1 and the processed left image (referred to as "left camera image" in the figure) 14 from the left camera module 2 are temporarily stored in the data holding area 5 of the image preprocessing unit 4 and input to the disparity image generation unit 6 in the appropriate processing cycle.

[0020] The parallax image generation unit 6 generates a parallax image 15 by combining the processed right image 13 and the processed left image 14. The parallax image 15 uses the difference between the processed right image 13 and the processed left image 14 to show distance information to an object in front of the vehicle.

[0021] In this embodiment, the parallax image generation unit 6 is composed of two parallax image generation unit blocks. Each parallax image generation unit block is configured to process half of the input image and output a parallax image of half of the input image. That is, the parallax image generation unit 6 is composed of a first-region parallax image generation unit 61 that processes half of the entire input image, and a second-region parallax image generation unit 62 that also processes half of the same region. The first and second regions may or may not overlap. If the first and second regions do not overlap at all, each can process exactly half of the entire input image, so the parallax image generation unit 6 can output a parallax image 15 that covers the entire input image. Half of the input image refers to the region when converted to area, and the shape of the region is arbitrary.

[0022] The first-region disparity image generation unit 61 and the second-region disparity image generation unit 62 may be configured using the exact same circuit configuration, differing only in their input data area (the range of memory addresses in the data holding area 5) and output data area (the range of memory addresses in the data holding area 7). In short, the difference lies in whether the area to be processed on the captured image is the first region or the second region. The input data area and the output data area should be configured to be easily switched using registers or the like. For example, the control core 84 that controls the overall operation of the image processing device 3 may be configured to allow the software to rewrite the memory address information recorded in registers (not shown). The memory is, for example, the RAM 1403 shown in Figure 14, which will be described later.

[0023] The selection ranges of the first and second regions are switched according to the disparity image processing cycle, as described later. Specifically, when using a disparity image processing cycle that matches the imaging cycle 41 (output transmission cycle) of camera modules 1 and 2 as the base, the disparity image 15 is generated across the entire region, combining the first and second regions. When using a disparity image processing cycle twice that of the base cycle, the selection ranges of the first and second regions are the same. Since the selection ranges of the first and second regions are the same and overlap, the size of the resulting disparity image is halved.

[0024] The object recognition processing unit 8 performs recognition processing to recognize the surrounding environment of the vehicle based on the disparity image 15 generated by the disparity image generation unit 6. The disparity image is image information to which distance information from stereo vision has been added. Based on this image information including distance information, the object recognition processing unit 8 detects objects that may be present in front of the vehicle. When an object is detected, the object recognition processing unit 8 stores information about the area in which the object exists as object detection information (object detection information group 16) in the data storage area 7. Based on the object detection information group 16, the object recognition processing unit 8 determines (identifies) the type of object, such as a vehicle, pedestrian, or obstacle, through object recognition identification processing, and stores the determination result as identification information (identification information group 17) in the data storage area 7. Machine learning methods such as neural networks and deep learning may be used in the object recognition identification processing.

[0025] Finally, the recognized object processing unit 8 (control core 84) transmits the identification information group 17 to a control device (not shown) inside the vehicle via the vehicle communication unit 20. The control device inside the vehicle controls safety-critical vehicle operations, such as the activation of collision mitigation brakes, according to the contents of the identification information group 17. For example, the control device inside the vehicle is an ECU (Electronic Control Unit). The vehicle communication unit 20 is an in-vehicle network such as a CAN (Controller Area Network).

[0026] The information in the object detection information group 16 and the identification information group 17 is limited to the size range and position information of the recognized object, and therefore its data size is significantly smaller compared to the image size of the original captured image groups 11 and 12. For this reason, the computational cost of processing this data is relatively small. Consequently, the recognized object processing unit 8 can be composed of programmable processing blocks such as a microcomputer or a signal processing processor, rather than dedicated hardware.

[0027] In this embodiment, the recognition object processing unit 8 is composed of a processor having four cores 81 to 84, each performing a specific calculation. A core 81 is assigned to handle recognition object detection processing, a first core 82 and a second core 83 are assigned to handle recognition object identification processing, and a control core 84 is assigned to handle the overall control processing of the image processing device 3. By providing multiple cores for recognition object identification processing, the throughput from the start of imaging to the identification of the recognized object can be increased.

[0028] As described above, the image processing apparatus (image processing apparatus 3) according to this embodiment is configured to include a disparity image generation unit (disparity image generation unit 6) that generates a disparity image from a pair of captured images, and a control unit (control core 84) that selects the region of the captured image to be used when generating the disparity image in the disparity image generation unit. This embodiment uses such a configuration to flexibly switch the object detection region of the captured image according to the number of captured images taken into image processing per unit time without increasing the amount of computation.

[0029] Furthermore, the image processing apparatus according to this embodiment includes a plurality of equivalent disparity image generation units (first region disparity image generation unit 61, second region disparity image generation unit 62) that generate a disparity image from a pair of captured images. In this embodiment, the plurality of disparity image generation units can generate a disparity image for each region of the captured image according to the region switching setting of the captured image.

[0030] [Operation during highway driving scenes] Here, the operation of the imaging device 10 in a scenario of driving on a highway (without any sudden obstacles) using the configuration of this embodiment will be explained with reference to Figures 3 and 4. The highway driving scenario is just one example of a high-speed driving scenario. Figure 3 shows examples of the captured images from the image groups 11 and 12 in the highway driving scenario.

[0031] Figure 3 shows an example of an image captured and input to the image processing device 3 while driving on a highway. In the highway driving scene 31, it is required to detect distant objects, such as the distant vehicle 30 in front in Figure 3. For this reason, the right images 11-1 to 11-4 and left images 12-1 to 12-4 captured by camera modules 1 and 2 utilize the entire field of view (the entire area of ​​the captured image) that includes the distant vehicle 30. However, the number of captured images input to the image preprocessing unit 4 from the image groups 11 and 12 is set to one every other image per imaging cycle (imaging period 41 in Figure 4), simplifying the calculation processing from the image preprocessing unit 4 onward. An example of the timing of each process in the highway driving scene is shown in Figure 4.

[0032] Figure 4 is a timing chart showing an example of the image processing operation of the image processing device 3 while it is traveling on the highway 31. The image preprocessing 42 of the image preprocessing unit 4 is configured to operate only on even or odd periods relative to the imaging period 41 (output transmission period) of the camera modules 1 and 2. That is, the image groups 11 and 12 captured during the unused period are discarded. In the example shown in Figure 4, every other image is discarded: the right image 11-2 and the left image 12-2, and the right image 11-4 and the left image 12-4 (for example, even frames 32 in intervals t2 and t4 (Figure 3)). This discarding process can be achieved simply by the image preprocessing unit 4 ignoring the input of the captured images.

[0033] The first-region disparity image generation unit 61 and the second-region disparity image generation unit 62, located after the image preprocessing unit 4, operate with a period (disparity image generation processing period 40) that is twice the length of the imaging period 41 (output transmission period) of the camera modules 1 and 2. Therefore, it is desirable that subsequent image processing operations be based on this disparity image generation processing period 40 and completed within the duration of this disparity image generation processing period 40.

[0034] In this embodiment, the first-region disparity image generation process 43 and the second-region disparity image generation process 44 in the first-region disparity image generation unit 61 and the second-region disparity image generation unit 62 are performed in parallel in terms of timing. At this time, the hardware configuration of the first-region disparity image generation unit 61 and the second-region disparity image generation unit 62 is such that the processing is completed within the disparity image generation processing cycle 40 (for example, interval t3 to t4). The hardware used for this disparity image generation process can be hardware that takes twice as long as the imaging cycle 41, so the hardware resources can be reduced compared to when the disparity image generation process is configured to be completed within the imaging cycle 41.

[0035] In configurations other than this embodiment, a single disparity image generation unit is used that completes processing within the imaging cycle 41. In contrast to such a configuration, this embodiment uses two disparity image generation units (first-region disparity image generation unit 61 and second-region disparity image generation unit 62) for parallel calculation. By thus doubling the disparity image generation processing cycle, the increase in hardware resources (circuit size) can be suppressed.

[0036] The recognition object detection process 45 for obtaining object detection information set 16 from the disparity image 15 is performed using the recognition object detection processing core 81 (Figure 1) of the recognition object processing unit 8. Generally, when the distance to the vehicle is obtained for each pixel in the disparity image, a method is known in which the disparity information (corresponding to distance information) is histogrammed to detect objects in order to obtain the object detection information set 16 from the disparity image. This method has a low computational load for detecting objects from a disparity image. For this reason, the recognition object detection process 45 by the recognition object detection processing core 81 is completed within the disparity image generation processing cycle 40 (for example, interval t5~t6).

[0037] Depending on the target value for increasing the object recognition rate, the computational load of the recognition object identification process, which obtains the identification information group 17 from the object detection information group 16, is generally heavier than the recognition object detection process 45, which detects objects from the disparity image. Therefore, if the recognition object identification process is performed using only the first core 82 for recognition object identification, it is highly likely that all processing will not be completed within the disparity image generation processing cycle 40, and the processing will have to be terminated midway and resumed in the next cycle. In this case, the calculation results of the recognition object identification process are temporarily saved to the data storage area 7, and the calculation is resumed in the next processing cycle. However, since the calculation is continued in the next processing cycle, the output results of the previous stage (recognition object detection process 45) are not used and are discarded.

[0038] Therefore, in this embodiment, the recognition object identification process is distributed to two cores, namely a first core 82 for recognition object identification processing and a second core 83 for recognition object identification processing. The preceding recognition object identification process by the first core 82 is designated as recognition object identification process (1) 46, and the subsequent recognition object identification process by the second core 83 is designated as recognition object identification process (2) 47, with each process being completed within the disparity image generation processing cycle 40 (for example, intervals t7~t8 and t9~t10). With this configuration, the throughput 48 from the time the camera modules 1 and 2 start imaging until the identification processing result is obtained from the target captured image is equivalent to 10 imaging cycles 41 (the period from interval t1 to interval t10).

[0039] [Operation in general road driving scenarios] The operation of the imaging device 10 in a scenario where the vehicle is driving on a public road (with an unexpected obstacle) according to the configuration of this embodiment will be explained using Figures 5 to 8. The public road driving scenario is an example of a low-speed driving scenario.

[0040] Figure 5 shows an example of the functional configuration of the imaging device 10 equipped with the image processing device 3, and an example of retained data during low-speed driving. The functional configuration of the image processing device 3 shown in Figure 5 is the same as the functional configuration of the image processing device 3 shown in Figure 1, except for region selection.

[0041] Figure 6 is an enlarged view of an example of retained data (image) during low-speed driving in the image processing device 3. In the examples in Figures 5 and 6, the area used for generating the disparity image of the captured image is divided into upper and lower sections.

[0042] The configuration of the image processing device 3 shown in Figure 5 is the same as the configuration of the image processing device 3 shown in Figure 1, except that the size of the processed captured images and parallax images being handled is different. The first-region parallax image generation unit 61 and the second-region parallax image generation unit 62 each take half a region (right captured image group 513, left captured image group 514) of the captured images from the right camera module 1's captured image group 11 and the left camera module 2's captured image group 12 as input data. The first-region parallax image generation unit 61 and the second-region parallax image generation unit 62 each generate a parallax image (parallax image group 515) of half a region using the processed right captured image group 513 and left captured image group 514, respectively.

[0043] Here, we will explain examples of the images captured in the image groups 11 and 12 during general road driving scenes using Figure 7. Figure 7 shows an example of an image captured and input to the image processing device 3 when driving on a public road. In general road driving scenarios, in order to respond to objects suddenly appearing from the side on the general road 71, it is necessary to detect and identify, for example, a vehicle 70 in front of the vehicle to the side as early as possible, based on its relative lateral speed to the vehicle itself. For this reason, all captured images from the right image group 11 and the left image group 12 (in the example in Figure 7, right image images 11-1 to 11-4 and left image images 12-1 to 12-4) captured by the camera modules 1 and 2 are input to the image preprocessing unit 4.

[0044] However, the area used for each captured image is halved, and the range of the area is adjusted so that the area in which the vehicle in front 70 is captured is captured. The range (position and size) of the area in which objects such as vehicles or people that pose a collision risk to the vehicle are captured generally changes depending on the speed of the vehicle. For this reason, it is desirable to be able to adjust the range of this area according to the speed. In the example in Figure 7, this set area is the lower half of the captured image (the lower half of the full field of view), and the lower area 72 is input to the image preprocessing unit 4.

[0045] As shown in Figure 6, the size of each processed image in the right image group 513 and left image group 514, which are stored in the data storage area 5 by the image preprocessing unit 4, and the size of each disparity image in the disparity image group 515, which are stored in the data storage area 7, are half of those in the highway driving scene. However, as mentioned above, since the image preprocessing unit 4 does not select which images to process, the number of images to process is twice the number of images shown in Figure 3, and there is no difference in the required data storage area capacity. An example of the timing of each process in the general road driving scene is shown in Figure 8.

[0046] Figure 8 is a timing chart showing an example of the image processing operation of the image processing device 3 when it is traveling on a public road 71. In relation to the imaging cycle 41 (output transmission cycle) of camera modules 1 and 2, the image preprocessing 42 of the image preprocessing unit 4 switches its settings to operate for all imaging cycles 41 (for example, intervals t1, t2, ...). This setting switching control is performed by the control core 84 executing setting switching software based on information provided by the vehicle while driving. The first-region disparity image generation unit 61 and the second-region disparity image generation unit 62 after the image preprocessing unit 4 operate at twice the period of the imaging cycle 41 (output transmission cycle) of camera modules 1 and 2. Therefore, if the first-region disparity image generation unit 61 and the second-region disparity image generation unit 62 process all regions of each image in the image group 11 and 12, the processing will not be completed in time, and one of the images will be discarded.

[0047] Therefore, in general road driving scenes, only half of the area of ​​each captured image is used. Specifically, only the lower area 72 of the captured image, which is important for detecting sudden appearances from the side in the general road driving scene in Figure 7, is processed, and the captured image groups 11 and 12 are alternately input to the first area disparity image generation unit 61 and the second area disparity image generation unit 62 for each imaging cycle 41. In this embodiment, as described above, the data areas (memory address ranges) to which the inputs of the first area disparity image generation unit 61 and the second area disparity image generation unit 62 are entered are configured to be rewritable by registers, so such switching can be easily performed.

[0048] With this configuration, the first-region disparity image generation process 43 (for example, section t2 to t3) of the first-region disparity image generation unit 61 and the second-region disparity image generation process 44 (for example, section t3 to t4) of the second-region disparity image generation unit 62 produce a group of disparity images 515 using the lower region 72.

[0049] The object detection information group 16 is obtained by the object detection processing 45 performed by the object detection processing core 81 of the object recognition processing unit 8. However, in general road driving scenes, the size of the disparity image is halved, so the computational processing load is halved. Therefore, the object detection processing 45 can obtain output at each imaging period 41 (for example, intervals t4, t5, ...).

[0050] The object recognition process has the same computational load as when driving on a highway, but it is not necessary to perform object recognition in every processing cycle. For example, in the example in Figure 8, the object recognition process based on the result of the object detection process 45 in section t5 is not executed, and the result of the object detection process 45 is discarded. This is because, as long as the object detection process 45 has detected the target object, it is easy to estimate that the object that has already been identified is the same object based on its location information.

[0051] In the example shown in Figure 8, during the first processing cycle, the first core 82 for object recognition processing performs object recognition processing (1) 46 in the interval t5-t6, followed by object recognition processing (2) 47 by the second core 83 for object recognition processing in the interval t7-t8. Then, for the forward vehicle 80 to the side that was initially identified by object recognition processing (1) 46 and object recognition processing (2) 47, the object in the object detection information group 16 at approximately the same position in object recognition processing (1) 46 (for example, interval t7-t8) and object recognition processing (2) 47 (for example, interval t9-t10) of the next processing cycle can be estimated to be this forward vehicle 80 to the side.

[0052] Furthermore, if there are other detected objects besides the forward vehicle 80 in the object detection information group 16, the recognition object identification process for the next object (an object other than the forward vehicle 80) is started sequentially after the recognition object identification process (1) 46 of the forward vehicle 80 by the first core 82 for recognition object identification processing and the recognition object identification process (2) 47 of the forward vehicle 80 by the second core 83 for recognition object identification processing are completed. In the general road driving scene of the configuration of this embodiment, the throughput 49 until the recognition object identification processing result is finally obtained is eight times the imaging cycle 41 (the period from section t1 to section t8), which is faster than the throughput 48 in the highway driving scene.

[0053] Although it was explained that the output result of the preceding stage of the recognized object identification process (for example, the result of the recognized object detection process 45 in section t5) is discarded when the calculation is resumed in the next processing cycle (for example, a cycle in which the imaging start timing is set to section t2), the output result of the preceding stage may be kept in the data retention area 7 instead of being discarded. Even just detecting the presence or absence of an object in front is useful for vehicle control such as collision avoidance, and can thus contribute to driving safety.

[0054] According to the image processing device (image processing device 3) of this embodiment described above, in processing overlapping regions of images captured by multiple cameras (camera modules 1 and 2), it is possible to switch the object detection region of the captured images as a stereo camera according to the number of captured images taken into image processing per unit time, without increasing the computational load. Therefore, this embodiment can achieve both suppression of an increase in the size of the electronic circuit and assurance of driving safety. For example, by suppressing an increase in the size of the electronic circuit, the ease of mounting on a vehicle is improved.

[0055] Furthermore, in this embodiment, the selection of the captured image region and the setting of the number of pairs of captured images to be incorporated into image processing per unit time are performed without changing the imaging period of the pair of camera modules. As a result, this embodiment contributes to both suppressing the increase in the size of the electronic circuit and ensuring driving safety without increasing the computational load.

[0056] <Second Embodiment> In the first embodiment, the operating modes of the image processing device 3 were described separately for highway driving scenes and general road driving scenes. However, in general road driving scenes, there are other requirements, such as wanting to switch the area of ​​the captured image (parallax image) to an area corresponding to the field of view for obtaining signal information from traffic lights when the vehicle is stopped. Therefore, in the second embodiment of the present invention, the operation of the image processing device 3 for more flexibly switching the area of ​​the captured image will be described with reference to Figure 9.

[0057] Figure 9 shows an example of an image captured and input to the image processing device 3 in the second embodiment. When the image preprocessing unit 4 of the image processing device 3 receives information from the control core 84 that the vehicle has stopped, it switches the area of ​​the captured image to an area corresponding to the field of view for obtaining signal information when the vehicle is stopped. In this case, an area 92 is set in each of the captured images in the image groups 11 and 12 (in the example in Figure 9, the right captured images 11-1 to 11-4 and the left captured images 12-1 to 12-4). The area 92 is an area that is large and positioned to accommodate the traffic light 90 installed at the intersection 91. Compared to the lower area 72 shown in Figure 7, the area 92 is longer in the vertical direction and shorter in the horizontal direction.

[0058] Such changes to the region settings can be achieved simply by switching the information indicating the input data region and the output data region of the first region disparity image generation unit 61 and the second region disparity image generation unit 62 in Figure 5, as described above.

[0059] Although an example has been described in which the area of ​​the captured image is switched to an area corresponding to the field of view for obtaining signal information when the vehicle is stopped, the configuration may also be such that the area setting switch according to this embodiment is performed when the vehicle speed decreases to the speed just before stopping (a preset extremely low speed).

[0060] <Third Embodiment> Furthermore, if throughput is a priority in highway driving scenarios, the data processing described with reference to Figures 5 to 8 may be used to detect distant and nearby vehicles. In this case, the area of ​​the captured image in which objects are actually detected will be half the size of the original captured image. To compensate for this, in the second embodiment of the present invention, areas where it is thought that no objects to be detected exist are excluded from processing from the image preprocessing unit 4 onward to secure the size of the area.

[0061] Figure 10 shows an example of an image captured and input to the image processing device 3 in the third embodiment. Here, we will explain using an example in which two vehicles 101 and 102 are traveling in front of the vehicle on the highway 31.

[0062] In Figure 10, if the vehicle is traveling in the rightmost lane of a three-lane road, a region for obtaining a disparity image, such as region 100, is selected for each of the captured images in the image groups 11 and 12 (in the example in Figure 10, the right captured images 11-1 to 11-4 and the left captured images 12-1 to 12-4). In this example, while monitoring the distant area, regions to the right of region 100 where no objects to be detected are thought to be absent are excluded from processing from the image preprocessing unit 4 onward.

[0063] Such changes to the region settings can be immediately addressed by simply rewriting the information indicating the input data region and output data region of the first region disparity image generation unit 61 and the second region disparity image generation unit 62 in the image processing apparatus 3 (Figures 1 and 5) according to the first embodiment, as described above. Furthermore, the first to third embodiments can be implemented by combining two or more embodiments.

[0064] <Fourth Embodiment> [Configuration of an imaging device equipped with an image processing device] Next, the configuration of an imaging device equipped with an image processing device according to the fourth embodiment of the present invention will be described with reference to Figures 11 and 12. Figure 11 shows an example of the functional configuration of an imaging device equipped with an image processing device according to the fourth embodiment. Figure 12 is an enlarged view of an example (image) of retained data during vehicle operation in the image processing apparatus according to the fourth embodiment.

[0065] The difference between the imaging device 10A according to this embodiment and the imaging device 10 according to the first embodiment (Figure 1) is that the parallax image generation unit 6 of the image processing device 3 is replaced by the parallax image generation unit 6A of the image processing device 3A. The parallax image generation unit 6A consists of a first-region parallax image generation unit 1161, a second-region parallax image generation unit 1162, and a third-region parallax image generation unit 1163.

[0066] The first-region disparity image generation unit 1161, the second-region disparity image generation unit 1162, and the third-region disparity image generation unit 1163 each use one-third of the region of each captured image from the captured image group 11 obtained by the right camera module 1 and the captured image group 12 obtained by the left camera module 2 as input data.

[0067] One-third of the area of ​​the captured image input to each of the disparity image generation units 1161 to 1163 is one-third of the area of ​​the captured image processed by the image preprocessing unit 4. That is, the processed right captured image group (referred to as "right camera image group" in the figure) 1113 from the right camera module 1 and the processed left captured image group (referred to as "left camera image group" in the figure) 1114 from the left camera module 2 are temporarily stored in the data holding area 5 of the image preprocessing unit 4 and input to the disparity image generation unit 6 in an appropriate processing cycle.

[0068] Furthermore, the parallax image generation units 1161 to 1163 are configured to generate a parallax image group 1115 for one-third of each region of the processed right image group 1113 and the processed left image group 1114. This configuration allows for flexibility in selecting the region of the captured image, and also enables the processing cycle, i.e., throughput, from the image preprocessing unit 4 onwards to be varied depending on the driving scene.

[0069] For example, in a highway driving scene with the highest speed range, the image preprocessing unit 4 discards two of the three captured images for each imaging cycle 41 (Figure 13). For example, the disparity image generation processing cycle 1300 after the image preprocessing unit 4 is set to three times the imaging cycle 41.

[0070] In situations such as highway congestion where the vehicle speed decreases, the image preprocessing unit 4 captures two of the three captured images in order to perform early detection of vehicles cutting in. An example of the timing of each process at this time is shown in Figure 13, which will be described later.

[0071] In the example shown in Figure 12 above, the captured image was divided into three sections vertically (top, center, bottom), but it may also be divided into three sections horizontally (left, center, right). Furthermore, when dividing the captured image into three regions, the regions do not have to be equal in size. However, since the time required to complete object recognition increases with the size of the region, dividing each region equally can shorten the time (throughput) required to complete object recognition for a pair of captured images. In this embodiment, three disparity image generation units 1161 to 1163 are provided to divide the captured image into three regions, but it is also possible to provide four or more disparity image generation units to divide the captured image into four or more regions.

[0072] [Operation in a highway traffic jam scenario] Figure 13 is a timing chart showing an example of the image processing operation of the image processing device 3A during congested driving on a highway. Here, an example of image processing operation is shown when the image preprocessing unit 4 acquires two of the three captured images.

[0073] From the image preprocessing unit 4 onward, the image preprocessing unit performs a disparity image generation process on only the lower two-thirds of the captured images in the captured image group 11, 12 to obtain a disparity image that is two-thirds the size of the original captured image. In the example in Figure 13, the first region disparity image generation unit 1161 performs the first region disparity image generation process 1301 on the lower region of the processed image (e.g., section t1) (e.g., section t2 to t4), and the second region disparity image generation unit 1162 performs the second region disparity image generation process 1302 on the central region of the processed image (e.g., section t2) (e.g., section t3 to t5). The image of the remaining upper one-third region (e.g., the image preprocessed in section t3) is discarded.

[0074] Next, a recognition object detection process 45 (for example, in section t5) is performed based on the result of the first-region disparity image generation process 1301, followed by a recognition object detection process 45 (for example, in section t6) based on the result of the second-region disparity image generation process 1302. Then, a recognition object identification process (1) 46 (for example, in sections t6 to t7) and a recognition object identification process (2) 47 (for example, in sections t8 to t9) are performed based on the result of the recognition object detection process 45 (section t5). With this configuration, the throughput 1310 from the time when the camera modules 1 and 2 start imaging until the identification processing result is obtained from the target captured image is equivalent to nine imaging cycles 41 (the period from section t1 to section t9). In the example in Figure 13, the result of the recognition object detection process 45 in section t6 is discarded.

[0075] In traffic jam scenarios, vehicle speeds are lower than during high-speed driving, eliminating the need to detect distant objects. Therefore, leaving one-third of the discarded image area as a region where distant objects are captured does not pose a safety problem. This reduces the processing time for generating the perspective image to obtain the parallax image to two-thirds of that when processing the entire captured image, enabling faster detection of vehicles approaching the vehicle's front.

[0076] In the example shown in Figure 13 above, the third-region disparity image generation unit 1163 performs the third-region disparity image generation process 1303 on the lower region of the captured image that was preprocessed in section t4 of the next disparity image generation processing cycle 1300. In addition, the first-region disparity image generation unit 1161 performs the first-region disparity image generation process 1301 on the central region of the captured image that was preprocessed in section t5.

[0077] Furthermore, if the traffic congestion worsens and the vehicle speed decreases, or if the vehicle moves onto a regular road, the image preprocessing unit 4 captures the image groups 11 and 12 for all cycles of the imaging cycle 41. The image preprocessing unit 4 sets the area to be used for the captured image to one-third of the area of ​​the original captured image. This makes it possible to perform object detection even earlier. Other operations are the same as in the first embodiment.

[0078] For highway driving scenes, highway congestion scenes, and general road driving scenes, the control core 84 makes a determination based on information reflecting the vehicle's driving scene obtained from the vehicle, and notifies the image preprocessing unit 4 of the determination result. The image preprocessing unit 4 switches the region setting based on this driving scene determination result. In this embodiment, the region selection of the captured image and the setting of the number of pairs of captured images to be incorporated into image processing per unit time are switched according to information reflecting the vehicle's driving scene.

[0079] The information reflecting the vehicle's driving scene may include speed information obtained from the vehicle, map information used for driving the vehicle, or road traffic information (congestion information). For example, the control core 84 calculates speed information based on information such as the vehicle's engine speed and wheel speed, or obtains speed information from other control devices within the vehicle. The control core 84 also obtains map information recorded in the vehicle's non-volatile storage or map information stored on an external server based on its current location. Furthermore, the control core 84 receives road traffic information, for example, from a Vehicle Information and Communication System.

[0080] Furthermore, the region setting may be switched not only based on information from the vehicles but also on information obtained from the images captured by the imaging device 10A. For example, if the vehicle information obtained from the processed right image 13 and left image 14 shows no oncoming vehicles and only vehicles traveling in the same direction are detected, it is determined to be a "highway driving scene". If the number of vehicles traveling in the same direction increases, it is determined to be a "highway congestion scene", etc.

[0081] Generally, to obtain the effects of the present invention, the size of the area used to generate the parallax image from the captured image (parallax image area size) and the period of the parallax image generation process should be proportional. For example, if the parallax image area size is set to one-third of the entire captured image, the parallax image generation process period can be set to one-third of the period when processing the entire captured image. Also, if the parallax image area size is set to two-thirds of the entire captured image, the parallax image generation process period can be set to two-thirds of the period when processing the entire captured image. The case where the entire area of ​​the captured image is selected as the parallax image area size is the base (1x) of the parallax image generation process period, which corresponds to the imaging period 41 (output transmission period) of camera modules 1 and 2.

[0082] In other words, the relationship between the region selection of the captured image and the number of pairs of captured images incorporated into image processing per unit time should be configured such that the product of the ratio of the size of the selected region to the total area of ​​the original captured image and the number of pairs of captured images incorporated into image processing per unit time is less than or equal to a certain value uniquely determined by the hardware. With such a configuration, it is possible to switch the object detection region of the captured image as a stereo camera without increasing the computational load.

[0083] Dividing the disparity image generation processing block in the disparity image generation unit 6 in this way can speed up object detection for high-priority areas of the captured image. However, this worsens the throughput when the entire captured image is used for generating disparity images. If the system is configured to achieve the same throughput even when the entire captured image is used for generating disparity images, the disparity image generation processing block that calculates disparity in parallel in the disparity image generation unit 6 will need to be equal to the number of divisions, which will increase the overall hardware size of the disparity image generation unit 6, thus negating the benefits of the present invention.

[0084] According to the embodiment described above, similar to the first to third embodiments, it is possible to suppress the increase in the size of the electronic circuit and ensure driving safety at the same time. Furthermore, according to this embodiment, since unused parallax image generation blocks (first region parallax image generation unit 1161, second region parallax image generation unit 1162, or third region parallax image generation unit 1163) and cores for recognition object identification processing (first core 82 for recognition object identification processing, or second core 83 for recognition object identification processing) are stopped, an image processing device with reduced power consumption can be realized.

[0085] [Hardware configuration of the image processing unit] Next, the hardware configuration of the computers in the image processing devices 3 and 3A of the imaging devices 10 and 10A will be explained with reference to Figure 14.

[0086] Figure 14 is a block diagram showing an example of the computer hardware configuration of the image processing devices 3 and 3A. The computer 1400 shown in Figure 14 is hardware used as a so-called computer.

[0087] The computer 1400 comprises a CPU (Central Processing Unit) 1401, ROM (Read Only Memory) 1402, RAM (Random Access Memory) 1403, non-volatile storage 1406, and a communication interface 1407, all connected to the system bus.

[0088] CPU 1401 is an example of an arithmetic unit (processor). The recognition object processing unit 8 may have four processor cores, each for a different purpose. ROM 1402 and RAM 1403 are examples of memory. For example, data storage area 5 and data storage area 7 are implemented by RAM 1403.

[0089] The non-volatile storage 1406 is a non-volatile memory element with a larger capacity than memory. Information such as programs, tables, and files that realize each function of the image processing devices 3 and 3A can be stored in memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, optical discs, or magneto-optical discs.

[0090] The programs that implement the functions of the image processing devices 3 and 3A in each of the embodiments described above are stored in their respective non-volatile storage devices 1406. This non-volatile storage device 1406 is an example of a computer-readable, non-transient recording medium. The programs may also be stored in the ROM 1402. At a minimum, when the functions of the processing blocks in the image processing devices 3 and 3A are implemented by software, the CPU 1401 executes the programs stored in the ROM 1402 or the non-volatile storage device 1406 to implement the functions of the processing blocks in the image processing devices 3 and 3A.

[0091] The communication interface 1407 consists of communication devices that control communication between the in-vehicle network and other control devices (e.g., ECUs) or external servers.

[0092] Furthermore, the present invention is not limited to the embodiments described above, and of course, various other applications and modifications can be taken as long as they do not depart from the gist of the invention as described in the claims. For example, the embodiments described above are described in detail and specifically in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those comprising all the components described. Also, it is possible to replace a part of the configuration of one embodiment with a component of another embodiment. It is also possible to add a component of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, replace, or delete other components for a part of the configuration of each embodiment.

[0093] Furthermore, some or all of the above configurations, functions, and processing units may be implemented in hardware, for example, by designing them as integrated circuits. For example, the image preprocessing unit 4 and the disparity image generation unit 6 may be configured with dedicated hardware such as a DSP (Digital Signal Processor). Alternatively, broad-based processor devices such as FPGAs (Field Programmable Gate Arrays) and ASICs (Application Specific Integrated Circuits) may be used as hardware.

[0094] Furthermore, in the embodiments described above, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In practice, it can be assumed that almost all components are interconnected. [Explanation of Symbols]

[0095] 1...Right camera module, 2...Left camera module, 3,3A...Image processing device, 4...Pre-processing unit for captured images, 5...Data storage area, 6...Disparity image generation unit, 6A...Disparity image generation unit, 7...Data storage area, 8...Recognition object processing unit, 10,10A...Imaging device, 11...Right captured image group, 11-1~11-4...Right captured image, 12...Left captured image group, 12-1~12-4...Left captured image, 13...Right captured image (after processing), 14...Left captured image (after processing), 15...Disparity image, 16...Object detection information group, 17...Identification information group, 40...Disparity image generation processing cycle, 41...Imaging cycle, 61...First region disparity image generation unit, 62...Second region disparity image generation unit, 81...Core for recognition object detection processing, 82...First core for recognition object identification processing, 83...Second core for object recognition processing, 84...Control core, 92...Region, 100...Region, 513...Right image group, 514...Left image group, 515...Disparity image group, 1113...Right image group, 1114...Left image group, 1115...Disparity image group, 1161...First region disparity image generation unit, 1162...Second region disparity image generation unit, 1163...Third region disparity image generation unit, 1300...Disparity image generation processing cycle

Claims

1. An image processing device for processing a series of images acquired by a pair of camera modules that capture images of the surrounding environment of a vehicle, and for obtaining surrounding environment recognition information for controlling the vehicle, The system comprises multiple equivalent disparity image generation units that generate a disparity image from the aforementioned pair of captured images, The system is configured to allow selection of the region of the captured images to be used when generating a disparity image from the pair of captured images, and the selection of the region is switched according to the number of images from the pair of captured images to be incorporated into the image processing per unit time. Image processing device.

2. The relationship between the region selection of the captured image and the number of pairs of captured images incorporated into image processing per unit time is configured such that the product of the ratio of the size of the selected region to the total area of ​​the original captured image and the number of pairs of captured images incorporated into image processing per unit time is less than or equal to a certain value uniquely determined by the hardware. The image processing apparatus according to claim 1.

3. The selection of the region of the captured image and the setting of the number of the pair of captured images to be incorporated into image processing per unit time are performed without changing the imaging period of the pair of camera modules. The image processing apparatus according to claim 1.

4. The region of the captured image is divided based on the number of disparity image generation units, and a region of the captured image to be processed is assigned to each of the disparity image generation units. The image processing apparatus according to claim 1.

5. The selection of the region of the captured image and the setting of the number of pairs of captured images to be incorporated into image processing per unit time are switched according to information that reflects the vehicle's driving scene. The image processing apparatus according to claim 1.

6. The selection of the region of the captured image and the setting of the number of pairs of captured images to be incorporated into image processing per unit time are switched according to the speed information obtained from the vehicle. The image processing apparatus according to claim 5.

7. The selection of the region of the captured image and the setting of the number of pairs of captured images to be incorporated into image processing per unit time are switched according to the information obtained from the captured image. The image processing apparatus according to claim 5.

8. An imaging device comprising: a pair of camera modules for capturing images of the surrounding environment of a vehicle; and an image processing device for processing a series of images acquired by the pair of camera modules and obtaining surrounding environment recognition information for controlling the vehicle, The image processing apparatus comprises a plurality of equivalent disparity image generation units that generate a disparity image from the pair of captured images, The image processing device is configured to select a region of the captured images to be used when generating a disparity image from the pair of captured images, and the selection of the region is switched according to the number of images of the pair of captured images to be incorporated into the image processing per unit time. Imaging device.

9. An image processing method performed in an image processing device for obtaining ambient environment recognition information for controlling a vehicle, which processes a series of captured images obtained by a pair of camera modules that capture images of the surrounding environment of a vehicle, The image processing apparatus comprises a plurality of equivalent disparity image generation units that generate a disparity image from the pair of captured images, The process includes selecting a region of the captured images to be used when generating a disparity image from the pair of captured images, and the selection of the region is switched according to the number of images of the pair of captured images to be incorporated into the image processing per unit time. Image processing methods.

Images