Image processing device

Abstract

To generate a highly reliable parallax in real time by a neural network while reducing a processing amount of a processor and power consumption thereof when reliability in parallax generated by stereo matching decreases.SOLUTION: An image processing device (10) of the present invention comprises: an image acquisition unit (20) that acquires one set of images having a parallax photographed by a plurality of cameras (21) loaded in a vehicle; a first parallax generation unit (30) that searches for an area with which a pixel block of a part of the image of one performs matching from the image of other, and thereby obtains a first parallax of the one set of images; an image range designation unit (40) that designates a processing range on the basis of reliability on an image in a specific area serving as an area of a part of the one or the one set of images; and a second parallax generation unit (50) that obtains a second parallax of the one set of images by neural network processing to the processing range designated by the image range designation unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device such as a stereo camera that uses multiple cameras to recognize obstacles outside a vehicle. [Background technology]

[0002] In order to improve vehicle driving safety, research is being conducted into systems that use cameras mounted on vehicles to detect obstacles ahead and warn the driver or automatically brake if there is a possibility of a collision with the obstacle.

[0003] There are various types of cameras, such as monocular cameras and stereo cameras that use multiple cameras. Stereo cameras can measure the distance to an object by using the parallax of the overlapping area of images taken by two cameras placed on the vehicle at a certain distance apart. This makes it possible to accurately grasp the risk of collision with the object.

[0004] As mentioned above, a stereo camera calculates the parallax between images captured by two cameras and converts the parallax into distance. However, depending on the image conditions, the parallax may not be calculated accurately. Therefore, Patent Document 1 proposes a method in which the reliability of the parallax calculated by the stereo matching processing method is calculated and the parallax with low reliability is combined with a pseudo-parallax calculated by another method. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-253666 Summary of the Invention [Problem to be solved by the invention]

[0006] However, in the technology of Patent Document 1, pixels with reduced reliability of disparity in the stereo matching method are covered by other disparity calculation methods, thereby improving the reliability of disparity for the entire image. However, since multiple disparity calculation methods that combine stereo matching processing with other methods are performed on the entire image, the processor processing load increases and power consumption increases. Embedded devices such as in-vehicle cameras require real-time processing, but there is a demand for reducing the processor processing load and power consumption to reduce power consumption. Therefore, it is necessary to generate highly reliable disparity while performing real-time processing by reducing the processor processing load and power consumption. [Means for solving the problem]

[0007] In order to solve the above problem, the present invention comprises an image acquisition unit that acquires a set of images having parallax captured by multiple cameras mounted on a vehicle; a first parallax generation unit that determines a first parallax of the set of images by searching for an area in one of the images that matches a portion of a pixel block in the other of the images; an image range designation unit that designates a processing range based on the reliability of an image of a specific area that is a portion of one or the set of images; and a second parallax generation unit that determines a second parallax of the set of images by neural network processing for the processing range designated by the image range designation unit. [Effects of the Invention]

[0008] According to the present invention, when it is determined that the reliability of a region required for recognition processing is reduced among the disparities calculated using the stereo matching method, disparity is generated for that region using neural network processing, which has the effect of reducing the amount of processing and power consumption of the processor, enabling real-time processing while generating highly reliable disparity for the required region.

[0009] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a hardware configuration diagram of a vehicle control system including an image processing device. [Figure 2] FIG. 2 is a functional block diagram of the image processing apparatus. [Figure 3] FIG. 10 is a diagram showing a parallax generation process using a stereo matching method. [Figure 4] FIG. 10 is a diagram showing an example of setting an area of interest. [Figure 5] FIG. 4 is a functional block diagram of an image area designation unit 40. [Figure 6] 3 is a processing flowchart of the image processing device 10. [Figure 7] 10 is a flowchart showing a process for evaluating the reliability of disparity. [Figure 8] 10 is a flowchart of a process for determining a parallax search range for the second parallax generating unit 50. [Figure 9] FIG. 10 is a diagram showing an example in which an area of interest is set at the left edge of an image. [Figure 10] FIG. 10 is a hardware configuration diagram of an information processing apparatus according to a third embodiment. [Figure 11] 10 is a flowchart for selecting an image for performing learning suitable for the second parallax generation unit 50. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description shows specific examples of the contents of the present invention, and the present invention is not limited to these descriptions. Various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, parts having the same function are designated by the same reference numerals, and repeated explanations thereof may be omitted.

[0012] <First Example> FIG. 1 shows a hardware configuration diagram of a vehicle control system 1 including an image processing device 10. The vehicle-mounted camera 21 has a pair of left and right cameras, a left camera 22 and a right camera 23. The image processing device 10 is composed of a processor 11 and a memory 12, such as a read-only memory (ROM) or a random access memory (RAM), which serves as a storage device. The vehicle-mounted camera 21, the image processing device 10, and the vehicle control device 13 are connected by a bus line 14, which is a communication path for exchanging information with each other. The vehicle control device 13 controls the vehicle by generating vehicle control command values using distance information of external objects generated by the image processing device 10. The left camera 22 and the right camera 23 constituting the vehicle-mounted camera 21 may be stereo cameras housed in a single housing, or may be separate cameras housed in separate housings and arranged at different positions on the vehicle. Furthermore, the vehicle-mounted camera 21 may be a multi-camera system consisting of three or more cameras arranged on the vehicle so as to have overlapping imaging areas. In the case of the multi-camera system described above, a set of images can be acquired from any combination of the three or more cameras. In the following examples, a stereo camera will be described as an example.

[0013] 2 is a functional block diagram of the image processing device 10. The image processing device 10 includes an image acquisition unit 20, a first parallax generation unit 30, an image range designation unit 40, a second parallax generation unit 50, a parallax synthesis unit 60, and a recognition processing unit 70. Note that the recognition processing unit 70 may not be included in the image processing device 10, but may be included in the vehicle control device 13.

[0014] The image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23. The image acquisition unit 20 transmits the acquired left image 2110 and right image 2210 to the first parallax generation unit 30, the image range designation unit 40, and the second parallax generation unit 50, respectively.

[0015] The first parallax generation unit 30 receives the left image 2110 and the right image 2210 as input and generates a first parallax image 35 by stereo matching. The first parallax generation unit 30 finds a region in one image where some pixel blocks of the other image match, thereby obtaining a first parallax between the left image 2110 and the right image 2210, which are a pair of images. In this embodiment, a parallax corresponding to each pixel of the right image 2210 is generated, but a parallax corresponding to each pixel of the left image 2110 may also be generated. The first parallax generation unit 30 generates the first parallax image 35 using the first parallax.

[0016] FIG. 3 shows the parallax generation process by the stereo matching method in the first parallax generation unit 30. A right image 2210 captured by the right camera 23 is used as a reference image, and a reference block image 221, such as 16 pixels by 16 pixels, is defined. The size of the block image is not limited to this example. Meanwhile, in the left image 2110 (reference image) captured by the left camera 22, a comparison block 211 is selected from a search area 212 with a search width r, using the same vertical position (Y coordinate) and horizontal position (X coordinate) as those of the reference block image 221 as a reference. Then, the difference between the reference block image 221 and the comparison block 211 is calculated. This difference calculation is called SAD (Sum of Absolute Difference), and the following calculation is performed: SAD=ΣΣ|I(i,j)-T(i,j)|...Equation (1)

[0017] In equation (1), I is the comparison block 211 in the reference image, T is the base block image 221, and i and j are coordinates within the image block. To calculate one disparity, the position of the comparison block 211 is shifted by one pixel at a time from the left edge of the search area 212 to the right edge, and calculations are performed for all pixels in the search area 212. The position d at which the SAD value is smallest is determined as the matching position. Because this stereo matching method compares left and right images, if the image features are few or the brightness is low, the matching position may not be accurately determined, and the reliability of the disparity may decrease. However, the stereo matching method is not limited to SAD. For example, matching may be performed by combining methods such as Sum of Squared Difference (SSD), Features from Accelerated Segment Test (FAST), and BRIEF (Binary Robust Independent Elementary Features), which extract corner feature points and check whether they are the same.

[0018] Returning to the explanation of FIG. 2 , the image range designation unit 40 designates a processing range based on the reliability of an image of a specific region, which is a partial region of one or a set of images. The image range designation unit 40 determines the reliability of the parallax of an image area (hereinafter referred to as an attention area 210) near an object to be recognized by the recognition processing unit 70 in the first parallax image 35 generated by the first parallax generation unit 30. If the image range designation unit 40 determines that the reliability of the parallax of the attention area 210 is low, it activates the second parallax generation unit 50. The second parallax generation unit 50 calculates the parallax within the attention area 210 and generates a second parallax image 55. The second parallax generation unit 50 calculates the second parallax of the set of images by neural network processing for the processing range designated by the image range designation unit 40. The image range designation unit 40 determines the search width of the parallax within the processing range based on the parallax determined by the first parallax generation unit 30, and the second parallax generation unit 50 determines the second parallax based on the search width of the parallax determined by the image range designation unit 40.

[0019] The image range designation unit 40 determines an area of interest 210, which is a specific area, according to the object of recognition processing in the recognition processing unit 70. The image range designation unit 40 determines the position of the specific area in the imaging field of view of the camera that captures a set of images, based on at least one of the vehicle's driving environment, the driving time zone, the weather, the installation position of the camera on the vehicle, and the deterioration state of the camera.

[0020] The attention area 210 indicates an area surrounding an object to be recognized, and the object to be recognized can be any object, such as a vehicle, a pedestrian, a bicycle, a sign, a traffic light, a road marking, or an obstacle on the road, and can be set according to a request from the recognition processing unit 70. For example, when the recognition processing application of the recognition processing unit 70 is preceding vehicle tracking, the image range designation unit 40 identifies an area in which the preceding vehicle is captured in at least one of the reference image 2110 or the base image 2210, and sets the rectangle containing the fewest number of pixels that contains the entire identified area as the attention area 210. Alternatively, the image range designation unit 40 identifies an area in which the preceding vehicle is captured, and sets the attention area 210 so that it contains a predetermined number of surrounding pixels. When the recognition processing application is road bump detection, the image range designation unit 40 identifies a road surface area in which a speed suppression bump is expected to be captured in at least one of the reference image 2110 or the base image 2210, and sets the rectangle containing the entire identified road surface area as the attention area 210. Alternatively, the area where the road surface area is imaged is identified, and the area of interest 210 is set so as to include a preset number of surrounding pixels. The method of setting the area of interest 210 and the shape, size, and number of the area of interest 210 are not limited to those described above, and any area may be used as long as it includes the object to be recognized by the recognition processing unit 70.

[0021] An example of setting the attention area 210 is shown in Figure 4. When the vehicle-mounted camera 21 is mounted facing forward so as to capture an image ahead of the vehicle in the traveling direction of the vehicle, for example, a preceding vehicle 200 present ahead of the vehicle can be set as the recognition target. A rectangular area surrounding the preceding vehicle 200 is set as the attention area 210.

[0022] The second parallax generation unit 50 generates highly reliable parallax within the area of interest 210 using a parallax generation method that uses a neural network. In the feature extraction step, the second parallax generation unit 50 extracts features from the left image 2110 and the right image 2210 acquired by the image acquisition unit 20 by performing two-dimensional convolution using a kernel determined by learning. In the matching cost calculation step, the second parallax generation unit 50 shifts the features of the left image 2110 relative to the features of the right image 2210 and repeatedly performs a process of combining them in the channel direction a number of times corresponding to a parallax search range s (described below) to generate a composite feature. The matching cost is calculated by repeatedly performing three-dimensional convolution processing using a kernel determined by learning on the composite feature. In the parallax estimation step, the second parallax generation unit 50 estimates a second parallax for each pixel based on the calculated matching cost by applying a known soft argmin operation or the like.

[0023] Although the neural network-based parallax generation by the second parallax generation unit 50 can generate parallax with higher reliability than the stereo matching-based parallax generation by the first parallax generation unit 30, it requires a larger amount of processing, resulting in increased processing time and power consumption in the processor 11. For this reason, in this embodiment, only when it is determined that the reliability of the first parallax in the area of interest 210 is low, the second parallax generation unit 50 is activated, and the second parallax generation unit 50 calculates a second parallax in the area of interest 210, thereby generating a second parallax image 55. In this manner, it is possible to generate a second parallax image 55 having a highly reliable second parallax in an area necessary for recognition processing while suppressing the processing amount and power consumption in the processor 11.

[0024] Returning to FIG. 2 , the parallax synthesis unit 60 synthesizes the first parallax image 35 generated by the first parallax generation unit 30 and the second parallax image 55 generated by the second parallax generation unit 50, and transmits the synthesized parallax image to the recognition processing unit 70. The recognition processing unit 70 recognizes the external environment based on the synthesized parallax image generated by the parallax synthesis unit 60. For example, when the recognition processing application is leading vehicle following, the recognition processing unit 70 calculates the distance to the leading vehicle using the second parallax image 55 and outputs the calculated distance to the vehicle control device 13. The vehicle control device 13 calculates control command values for the accelerator, brake, actuator, etc. of the host vehicle based on the distance to the leading vehicle calculated by the recognition processing unit 70, and performs leading vehicle following control. When the recognition processing application is road bump detection, the recognition processing unit 70 calculates the distance to a speed suppression bump present on the road using the second parallax image 55 and outputs the calculated distance to the vehicle control device 13. The vehicle control device 13 performs deceleration control and suspension control of the host vehicle based on the distance to the speed suppression bump calculated by the recognition processing unit 70. The recognition processing unit 70 may not only calculate the distance to the object using the second parallax image 55, but may also output the object identification results, such as the vehicle type and sign type. Furthermore, the control in the vehicle control device 13 is not limited to the above example, and may output information for presenting the distance to the object and vehicle control details to a display installed in the vehicle or an information presentation terminal carried by the occupants or passengers, for example. This allows the recognition processing unit 70 to recognize the external environment using highly accurate parallax in the area where the object to be recognized is captured, thereby improving the accuracy of the recognition processing of the external environment.

[0025] However, when the recognition processing unit 70 (described later) processes the parallax within the attention area 210 and the first parallax image 35 of the entire image excluding the attention area 210 separately, the parallax synthesis unit 60 can transmit both or either of the first parallax image 35 and the second parallax image 55 to the recognition processing unit 70 without performing synthesis processing. This makes it possible to transmit only necessary data according to the processing state of the recognition processing unit 70, thereby reducing the communication load.

[0026] FIG. 5 is a functional block diagram of the image area designation unit 40. As shown in FIG. The image range designation unit 40 calculates the reliability of the image of the area of interest 210 based on at least one of the number of edges of the edge image located at coordinates in the edge image corresponding to the area of interest 210 (specific region), the edge strength of the edge image, or the average value of the image brightness. If the reliability of the image of the area of interest 210 is below a threshold, the image range designation unit 40 causes the second parallax generation unit 50 to generate a second parallax. If the reliability of the image of the area of interest 210 is above the threshold, the image range designation unit 40 does not cause the second parallax generation unit 50 to generate a second parallax.

[0027] The attention area determination unit 410 determines the coordinates of the attention area 210 in the right image 2210 according to the object that the recognition processing unit 70 is to recognize.

[0028] In order to evaluate the parallax reliability within the area of interest 210, the parallax reliability evaluation unit 420 converts at least one of the left image 2110 and the right image 2210 into an edge image, and calculates the total number of edge pixels (edge number), edge strength, and average brightness within the area of interest 210.

[0029] The parallax reliability determination unit 430 determines whether the evaluation results of the number of edges and edge strength of the edge image and the average value of luminance within the attention area 210 satisfy reference values. If at least one evaluation result or all evaluation results are equal to or less than the reference value, the parallax reliability determination unit 430 activates the second parallax generation unit 50, which then generates parallax for the attention area 210. Details of the determination process will be described later.

[0030] Furthermore, the search range determination unit 440 extracts the maximum and minimum values of parallax in the first parallax image 35 within the area of interest 210, and determines a search range s for the second parallax generation unit 50 based on the maximum and minimum values of parallax. The search range s for the second parallax generation unit 50 can be set to a generous range, such as a range from the minimum value -α to the maximum value +β. Based on the determined parallax search range s, the second parallax generation unit 50 shifts the feature amounts of the left image 2110 relative to the feature amounts of the right image 2210 in the matching cost calculation step, and repeatedly performs a process of combining the feature amounts in the channel direction a number of times corresponding to the parallax search range s to generate a combined feature amount.

[0031] As described above, by limiting the range of disparity generation by the second parallax generation unit 50 to a three-dimensional range using the parallax search range s, it is possible to significantly reduce the amount of processing by the second parallax generation unit 50. Furthermore, the parallax reliability assessment unit 420 does not evaluate the reliability of disparity based on the parallax generated by the first parallax generation unit 30, but evaluates the reliability of disparity based on the input images (left image 2110 and right image 2210), thereby enabling the first parallax generation unit 30 and the second parallax generation unit 50 to be processed in parallel, thereby reducing the processing time.

[0032] FIG. 6 shows a processing flowchart of the image processing device 10. First, the image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23 (S100). The first parallax generation unit 30 generates a first parallax image 35 based on the left image 2110 and the right image 2210 (S110). Next, the image range designation unit 40 determines the area of interest 210 according to the object to be recognized by the recognition processing unit 70 (S120). The parallax reliability evaluation unit 420 evaluates the reliability of the parallax of the area of interest 210 based on the total number of pixels in the edge image, the edge strength, and the average brightness value within the area of interest 210 (S130). The parallax reliability evaluation process will be described later. Next, it is determined whether the reliability of the parallax of the area of interest 210 is equal to or less than a reference value (S140). In this embodiment, the reliability of the parallax is expressed as a value between 0 and 3, with 2 being the reference value. The method for setting the reliability and reference value of the disparity is not limited to this example.

[0033] When the reliability of the disparity for the area of interest 210 is 2 or less, the disparity reliability evaluation unit 420 determines a search range s for disparity in the second parallax generation unit 50 in order for the second parallax generation unit 50 to generate disparity for the area of interest 210 (S150). The process of determining this search range s will be described later. The second parallax generation unit 50 generates disparity for the area of interest 210 based on the determined search range (S160). Finally, the parallax synthesis unit 60 generates a synthesized parallax image by synthesizing the second parallax image 55 with the area of interest 210 in the first parallax image 35, and transmits the synthesized parallax image to the recognition processing unit 70 (S170). However, as described above, depending on the processing content of the recognition processing unit 70, the first parallax image 35 and the second parallax image 55 may be transmitted separately to the recognition processing unit 70 without performing synthesis processing.

[0034] FIG. 7 is a flowchart of the parallax reliability evaluation process. When the reliability evaluation process is called in S130 of FIG. 6, the parallax reliability evaluation unit 420 first clears the reliability value to zero (S200). Next, the parallax reliability evaluation unit 420 converts the image of the area of interest 210 into an edge image and measures the number of edge pixels (number of edges). If the number of edge pixels is equal to or greater than a predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S210). The predetermined value for the number of edge pixels can be set to any value, such as 30% of the number of pixels in the area of interest 210. Next, the parallax reliability evaluation unit 420 calculates the average value of the edge strength of the edge image, and if the average value of the edge strength is equal to or greater than the predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S220). The predetermined value for the average value of the edge strength can be set to any value, such as 50%. Finally, if the average brightness value of the image of the area of interest 210 is equal to or greater than a predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S230). The reference value for the average brightness can be set to any value, such as 40% of the maximum brightness.

[0035] FIG. 8 is a flowchart showing a process for determining the parallax search range s in the second parallax generation unit 50, which is determined by the search range determination unit 440. When the process is called in S150 of FIG. 6, the search range determination unit 440 selects the maximum parallax value in the first parallax image 35 within the area of interest 210 (S300). Next, the search range determination unit 440 selects the minimum parallax value in the first parallax image 35 within the area of interest 210 (S310). Finally, the search range determination unit 440 determines the minimum search width by subtracting a fixed value α from the minimum parallax value, and the maximum search width by adding a fixed value β to the maximum parallax value. Because the maximum and minimum parallax values in the first parallax image 35 may have low reliability, the search range s is set to have a margin of α and β. The fixed values α and β can be values stored in advance in a memory (not shown) within the image processing device 10.

[0036] According to the first embodiment, when it is determined that the reliability of an area required for recognition processing will decrease, a highly reliable disparity can be generated for that area using a neural network. This reduces the amount of processing and power consumption of the processor, allowing for real-time processing, while generating highly reliable disparity for the required area.

[0037] <Second Example> In the second embodiment, the attention area 201 is determined in advance at a predetermined position within the image, rather than being determined according to the object to be recognized by the recognition processing unit 70. Only the configuration that has been changed from the first embodiment will be explained below.

[0038] FIG. 9 shows an example in which an area of interest 500 is set at a predetermined position in an image. In S120 of FIG. 6 , the image range designation unit 40 sets the area of interest 500 at a predetermined position within the image, regardless of the object to be recognized by the recognition processing unit 70. For example, when the vehicle is traveling on a highway at night, the camera exposure time is longer than during the day because the image is darker. Furthermore, the amount of movement of the subject at the edge of the image increases as the vehicle travels at high speed. Therefore, the longer exposure time and the greater amount of movement of the subject at the edge of the image result in greater image blurring in the edge regions of the image. Therefore, the reliability of the parallax generated by the first parallax generation unit 30 using a stereo matching method may decrease in the edge regions of the image. Examples of factors that can decrease the reliability of the parallax include the driving environment and time of day of the vehicle, weather, camera placement, and camera deterioration over time.

[0039] 6, the image range designation unit 40 can set the reliability of the attention area 500 to a reference value or less in advance. When it is expected that the reliability of the disparity in a predetermined area will decrease, by setting the attention area 500 to a predetermined area in advance and setting the reliability of the attention area 500 to a reference value or less, the second disparity generation unit 50 can generate a highly reliable disparity image without evaluating the reliability of the disparity for each frame, thereby reducing the processing load and power consumption on the processor.

[0040] <Third Example> The third embodiment shows a method for efficiently performing training of the neural network used in the second parallax generation unit 50.

[0041] The reliability evaluation of disparity shown in this example can also be used to select images for training a neural network by making a judgment based on the image rather than the disparity itself. In order to generate disparity using a neural network, it is necessary to load a large number of images in advance and perform training on disparity generation.

[0042] 10 shows a configuration diagram of an information processing device 90 that executes learning of a neural network used in the second disparity generation unit 50 in the image processing device 10. The information processing device 90 may be implemented as a computer in a development environment, or may be implemented as an ECU in a vehicle. Furthermore, this embodiment may be executed within the image processing device 10. In the following description, a case will be described in which the information processing device 90 is implemented as a computer in a development environment.

[0043] The information processing device 90 has a memory 912 such as a read-only memory (ROM) or a random access memory (RAM) that serves as a storage device, a processor 911, and an interface 913. The information processing device 90 is connected to an input device, an output device, etc. (external device 920) via the interface 913. The memory 912, the processor 911, and the interface 913 are connected by a bus line 914. The information processing device 90 is further connected by wire or wirelessly to an image database 930 that stores learning images.

[0044] FIG. 11 shows a flowchart for selecting images for performing learning suitable for the disparity estimation neural network model used in the second disparity generating unit 50. In FIG.

[0045] First, processor 911 reads at least one of a training image and an edge image generated from the training image (hereinafter referred to as training image, etc.) from image database 930, which stores a large number of training images (S400). Next, processor 911 evaluates the reliability of the read training image, etc. (S410). The reliability evaluation here is the same as the process described in FIG. 7, but the range on the image for which the reliability is evaluated is the entire training image, etc., rather than just a portion of the read training image, etc. The calculation of the reliability of the training image is also the same as in the above-described embodiment, and is performed based on at least one of the number of edges in the edge image, the edge strength of the edge image, or the average brightness of the training image.

[0046] Next, it is determined whether the reliability of the disparity of the entire training images is equal to or less than a reference value (S420). This determination process corresponds to the step of selecting training images to be used for training the disparity estimation neural network model based on the reliability of the training images. The details of this determination process are the same as those described in the first embodiment, and therefore will not be repeated here.

[0047] Next, the disparity estimation neural network model is trained on the training images etc. whose disparity reliability is equal to or lower than a reference value (S430). This training process corresponds to the step of training the disparity estimation neural network model using the selected training images. Steps S400 to S430 are repeated until all images contained in the image database 930 or all images set as training targets have been read (S440). The trained disparity estimation neural network model is stored in memory.

[0048] The processing described in the third embodiment makes it possible to select images with low feature values from a large number of images and have the neural network learn them, thereby enabling efficient learning to be performed as a neural network used in the second disparity generation unit 50.

[0049] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0050] 1: vehicle control system, 10: image processing device, 11: processor, 12: memory, 13: vehicle control device, 14: bus line, 20: image acquisition unit, 21: on-board camera, 22: left camera, 23: right camera, 30: first parallax generation unit, 35: first parallax image, 40: image range designation unit, 50: second parallax generation unit, 55: second parallax image, 60: parallax synthesis unit, 70: recognition processing unit, 200: forward vehicle, 210: attention area in first embodiment, 410: attention area determination unit, 420: parallax reliability evaluation unit, 430: parallax reliability determination unit, 440: search range determination unit, 500: attention area in second embodiment, 2110: left image, 2210: right image, 90: information processing device, 911: processor, 912: memory

Claims

1. an image acquisition unit that acquires a set of images having parallax captured by a plurality of cameras mounted on the vehicle; a first disparity generator that calculates a first disparity between the pair of images by searching for an area in the other image that matches a partial pixel block of one of the images; an image area designation unit that designates a processing area based on the reliability of an image of a specific area that is a part of one or a set of the images; a second parallax generating unit that calculates a second parallax of the set of images by neural network processing for the processing range designated by the image range designating unit; An image processing device comprising:

2. 2. The image processing device according to claim 1, an image processing device comprising: a recognition processing unit that recognizes an external environment based on at least one of the first parallax generated by the first parallax generation unit and the second parallax generated by the second parallax generation unit;

3. 3. The image processing device according to claim 2, a parallax synthesis unit that generates a synthetic parallax image by synthesizing the first parallax and the second parallax, The recognition processing unit recognizes an external environment based on the synthetic parallax image generated by the parallax synthesis unit.

4. 3. The image processing device according to claim 2, The image processing device wherein the image range designation unit determines the specific area according to an object to be recognized by the recognition processing unit.

5. 3. The image processing device according to claim 2, The image range designation unit converting one or a set of said images into an edge image; An image processing device that calculates a reliability for the image of the specific region based on at least one of the number of edges of the edge image located at coordinates corresponding to the specific region in the edge image, the edge strength of the edge image, or the average brightness of the image.

6. 3. The image processing device according to claim 2, The image processing device is configured such that, when a reliability of the image of the specific region is lower than a threshold, the image range designation unit causes the second parallax generation unit to generate the second parallax.

7. 3. The image processing device according to claim 2, The image processing device, wherein the image range designation unit does not cause the second parallax generation unit to generate the second parallax when a reliability of the image of the specific region exceeds a threshold.

8. 2. The image processing device according to claim 1, The image processing device wherein the image range designation unit designates the processing range based on a reliability of an image of a specific region that is set in advance with respect to an imaging field of view of a camera that captures the set of images.

9. 9. The image processing device according to claim 8, The image range designation unit determines the position of a specific area in the imaging field of view of the camera that captures the set of images based on at least one of the vehicle's driving environment, driving time zone, weather, the installation position of the camera on the vehicle, and the deterioration state of the camera.

10. 2. The image processing device according to claim 1, the image range designation unit determines a parallax search width within the processing range based on the parallax generated by the first parallax generation unit; The second parallax generating unit calculates the second parallax based on a parallax search width determined by the image range specifying unit.

11. 1. A method performed by an image processing device having a processor and a memory, comprising: the processor: acquiring a set of images with parallax taken by a plurality of cameras mounted on a vehicle; determining a first disparity between the pair of images by searching for an area in the other image that matches a partial pixel block of one of the images; A step of specifying a processing range based on a reliability of an image of a specific region that is a part of one or a set of the images; and calculating a second disparity between the set of images by neural network processing for the processing range specified in the step.

12. A method for training a disparity estimation neural network model, which is executed by an information processing device having a processor and a memory, the processor: receiving at least one of a training image and an edge image generated from the training image from a database; calculating a reliability of the learning image based on at least one of the number of edges in the edge image, the edge strength of the edge image, or the average value of the luminance of the learning image; selecting training images to be used for training a disparity estimation neural network model based on the reliability of the training images; training a disparity estimation neural network model using the selected training images; storing the trained disparity estimation neural network model in the memory.

Images