Image processing device

Abstract

The problem addressed by the present invention is to generate highly reliable parallax in real time by a neural network while reducing processing amount and power consumption of a processor, when reliability of parallax generated by stereo matching decreases. An image processing device (10) according to the present invention includes an image acquisition unit (20) that acquires a set of images having parallax, image-captured by a plurality of cameras (21) mounted on a vehicle, a first parallax generation unit (30) that finds a first parallax of the set of images by searching for a region in which a partial pixel block of one of the images matches in the other image, an image range specifying unit (40) that specifies a processing range on the basis of reliability regarding an image of a specific region, which is a partial region of one or a set of the images, and a second parallax generation unit (50) that finds a second parallax of the set of images by neural network processing with respect to the processing range specified by the image range specifying unit.

Description

Image processing device

[0001] The present invention relates to an image processing device such as a stereo camera that uses multiple cameras to recognize obstacles outside a vehicle.

[0002] In order to improve vehicle driving safety, research is being conducted into systems that use cameras mounted on vehicles to detect obstacles ahead and warn the driver or automatically brake if there is a possibility of a collision with the obstacle.

[0003] There are various types of cameras, such as a monocular camera and a stereo camera that uses multiple cameras. A stereo camera can measure the distance to an object by utilizing the parallax of the overlapping area of ​​images taken by two cameras placed at a predetermined distance from each other on the vehicle. This allows the risk of collision with the object to be accurately determined.

[0004] As mentioned above, a stereo camera calculates the parallax between images captured by two cameras and converts the parallax into distance. However, depending on the image conditions, the parallax may not be calculated accurately. Therefore, Patent Document 1 proposes a method in which the reliability of the parallax calculated by the stereo matching processing method is calculated and the parallax with low reliability is combined with a pseudo-parallax calculated by another method.

[0005] Japanese Patent Application Laid-Open No. 2012-253666

[0006] However, in the technology of Patent Document 1, pixels with reduced reliability of parallax in the stereo matching method are covered by other parallax calculation methods, thereby improving the reliability of parallax for the entire image. However, since multiple parallax calculation methods that combine stereo matching processing with other methods are performed on the entire image, the processor processing load increases and power consumption increases. In embedded devices such as in-vehicle cameras, real-time processing is essential, but there is a demand for reducing the processor processing load and power consumption to reduce power consumption. Therefore, it is necessary to generate highly reliable parallax while performing real-time processing by reducing the processor processing load and power consumption.

[0007] In order to solve the above problem, the present invention comprises an image acquisition unit that acquires a set of images having parallax captured by multiple cameras mounted on a vehicle; a first parallax generation unit that determines a first parallax of the set of images by searching for an area in one of the images that matches a portion of a pixel block in the other of the images; an image range designation unit that designates a processing range based on the reliability of an image of a specific area that is a portion of one or the set of images; and a second parallax generation unit that determines a second parallax of the set of images by neural network processing for the processing range designated by the image range designation unit.

[0008] According to the present invention, when it is determined that the reliability of a region required for recognition processing is reduced among the disparities calculated using the stereo matching method, disparity is generated for that region using neural network processing, which has the effect of reducing the amount of processing and power consumption of the processor, enabling real-time processing while generating highly reliable disparity for the required region.

[0009] Further features related to the present invention will become apparent from the description of the present specification and the accompanying drawings. In addition, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments.

[0010] 1. A hardware configuration diagram of a vehicle control system including an image processing device. A functional block diagram of the image processing device. A diagram showing disparity generation processing using a stereo matching method. A diagram showing an example of setting an area of ​​interest. A functional block diagram of an image range designation unit 40. A processing flowchart of the image processing device 10. A processing flowchart of disparity reliability evaluation. A flowchart of processing to determine a disparity search range for the second disparity generation unit 50. A diagram showing an example of setting an area of ​​interest in the left edge portion of an image. A hardware configuration diagram of an information processing device according to a third embodiment. A flowchart for selecting images for learning suitable for the second disparity generation unit 50.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The following description shows specific examples of the contents of the present invention, and the present invention is not limited to these descriptions. Various changes and modifications can be made by those skilled in the art within the scope of the technical ideas disclosed in this specification. Furthermore, in all drawings used to explain the present invention, parts having the same function are designated by the same reference numerals, and repeated explanations thereof may be omitted.

[0012] First Embodiment FIG. 1 shows a hardware configuration diagram of a vehicle control system 1 including an image processing device 10. The vehicle-mounted camera 21 has a pair of left and right cameras, a left camera 22 and a right camera 23. The image processing device 10 includes a processor 11 and a memory 12, such as a read-only memory (ROM) or a random access memory (RAM), which serves as a storage device. The vehicle-mounted camera 21, the image processing device 10, and the vehicle control device 13 are connected by a bus line 14, which serves as a communication path for exchanging information with each other. The vehicle control device 13 controls the vehicle by generating vehicle control command values ​​using distance information of external objects generated by the image processing device 10. The left camera 22 and the right camera 23 constituting the vehicle-mounted camera 21 may be stereo cameras housed in a single housing, or separate cameras housed in separate housings and arranged at different positions on the vehicle. Furthermore, the vehicle-mounted camera 21 may be a multi-camera system consisting of three or more cameras arranged on the vehicle so as to have overlapping imaging areas. In the case of the multi-camera system described above, a set of images can be acquired from any set of three or more cameras. In the following embodiments, a stereo camera will be described as an example.

[0013] 2 is a functional block diagram of the image processing device 10. The image processing device 10 includes an image acquisition unit 20, a first parallax generation unit 30, an image range designation unit 40, a second parallax generation unit 50, a parallax synthesis unit 60, and a recognition processing unit 70. Note that the recognition processing unit 70 may not be included in the image processing device 10, but may be included in the vehicle control device 13.

[0014] The image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23. The image acquisition unit 20 transmits the acquired left image 2110 and right image 2210 to the first parallax generation unit 30, the image range designation unit 40, and the second parallax generation unit 50, respectively.

[0015] The first parallax generation unit 30 receives the left image 2110 and the right image 2210 as input and generates a first parallax image 35 by stereo matching. The first parallax generation unit 30 finds a region in one image where some pixel blocks of the other image match, thereby obtaining a first parallax between the left image 2110 and the right image 2210, which are a pair of images. In this embodiment, parallax corresponding to each pixel of the right image 2210 is generated, but parallax corresponding to each pixel of the left image 2110 may also be generated. The first parallax generation unit 30 generates the first parallax image 35 using the first parallax.

[0016] FIG. 3 shows the parallax generation process by the stereo matching method in the first parallax generation unit 30. A right image 2210 captured by the right camera 23 is used as a reference image, and a reference block image 221, such as 16 pixels by 16 pixels, is defined. The size of the block image is not limited to this example. Meanwhile, in the left image 2110 (reference image) captured by the left camera 22, a comparison block 211 is selected within a search area 212 of search width r, using the same vertical position (Y coordinate) and horizontal position (X coordinate) as those of the reference block image 221 as a reference. Then, the difference between the reference block image 221 and the comparison block 211 is calculated. This difference calculation is called SAD (Sum of Absolute Difference), and is calculated as follows: SAD=ΣΣ|I(i,j)-T(i,j)|...Equation (1)

[0017] In equation (1), I is the comparison block 211 in the reference image, T is the base block image 221, and i and j are coordinates within the image block. To calculate one disparity, the position of the comparison block 211 is shifted by one pixel at a time in the direction from the left end to the right end of the search area 212, and calculations are performed for all pixels in the search area 212, and the position d at which the SAD value is smallest is determined as the matching position. Because this stereo matching method compares left and right images, if the image features are few or the brightness is low, the matching position cannot be accurately determined, and the reliability of the disparity may decrease. However, the stereo matching technique is not limited to SAD, and matching may be performed by combining techniques such as Sum of Squared Difference (SSD) or Features from Accelerated Segment Test (FAST), which extracts corner feature points and checks whether the feature points are the same, or Binary Robust Independent Elementary Features (BRIEF).

[0018] Returning to the description of FIG. 2 , the image range designation unit 40 designates a processing range based on the reliability of an image of a specific region, which is a partial region of one or a set of images. The image range designation unit 40 determines the reliability of the parallax of an image area (hereinafter referred to as an attention area 210) near an object to be recognized by the recognition processing unit 70 in the first parallax image 35 generated by the first parallax generation unit 30. If the image range designation unit 40 determines that the reliability of the parallax of the attention area 210 is low, it activates the second parallax generation unit 50. The second parallax generation unit 50 calculates the parallax within the attention area 210 and generates a second parallax image 55. The second parallax generation unit 50 calculates the second parallax of the set of images by neural network processing for the processing range designated by the image range designation unit 40. The image range designation unit 40 determines the search width of the parallax within the processing range based on the parallax determined by the first parallax generation unit 30, and the second parallax generation unit 50 determines the second parallax based on the search width of the parallax determined by the image range designation unit 40.

[0019] The image range designation unit 40 determines an area of ​​interest 210, which is a specific area, according to the object of recognition processing in the recognition processing unit 70. The image range designation unit 40 determines the position of the specific area in the imaging field of view of the camera that captures the set of images, based on at least one of the vehicle's driving environment, the driving time zone, the weather, the installation position of the camera on the vehicle, and the deterioration state of the camera.

[0020] The attention area 210 indicates an area surrounding an object to be recognized, and the object to be recognized can be any object, such as a vehicle, a pedestrian, a bicycle, a sign, a traffic light, a road marking, or an obstacle on the road, that can be set according to a request from the recognition processing unit 70. For example, when the recognition processing application of the recognition processing unit 70 is preceding vehicle tracking, the image range designation unit 40 identifies an area in which the preceding vehicle is captured in at least one of the reference image 2110 or the base image 2210, and sets the rectangle containing the smallest number of pixels that contains the entire identified area as the attention area 210. Alternatively, the image range designation unit 40 identifies an area in which the preceding vehicle is captured, and sets the attention area 210 so as to include a predetermined number of surrounding pixels. When the recognition processing application is road bump detection, the image range designation unit 40 identifies a road surface area in which a speed suppression bump is expected to be captured in at least one of the reference image 2110 or the base image 2210, and sets the rectangle containing the entire identified road surface area as the attention area 210. Alternatively, the area where the road surface area is imaged is identified, and the area of ​​interest 210 is set so as to include a preset number of surrounding pixels. The method of setting the area of ​​interest 210 and the shape, size, and number of the area of ​​interest 210 are not limited to those described above, and any area may be used as long as it includes the object to be recognized by the recognition processing unit 70.

[0021] An example of setting the attention area 210 is shown in Figure 4. When the vehicle-mounted camera 21 is mounted facing forward so as to capture images ahead of the vehicle in the traveling direction of the vehicle, for example, a preceding vehicle 200 present ahead of the vehicle can be set as the recognition target. A rectangular area surrounding the preceding vehicle 200 is set as the attention area 210.

[0022] The second parallax generation unit 50 generates highly reliable parallax within the area of ​​interest 210 using a parallax generation method that uses a neural network. In the feature extraction step, the second parallax generation unit 50 extracts features from the left image 2110 and the right image 2210 acquired by the image acquisition unit 20 by performing two-dimensional convolution using a kernel determined by learning. In the matching cost calculation step, the second parallax generation unit 50 shifts the features of the left image 2110 relative to the features of the right image 2210 and repeatedly performs a process of combining them in the channel direction a number of times corresponding to a parallax search range s (described below) to generate a composite feature. The matching cost is calculated by repeatedly performing three-dimensional convolution processing using the kernel determined by learning on the composite feature. In the parallax estimation step, the second parallax generation unit 50 estimates a second parallax for each pixel based on the calculated matching cost by applying a known soft argmin operation or the like.

[0023] While the neural network-based parallax generation by the second parallax generation unit 50 can generate parallax with higher reliability than the stereo matching-based parallax generation by the first parallax generation unit 30, it requires a larger amount of processing, resulting in increased processing time and power consumption in the processor 11. For this reason, in this embodiment, only when it is determined that the reliability of the first parallax in the area of ​​interest 210 is low, the second parallax generation unit 50 is activated, and the second parallax generation unit 50 calculates a second parallax in the area of ​​interest 210, thereby generating a second parallax image 55. In this manner, it is possible to generate a second parallax image 55 having a highly reliable second parallax in an area necessary for recognition processing while suppressing the processing amount and power consumption in the processor 11.

[0024] Returning to FIG. 2 , the parallax synthesis unit 60 synthesizes the first parallax image 35 generated by the first parallax generation unit 30 and the second parallax image 55 generated by the second parallax generation unit 50, and transmits the synthesized image to the recognition processing unit 70. The recognition processing unit 70 recognizes the external environment based on the synthesized parallax image generated by the parallax synthesis unit 60. For example, when the recognition processing application is leading vehicle following, the recognition processing unit 70 calculates the distance to the leading vehicle using the second parallax image 55 and outputs the calculated distance to the vehicle control device 13. The vehicle control device 13 calculates control command values ​​for the accelerator, brake, actuator, etc. of the host vehicle based on the distance to the leading vehicle calculated by the recognition processing unit 70, and performs leading vehicle following control. When the recognition processing application is road bump detection, the recognition processing unit 70 calculates the distance to a speed suppression bump present on the road surface using the second parallax image 55 and outputs the calculated distance to the vehicle control device 13. The vehicle control device 13 performs deceleration control and suspension control of the host vehicle based on the distance to the speed suppression bump calculated by the recognition processing unit 70. Note that the recognition processing unit 70 may not only calculate the distance to the object using the second parallax image 55, but may also output the object identification results, such as the vehicle type and sign type, together. The control in the vehicle control device 13 is not limited to the above example. For example, the control may be performed by the vehicle control device 13 by outputting information for presenting the distance to the object and vehicle control details to a display installed in the vehicle or an information presentation terminal carried by a passenger or the like. This allows the recognition processing unit 70 to recognize the external environment using highly accurate parallax in the area where the object to be recognized is captured, thereby improving the accuracy of the external environment recognition process.

[0025] However, when a later-described recognition processing unit 70 separately processes the parallax within the attention area 210 and the first parallax image 35 of the entire image excluding the attention area 210, the parallax synthesis unit 60 can transmit both or one of the first parallax image 35 and the second parallax image 55 to the recognition processing unit 70 without performing synthesis processing. This makes it possible to transmit only necessary data according to the processing state of the recognition processing unit 70, thereby reducing the communication load.

[0026] 5 is a functional block diagram of the image range designation unit 40. The image range designation unit 40 calculates the reliability of the image of the area of ​​interest 210 (specific region) based on at least one of the number of edges in the edge image located at coordinates in the edge image corresponding to the area of ​​interest 210, the edge strength of the edge image, or the average brightness of the image. If the reliability of the image of the area of ​​interest 210 is below a threshold, the image range designation unit 40 causes the second parallax generation unit 50 to generate a second parallax. If the reliability of the image of the area of ​​interest 210 is above the threshold, the image range designation unit 40 does not cause the second parallax generation unit 50 to generate a second parallax.

[0027] The attention area determination unit 410 determines the coordinates of the attention area 210 in the right image 2210 according to the object that the recognition processing unit 70 is targeting for recognition.

[0028] In order to evaluate the parallax reliability within the area of ​​interest 210, the parallax reliability evaluation unit 420 converts at least one of the left image 2110 and the right image 2210 into an edge image, and calculates the total number of edge pixels (number of edges), edge strength, and average brightness within the area of ​​interest 210.

[0029] The parallax reliability determination unit 430 determines whether the evaluation results of the number of edges and edge strength of the edge image and the average value of luminance within the attention area 210 satisfy reference values. If at least one evaluation result or all evaluation results are equal to or less than the reference value, the parallax reliability determination unit 430 activates the second parallax generation unit 50, which then generates parallax for the attention area 210. Details of the determination process will be described later.

[0030] Furthermore, the search range determination unit 440 extracts the maximum and minimum parallax values ​​in the first parallax image 35 within the area of ​​interest 210, and determines a search range s for the second parallax generation unit 50 based on the maximum and minimum parallax values. The search range s for the second parallax generation unit 50 can be set to a generous range, such as a range from the minimum value −α to the maximum value +β. Based on the determined parallax search range s, the second parallax generation unit 50 shifts the feature amounts of the left image 2110 relative to the feature amounts of the right image 2210 in the matching cost calculation step, and repeatedly performs a process of combining the feature amounts in the channel direction a number of times corresponding to the parallax search range s to generate a combined feature amount.

[0031] As described above, by limiting the range of parallax generation by the second parallax generation unit 50 to a three-dimensional range using the parallax search range s, it is possible to significantly reduce the amount of processing by the second parallax generation unit 50. Furthermore, the parallax reliability assessment unit 420 evaluates the reliability of parallax based on the input images (the left image 2110 and the right image 2210) rather than evaluating the reliability of parallax based on the parallax generated by the first parallax generation unit 30. This allows the first parallax generation unit 30 and the second parallax generation unit 50 to be processed in parallel, thereby reducing the processing time.

[0032] FIG. 6 shows a processing flowchart of the image processing device 10. First, the image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23 (S100). The first parallax generation unit 30 generates a first parallax image 35 based on the left image 2110 and the right image 2210 (S110). Next, the image range designation unit 40 determines the area of ​​interest 210 based on the object to be recognized by the recognition processing unit 70 (S120). The parallax reliability evaluation unit 420 evaluates the reliability of the parallax of the area of ​​interest 210 based on the total number of pixels in the edge image, the edge strength, and the average luminance value within the area of ​​interest 210 (S130). The parallax reliability evaluation process will be described later. Next, it is determined whether the reliability of the parallax of the area of ​​interest 210 is equal to or less than a reference value (S140). In this embodiment, the parallax reliability is expressed as a value between 0 and 3, with 2 being the reference value. The method for setting the reliability and reference value of the disparity is not limited to this example.

[0033] When the reliability of the parallax for the area of ​​interest 210 is 2 or less, the parallax reliability evaluation unit 420 determines a search range s for the parallax in the second parallax generation unit 50 so that the second parallax generation unit 50 can generate parallax for the area of ​​interest 210 (S150). The process of determining the search range s will be described later. The second parallax generation unit 50 generates parallax for the area of ​​interest 210 based on the determined search range (S160). Finally, the parallax synthesis unit 60 generates a synthesized parallax image by synthesizing the second parallax image 55 with the area of ​​interest 210 in the first parallax image 35, and transmits the synthesized parallax image to the recognition processing unit 70 (S170). However, as described above, depending on the processing content of the recognition processing unit 70, the first parallax image 35 and the second parallax image 55 may be transmitted separately to the recognition processing unit 70 without performing synthesis processing.

[0034] FIG. 7 is a flowchart of the parallax reliability evaluation process. When the reliability evaluation process is called in S130 of FIG. 6, the parallax reliability evaluation unit 420 first clears the reliability value to zero (S200). Next, the parallax reliability evaluation unit 420 converts the image of the area of ​​interest 210 into an edge image and measures the number of edge pixels (number of edges). If the number of edge pixels is equal to or greater than a predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S210). The predetermined value for the number of edge pixels can be set to any value, such as 30% of the number of pixels in the area of ​​interest 210. Next, the parallax reliability evaluation unit 420 calculates the average edge strength of the edge image, and if the average edge strength is equal to or greater than the predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S220). The predetermined value for the average edge strength can be set to any value, such as 50%. Finally, if the average brightness of the image of the area of ​​interest 210 is equal to or greater than a predetermined value, the parallax reliability evaluation unit 420 adds 1 to the reliability (S230). The reference value of the average brightness can be set to any value, such as 40% of the maximum brightness.

[0035] FIG. 8 is a flowchart showing a process for determining the parallax search range s in the second parallax generation unit 50, as determined by the search range determination unit 440. When the process is called in S150 of FIG. 6 , the search range determination unit 440 selects the maximum parallax value in the first parallax image 35 within the area of ​​interest 210 (S300). Next, the search range determination unit 440 selects the minimum parallax value in the first parallax image 35 within the area of ​​interest 210 (S310). Finally, the search range determination unit 440 determines the minimum search width by subtracting a fixed value α from the minimum parallax value, and the maximum search width by adding a fixed value β to the maximum parallax value. Because the maximum and minimum parallax values ​​in the first parallax image 35 may have low reliability, the search range s is set to have a margin of α and β. The fixed values ​​α and β can be values ​​stored in advance in a memory (not shown) within the image processing device 10.

[0036] According to the first embodiment, when it is determined that the reliability of an area required for recognition processing will decrease, a highly reliable disparity can be generated for that area using a neural network. This reduces the amount of processing and power consumption of the processor, allowing for real-time processing, while generating highly reliable disparity for the required area.

[0037] Second Example In the second example, the attention area 201 is determined in advance at a predetermined position within the image, rather than being determined according to the object to be recognized by the recognition processing unit 70. Only the configuration that has been changed from the first example will be described below.

[0038] FIG. 9 shows an example in which an area of ​​interest 500 is set at a predetermined position in an image. In S120 of FIG. 6 , the image range designation unit 40 sets the area of ​​interest 500 at a predetermined position within the image, regardless of the object to be recognized by the recognition processing unit 70. For example, when the vehicle is traveling on a highway at night, the camera exposure time is longer than during the day because the image is darker. Furthermore, the amount of movement of the subject at the edge of the image increases as the vehicle travels at high speed. Therefore, the longer exposure time and the greater amount of movement of the subject at the edge of the image result in greater image blurring in the edge regions of the image. Therefore, the reliability of the parallax generated by the first parallax generation unit 30 using a stereo matching method may decrease in the edge regions of the image. Examples of factors that can decrease the reliability of the parallax include the vehicle's driving environment, time of day, weather, camera placement, and camera deterioration over time.

[0039] 6, the image range designation unit 40 can set the reliability of the attention area 500 to a reference value or less in advance. When it is expected that the reliability of the parallax in a predetermined area will decrease, by setting the attention area 500 to a predetermined area in advance and setting the reliability of the attention area 500 to a reference value or less, the second parallax generation unit 50 can generate a highly reliable parallax image without evaluating the reliability of the parallax for each frame, thereby reducing the processing load and power consumption on the processor.

[0040] Third Example The third example shows a method for efficiently performing training of the neural network used in the second parallax generation unit 50.

[0041] The reliability evaluation of disparity shown in this example can also be used to select images for training a neural network by making a judgment based on the image rather than the disparity itself. In order to generate disparity using a neural network, it is necessary to load a large number of images in advance and perform training on disparity generation.

[0042] 10 shows a configuration diagram of an information processing device 90 that executes learning of a neural network used in the second disparity generation unit 50 in the image processing device 10. The information processing device 90 may be implemented as a computer in a development environment, or may be implemented as an ECU in a vehicle. Furthermore, this embodiment may be executed within the image processing device 10. In the following description, a case will be described in which the information processing device 90 is implemented as a computer in a development environment.

[0043] The information processing device 90 has a memory 912 such as a read-only memory (ROM) or a random access memory (RAM) that serves as a storage device, a processor 911, and an interface 913. The information processing device 90 is connected to an input device, an output device, etc. (external device 920) via the interface 913. The memory 912, the processor 911, and the interface 913 are connected by a bus line 914. The information processing device 90 is further connected by wire or wirelessly to an image database 930 that stores learning images.

[0044] FIG. 11 shows a flowchart for selecting images for performing learning suitable for the disparity estimation neural network model used in the second disparity generation unit 50.

[0045] First, the processor 911 reads at least one of a training image and an edge image generated from the training image (hereinafter referred to as the training image, etc.) from an image database 930 storing a large number of training images (S400). Next, the processor 911 evaluates the reliability of the read training image, etc. (S410). The reliability evaluation here is the same as the process described in FIG. 7, but the range on the image for which the reliability is evaluated is the entire training image, etc., rather than just a portion of the read training image, etc. The calculation of the reliability of the training image is also the same as in the above embodiment, and is performed based on at least one of the number of edges in the edge image, the edge strength of the edge image, or the average brightness of the training image.

[0046] Next, it is determined whether the reliability of the disparity of the entire training images is equal to or less than a reference value (S420). This determination process corresponds to the step of selecting training images to be used for training the disparity estimation neural network model based on the reliability of the training images. The details of this determination process are the same as those described in the first embodiment, and therefore will not be repeated here.

[0047] Next, the parallax estimation neural network model is trained on training images or the like whose disparity reliability is equal to or lower than a reference value (S430). This training process corresponds to training the parallax estimation neural network model using the selected training images. Steps S400 to S430 are repeated until all images contained in the image database 930 or all images set as training targets have been read (S440). The trained parallax estimation neural network model is stored in memory.

[0048] The processing described in the third embodiment makes it possible to select images with low feature values ​​from a large number of images and have the neural network learn them, thereby enabling efficient learning to be performed as a neural network used in the second parallax generation unit 50.

[0049] The present invention is not limited to the above-described embodiments, but includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0050] 1: Vehicle control system, 10: Image processing device, 11: Processor, 12: Memory, 13: Vehicle control device, 14: Bus line, 20: Image acquisition unit, 21: In-vehicle camera, 22: Left camera, 23: Right camera, 30: First parallax generation unit, 35: First parallax image, 40: Image range designation unit, 50: Second parallax generation unit, 55: Second parallax image, 60: Parallax synthesis unit, 70: Recognition processing unit, 200: Forward vehicle, 210: Attention area in first embodiment, 410: Attention area determination unit, 420: Parallax reliability evaluation unit, 430: Parallax reliability determination unit, 440: Search range determination unit, 500: Attention area in second embodiment, 2110: Left image, 2210: Right image, 90: Information processing device, 911: Processor, 912: Memory

Claims

1. An image processing apparatus comprising: an image acquisition unit that acquires a set of images having parallax imaged by a plurality of cameras mounted on a vehicle; a first parallax generation unit that obtains a first parallax of the set of images by searching for a region in which a pixel block of a part of one of the images matches in the other image; an image range specifying unit that specifies a processing range based on a reliability regarding an image of a specific region that is a part of one or the set of images; and a second parallax generation unit that obtains a second parallax of the set of images by performing neural network processing on the processing range specified by the image range specifying unit.

2. The image processing apparatus according to claim 1, further comprising a recognition processing unit that recognizes an external environment based on at least one of the first parallax generated by the first parallax generation unit or the second parallax generated by the second parallax generation unit.

3. The image processing apparatus according to claim 2, comprising a parallax synthesis unit that generates a synthesized parallax image by synthesizing the first parallax and the second parallax, wherein the recognition processing unit recognizes an external environment based on the synthesized parallax image generated by the parallax synthesis unit.

4. The image processing apparatus according to claim 2, wherein the image range specifying unit determines the specific region according to an object to be recognized in the recognition processing by the recognition processing unit.

5. The image processing apparatus according to claim 2, wherein the image range specifying unit converts one or the set of images into an edge image, and obtains a reliability regarding the image of the specific region based on at least one of the number of edges of the edge image located at coordinates corresponding to the specific region in the edge image, the edge intensity of the edge image, or the average value of the luminance of the image.

6. The image processing apparatus according to claim 2, wherein the image range specifying unit causes the second parallax generation unit to generate the second parallax when the reliability regarding the image of the specific region is less than a threshold value.

7. The image processing apparatus according to claim 2, wherein the image range specifying unit does not cause the second parallax generation unit to generate the second parallax when the reliability regarding the image of the specific region is greater than a threshold value.

8. The image processing apparatus according to claim 1, wherein the image range specifying unit specifies the processing range based on the reliability of an image of a specific area preset with respect to the imaging field of view of a camera that captures the set of images.

9. The image processing apparatus according to claim 8, wherein the image range specifying unit determines the position of a specific area in the imaging field of view of the camera that captures the set of images based on at least one of the driving environment of the vehicle, the driving time zone, the weather, the installation position of the camera on the vehicle, and the deterioration state of the camera.

10. The image processing apparatus according to claim 1, wherein the image range specifying unit determines a search range of the disparity within the processing range based on the disparity generated by the first disparity generation unit, and the second disparity generation unit obtains the second disparity based on the search range of the disparity determined by the image range specifying unit.

11. A method executed by an image processing apparatus having a processor and a memory, the method including steps of: the processor obtaining a set of images having disparity captured by a plurality of cameras mounted on a vehicle; obtaining a first disparity of the set of images by searching for an area in which a pixel block of a part of one of the images matches in the other image; specifying a processing range based on the reliability of an image of a specific area that is a part of one or the set of images; and obtaining a second disparity of the set of images by performing neural network processing on the processing range specified in the step.

12. A method for training a disparity estimation neural network model executed by an information processing apparatus having a processor and a memory, the method comprising the steps of: receiving, by the processor, at least one of a training image or an edge image generated from the training image from a database; calculating a reliability of the training image based on at least one of the number of edges in the edge image, the edge intensity of the edge image, or the average value of the luminance of the training image; selecting a training image to be used for training the disparity estimation neural network model based on the reliability of the training image; training the disparity estimation neural network model using the selected training image; and storing the trained disparity estimation neural network model in the memory.

Images