Image processing apparatus

Abstract

The objective of this invention is to reduce the processing load and power consumption of the processor while generating highly reliable disparities in real time using a neural network, in the case of reduced reliability of disparities generated by stereo matching. The image processing apparatus (10) of this invention comprises: an image acquisition unit (20) that acquires a set of images with disparities captured by a plurality of cameras (21) mounted on a vehicle; a first disparity generation unit (30) that searches for a region in another image that matches a partial pixel block of one image, thereby determining a first disparity of the set of images; an image range specification unit (40) that specifies a processing range based on the reliability of an image in a specific region, the specific region being a partial region of one or a set of images; and a second disparity generation unit (50) that, for the processing range specified by the image range specification unit, determines a second disparity of the set of images by neural network processing.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, such as a stereo camera, that uses multiple cameras to identify obstacles outside a vehicle. Background Technology

[0002] To improve vehicle driving safety, a system is being researched that uses cameras mounted on the vehicle to detect obstacles ahead and alert the driver or automatically brake if there is a possibility of collision with the obstacle.

[0003] As types of cameras, there are monocular cameras and stereo cameras that use multiple cameras. Stereo cameras utilize the parallax of overlapping areas in images captured by two cameras positioned at specified intervals on a vehicle to measure the distance to the object being photographed. Therefore, the collision risk of an object can be accurately determined.

[0004] As described above, a stereo camera calculates the disparity between images captured by two cameras and converts this disparity into distance. However, the disparity cannot always be accurately calculated based on the condition of the images. Therefore, Patent Document 1 proposes a method that calculates the reliability of the disparity obtained through stereo matching processing and then synthesizes the disparity with low reliability with pseudo-disparities calculated through other methods.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-253666 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] However, in the technology of Patent Document 1 mentioned above, pixels with reduced disparity reliability due to stereo matching are covered by other disparity calculation methods, thus improving the overall image disparity reliability. However, because multiple disparity calculation methods, including stereo matching and others, are combined across the entire image, the processor's workload and power consumption increase. Embedded devices such as vehicle cameras are required to reduce processor workload and power consumption while performing real-time processing. Therefore, it is necessary to reduce processor workload and power consumption to generate highly reliable disparities while performing real-time processing.

[0010] Methods for solving problems

[0011] To address the aforementioned issues, the present invention comprises: an image acquisition unit that acquires a set of images with parallax captured by a plurality of cameras mounted on a vehicle; a first parallax generation unit that searches for a region in another image that matches a partial pixel block of one image, thereby determining a first parallax of the set of images; an image range specification unit that specifies a processing range based on reliability related to an image of a specific region, the specific region being a partial region of one or a set of images; and a second parallax generation unit that, for the processing range specified by the image range specification unit, determines a second parallax of the set of images by neural network processing.

[0012] Invention Effects

[0013] According to the present invention, when the reliability of the region required for recognition processing is reduced in the parallax calculated by stereo matching, the parallax is generated for the region by neural network processing. Therefore, it has the following effects: reducing the processing load and power consumption of the processor, and generating highly reliable parallax for the required region while performing real-time processing.

[0014] Further features related to this invention will become clear from the description and drawings in this specification. Furthermore, issues, configurations, and effects other than those described above will be clarified through the following description of embodiments. Attached Figure Description

[0015] Figure 1 This is a hardware configuration diagram of a vehicle control system that includes an image processing device.

[0016] Figure 2 This is a function block diagram of an image processing device.

[0017] Figure 3 This is a diagram illustrating the disparity generation process based on stereo matching.

[0018] Figure 4 This is a diagram showing an example of the setting of the region of interest.

[0019] Figure 5 This is a function block diagram of the image range specification section 40.

[0020] Figure 6 This is a flowchart of the image processing device 10.

[0021] Figure 7 This is a flowchart of the parallax reliability evaluation process.

[0022] Figure 8 This is a flowchart of the process for determining the search range of the disparity of the second disparity generation unit 50.

[0023] Figure 9This is a diagram showing an example where the region of interest is defined in the left part of the image.

[0024] Figure 10 This is a hardware configuration diagram of the information processing device according to the third embodiment.

[0025] Figure 11 This is a flowchart for selecting images suitable for training the second disparity generation unit 50. Detailed Implementation

[0026] Hereinafter, embodiments of the present invention will be described using the accompanying drawings. The following description illustrates specific examples of the content of the present invention; however, the present invention is not limited to these descriptions, and various changes and modifications can be made by those skilled in the art within the scope of the technical concept disclosed in this specification. Furthermore, in all the drawings used to describe the present invention, parts having the same function are given the same reference numerals, and sometimes repeated descriptions are omitted.

[0027] <First Embodiment>

[0028] Figure 1 This is a hardware configuration diagram of a vehicle control system 1 including an image processing unit 10. The vehicle-mounted camera 21 has a pair of cameras, a left camera 22 and a right camera 23. The image processing unit 10 consists of a processor 11 and a memory 12, such as a read-only memory (ROM) or random access memory (RAM), which serves as a storage device. The vehicle-mounted camera 21, the image processing unit 10, and the vehicle control unit 13 are connected via a bus 14, which serves as a communication line for transmitting and receiving information between them. The vehicle control unit 13 generates vehicle control command values ​​using distance information of external objects generated by the image processing unit 10, thereby controlling the vehicle. Furthermore, the left camera 22 and the right camera 23 constituting the vehicle-mounted camera 21 can be stereo cameras housed in a single housing, or they can be separate cameras housed in another housing and positioned at different locations within the vehicle. Alternatively, they can be a multi-camera system consisting of three or more cameras arranged in the vehicle to have repeating imaging areas. In the case of the aforementioned multi-camera system, a set of images can be acquired from any group of three or more cameras. In the following embodiments, a stereo camera will be used as an example for illustration.

[0029] Figure 2 A functional block diagram of the image processing apparatus 10 is shown. The image processing apparatus 10 includes an image acquisition unit 20, a first parallax generation unit 30, an image range specification unit 40, a second parallax generation unit 50, a parallax synthesis unit 60, and a recognition processing unit 70. Furthermore, the recognition processing unit 70 may be included in the vehicle control device 13, but not in the image processing apparatus 10.

[0030] The image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23. The image acquisition unit 20 sends the acquired left image 2110 and right image 2210 to the first parallax generation unit 30, the image range specification unit 40, and the second parallax generation unit 50, respectively.

[0031] The first disparity generation unit 30 takes the left image 2110 and the right image 2210 as input and generates a first disparity image 35 through stereo matching processing. The first disparity generation unit 30 searches for regions in another image that match partial pixel blocks of one image, thereby determining the first disparity of the left image 2110 and the right image 2210 as a set of images. In this embodiment, the disparity corresponding to each pixel of the right image 2210 is generated, but it is also possible to generate the disparity corresponding to each pixel of the left image 2110. The first disparity generation unit 30 uses the first disparity to generate the first disparity image 35.

[0032] Figure 3 The parallax generation process based on stereo matching in the first parallax generation unit 30 is shown. Using the right image 2210 captured by the right camera 23 as a reference image, a reference block image 221, for example, 16 pixels × 16 pixels, is defined. The size of the block image is not limited to this example. On the other hand, in the left image 2110 captured by the left camera 22 (referencing the image), a comparison block 211 is selected within the search region 212 of the search range r, based on the same vertical (Y-coordinate) and horizontal (X-coordinate) positions as the reference block image 221. Then, the difference between the reference block image 221 and the comparison block 211 is calculated. This difference calculation is called SAD (Sum of Absolute Difference) and is performed as follows.

[0033] SAD=ΣΣ|I(i,j)-T(i,j)|···Formula (1)

[0034] In equation (1), I is the comparison block 211 in the reference image, T is the reference block image 221, and i and j are the coordinates within the image block. To calculate a disparity, the position of the comparison block 211 is shifted pixel by pixel from the left end to the right end of the search area 212. The calculation is performed on all pixels of the search area 212, and the position d with the minimum SAD value is determined as the matching position. This stereo matching method compares the left and right images. Therefore, when the number of image features is small or the brightness is low, the matching position cannot be correctly determined, and the reliability of the disparity may be reduced. However, the stereo matching method is not limited to SAD. For example, it can also combine SSD (Sum of Squared Difference) or perform corner feature point extraction to check whether they are the same feature point, such as FAST (Features from Accelerated Segment Test) or BRIEF (BinaryRobust Independent Elementary Features), to perform matching.

[0035] return Figure 2 As explained, the image range designation unit 40 designates a processing range based on the reliability of an image relating to a portion of a region, or a specific region, of one or a set of images. Within the first disparity image 35 generated by the first disparity generation unit 30, the image range designation unit 40 determines the reliability of the disparity of an image region (hereinafter referred to as the region of interest 210) near an object that the recognition processing unit 70 is targeting. If the image range designation unit 40 determines that the reliability of the disparity in the region of interest 210 is low, it activates the second disparity generation unit 50. The second disparity generation unit 50 calculates the disparity within the region of interest 210 and generates a second disparity image 55. The second disparity generation unit 50 calculates the second disparity of a set of images using neural network processing within the processing range designated by the image range designation unit 40.

[0036] The image range specification unit 40 determines the search range of disparity within the processing range based on the disparity obtained by the first disparity generation unit 30, and the second disparity generation unit 50 obtains the second disparity based on the search range of disparity determined by the image range specification unit 40.

[0037] The image range designation unit 40 determines the region of interest 210 as a specific region based on the object being identified in the recognition processing unit 70. The image range designation unit 40 determines the position of the specific region in the field of view of the camera that captures a set of images based on at least one of the following: the vehicle's driving environment, the driving time period, the weather, the camera's installation position on the vehicle, and the camera's degradation state.

[0038] Furthermore, the area of ​​interest 210 represents the area surrounding the object to be identified. The object to be identified can be arbitrarily set to vehicles, pedestrians, bicycles, signs, traffic lights, road markings, road obstacles, etc., according to the requirements of the identification processing unit 70. For example, when the identification processing application of the identification processing unit 70 is a vehicle following, the image range designation unit 40 identifies the area where the vehicle is captured in front, referring to at least one of the reference images 2110 or 2210, and sets the rectangle containing the fewest pixels among the rectangles that encompass the entire identified area as the area of ​​interest 210. Alternatively, the area where the vehicle is captured is identified, and the area of ​​interest 210 is set to include a predetermined number of surrounding pixels. Additionally, when the identification processing application is a road bump detection, referring to at least one of the reference images 2110 or 2210, the road area where the speed bump is expected to be captured is identified, and a rectangle encompassing the entire identified road area is set to the area of ​​interest 210. Alternatively, the area where the road area is captured is identified, and the area of ​​interest 210 is set to include a predetermined number of surrounding pixels. The method for setting the attention area 210 and the shape, size, and number of attention areas 210 are not limited to this, as long as it is an area containing the recognition object of the recognition processing unit 70.

[0039] Figure 4 An example setting of the region of interest 210 is shown. When the vehicle-mounted camera 21 is mounted facing forward of the vehicle to capture images in the direction of travel of the vehicle, for example, a vehicle 200 located in front of the vehicle can be identified as the object of recognition. The rectangular area surrounding the vehicle 200 is designated as the region of interest 210.

[0040] The second disparity generation unit 50 generates highly reliable disparities within the region of interest 210 using a neural network disparity generation method. In the feature extraction step, the second disparity generation unit 50 extracts features from the left image 2110 and right image 2210 acquired by the image acquisition unit 20 by performing two-dimensional convolution using a training-determined convolution kernel. In the matching cost calculation step, the second disparity generation unit 50 repeatedly performs the following process a number of times corresponding to the disparity search range s (described later): shifting the feature values ​​of the left image 2110 relative to the feature values ​​of the right image 2210 and combining them in the channel direction to generate synthetic feature values. The matching cost is calculated by repeatedly performing three-dimensional convolution processing using a training-determined convolution kernel on the synthetic feature values. In the disparity estimation step, the second disparity generation unit 50 estimates the second disparity pixel-by-pixel based on the calculated matching cost by applying known softargmin operations, etc.

[0041] Compared to the stereo matching-based disparity generation performed by the first disparity generation unit 30, the neural network-based disparity generation performed by the second disparity generation unit 50 can generate highly reliable disparities. However, it requires more processing power, increasing the processing time or power consumption in the processor 11. Therefore, in this embodiment, the second disparity generation unit 50 is activated only when the reliability of the first disparity within the region of interest 210 is determined to be low. The second disparity generation unit 50 calculates the second disparity within the region of interest 210 and generates a second disparity image 55. Thus, while suppressing the processing power or power consumption in the processor 11, a second disparity image 55 with highly reliable second disparities in the region required for recognition processing can be generated.

[0042] return Figure 2 The parallax synthesis unit 60 synthesizes the first parallax image 35 generated by the first parallax generation unit 30 and the second parallax image 55 generated by the second parallax generation unit 50, and sends the result to the recognition processing unit 70. The recognition processing unit 70 identifies the external environment based on the synthesized parallax image generated by the parallax synthesis unit 60. For example, in the case of a vehicle following application, the recognition processing unit 70 calculates the distance to the vehicle in front using the second parallax image 55 and outputs it to the vehicle control device 13. In the vehicle control device 13, based on the distance to the vehicle in front calculated by the recognition processing unit 70, it calculates control command values ​​for the accelerator, brakes, actuators, etc., of its own vehicle and executes vehicle following control. In the case of a road bump detection application, the recognition processing unit 70 calculates the distance to a speed bump on the road surface using the second parallax image 55 and outputs it to the vehicle control device 13. In the vehicle control device 13, based on the distance to the speed bump calculated by the recognition processing unit 70, it executes deceleration control or suspension control of its own vehicle. Furthermore, the recognition processing unit 70 can not only calculate the distance to the object using the second parallax image 55, but also output the object recognition results, such as the type of vehicle and the type of sign. Additionally, the control in the vehicle control device 13 is not limited to the examples described above; for instance, it can output information indicating the distance to the object or vehicle control content to a display installed in the vehicle or an information display terminal carried by the driver or passengers. Therefore, the recognition processing unit 70 can perform high-precision parallax recognition of the external environment in the area where the object being recognized is captured, improving the accuracy of the external environment recognition processing.

[0043] However, when the recognition processing unit 70, described later, independently processes the parallax within the region of interest 210 and the first parallax image 35 of the entire image excluding the region of interest 210, the parallax synthesis unit 60 does not perform synthesis processing, but can send either or both of the first parallax image 35 and the second parallax image 55 to the recognition processing unit 70. Therefore, only the necessary data can be sent according to the processing state of the recognition processing unit 70, reducing the communication load.

[0044] Figure 5 This is a function block diagram of the image range specification section 40.

[0045] The image range designation unit 40 calculates the reliability of the image related to the region of interest 210 based on at least one of the following: the number of edges in the edge image located at coordinates corresponding to the region of interest 210 (specific region), the edge intensity of the edge image, or the average brightness of the image. If the reliability related to the image of the region of interest 210 is lower than a threshold, the image range designation unit 40 causes the second disparity generation unit 50 to generate a second disparity. If the reliability related to the image of the region of interest 210 is higher than the threshold, the image range designation unit 40 does not cause the second disparity generation unit 50 to generate a second disparity.

[0046] The region of interest determination unit 410 determines the coordinates of the region of interest 210 in the right image 2210 based on the object identified by the recognition processing unit 70.

[0047] In order to evaluate the parallax reliability within the region of interest 210, the parallax reliability evaluation unit 420 converts at least one image of the left image 2110 and the right image 2210 into an edge image, and calculates the total number of pixels of the edge (number of edges), the edge intensity, and the average value of the brightness within the region of interest 210.

[0048] The disparity reliability determination unit 430 determines whether the evaluation results of the number of edges and edge intensity of the edge image, as well as the average brightness within the region of interest 210, meet the benchmark values. If at least one evaluation result or all evaluation results are below the benchmark value, the disparity reliability determination unit 430 activates the second disparity generation unit 50, which generates the disparity of the region of interest 210. Details of the determination process will be described later.

[0049] Furthermore, the search range determination unit 440 extracts the maximum and minimum values ​​of disparity in the first disparity image 35 within the region of interest 210, and determines the search range s of the second disparity generation unit 50 based on the maximum and minimum values ​​of disparity. The search range s of the second disparity generation unit 50 can be set to a range with margin, such as from the minimum value - α to the maximum value + β. Based on the disparity search range s determined here, the second disparity generation unit 50 repeatedly performs the following process in the matching cost calculation step according to the number of times corresponding to the disparity search range s: shifting the feature quantity of the left image 2110 relative to the feature quantity of the right image 2210, combining them in the channel direction, thereby generating a synthetic feature quantity.

[0050] Therefore, by limiting the generation range of the disparity of the second disparity generation unit 50 to a three-dimensional range through the disparity search range s, the processing load of the second disparity generation unit 50 can be significantly reduced. Furthermore, the disparity reliability evaluation unit 420 evaluates the reliability of the disparity not based on the disparity generated by the first disparity generation unit 30, but based on the input images (left image 2110 and right image 2210), thereby enabling the first disparity generation unit 30 and the second disparity generation unit 50 to be processed in parallel, thus shortening the processing time.

[0051] Figure 6 A flowchart of the image processing apparatus 10 is shown. First, the image acquisition unit 20 acquires a left image 2110 captured by the left camera 22 and a right image 2210 captured by the right camera 23 (S100). The first parallax generation unit 30 generates a first parallax image 35 based on the left image 2110 and the right image 2210 (S110). Next, the image range designation unit 40 determines the region of interest 210 based on the object to be identified by the recognition processing unit 70 (S120). The parallax reliability evaluation unit 420 evaluates the reliability of the parallax in the region of interest 210 based on the evaluation results of the total number of pixels in the edge image, the edge intensity, and the average brightness within the region of interest 210 (S130). The parallax reliability evaluation process will be described later. Next, it is determined whether the reliability of the parallax in the region of interest 210 is below a reference value (S140). In this embodiment, the reliability of the parallax is represented by any value from 0 to 3, and 2 is used as the reference value. The method for setting the reliability or reference value of parallax is not limited to this example.

[0052] When the reliability of the disparity in the region of interest 210 is 2 or less, the disparity reliability evaluation unit 420 determines the search range s of the disparity in the second disparity generation unit 50, so that the second disparity generation unit 50 generates the disparity of the region of interest 210 (S150). The process of determining the search range s will be described later. The second disparity generation unit 50 generates the disparity of the region of interest 210 based on the determined search range (S160). Finally, the disparity synthesis unit 60 synthesizes the second disparity image 55 from the region of interest 210 in the first disparity image 35 and sends it to the recognition processing unit 70 (S170). However, as mentioned above, depending on the processing content of the recognition processing unit 70, sometimes the synthesis process is not performed, and the first disparity image 35 and the second disparity image 55 are sent to the recognition processing unit 70 independently.

[0053] Figure 7 This is a flowchart of the parallax reliability evaluation process. When in Figure 6 When the reliability evaluation process is invoked in S130, the parallax reliability evaluation unit 420 first resets the reliability value to zero (S200). Next, the parallax reliability evaluation unit 420 converts the image of the region of interest 210 into an edge image and measures the number of pixels at the edges (edge ​​count). If the number of pixels at the edges is above a predetermined value, the reliability is incremented by 1 (S210). The predetermined value for the number of pixels at the edges can be set to any value, such as 30% of the number of pixels in the region of interest 210. Next, the parallax reliability evaluation unit 420 calculates the average edge intensity of the edge image. If the average edge intensity is above a predetermined value, the reliability is incremented by 1 (S220). The predetermined value for the average edge intensity can be set to any value, such as 50%. Finally, if the average brightness of the image of the region of interest 210 is above a predetermined value, the reliability is incremented by 1 (S230). The baseline value for the average brightness can be set to any value, such as 40% of the maximum brightness.

[0054] Figure 8 This is a flowchart illustrating the process of determining the search range s of the disparity in the second disparity generation unit 50, as determined by the search range determination unit 440. When in Figure 6During the S150 call processing, the search range determination unit 440 selects the maximum value of the disparity in the first disparity image 35 within the region of interest 210 (S300). Next, it selects the minimum value of the disparity in the first disparity image 35 within the region of interest 210 (S310). Finally, the search range determination unit 440 uses the value obtained by subtracting a fixed value α from the minimum disparity as the minimum value of the search range, and uses the value obtained by adding a fixed value β to the maximum value of the disparity as the maximum value of the search range. Since the maximum and minimum values ​​of the disparity in the first disparity image 35 may have low reliability, a search range s with margins of α and β is set. The fixed values ​​α and β can be values ​​pre-stored in a memory (not shown) within the image processing device 10.

[0055] According to the first embodiment, when it is determined that the reliability of the area required for identification processing has decreased, a highly reliable parallax can be generated for that area through a neural network. Therefore, while reducing the processing power and power consumption of the processor, a highly reliable parallax for the required area can be generated.

[0056] <Second Embodiment>

[0057] The second embodiment does not determine the region of interest 201 based on the object of the recognition object identified by the recognition processing unit 70, but instead predetermines a predetermined position within the image. The following description only covers configurations that differ from the first embodiment.

[0058] Figure 9 This is an example of setting a region of interest of 500 pixels at a specified location in the image. Figure 6 In S120, the image range designation unit 40 sets a region of interest 500 at a predetermined position within the image, independent of the object to be identified by the recognition processing unit 70. For example, when a vehicle is traveling on a highway at night, the camera's exposure time becomes longer than during the day due to the darker image, and the amount of movement of the subject at the image edge increases with high speed. Therefore, due to the longer exposure time and the increased movement of the subject at the image edge, image ghosting becomes greater in the image edge region. Consequently, the reliability of the parallax generated by the first parallax generation unit 30 using stereo matching may decrease in the image edge region. Examples of decreased parallax reliability include the vehicle's driving environment or time of day, weather, camera configuration, and camera degradation over time.

[0059] Furthermore, in Figure 6In S130, the image range designation unit 40 can preset the reliability of the region of interest 500 to be below a reference value. When a decrease in parallax reliability is anticipated within a specified region, by presetting the region of interest 500 within the specified region and setting its reliability below the reference value, a highly reliable parallax image can be generated by the second parallax generation unit 50 without performing parallax reliability evaluation frame by frame, thereby reducing the processing load or power consumption in the processor.

[0060] <Third Embodiment>

[0061] The third embodiment illustrates a method for efficiently implementing the training of the neural network used in the second disparity generation unit 50.

[0062] The disparity reliability evaluation shown in this embodiment, which uses images instead of disparity itself, can also be applied to the selection of training images for neural networks. To generate disparity through a neural network, it needs to be trained beforehand by reading multiple images related to disparity generation.

[0063] Figure 10 A configuration diagram of an information processing device 90 is shown, which performs training of the neural network used in the second disparity generation unit 50 within the image processing device 10. The information processing device 90 can be installed as a computer in a development environment or as an ECU mounted in a vehicle. Alternatively, this embodiment can also be executed within the image processing device 10. The following description will focus on the case where the information processing device 90 is installed as a computer in a development environment.

[0064] The information processing device 90 includes a memory 912, such as a read-only memory (ROM) or random access memory (RAM), a processor 911, and an interface 913. The information processing device 90 is connected to an input device or an output device (external device 920) via the interface 913. The memory 912, the processor 911, and the interface 913 are connected via a bus cable 914. Furthermore, it is connected to an image database 930 storing training images via a wired or wireless connection.

[0065] Figure 11 A flowchart is shown for selecting an image for training the disparity estimation neural network model used in the second disparity generation unit 50.

[0066] First, the processor 911 reads at least one training image or edge image generated from the training image (hereinafter referred to as the training image, etc.) from the image database 930, which stores multiple training images (S400). Next, the processor 911 performs a reliability evaluation on the read training image, etc. (S410). The reliability evaluation here is consistent with that described in... Figure 7 The processing is the same, but the range on the image used to evaluate reliability is not a part of the training image or the like, but the entirety of the training image or the like. The reliability of the training image is also calculated in the same way as in the above embodiment, based on at least one of the edge number of the edge image, the edge intensity of the edge image, or the average brightness of the training image.

[0067] Next, it is determined whether the reliability of the disparity of the training images as a whole is below a reference value (S420). This determination process is equivalent to the step of selecting training images for training the disparity estimation neural network model based on the reliability of the training images. The details of this determination process are omitted here because they are the same as those described in the first embodiment.

[0068] Next, the disparity estimation neural network model is trained using training images whose disparity reliability is below a baseline value (S430). This training process is equivalent to training the disparity estimation neural network model using the selected training images. Steps from S400 to S430 are repeated until all images contained in the image database 930 or all images set as training objects are read (S440). The trained disparity estimation neural network model is stored in memory.

[0069] By means of the processing described in the third embodiment, images with low feature values ​​can be selected from multiple images to train the neural network, thus enabling efficient training of the neural network used in the second disparity generation unit 50.

[0070] Furthermore, the present invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above have been explained in detail to facilitate understanding of the present invention, and are not limited to necessarily possessing all the described configurations. In addition, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of one embodiment can also be added to the configuration of another embodiment. Furthermore, regarding a part of the configuration of each embodiment, other configurations can be added, deleted, or replaced.

[0071] Symbol Explanation

[0072] 1: Vehicle control system; 10: Image processing device; 11: Processor; 12: Memory; 13: Vehicle control device; 14: Bus cable; 20: Image acquisition unit; 21: Vehicle-mounted camera; 22: Left camera; 23: Right camera; 30: First parallax generation unit; 35: First parallax image; 40: Image range specification unit; 50: Second parallax generation unit; 55: Second parallax image; 60: Parallax synthesis unit; 70: Recognition processing unit; 200: Vehicle ahead; 210: Region of interest in the first embodiment; 410: Region of interest determination unit; 420: Parallax reliability evaluation unit; 430: Parallax reliability determination unit; 440: Search range determination unit; 500: Region of interest in the second embodiment; 2110: Left image; 2210: Right image; 90: Information processing device; 911: Processor; 912: Memory.

Claims

1. An image processing apparatus, characterized in that, have: The image acquisition unit acquires a set of images with parallax captured by multiple cameras mounted on the vehicle; The first disparity generation unit searches for regions in another image that match a portion of pixel blocks in one image, thereby determining the first disparity of the set of images; The image range specification unit specifies the processing range based on the reliability of the image in relation to a specific region, wherein the specific region is a partial region of one or a group of images; as well as The second disparity generation unit calculates a set of second disparities of the images by means of neural network processing for the processing range specified by the image range specification unit.

2. The image processing apparatus as claimed in claim 1, characterized in that, The device includes an identification processing unit that identifies the external environment based on at least one of a first disparity generated by a first disparity generation unit or a second disparity generated by a second disparity generation unit.

3. The image processing apparatus as described in claim 2, characterized in that, The device includes a disparity synthesis unit that generates a synthesized disparity image by synthesizing the first disparity and the second disparity. The recognition processing unit identifies the external environment based on the synthetic parallax image generated by the parallax synthesis unit.

4. The image processing apparatus as claimed in claim 2, characterized in that, The image range designation unit determines the specific region based on the object being identified in the recognition processing unit.

5. The image processing apparatus as claimed in claim 2, characterized in that, The image range specification unit converts one or a group of images into edge images. The reliability of the image related to the specific region is determined based on at least one of the following: the number of edges in the edge image located at coordinates corresponding to the specific region, the edge intensity of the edge image, or the average value of the brightness of the image.

6. The image processing apparatus as claimed in claim 2, characterized in that, If the reliability of the image related to the specific region is below a threshold, the image range specifying unit causes the second disparity generation unit to generate the second disparity.

7. The image processing apparatus as claimed in claim 2, characterized in that, If the reliability associated with the image of the specific region is higher than a threshold, the image range specification unit does not cause the second disparity generation unit to generate the second disparity.

8. The image processing apparatus as claimed in claim 1, characterized in that, The image range specification unit specifies the processing range based on the reliability of images related to a specific region, which is preset for the field of view of the camera capturing the set of images.

9. The image processing apparatus as claimed in claim 8, characterized in that, The image range specification unit determines the location of a specific area in the field of view of the camera that captured the set of images based on at least one of the vehicle's driving environment, driving time period, weather, the camera's position relative to the vehicle, and the camera's degradation status.

10. The image processing apparatus as claimed in claim 1, characterized in that, The image range specifying unit determines the search range of disparity within the processing range based on the disparity generated by the first disparity generation unit. The second disparity generation unit calculates the second disparity based on the disparity search range determined by the image range specification unit.

11. An image processing method, which is executed by an image processing apparatus having a processor and a memory, characterized in that, The processor executes: The steps of acquiring a set of images with parallax captured by multiple cameras mounted on a vehicle; The step of searching in another image for a region that matches a partial pixel block of one image to determine the first disparity of the set of images; The step of specifying the processing range based on the reliability associated with an image of a specific region, wherein the specific region is a partial region of one or a group of images; as well as The step of determining the second disparity of the set of images by means of a neural network processing method for the processing range specified by the steps described above.

12. A training method for training a disparity inference neural network model executed by an information processing device having a processor and a memory, characterized in that, The processor executes: The step of receiving at least one training image or an edge image generated from the training image from a database; The step of calculating the reliability of the training image based on at least one of the edge count of the edge image, the edge intensity of the edge image, or the average brightness of the training image; The step of selecting training images for training the disparity estimation neural network model based on the reliability of the training images. The step of training a disparity estimation neural network model using the selected training images; as well as The step of storing the trained disparity estimation neural network model into the memory.

Images