Image processing apparatus

Abstract

The purpose of the present invention is to provide an image processing device capable of calculating correct parallax. An image processing device (20) is provided with: a parallax calculation unit (30) that calculates the parallax of images (111, 112) captured by a plurality of cameras (11, 12); a marking unit (40) that classifies the subject captured in the images (111, 112), and assigns a label corresponding to the type of the classified subject to each pixel of the subject; a reliability calculation unit (50) that calculates the reliability of the parallax for each given tag; and a parallax correction unit (60) that corrects the parallax on the basis of the calculated reliability.

Description

Technical Field

[0001] The present invention relates to an image processing device. Background Art

[0002] In order to improve the driving safety of vehicles, a system that detects objects around the vehicle using sensors installed in the vehicle has been studied. When there is a possibility of collision with an object, such a system notifies the driver of an alarm or activates an automatic emergency brake.

[0003] Sensors for monitoring the surroundings of a vehicle include millimeter wave radars, laser radars, or cameras. The types of cameras include monocular cameras or stereo cameras with multiple cameras. A stereo camera can measure the distance to an object using the parallax of the overlapping area of ​​images captured by two cameras arranged at a predetermined interval. Therefore, a stereo camera can accurately grasp the risk of collision with an object.

[0004] A stereo camera calculates the parallax of images captured by a plurality of cameras and converts the calculated parallax into a distance, so it is desirable to calculate accurate parallax. As a technology for calculating accurate parallax, there is Patent Document 1, for example.

[0005] Patent document 1 discloses a stereo image processing device, characterized in that it includes: a shooting unit, which shoots an object; a stereo processing unit, which performs stereo matching processing on each small area of ​​a pair of images shot by the shooting unit by determining the corresponding area by calculating the correlation between each other, and obtains an evaluation function as a result of the stereo matching processing, distance information to the object obtained based on the evaluation function, and reliability information indicating its reliability; and a parallax correction unit, which sets a re-search range around the small area that becomes the object according to the reliability, corrects the evaluation function calculated for the small area that becomes the object according to the evaluation function calculated for the small area within the re-search range, calculates a corrected evaluation function, and corrects the distance information according to the calculated corrected evaluation function. Prior art literature Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2006-90896 Summary of the invention Technical problem to be solved by the invention

[0007] The device disclosed in Patent Document 1 corrects the parallax by correcting the evaluation function calculated in units of matching blocks set in stereo matching. Therefore, in the device disclosed in Patent Document 1, even the corrected parallax may contain a large error depending on the state of the captured image, so there is room for improvement.

[0008] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing device capable of calculating accurate parallax. Technical solutions to technical problems

[0009] In order to solve the above-mentioned technical problems, the image processing device of the present invention is characterized in that it includes: a parallax calculation unit, which calculates the parallax of an image captured by multiple cameras; a labeling unit, which classifies the objects captured in the image and assigns a label corresponding to the type of the classified object to each pixel of the object; a reliability calculation unit, which calculates the reliability of the parallax for each assigned label; and a parallax correction unit, which corrects the parallax according to the calculated reliability. Effects of the Invention

[0010] According to the present invention, it is possible to provide an image processing device capable of calculating accurate parallax. Problems, structures, and effects other than those described above will become more apparent from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a diagram showing the structure of a stereo camera device including an image processing device according to the present embodiment. Figure 2 A diagram showing a parallax image. Figure 3 is a diagram showing a label image. Figure 4 It is a diagram for explaining the blocked part. Figure 5 is a diagram showing a reliability table. Figure 6 is a diagram showing a reliability image. Figure 7 This is a flowchart showing the operation of the image processing device. Figure 8 : is a flowchart showing the processing performed by the reliability calculation unit. Fig. 9 1 is a diagram showing a structural example of data for one pixel written in the reliability image. Fig.10 is a flowchart showing the processing performed by the interpolation unit. Fig.11 Yes Description Fig.10 The diagram of step S34 is shown. Fig.12 This is a flowchart showing the processing performed by the recalculation unit. DETAILED DESCRIPTION

[0012] Hereinafter, embodiments of the present invention will be described using the drawings. Unless otherwise specified, structures denoted by the same reference numerals in each embodiment have the same functions in each embodiment, and thus description thereof will be omitted.

[0013] Figure 1 1 is a diagram showing a configuration of a stereo camera device 1 including an image processing device 20 according to the present embodiment. Figure 2 is a diagram showing the parallax image 130 . Figure 3 is a diagram showing the label image 140 . Figure 4 It is a diagram for explaining the shielding portion 241. Figure 5 : is a diagram showing the reliability table 51 . Figure 6 is a diagram showing the reliability image 150 .

[0014] The stereo camera device 1 is mounted on a vehicle and is one of the sensors used to monitor the periphery of the vehicle. The stereo camera device 1 detects objects around the vehicle and measures the distance to the objects by using the parallax of overlapping areas of images 111 and 112 captured by a plurality of cameras 11 and 12 arranged at predetermined intervals in the horizontal direction.

[0015] Figure 1 The stereo camera device 1 shown includes a left camera 11 and a right camera 12 , and an image processing device 20 that performs image processing on a left image 111 captured by the left camera 11 and a right image 112 captured by the right camera 12 .

[0016] The image processing device 20 is constituted by, for example, a computer including a processor such as a CPU and a storage device such as a ROM and a RAM. The image processing device 20 implements various functions of the image processing device 20 by the CPU executing a program stored in the ROM.

[0017] The image processing device 20 includes a parallax calculation unit 30 , a labeling unit 40 , a reliability calculation unit 50 , a parallax correction unit 60 , and a recognition processing unit 70 .

[0018] The parallax calculation unit 30 calculates the parallax of the left image 111 and the right image 112 captured by the left camera 11 and the right camera 12 for each pixel. The parallax calculation unit 30 of the present embodiment has a neural network model. The left image 111 and the right image 112 are set in the input layer of the neural network model. The parallax of the left image 111 and the right image 112 is set in the output layer of the neural network model. In the middle layer of the neural network model, the weight, bias and activation function of each neuron are set. The neural network model is constructed by performing machine learning on the weight and bias of each neuron using the error back propagation method, etc., and is pre-assembled in the parallax calculation unit 30. The machine learning can be learning with a teacher. In addition, the parallax calculation unit 30 can also have a model constructed by machine learning other than the neural network model. The model possessed by the parallax calculation unit 30 can also be, for example, an optical flow model.

[0019] The parallax calculation unit 30 generates a parallax image 130 in which the calculated parallax is stored in each pixel. Figure 2 As shown, the parallax image 130 generates the parallax corresponding to each pixel of the right image 112 as an image stored in each pixel of the right image 112 . Figure 2 The parallax image 130 shown shows that the darker the color, the larger the parallax and the closer the distance.

[0020] The labeling unit 40 classifies the subject captured in the right image 112, and assigns a label corresponding to the type of the classified subject to each pixel of the subject. The labeling unit 40 can use a neural network model to classify the subject and assign labels. The labeling unit 40 of the present embodiment classifies the subject and assigns labels through panoptic segmentation. The processing overview of panoptic segmentation is introduced in the paper "Panoptic Feature Pyramid Networks" by Kirillov et al., 2019 IEEE / CVF Conference on CVPR. The labeling unit 40 can cite this paper.

[0021] The marking portion 40 is, for example, Figure 3 As shown, based on the learned label information, the objects captured in the right image 112 are classified in units of pixels, such as road surface 210, water 220, ditch 230, vehicle 240, vehicle 250, and zebra crossing 260. Then, the labeling unit 40 writes the label number of the label assigned to each object to each pixel of the object. In this way, the labeling unit 40 generates a label image 140 in which the label number is stored in each pixel.

[0022] In addition, in the area captured by the right camera 12, the left camera 11 may have a blind spot and not capture a portion. Since the correct parallax cannot be calculated in such an obstructed portion 241, it is necessary to clearly indicate that the parallax is invalid. In order to identify the obstructed portion 241, the marking unit 40 performs the following processing. For example, Figure 4 As shown, in the right image 112 and the left image 111, the overlapping conditions of the vehicle 240 and the vehicle 250 are different, and the blocked portion 241 is not captured in the left image 111. In this case, the marking unit 40 compares the parallax of the vehicle 240 and the vehicle 250, and determines the subject with a large parallax, that is, the area existing on the right side of the vehicle 240, as the blocked portion 241. The lateral width of the blocked portion 241 is the number of pixels of the difference between the parallax of the vehicle 240 and the parallax of the vehicle 250.

[0023] In addition, the labeling unit 40 can verify the consistency between the type of the classified subject and the label assigned to the subject based on the parallax calculated by the parallax calculation unit 30. For example, in the case where a billboard with a photo of a person's portrait is the subject, each parallax of the pixel group constituting the subject represents a value that is approximately constant or linearly changes in the pixel group, and the subject is recognized as a plane. On the other hand, in the case where an actual person is the subject, each parallax of the pixel group constituting the subject represents a value that changes nonlinearly in the pixel group, and the subject is recognized as a three-dimensional object. In addition, the labeling unit 40 uses not only the right image 112 but also the parallax image 130, so that the consistency between the type of the classified subject and the label assigned to the subject can be verified.

[0024] The reliability calculation unit 50 calculates the reliability of the parallax for each label assigned by the labeling unit 40. The reliability is an index that digitizes the size of the error included in the parallax value. As the error included in the parallax value becomes smaller, the reliability becomes higher. In this embodiment, if the error included in the parallax value is greater than 20%, the reliability is "low", if the error is greater than 10% and less than 20%, the reliability is "medium", and if the error is less than 10%, the reliability is "high".

[0025] Depending on the conditions around the vehicle where the images 111 and 112 are taken, there is a case where the parallax is erroneous and the correct parallax cannot be calculated. If the incorrect parallax is directly used for recognition processing, it is possible that the object is misdetected or cannot be detected. Therefore, the reliability calculation unit 50 calculates the reliability to clarify the subject assumed to have an incorrect parallax.

[0026] The reliability calculation unit 50 calculates the reliability of the parallax using a reliability table 51 prepared in advance. Figure 5 As shown in FIG. 5 , the reliability table 51 is a table showing the correspondence between labels and reliability. Figure 5In the reliability table 51 shown, when the reliability item is "0", it means that the reliability is "invalid", when the reliability item is "1", it means that the reliability is "low", when the reliability item is "2", it means that the reliability is "medium", and when the reliability item is "3", it means that the reliability is "high".

[0027] The reliability calculation unit 50 refers to the reliability table 51 and determines the reliability corresponding to the label assigned by the labeling unit 40. Then, the reliability calculation unit 50 writes the determined reliability to each pixel of the object assigned the label. Figure 6 As shown, a reliability image 150 is generated in which a label number and reliability are stored in each pixel.

[0028] The parallax correction unit 60 corrects the parallax based on the reliability calculated by the reliability calculation unit 50. The parallax correction unit 60 includes an interpolation unit 61 and a recalculation unit 62.

[0029] The interpolation unit 61 interpolates the parallax whose reliability does not satisfy the reference based on the parallax calculated for the images 111 and 112 taken in the past. The interpolation unit 61 can correct the parallax image 130 by interpolating the parallax whose reliability does not satisfy the reference. The interpolation unit 61 of the present embodiment determines whether to interpolate the parallax whose reliability is "low" or "invalid" based on the parallax calculated for the images 111 and 112 taken in the past, and when it is determined to interpolate, the parallax whose reliability is "high" is used to interpolate the parallax whose reliability is "low" or "invalid". For example, the area whose reliability is calculated as "low" or "invalid" for the currently taken images 111 and 112 is set to the area whose reliability is calculated as "high" in the images 111 and 112 taken in the past. In this case, the interpolation unit 61 of this embodiment determines to interpolate the parallax with "low" or "invalid" reliability in the area, and interpolates the parallax with "low" or "invalid" reliability in the area using the parallax with "high" reliability adjacent to the area.

[0030] The recalculation unit 62 recalculates the parallax whose reliability does not meet the reference in a different manner from that of the parallax calculation unit 30. The recalculation unit 62 can correct the parallax image 130 by replacing the parallax whose reliability does not meet the reference with the recalculated parallax. The recalculation unit 62 of the present embodiment recalculates the parallax whose reliability is "medium" or "low" by, for example, stereo matching. The recalculation unit 62 of the present embodiment may also use the parallax interpolated by the interpolation unit 61 as the object of recalculation. In addition, when the parallax calculation unit 30 calculates the parallax by stereo matching, the recalculation unit 62 may recalculate the parallax using a neural network model.

[0031] The recognition processing unit 70 detects objects around the vehicle based on the parallax corrected by the parallax correction unit 60, the reliability calculated by the reliability calculation unit 50, and the images 111 and 112. Specifically, the recognition processing unit 70 detects objects around the vehicle using the corrected parallax image 130, the reliability image 150, the left image 111, and the right image 112.

[0032] For example, the area of ​​the road surface with a reliability of "high" has the correct parallax and can detect small objects on the road surface. On the other hand, the area of ​​the road surface with a reliability of "low" makes it difficult to distinguish between small objects on the road surface and the error contained in the parallax. Therefore, in order to avoid false detection in the area with a reliability of "low", the recognition processing unit 70 changes the size of the object to be detected or the distance to the object according to the reliability. For example, the recognition processing unit 70 uses the objects on the road surface with a height larger than the error contained in the parallax of the road surface as the detection object in the area with a reliability of "low". The height of the object is high when the object photographed as the object is located at a position close to the cameras 11 and 12 or has a height higher than a specified value. Therefore, the recognition processing unit 70 uses the objects with a height larger than the error contained in the parallax of the road surface as the detection object in the area of ​​the road surface with a reliability of "low", and uses the objects with a height smaller than the area of ​​the road surface with a reliability of "high" as the detection object in the area of ​​the road surface with a reliability of "high". In this way, the recognition processing unit 70 can change the size of the object to be detected or the distance to the object according to the reliability. As for the method of object detection itself, the recognition processing unit 70 can adopt a well-known method.

[0033] In addition, the parallax image 130 and the reliability image 150 are input to the recognition processing unit 70 in correspondence with each other for each pixel. In addition, the information of the parallax, reliability, and label is input to the recognition processing unit 70 in correspondence with each other for each pixel. Thus, the recognition processing unit 70 can, for example, assign labels to pixels whose reliability does not meet the reference after the fact, or calculate the reliability after the fact. That is, the recognition processing unit 70 can re-modify the data (reliability or label) acquired in the previous stage of object detection, and can improve the accuracy of object detection.

[0034] Figure 7 It is a flowchart showing the operation of the image processing device 20 .

[0035] In step S11 , the parallax calculation unit 30 of the image processing device 20 generates a parallax image 130 from the left image 111 and the right image 112 .

[0036] In step S12 , the labeling section 40 of the image processing device 20 classifies the subject using the right image 112 and the parallax image 130 , and assigns a label to each pixel of the subject to generate a label image 140 .

[0037] In step S13 , the reliability calculation unit 50 of the image processing device 20 calculates the reliability of the parallax for each label using the reliability table 51 , and generates the reliability image 150 .

[0038] In step S14 , the parallax correction unit 60 of the image processing device 20 corrects the parallax whose reliability does not satisfy the reference.

[0039] In step S15, the recognition processing unit 70 of the image processing device 20 detects an object using the parallax image 130, the reliability image 150, the right image 112, and the left image 111. Thereafter, as a process corresponding to the object detection result, the image processing device 20 performs a process such as sending a warning notification or a signal urging the operation of the automatic emergency brake to the vehicle control device, and ends the present process.

[0040] Figure 8 : is a flowchart showing the processing performed by the reliability calculation unit 50 . Fig. 9 1 is a diagram showing a structural example of data for one pixel written in the reliability image 150 .

[0041] In step S21, the reliability calculation unit 50 initializes (Y=0) the Y coordinate value of the coordinates (X, Y) used to access (read and write) the label image 140 and the reliability image 150. The coordinates (X, Y) represent the positions of the pixels of the label image 140 and the reliability image 150. The Y coordinate value represents the vertical coordinate value of the label image 140 and the reliability image 150.

[0042] In step S22 , the reliability calculation unit 50 initializes the X coordinate value of the coordinate (X, Y) (X=0). The X coordinate value indicates the horizontal coordinate value of the label image 140 and the reliability image 150 .

[0043] In step S23 , the reliability calculation section 50 reads the label number of the coordinates (X, Y) from the label image 140 pixel by pixel.

[0044] In step S24 , the reliability calculation section 50 acquires the reliability of the parallax corresponding to the label number from the reliability table 51 .

[0045] In step S25, the reliability calculation unit 50 writes the tag number and reliability into the coordinates (X, Y) of the reliability image 150. The data written at this time is, for example, Fig. 9 As shown in the figure. Fig. 9 In the example, one pixel of the written data is 8 bits, bit0 to bit3 are composed of reliability, and bit4 to bit7 are composed of label numbers.

[0046] In step S26 , the reliability calculation section 50 updates the X coordinate value (X=X+1).

[0047] In step S27, the reliability calculation unit 50 determines whether the X coordinate value has reached a maximum value. The maximum value of the X coordinate value indicates the coordinate value of the lateral end of the label image 140 and the reliability image 150. When the X coordinate value has reached a maximum value, the reliability calculation unit 50 transfers to step S28. When the X coordinate value has not reached a maximum value, the reliability calculation unit 50 transfers to step S23.

[0048] In step S28 , the reliability calculation section 50 updates the Y coordinate value (Y=Y+1).

[0049] In step S29, the reliability calculation unit 50 determines whether the Y coordinate value has reached a maximum value. The maximum value of the Y coordinate value represents the coordinate value of the longitudinal end of the label image 140 and the reliability image 150. When the Y coordinate value has reached the maximum value, the reliability calculation unit 50 ends the present process. When the Y coordinate value has not reached the maximum value, the reliability calculation unit 50 transfers to step S22.

[0050] Fig.10 : is a flowchart showing the processing performed by the interpolation unit 61 . Fig.11 Yes Description Fig.10 FIG. 10 is a diagram of step S34 shown in FIG. Fig.10 and Fig.11 In the description, a case where the road surface between the preceding vehicle and the host vehicle is made a processing target of the interpolation unit 61 is taken as an example.

[0051] In step S31, the interpolation unit 61 determines whether there is a preceding vehicle in the reliability image 150. If there is a preceding vehicle, the interpolation unit 61 proceeds to step S32. If there is no preceding vehicle, the interpolation unit 61 ends the present process.

[0052] In step S32, the interpolation unit 61 determines whether there is an area with "low" or "invalid" reliability on the road surface between the preceding vehicle and the host vehicle. If there is an area with "low" or "invalid" reliability, the interpolation unit 61 transfers to step S33. If there is no area with "low" or "invalid" reliability, the interpolation unit 61 ends the present process.

[0053] In step S33, the interpolation unit 61 determines whether there is a past image (label image 140 or reliability image 150) that captures a preceding vehicle traveling in an area with a reliability of "low" or "invalid". If there is a past image that captures a preceding vehicle, the interpolation unit 61 transfers to step S34. If there is no past image that captures a preceding vehicle, the interpolation unit 61 ends the present process.

[0054] In step S34, the interpolation unit 61 interpolates the parallax of the road surface with the reliability of "high" in units of rasters for the parallax of the road surface with the reliability of "low" or "invalid". After step S34, the interpolation unit 61 ends the present process.

[0055] For example, Fig.11 As shown, it is assumed that there is an area 310 (e.g., an area of ​​water accumulation) with a "low" or "invalid" reliability on the road surface between the preceding vehicle and the host vehicle. In this case, the interpolation unit 61 interpolates using the parallax of the road surface 300 with a "high" reliability adjacent to the area 310. Specifically, the interpolation unit 61 interpolates the parallaxes B to E of the pixels 311 to 314 constituting the area 310 using the parallax A of the pixel 301 of the road surface 300 adjacent to the pixel 311 in the horizontal direction and the parallax F of the pixel 302 of the road surface 300 adjacent to the pixel 314 in the horizontal direction.

[0056] First, the interpolation unit 61 calculates equation (1). Interpolation value = (parallax F - parallax A) / (X coordinate value of pixel 302 - X coordinate value of pixel 301) ... (1)

[0057] Next, the interpolation unit 61 calculates equation (2). Parallax B = parallax A + interpolation value × (X coordinate value of pixel 311 - X coordinate value of pixel 301) (2)

[0058] The parallaxes C to E are also calculated in the same manner as in equation (2). By performing the above calculation in raster units, the interpolation unit 61 can interpolate the parallax of the road surface with "high" reliability using the parallax of the road surface with "low" reliability or "invalid" reliability.

[0059] Fig.12 : is a flowchart showing the processing performed by the recalculation unit 62 .

[0060] In step S41, the recalculation unit 62 initializes (Y=0) the Y coordinate value of the coordinates (X, Y) used to access (read and write) the reliability image 150 and the parallax image 130. The coordinates (X, Y) represent the position of each pixel of the reliability image 150 and the parallax image 130. The Y coordinate value represents the vertical coordinate value of the reliability image 150 and the parallax image 130.

[0061] In step S42 , the recalculation unit 62 initializes the X coordinate value of the coordinate (X, Y) (X=0). The X coordinate value indicates the horizontal coordinate value of the reliability image 150 and the parallax image 130 .

[0062] In step S43 , the recalculation unit 62 reads the reliability of the coordinates (X, Y) from the reliability image 150 pixel by pixel.

[0063] In step S44, the recalculation unit 62 determines whether the reliability is "medium" or "low". If the reliability is "medium" or "low", the recalculation unit 62 moves to step S45. If the reliability is not "medium" or "low", the recalculation unit 62 moves to step S46.

[0064] In step S44, the case where the reliability is not "medium" or "low" includes the case where the reliability is "high" and the case where the reliability is "invalid". In the case where the reliability is "high", it is not necessary to correct the parallax, so the recalculation unit 62 transfers to step S46. In the case where the reliability is "invalid", even if the recalculation is performed, the possibility of calculating the parallax with a reliability of "high" is low, so the recalculation unit 62 transfers to step S46 to retain the correction of the parallax. In addition, for pixels having a parallax whose reliability is calculated as "invalid", since there is a sufficient possibility that a parallax with a reliability of "high" is calculated in the images 111 and 112 to be captured next time, there is no problem even if the recalculation unit 62 retains the correction of the parallax.

[0065] In step S45, the recalculation unit 62 recalculates the disparity of the coordinates (X, Y) using a method different from that of the disparity calculation unit 30. When the disparity calculation unit 30 calculates the disparity using a neural network model, the recalculation unit 62 may recalculate the disparity by stereo matching.

[0066] In step S46, the recalculation unit 62 updates the X coordinate value (X=X+1).

[0067] In step S47, the recalculation unit 62 determines whether the X coordinate value has reached the maximum value. The maximum value of the X coordinate value indicates the coordinate value of the lateral end of the reliability image 150 and the parallax image 130. If the X coordinate value has reached the maximum value, the recalculation unit 62 transfers to step S48. If the X coordinate value has not reached the maximum value, the recalculation unit 62 transfers to step S43.

[0068] In step S48, the recalculation unit 62 updates the Y coordinate value (Y=Y+1).

[0069] In step S49, the recalculation unit 62 determines whether the Y coordinate value has reached the maximum value. The maximum value of the Y coordinate value indicates the coordinate value of the longitudinal end of the reliability image 150 and the parallax image 130. When the Y coordinate value reaches the maximum value, the recalculation unit 62 ends the present process. When the Y coordinate value does not reach the maximum value, the recalculation unit 62 transfers to step S42.

[0070] As described above, the image processing device 20 of the present embodiment includes: a parallax calculation unit 30, which calculates the parallax of images 111 and 112 captured by multiple cameras 11 and 12; a labeling unit 40, which classifies the objects captured in the images 111 and 112, and assigns a label corresponding to the type of the classified object to each pixel of the object; a reliability calculation unit 50, which calculates the reliability of the parallax for each assigned label; and a parallax correction unit 60, which corrects the parallax based on the calculated reliability.

[0071] Thus, the image processing device 20 of the present embodiment can calculate the reliability of the parallax that changes according to the conditions of the captured images 111 and 112 in units of pixels, and can clarify the incorrect parallax in units of pixels. Therefore, the image processing device 20 of the present embodiment can correct the parallax according to the conditions of the captured images 111 and 112, and can calculate the correct parallax. Therefore, according to the present invention, it is possible to provide an image processing device that can calculate the correct parallax.

[0072] In the image processing device 20 according to the present embodiment, the parallax correction unit 60 includes a recalculation unit 62 that recalculates the parallax whose reliability does not satisfy the reference using a method different from that of the parallax calculation unit 30 .

[0073] Thus, the image processing device 20 of this embodiment can calculate parallax using a plurality of calculation methods, so even if an error occurs in parallax due to the characteristics of one calculation method, the parallax can be compensated by another calculation method. Therefore, the image processing device 20 of this embodiment can calculate a more accurate parallax.

[0074] In the image processing device 20 of this embodiment, the parallax calculation unit 30 calculates the parallax using a neural network model in which the images 111 and 112 are set as an input layer and the parallax is set as an output layer, and the recalculation unit 62 recalculates the parallax by stereo matching.

[0075] Thus, the image processing device 20 of the present embodiment can compensate for the parallax by stereo matching even if an error occurs in the parallax due to the characteristics of the calculation method using the neural network model. Therefore, the image processing device 20 of the present embodiment can calculate a more accurate parallax.

[0076] In the image processing device 20 of the present embodiment, the parallax correction unit 60 includes the interpolation unit 61 that interpolates the parallax whose reliability does not satisfy the reference based on the parallax calculated for the images 111 and 112 captured in the past.

[0077] Thus, even if the conditions of the currently captured images 111 and 112 are not suitable for parallax calculation, the image processing device 20 of this embodiment can determine the appropriateness of interpolating the parallax and interpolate the parallax based on the parallax of the previously captured images 111 and 112. Therefore, the image processing device 20 of this embodiment can calculate a more accurate parallax.

[0078] In the image processing device 20 according to the present embodiment, the labeling unit 40 verifies the consistency between the type of the classified object and the label assigned to the object based on the parallax.

[0079] As a result, the image processing device 20 of the present embodiment can more accurately assign labels, and thus can accurately calculate the reliability. Therefore, the image processing device 20 of the present embodiment can calculate more accurate parallax.

[0080] Furthermore, in the image processing device 20 of the present embodiment, the reliability calculation unit 50 calculates the reliability using the reliability table 51 indicating the correspondence relationship between the label and the reliability.

[0081] Thus, the image processing device 20 of the present embodiment can reduce the amount of calculation for calculating the reliability, and thus can easily calculate the reliability even with a simple configuration. Therefore, the image processing device 20 of the present embodiment can calculate a more accurate parallax.

[0082] In addition, the image processing device 20 of the present embodiment also includes a recognition processing unit 70, which detects objects around a vehicle equipped with multiple cameras 11 and 12 based on the corrected parallax, reliability and images 111 and 112. The recognition processing unit 70 changes the size of the subject to be detected or the distance to the subject according to the reliability.

[0083] Thus, the image processing device 20 of the present embodiment can exclude a subject having a parallax whose reliability does not satisfy a reference from detection targets. Therefore, the image processing device 20 of the present embodiment can calculate an accurate parallax and suppress erroneous detection of an object.

[0084] In the image processing device 20 of the present embodiment, information on the parallax, reliability, and label are input to the recognition processing unit 70 in association with each other for each pixel.

[0085] Thus, the image processing device 20 of this embodiment can re-modify the reliability and label acquired in the previous stage of object detection, thereby improving the accuracy of object detection. Therefore, the image processing device 20 of this embodiment can calculate the correct parallax and improve the accuracy of object detection.

[0086] The present invention is not limited to the above-mentioned embodiments, but also includes various modified examples. For example, the above-mentioned embodiments are detailed descriptions for the purpose of explaining the present invention in an understandable manner, and the present invention is not necessarily limited to including all the structures described. In addition, a part of the structure of a certain embodiment may be replaced with the structure of another embodiment, and in addition, the structure of another embodiment may be added to the structure of a certain embodiment. In addition, other structures may be added, deleted, or replaced with a part of the structure of each embodiment.

[0087] In addition, the above-mentioned structures, functions, processing parts, processing units, etc. can also be implemented in hardware by designing a part or all of them, such as through integrated circuits. In addition, the above-mentioned structures, functions, etc. can also be implemented in software by interpreting and executing programs that implement the respective functions through a processor. Information such as programs, tables, files, etc. that implement the various functions can be stored in a memory or a recording device such as a hard disk, SSD (Solid State Drive), or a recording medium such as an IC card, SD card, DVD, etc.

[0088] In addition, although control lines and information lines necessary for explanation are shown, this does not limit the display to all control lines and information lines necessary for the product. In fact, it can be considered that almost all structures are connected to each other. Description of symbols

[0089] 11.12 Camera 20 Image processing device 30 Parallax calculation unit 40 Marking Department 50 Reliability Calculation Department 51 Reliability Table 60 Parallax correction unit 61 Interpolation Department 62 Recalculation Unit 70 Identification processing unit.

Claims

1. An image processing device, characterized in that: include: a parallax calculation unit that calculates the parallax of images captured by the plurality of cameras; a labeling unit, which classifies the subject captured in the image and assigns a label corresponding to the type of the classified subject to each pixel of the subject; a reliability calculation unit that calculates the reliability of the parallax for each of the assigned labels; as well as A parallax correction unit is configured to correct the parallax based on the calculated reliability.

2. The image processing device according to claim 1, wherein: The parallax correction unit includes a recalculation unit configured to recalculate the parallax whose reliability does not satisfy a reference by using a method different from that of the parallax calculation unit.

3. The image processing device according to claim 2, wherein: The parallax calculation unit calculates the parallax using a neural network model that sets the image as an input layer and the parallax as an output layer, The recalculation unit recalculates the parallax by stereo matching.

4. The image processing device according to claim 1, wherein: The parallax correction unit includes an interpolation unit that interpolates the parallax whose reliability does not satisfy a reference based on the parallax calculated for the image captured in the past.

5. The image processing device according to claim 1, wherein: The labeling unit verifies consistency between the type of the classified object and the label assigned to the object based on the parallax.

6. The image processing device according to claim 1, wherein: The reliability calculation unit calculates the reliability using a reliability table indicating a correspondence relationship between the label and the reliability.

7. The image processing device according to claim 1, wherein: The system further includes a recognition processing unit that detects objects around the vehicle equipped with the plurality of cameras based on the corrected parallax, the reliability, and the image. The recognition processing unit changes the size of the object to be detected or the distance to the object according to the reliability.

8. The image processing device according to claim 7, wherein: The parallax, the reliability, and the label information are input to the recognition processing unit in correspondence with each other for each pixel.

Images