Image processing apparatus and image processing method

Abstract

An image processing device is provided with: a region division unit that divides each of a plurality of captured images captured by an in-vehicle camera into a plurality of divided regions; a feature point detection unit that detects a portion exceeding a threshold value as a feature point for each divided region; a threshold value setting unit that sets a threshold value for each divided region such that the distribution of feature points detected for each divided region becomes uniform; a feature point matching unit that matches the feature points detected in the plurality of captured images in association with each other among the plurality of captured images; and a relative parameter calculation unit that calculates a relative parameter on the basis of the feature points that correspond to each other among the plurality of captured images.

Description

Technical Field

[0001] The present invention relates to an image processing device and an image processing method. Background Art

[0002] In order to recognize the three-dimensional position of the outside world reflected in the image using the image processing device, the image processing device needs to be calibrated. Calibration is performed by calculating camera parameters and relative parameters. Camera parameters are parameters that represent the lens distortion and focal length of the camera used in the image processing device. Relative parameters are parameters that represent the relationship between the relative position and posture of two cameras or a moving camera before and after movement.

[0003] In the case where the camera that captures the image is a stereo camera composed of two cameras, one of the cameras is used as a reference, and the relationship between the relative position and posture of the other camera relative to the reference camera is calculated as a relative parameter. In addition, the process in which the image processing device calculates the relative three-dimensional position ((x, y, z) axis position) and relative orientation ((x, y, z) axis angle) of the left camera relative to the reference with the position of the right camera as a reference is called calibration.

[0004] As a technique for calibrating an image processing device, there is known a technique described in Patent Document 1. Patent Document 1 describes "selecting an image in which feature points are sufficiently distributed at a distance corresponding to the purpose of use of two cameras, and using the feature points in the image distributed at a distance corresponding to the purpose of use of the two cameras and the image coordinates of corresponding points to calculate external parameters of the two cameras." Prior art literature Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2009-276233 Summary of the invention Problem that the invention aims to solve

[0006] In order to perform high-precision correction of an image processing device, the distribution of feature points detected from a captured image is important. In a captured image whose feature point distribution is not suitable for correction, the accuracy of correction is reduced. In the method described in Patent Document 1, in which a captured image is judged by the distance distribution of feature points and only captured images for which a high-precision correction result can be expected are extracted to correct a photographic device, only captured images for which a high-precision correction result can be expected are extracted. However, in the technology disclosed in Patent Document 1, a captured image for which the calculated number of feature points is less than a threshold value is not suitable for correction. Since captured images that are not suitable for such a configuration are discarded, there is a problem in that the opportunity for correction is reduced.

[0007] The present invention has been made in view of such circumstances, and an object of the present invention is to improve the accuracy of correction performed by an image processing device. Technical means of solving problems

[0008] The image processing device involved in the present invention comprises: a region segmentation unit, which segments a plurality of captured images captured by a vehicle-mounted camera into a plurality of segmented regions; a feature point detection unit, which detects a portion exceeding a threshold value as a feature point in each segmented region; a threshold setting unit, which sets a threshold value in each segmented region in such a way that the distribution of the feature points detected in the segmented region becomes uniform; a feature point matching unit, which matches the feature points detected in the plurality of captured images in a correspondence between the plurality of captured images; and a relative parameter calculation unit, which calculates relative parameters based on the feature points corresponding to each other in the plurality of captured images. Effects of the Invention

[0009] According to the present invention, by improving the accuracy of the correction performed by the image processing device, the image processing device can more accurately recognize the three-dimensional position of the outside world. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing an example of the internal structure of an image processing device according to one embodiment of the present invention. Figure 2 This is a flowchart showing an example of overall processing of the image processing device according to one embodiment of the present invention. Figure 3 1 is a flowchart showing an example of uniform feature point detection processing performed by the uniform feature point detection unit according to one embodiment of the present invention. Figure 4 This is a diagram showing an example of an image divided into a uniform grid according to one embodiment of the present invention. Figure 5 This is a diagram showing an example of an image divided into a grid shape by increasing the number of divisions according to one embodiment of the present invention. Figure 6 It is a diagram showing an example of a captured image divided according to a vehicle state according to one embodiment of the present invention. Figure 7 This is a diagram showing how a captured image captured A before in time series is divided according to one embodiment of the present invention. Figure 8 This is a diagram showing how a captured image captured at the time B later in the time series is divided according to one embodiment of the present invention. Fig. 9 FIG. 1 is a diagram showing an example of feature points detected from a first captured image according to an embodiment of the present invention. Fig.10FIG. 1 is a diagram showing an example of feature points detected from the second captured image according to one embodiment of the present invention. Fig.11 This is a diagram showing an example of a parallelized image in which the distribution of feature points is biased after the conventional technique is applied. Fig.12 This is an example of a parallelized image obtained by uniformizing the distribution of feature points using the technology of one embodiment of the present invention. Fig.13 This is a block diagram showing an example of the hardware configuration of a computer according to one embodiment of the present invention. DETAILED DESCRIPTION

[0011] Hereinafter, the method for implementing the present invention will be described with reference to the accompanying drawings. In this specification and the accompanying drawings, for components having substantially the same function or configuration, duplicate descriptions are omitted by assigning the same symbols. For example, the present invention can be applied to a vehicle control computing device that can communicate with an on-board ECU (Electronic Control Unit) for an advanced driver assistance system (ADAS) or an autonomous driving (AD).

[0012] [One embodiment] Figure 1 This is a block diagram showing an example of the internal configuration of the image processing device 100 according to one embodiment. The image processing device 100 is one of the devices mounted on an image processing ECU (not shown) mounted on a vehicle. The image processing device 100 has a high-precision correction function.

[0013] The vehicle-mounted camera 101 is an example of a camera mounted on the vehicle in order for the image processing device 100 to recognize the outside of the vehicle, and captures the outside at a predetermined timing. The vehicle-mounted camera 101 outputs the captured image 10 of the vehicle surroundings captured to recognize the three-dimensional distance of the outside to the uniform feature point detection unit 102 of the image processing device 100. The vehicle-mounted camera 101 may be two stereo cameras or a single monocular camera.

[0014] The captured images 10 outputted by the vehicle-mounted camera 101 to the uniform feature point detection unit 102 may be two captured images 10 captured simultaneously by two cameras (stereo cameras), or may be two captured images 10 captured by one camera (monocular camera) at different times. When a monocular camera is used as the vehicle-mounted camera 101, for example, one captured image 10 of an object captured by the monocular camera while the vehicle is stationary and one captured image 10 of the same object captured after the vehicle has slightly moved are outputted.

[0015] The image processing device 100 corrects the relative parameters of the vehicle-mounted camera 101 based on the two captured images 10, and performs processing to identify the three-dimensional distance of the outside world from the captured images 10. Here, the calibration of the image processing device 100 is performed each time the vehicle-mounted camera 101 is activated. For example, the calibration is performed at a predetermined time interval after the vehicle power is turned on and the vehicle-mounted camera 101 is just activated, or while the vehicle is driving. The image processing device 100 includes a uniform feature point detection unit 102, a feature point matching unit 103, a relative parameter calculation unit 104, a parallax image generation unit 105, and a three-dimensional distance recognition unit 106. In the following, refer to Figure 2 The processing of each functional unit is described.

[0016] Figure 2 2 is a flowchart showing an example of the overall processing of the image processing apparatus 100. Figure 2 , shows an example of an image processing method of the image processing device 100.

[0017] First, the uniform feature point detection unit 102 acquires the captured image 10 from the vehicle-mounted camera 101 (S1). The received captured image 10 is stored in a Fig.13 The storage devices such as the RAM 53 and the nonvolatile memory 55 are appropriately read from and rewritten in the processing of each functional unit of the image processing device 100. In the following description, the input and output processing of the captured image 10 to the storage device is omitted.

[0018] Then, in step S1, the uniform feature point detection unit 102 analyzes the captured image 10 received from the vehicle-mounted camera 101 and detects the contours of the objects (lanes, sidewalks, other vehicles, signs, etc.) reflected in the captured image 10. Figure 3 In the uniform feature point detection process (S2) shown, characteristic points such as corners of contour lines or intersections of contour lines are detected as feature points.

[0019] The uniform feature point detection unit 102 applies a unique method in the uniform feature point detection process shown in step S2, and operates so as to detect feature points with high accuracy of the relative parameters calculated by the relative parameter calculation unit 104 thereafter. The operations of the uniform feature point detection unit 102 include segmentation of the captured image 10, detection of feature points in the regions segmented by the segmentation process, counting of the number of feature points, and setting of thresholds (individual thresholds) for feature point detection. The operations of the uniform feature point detection unit 102 allow the number of feature points in each segmented region after the captured image 10 is segmented by the segmentation process to be uniformly detected.

[0020] Here, if Figure 1 As shown, the uniform feature point detection unit 102 includes a region segmentation unit 1021 , a feature point detection unit 1022 , and a threshold setting unit 1023 . The region dividing unit (region dividing unit 1021 ) divides each of the plurality of captured images 10 captured by the vehicle-mounted camera (vehicle-mounted camera 101 ) into a plurality of divided regions. The feature point detection unit (feature point detection unit 1022 ) detects a portion exceeding a threshold value (individual threshold value) as a feature point for each segmented area. The threshold setting unit (threshold setting unit 1023 ) sets a threshold (individual threshold) for each segmented area so that the distribution of the feature points detected for each segmented area becomes uniform. Below, refer to Figure 3 The following will describe uniform feature point detection processing executed by each functional unit of the uniform feature point detection unit 102 .

[0021] Figure 3 1 is a flowchart showing an example of uniform feature point detection processing performed by the uniform feature point detection unit 102. The uniform feature point detection processing is Figure 1 The image processing apparatus 100 shown in FIG. 1 is a part of the image processing method performed by each functional unit.

[0022] First, the region dividing unit 1021 divides the captured image 10 received from the vehicle-mounted camera 101 into a predetermined size and a predetermined number of divisions (S11). Figures 4 to 8 An example of processing performed by the region segmentation unit 1021 and an example of segmenting regions will be described.

[0023] For example, the region segmentation unit 1021 reads two captured images 10 acquired from the vehicle-mounted camera 101 one by one. The region segmentation unit (region segmentation unit 1021) increases the number of segments of the captured image 10 and reduces the size of the segmented regions to perform segmentation so that at least one feature point is detected in each segmented region. As an example of segmenting the captured image 10, for example, Figure 4 and Figure 5 Example shown.

[0024] Figure 4 The diagram shows an example of an image (captured image 10 ) divided into a uniform grid. Figure 4The captured image 10 captured by the vehicle-mounted camera 101 shown is divided into 4 evenly spaced grid-like divided areas of 2 rows × 2 columns by the area division unit 1021. In this case, the number of divisions of the captured image 10 is "4". In addition, the size of each divided area is the same. By dividing the captured image 10 into 4, the size of the divided area becomes one-fourth the size of the original captured image 10. Even when the captured image 10 is divided into 4 like this, it can be expected that the feature points in the captured image 10 are evenly arranged in each divided area depending on the object.

[0025] Figure 5 The figure shows an example of an image divided into a grid shape by increasing the number of divisions. Figure 5 The captured image 10 captured by the vehicle-mounted camera 101 is divided into 16 evenly spaced grid-shaped regions of 4 rows x 4 columns by the region division unit 1021. In this case, the number of divisions of the captured image 10 is "16". In addition, the size of each division region is the same. And, by increasing the number of divisions, Figure 5 The size of the segmented area shown becomes Figure 4 The area dividing unit 1021 can evenly arrange the feature points included in the captured image 10 in each divided area by increasing the number of divisions.

[0026] The reason why the region segmentation unit 1021 uniformly arranges the feature points in the captured image 10 by increasing the number of segmentations of the captured image 10 is described. For example, the deviation of the feature points existing in one region cannot be corrected later. Therefore, in the case where the upper half of the captured image 10 before segmentation reflects a sign or a vehicle, and the lower half reflects a lane, it is expected that more feature points will be detected in the upper half of the captured image 10 than in the lower half.

[0027] Therefore, the region segmentation unit 1021 increases the number of segments of the captured image 10 and divides the captured image 10 into segmented regions of small sizes, thereby even if there is a deviation in the feature points in each segmented region, the feature points can be evenly arranged in each segmented region as the entire captured image 10. For example, if 100 feature points are randomly arranged on the entire captured image 10, the arrangement positions of the feature points will be deviated. This is because, if 100 feature points are randomly arranged on the entire captured image 10 and then the entire captured image 10 is evenly divided into 100 equal parts, it can be expected that segmented regions containing one or more feature points and segmented regions not containing feature points will be generated. On the other hand, if one feature point is arranged in each segmented region after the entire captured image 10 is evenly divided into 100 equal parts, the feature points can be evenly arranged relative to the entire captured image 10. Therefore, after the captured image 10 is evenly divided, the feature points can be detected in each segmented region, so that the image processing device 100 can use the feature points detected in the entire captured image 10 for correction.

[0028] Figure 6 FIG. 1 is a diagram showing an example of a captured image 10 divided according to the state of the vehicle. Considering that the vehicle-mounted camera 101 is mounted on the vehicle, relatively close objects may be reflected in a larger size at the lower side of the captured image 10, while distant objects may be reflected in a smaller size at the upper side of the captured image 10. In addition, when there are no other vehicles traveling in front of the vehicle, a lane with uniform density is reflected in a large range at the lower side of the captured image 10.

[0029] Therefore, the region division unit (region division unit 1021) changes the position and size of the region for dividing the captured image 10 according to the mounting position of the vehicle-mounted camera (vehicle-mounted camera 101). For example, considering the condition of the vehicle and the characteristics of the captured image 10, the region division unit 1021 reduces the number of divisions for the region where a close object is reflected in a large size, and divides the captured image 10 into a large-sized region. On the other hand, for the region where a far-distance object is reflected in a small size, the region division unit 1021 increases the number of divisions for the captured image 10 into a small-sized region.

[0030] exist Figure 6 In the example of the segmentation process shown in FIG. 1 , the long-distance portion (the upper half of the captured image 10), that is, the portion corresponding to the traveling direction of the vehicle, needs to be processed by the three-dimensional distance recognition unit 106 (see FIG. 1 ). Figure 1) identifies the distance from the vehicle to the object in detail. In order to identify feature points from the captured image 10, the region segmentation unit 1021 of the uniform feature point detection unit 102 determines the segmentation mode (the number of segmentation areas and the segmentation size of each segmentation area) of the captured image 10, taking into account driving scenarios such as what size of object is reflected in which area on average, and how much the feature points located in that part will move in which direction over time. Therefore, the region segmentation unit (region segmentation unit 1021) segments the captured image 10 in a portion where more feature points are expected to be, with a smaller size than the portion where fewer feature points are expected to be. For example, in Figure 6 In the image, the portion near the center of the long-distance portion (the portion reflecting the lane) is divided into smaller sizes, and the portions near both ends of the long-distance portion (the portions reflecting the sidewalk) are divided into larger sizes.

[0031] In this way, the area segmentation unit 1021 can segment the captured image 10 into a larger size for an area with fewer expected feature points (a road or sidewalk at a short distance) than for an area with more expected feature points (a road or sidewalk at a long distance).

[0032] By changing the number of divisions and the size of divisions according to the characteristics of the object reflected in the captured image 10 by the region division unit 1021, the number of feature points detected from the segmented region reflecting the object at a short distance and the number of feature points detected from the segmented region reflecting the object at a long distance can be made close to uniform. As a result, the correction accuracy of the captured image 10 by the image processing device 100 in the depth direction can be improved.

[0033] Next, an example will be described in which the region segmentation unit 1021 performs different segmentation based on the time-series images before and after the traveling vehicle (time-series before A and time-series after B). Figure 7 and Figure 8 1 is a diagram showing examples of captured images 10 captured by the vehicle-mounted camera 101 at different times. Figure 7 1 is a diagram showing how a captured image captured A time ago in the time series is divided. Figure 8 1 and 2 are diagrams showing how the captured image captured after B in the time series is segmented. Figure 8 The captured image 10 shown is relative to Figure 7 The captured image 10 is an image captured only after time t. That is, Figure 8 The captured image 10 shown is an image captured by the vehicle-mounted camera 101 when the vehicle has moved forward for a period of time t.

[0034] The region segmentation unit (region segmentation unit 1021) segments the plurality of captured images 10 at different sizes, based on the prediction result of the feature points corresponding to the three-dimensional positions of the outside world, which are detected from the plurality of captured images 10 captured by the vehicle-mounted camera (vehicle-mounted camera 101) as the vehicle moves. For example, the region segmentation unit 1021 predicts where the feature points corresponding to the three-dimensional positions detected from the captured images 10 captured before A in the time series and the captured images 10 captured after B in the time series are located. Then, the region segmentation unit 1021 segments the two captured images 10 at different segmentation sizes in a manner that includes the positions of the predicted feature points.

[0035] Compare Figure 7 and Figure 8 The captured image 10 shown in FIG. Figure 7 The height H1 of the upper area of ​​the image divided into 8 divided areas is Figure 8 In the captured image 10 shown, the height H2 is reached, and the upper area of ​​the captured image 10 becomes wider. Figure 7 The area on the left side of the image shown is divided into a width W11 including a lane located far away and a width W12 including a sidewalk located far away. Figure 8 When the image 10 is captured, the vehicle is moving forward and approaching the object. Figure 8 The area on the left side of the image shown, as shown by the width W21 of the lane located far away, is larger than Figure 7 The width W11 shown is wide. On the other hand, Figure 8 The width W22 of the sidewalk in the far distance in the left area of ​​the image shown becomes larger than Figure 7 Width W12 is shown to be narrow.

[0036] In order to match the segmented regions of the captured image 10 captured before the time series A and the captured image 10 captured after the time series B, it is necessary to have the segmented regions of the captured image 10 captured before the time series A and the segmented regions of the captured image 10 captured after the time series B. Therefore, the number of segments of the two captured images 10 is set to be the same.

[0037] In this way, the feature point detection unit 1022 can reliably detect feature points of a region (e.g., a lane) where an object of interest is located in the direction of travel of the vehicle by changing the segmentation size of the captured image 10 before and after the time series by the region segmentation unit 1021. In addition, when the feature point matching unit 103 described later matches the segmented regions, it is possible to reduce erroneous operations of matching wrong segmented regions.

[0038] Return again Figure 3 Continue with the explanation. After the process of step S11, the feature point detection unit 1022 selects one segmented region segmented by the segmentation unit 1021 (S12). Then, the feature point detection unit 1022 detects feature points of the segmented region selected in step S12 using individual thresholds (S13).

[0039] In the feature point detection process, a detection threshold is used to increase or decrease the number of feature points detected by the feature point detection unit 1022 in the segmented area. Generally, the feature point detection threshold is set to one value for one captured image 10. On the other hand, the feature point detection unit 1022 of the present embodiment can set the detection threshold separately for each segmented area. Therefore, the detection threshold set for each segmented area is called an "individual threshold". The individual threshold is a value set for each segmented area by the threshold setting unit 1023 described later.

[0040] However, the individual threshold value set by the threshold setting unit 1023 has a range of settable values. Here, a brightness threshold value is used as the individual threshold value. By using the brightness threshold value, it is possible to perform binarization processing on each pixel such that the area above a certain brightness is white and the area below a certain brightness is black. For example, if the brightness threshold value is increased, the black area in the captured image 10 increases, and if the brightness threshold value is lowered, the white area in the captured image 10 increases. Furthermore, when the brightness threshold value is increased to the maximum value, the captured image 10 is completely painted black, and when the brightness threshold value is lowered to the minimum value, the captured image 10 is completely painted white, so in both cases, feature points can no longer be detected. Therefore, a range of effective brightness threshold values ​​is determined for the detection of feature points in each driving scene. Then, the threshold setting unit 1023 increases or decreases the individual threshold value within the range.

[0041] After step S13, the feature point detection unit 1022 counts the number of feature points of the segmented area detected in step S13 (S14). The number of feature points of the segmented area counted by the feature point detection unit 1022 is temporarily stored in the RAM 53 (refer to Fig.13 ) etc.

[0042] After the processing (S12 to S14) of the feature point detection unit 1022, the processing of the threshold setting unit 1023 begins. The threshold setting unit 1023 sets the threshold in order to make the feature points detected by the feature point detection unit 1022 uniform. Therefore, the threshold setting unit 1023 compares the number of feature points with the target value for each segmented area whose number of feature points is counted in step S14, and determines whether the number of feature points is more than the target value (S15). In steps S15 to S17, the threshold setting unit (threshold setting unit 1023) increases the set threshold for the segmented area whose number of feature points is more than the target value, and lowers the set threshold for the segmented area whose number of feature points is less than the target value. Therefore, individual thresholds are set so that the number of feature points detected by the feature point detection unit 1022 is close to the target value. Here, refer to Fig. 9 and Fig.10 The number of feature points will be described.

[0043] Fig. 9 1 is a diagram showing an example of feature points detected from the first captured image 10 . Fig.10 1 is a diagram showing an example of feature points detected from the second captured image 10 . Fig. 9 and Fig.10 The solid and dotted circles represent feature points. The solid circles represent feature points detected and matched to the same part in the two captured images 10 , while the dotted circles represent feature points not detected from the same part in the two captured images 10 .

[0044] Usually, the matching of feature points is performed by comparing the feature value of a feature point in one captured image 10 (base image) with the feature values ​​of all feature points in the other captured image 10 (reference image), and searching for the feature point with the closest feature value. The feature value is a numerical value representing the shape of the feature point, and is obtained by converting the configuration of pixels (brightness values) around the feature point into numerical values ​​in a certain order. Although the feature points can be conceived of various shapes, if they are the same object, they can be assumed to be the same feature points. However, since the matching of feature points is performed on the segmented areas at the same position in the two captured images 10, it can reduce the load and shorten the time until the matching is completed compared to the process of matching the feature points detected in the entire captured image 10.

[0045] Back to Figure 3 Description. Assumptions Fig. 9 The upper left side (inner side) of the captured image 10 shown is Figure 6The segmented area is segmented by the segmentation pattern shown. In this case, since many feature points are detected in the segmented area on the upper left side (inner side) of the captured image 10, the threshold setting unit 1023 determines that the number of feature points in the segmented area is more than the target value (No in S15). Therefore, the threshold setting unit 1023 performs a process of increasing the individual threshold (S16). As a result, the detection threshold in the segmented area becomes the maximum value. That is, it becomes the maximum detection threshold within the settable range. However, it is also possible to set the individual threshold at the divided interval in step S16 by dividing from the current individual threshold to the maximum value at a specified interval. Therefore, the individual threshold can also be set to a value smaller than the maximum value.

[0046] On the other hand, assuming Fig. 9 The lower side (near front side) of the captured image 10 shown is formed by Figure 6 The segmented area is segmented by the segmentation pattern shown. In this case, since almost no feature points are detected in the segmented area on the lower side (near front side) of the captured image 10, the threshold setting unit 1023 determines that the number of feature points in the segmented area is below the target value (No in S15). Therefore, the threshold setting unit 1023 performs a process of lowering the feature point detection threshold (S17). As a result, the detection threshold in the segmented area becomes the minimum value. That is, the detection threshold becomes the minimum value within the settable range. However, it is also possible to set the individual threshold at the divided interval in step S17 by dividing from the current individual threshold to the minimum value at a specified interval. Therefore, the individual threshold can also be set to a value larger than the minimum value.

[0047] After step S16 or S17, the uniform feature point detection unit 102 determines whether the processing after step S12 has been performed on all the segmented areas segmented by the area segmentation unit 1021 (S18). When the uniform feature point detection unit 102 determines that there are unprocessed segmented areas (No in S18), it again performs the processing starting from step S12 on the unprocessed segmented areas. Here, the feature point detection unit (feature point detection unit 1022) detects feature points using the threshold value set by the threshold setting unit (threshold setting unit 1023). Therefore, the number of feature points detected by the feature point detection unit 1022 becomes a number consistent with the target value. On the other hand, when the uniform feature point detection unit 102 determines that the processing has been performed on all the segmented areas (Yes in S18), it ends. Figure 3 The uniform feature point detection process is shown.

[0048] In addition, Figure 3In the uniform feature point detection process shown in FIG. 1 , after setting an individual threshold value for a certain segmented area through the process of steps S16 and S17, the feature point detection unit 1022 performs feature point detection processing on other segmented areas using the individual threshold value set in the previous process. This processing is to install Figure 3 In the case of uniform feature point detection processing shown above, the processing time for all segmented areas is shortened.

[0049] However, if the processing capacity of the image processing device 100 is sufficient, it is more effective for the feature point detection unit 1022 to perform feature point detection again in the same segmented area. In this case, based on the individual threshold value set by the threshold setting unit 1023 for a certain segmented area, the feature point detection unit 1022 detects feature points again for the same segmented area with the set single threshold value. After that, the individual threshold value setting process performed by the threshold setting unit 1023 is not performed, and the process starting from step S12 can be performed on other segmented areas.

[0050] Back to Figure 1 and Figure 2 Continue with the explanation. exist Figure 3 After the processing of the uniform feature point detection unit 102 shown in FIG. 1 , the feature point matching unit (feature point matching unit 103) matches the feature points detected in the plurality of captured images 10 by establishing correspondence between the plurality of captured images 10 (S3). At this time, the feature point matching unit 103 performs matching by establishing correspondence between the feature points detected as the same part in the two captured images 10 output from the uniform feature point detection unit 102. Therefore, the feature point matching unit 103 can match the feature points detected in the two captured images 10 before segmentation as in the conventional method.

[0051] In addition, when segmenting two captured images 10 to generate segmented areas, the feature point matching unit (feature point matching unit 103) matches the feature points in the segmented areas at the same position in the plurality of captured images 10. Therefore, compared with the conventional process of matching the feature points by matching the entire captured image 10, the feature point matching unit 103 can match the feature points uniformly detected in the entire captured image 10.

[0052] The matching process of feature points is as described above. Fig. 9 and Fig.10For example, the feature point matching unit 103 uses the segmented regions segmented by the region segmentation unit 1021 to compare a feature point in a segmented region in one captured image 10 with a feature point in a corresponding segmented region in another captured image 10. By such processing, since the range of the segmented region to be matched is limited, the feature point matching unit 103 can avoid erroneous matching with feature points detected in other segmented regions, and can shorten the time required for the matching operation.

[0053] The relative parameter calculation unit (relative parameter calculation unit 104) calculates the relative parameters based on the corresponding feature points obtained between the plurality of captured images 10 (S4). Therefore, the relative parameter calculation unit 104 calculates the relative parameters using the camera parameters and the matched feature points. The relative parameters are parameters that represent the relationship between the relative position and posture of the other camera observed from one camera before and after the movement of the two vehicle-mounted cameras 101 or a moving vehicle-mounted camera 101. Then, when calculating the relative parameters, the relative parameter calculation unit 104 calculates in a manner that minimizes the sum of the re-projection errors of all the matched feature points in the two captured images 10.

[0054] Here, when the vehicle-mounted camera (vehicle-mounted camera 101) is a stereo camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses the axis position and axis angle of one camera when one of the two cameras constituting the stereo camera captures the captured image 10 as a reference, and uses the relative axis position and axis angle of the other camera when the other camera captures the other captured image 10 as relative parameters. For example, the position of the right camera is used as a reference, and the relative three-dimensional position ((x, y, z) axis position) and relative orientation ((x, y, z) axis angle) of the left camera relative to the reference are used as relative parameters.

[0055] In addition, when the vehicle-mounted camera 101 is a monocular camera, the captured images 10 captured before and after the vehicle-mounted camera 101 moves are used. However, it is difficult for the posture of the monocular camera to remain constant before and after the movement. For example, under ideal conditions where the vehicle is traveling straight on a straight line, the posture of the vehicle-mounted camera 101 is constant. However, in reality, the posture of the vehicle-mounted camera 101 may change due to slight steering wheel operations causing the angle of the vehicle-mounted camera 101 to change to the left or right, or due to the operation of the brake or accelerator causing the angle of the vehicle-mounted camera 101 to change to the up or down, so it is conceivable that the posture of the vehicle-mounted camera 101 may change.

[0056] Therefore, when the vehicle-mounted camera (vehicle-mounted camera 101) is a monocular camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses the axis position and axis angle of the monocular camera when capturing one captured image 10 among the multiple captured images 10 captured by the monocular camera as the vehicle moves as a reference, and uses the relative axis position and axis angle of the monocular camera when capturing other captured images 10 as relative parameters. For example, using the position of the monocular camera captured at a certain moment as a reference, the relative three-dimensional position ((x, y, z) axis position) and relative orientation ((x, y, z) axis angle) of the monocular camera captured after time t relative to the reference are used as relative parameters. As a result, even when a monocular camera is used as the vehicle-mounted camera 101, relative parameters can be correctly calculated.

[0057] Next, the parallax image generating unit (parallax image generating unit 105) generates parallax images of the plurality of captured images 10 based on the relative parameters (S5). Therefore, the parallax image generating unit 105 generates a parallelized image by parallelizing the two captured images 10 based on the relative parameters. In the parallelized image, the epipolar lines are horizontal and have the same Y coordinate. Then, the parallax image generating unit 105 performs stereo matching on the parallelized images to generate a parallax image.

[0058] Next, the three-dimensional distance recognition unit (three-dimensional distance recognition unit 106) recognizes the three-dimensional distance of the outside world based on the parallax image (S6). For example, the three-dimensional distance recognition unit 106 calculates the three-dimensional distance of the outside world based on the parallax value of the parallax image generated by the parallax image generation unit 105, the focal length of the camera included in the camera parameters, and the pixel size of the light receiving element configured in the vehicle-mounted camera 101, and recognizes the external object. After that, the processing of the image processing device 100 ends.

[0059] As described above, even when the vehicle-mounted camera 101 is composed of a single monocular camera, the three-dimensional distance recognition unit 106 can calculate the three-dimensional distance of the outside world and recognize objects in the outside world. In this case, the vehicle-mounted camera 101 is required to take regular photos of the outside world before and after the vehicle moves in chronological order, and accurately know how the vehicle-mounted camera 101 moves between the two captured images 10. In addition, if the vehicle-mounted camera 101 is composed of two stereo cameras, the camera positions of the two captured images 10 taken by the stereo camera are fixed and accurately known. On the other hand, since the camera position before and after the movement of a monocular camera that moves with the vehicle is prone to large errors, the recognition accuracy of the three-dimensional distance of the outside world will be reduced compared to the case of using a stereo camera, but it has the effect of being able to recognize the three-dimensional distance using one camera.

[0060] So far, the functional units configured in the image processing device 100 have been described. Now, the effect of uniform feature point detection unit 102 uniformizing feature points by applying the technology of this embodiment will be described. Fig.11 and Fig.12 An example of a parallelized video generated by the parallax image generating unit 105 after the relative parameter calculating unit 104 calculates the relative parameter will be described.

[0061] Fig.11 This is a diagram showing an example of a parallelized image in which the distribution of feature points is biased, to which the conventional technology is applied. Fig.11 The horizontal axis is the X-axis of the captured image 10 , and the vertical axis is the Y-axis of the captured image 10 .

[0062] The black circles and dotted grids shown in the figure represent the feature points detected from the captured image 10 (reference image) captured by one camera and the tilt of the parallelized image. In addition, the white circles and solid grids shown in the figure represent the feature points detected from the captured image 10 (reference image) captured by the other camera and the tilt of the parallelized image. Ideally, it can be expected that the two parallelized parallelized images will have the same tilt. Here, the two captured images 10 are divided into 2 rows × 2 columns of areas 21 to 24.

[0063] exist Fig.11 In the example of the parallelized image shown, when calculating the relative parameter that minimizes the reprojection error, the weight is calculated for the feature point. The weight is a value obtained by accumulating the errors of the feature points detected from the two-dimensional captured image 10 (the errors of the white and black circles shown in the figure). In an area with many feature points, even if the error of one feature point is small, the weight of the feature point for the evaluation value becomes larger because the errors of multiple feature points are accumulated. On the other hand, in an area with fewer feature points, only the errors of a smaller number of feature points are accumulated, so even if the error of one feature point is large, the weight of the feature point for the evaluation value becomes smaller.

[0064] Based on the relationship between feature points and weights, Fig.11 The weight in the relative parameter calculation of the region 22 with dense feature points becomes larger, and the weight in the region 23 with sparse feature points becomes smaller. Therefore, in the relative parameter calculation, the relative parameters optimized for the region with large weight and dense feature points are calculated, and a large deviation occurs in the region 23 with sparse feature points. As a result, the captured image 10 (reference image) shown by the solid line is tilted more with respect to the captured image 10 (baseline image) shown by the dotted line, and parallelization is not achieved.

[0065] Fig.12 This is an example of a parallelized image in which the distribution of feature points is made uniform by applying the technology of this embodiment. exist Fig.12 The black circles, white circles, dotted lines and solid lines in the figure are also consistent with Fig.11 Same as shown. exist Fig.12 In the example of the parallelized image shown, the feature points are evenly distributed in the areas 21 to 24 (the entire image) of the two captured images 10. Therefore, the weights in the calculation of the relative parameters are also uniformed in the two captured images 10. As a result, a calculation result with less distortion can be obtained. In addition, the tilt of the captured image 10 (reference image) shown by the solid line is suppressed relative to the captured image 10 (baseline image) shown by the dotted line, and parallelization is achieved.

[0066] Next, the hardware configuration of the computer 50 constituting the image processing apparatus 100 will be described. Fig.13 1 is a block diagram showing an example of the hardware configuration of a computer 50. The computer 50 is an example of hardware that can be used as a computer that can operate as the image processing device 100 of this embodiment. The image processing device 100 of this embodiment implements the image processing device 100 by executing a program on the computer 50 (computer). Figure 1 and Figure 3 The image processing method is performed by the cooperation of the functional blocks shown.

[0067] The computer 50 includes a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM (Random Access Memory) 53, each of which is connected to a bus 54. In addition, the computer 50 includes a nonvolatile memory 55 and a network interface 56.

[0068] The CPU 51 reads the program code of the software that realizes each function of the present embodiment from the ROM 52, loads it into the RAM 53, and executes it. In the RAM 53, variables or parameters generated in the middle of the operation processing of the CPU 51 are temporarily written, and these variables or parameters are read appropriately by the CPU 51. However, an MPU (Micro Processing Unit) may be used instead of the CPU 51.

[0069] As the nonvolatile memory 55, for example, a HDD (Hard Disk Drive), an SSD (Solid State Drive), a floppy disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, or a nonvolatile memory can be used. In addition to an OS (Operating System) and various parameters, the nonvolatile memory 55 also records a program for making the computer 50 function. The ROM 52 and the nonvolatile memory 55 record programs and data required for the CPU 51 to operate, and are used as an example of a computer-readable non-transient storage medium that stores programs executed by the computer 50.

[0070] The network interface 56 uses, for example, a NIC (Network Interface Card) and can transmit and receive various data between devices via a LAN (Local Area Network) or a dedicated line connected to a NIC port.

[0071] In the image processing device 100 of one embodiment described above, when feature points are detected from the captured image 10, the detected feature points are operated so that the distribution within the captured image 10 is suitable for correction, so that the correction (calculation of relative parameters) can be made more accurate. Therefore, the three-dimensional distance recognition unit 106 provided in the image processing device 100 can recognize the three-dimensional position represented by the three-dimensional distance of the outside world more accurately than before.

[0072] Here, the two captured images 10 for matching are segmented by the region segmentation unit 1021 with the same size and number of segments. The captured image 10 is segmented so that at least one feature point is detected in the segmented region. Therefore, by evenly arranging the feature points in each segmented region, the feature points are evenly arranged in the entire captured image 10. In addition, it is also easier to match the feature points of each segmented region at the same position in each captured image 10.

[0073] In addition, the size of the segmented area is changed according to the number of feature points predicted for each part of the captured image 10. The captured image 10 is segmented in such a way that the size of the part predicted to have a large number of feature points is small, and the size of the part predicted to have a small number of feature points is large. As a result, even in the part predicted to have a small number of feature points, at least one feature point can be detected.

[0074] In addition, the individual thresholds for detecting feature points can be changed for each segmented area. Therefore, by setting the individual thresholds for the portion predicted to have a large number of feature points higher, the number of detected feature points can be reduced. Conversely, by setting the individual thresholds for the portion predicted to have a small number of feature points lower, the number of detected feature points can be increased. As a result, the number of feature points detected for each segmented area is uniformed.

[0075] Furthermore, since the arrangement of feature points varies, even captured images 10 that were previously discarded can be used for correction by the image processing device 100. Therefore, the image processing device 100 can increase the chances of performing correction and can perform correction more frequently and accurately than before.

[0076] [Modifications] In addition, if a malfunction occurs in one of the cameras constituting the stereo camera, correction can be performed using only the other camera (for example, the right camera) using the captured images 10 captured before and after the vehicle moves. In this case, the camera on one side can be corrected by the same processing as when a monocular camera is used as the vehicle-mounted camera 101.

[0077] In addition, the same segmented areas of two captured images 10 captured at different shooting times can also be matched by acquiring two captured images 10 captured by a stereo camera at different timings. For example, after the uniform feature point detection unit 102 detects feature points from the segmented areas of the two captured images 10 acquired at the initial timing, it sets an individual threshold. Then, the uniform feature point detection unit 102 detects feature points for the segmented areas of the two captured images 10 acquired at the next timing using the individual threshold set at the initial timing. In this way, for the captured images 10 captured twice within a short period of time, by using the individual threshold set initially in the next feature point detection, the three-dimensional distance recognition unit 106 can more accurately recognize the three-dimensional distance of the outside world.

[0078] In addition, the present invention is not limited to the above-mentioned embodiment, and various other application examples and modified examples can of course be adopted as long as they do not depart from the gist of the present invention described in the claims. For example, the above-mentioned embodiment is an embodiment that describes the configuration of the system in detail and specifically in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to having all the configurations described. In addition, other configurations may be added, deleted, or replaced for a part of the configuration of the present embodiment. In addition, the control lines and information lines are those that are considered necessary for the purpose of explanation, and not all control lines and information lines are necessarily shown on the product. In reality, it can be considered that almost all components are connected to each other. Explanation of symbols

[0079] 10…capturing an image, 100…image processing device, 101…vehicle-mounted camera, 102…uniform feature point detection unit, 103…feature point matching unit, 104…relative parameter calculation unit, 105…parallax image generation unit, 106…three-dimensional distance recognition unit, 1021…region segmentation unit, 1022…feature point detection unit, 1023…threshold setting unit.

Claims

1. An image processing device, characterized in that: have: A region segmentation unit, which segments a plurality of images captured by the vehicle-mounted camera into a plurality of segmentation regions; a feature point detection unit, which detects a portion exceeding a threshold value according to the segmented area as a feature point; a threshold setting unit that sets the threshold for each segmented area so that distribution of the feature points detected for each segmented area becomes uniform; a feature point matching unit for matching the feature points detected in the plurality of captured images so as to establish correspondence between the plurality of captured images; and A relative parameter calculation unit calculates a relative parameter based on the feature points associated with the plurality of captured images.

2. The image processing device according to claim 1, characterized in that The threshold setting unit sets a higher threshold for the segmented area having a larger number of feature points than a target value, and sets a lower threshold for the segmented area having a smaller number of feature points than a target value, The feature point detection unit detects the feature point using the threshold value set by the threshold value setting unit.

3. The image processing device according to claim 2, characterized in that: The region dividing unit divides the captured image by increasing the number of divisions and reducing the size of the divided regions so that at least one of the feature points is detected in each of the divided regions.

4. The image processing device according to claim 3, characterized in that: The region dividing unit changes the position and size of the region for dividing the captured image according to the mounting position of the vehicle-mounted camera.

5. The image processing device according to claim 4, characterized in that: The region dividing unit divides the captured image, which is predicted to be a portion with many feature points, into a smaller size than a portion predicted to be a portion with few feature points.

6. The image processing device according to claim 4, characterized in that: The region segmentation unit segments the plurality of captured images into different sizes based on a prediction result of feature points, wherein the feature points are feature points corresponding to three-dimensional positions of the outside world detected from the plurality of captured images captured by the vehicle-mounted camera as the vehicle moves.

7. The image processing device according to claim 5, characterized in that: The feature point matching unit matches the feature points of the divided regions having the same position in the plurality of captured images in a corresponding manner.

8. The image processing device according to claim 6, characterized in that: When the vehicle-mounted camera is a stereo camera, the relative parameter calculation unit uses the axis position and axis angle of one of the two cameras constituting the stereo camera when capturing the captured image as a reference, and uses the relative axis position and axis angle of the other camera when capturing the other captured image as the relative parameters.

9. The image processing device according to claim 6, characterized in that: In the case where the vehicle-mounted camera is a monocular camera, the relative parameter calculation unit uses the axis position and axis angle of the monocular camera when capturing one of the multiple images captured by the monocular camera as the vehicle moves as a reference, and uses the relative axis position and axis angle of the monocular camera when capturing other images as the relative parameters.

10. The image processing device according to claim 8, characterized in that: Further possesses: a parallax image generating unit configured to generate a plurality of parallax images of the captured images based on the relative parameters; and A three-dimensional distance recognition unit recognizes the three-dimensional distance of the outside world based on the parallax image.

11. An image processing method, characterized in that: include: The processing of dividing the multiple images captured by the vehicle-mounted camera into multiple divided areas respectively; The portion exceeding the threshold value detected in the segmented area is processed as a feature point; A process of setting the threshold value for each segmented area in such a manner that distribution of the feature points detected for each segmented area becomes uniform; A process of establishing correspondence between the feature points detected in the plurality of captured images; A process of calculating relative parameters based on the feature points corresponding to the plurality of captured images.

Images