Image processing device and image processing method

Abstract

An image processing device comprises: an area subdivision unit that divides each of a plurality of captured images taken by an in-vehicle camera into a plurality of subdivided areas; a feature point detection unit that detects as a feature point a section exceeding a threshold for each of the subdivided areas; a threshold setting unit that sets the threshold for each of the subdivided areas to evenly distribute the feature points detected in each of the subdivided areas; a feature point matching unit that performs a matching process by associating the feature points detected in the plurality of captured images with each other;and a relative parameter calculation unit that calculates a relative parameter based on the feature points linked between the multitude of captured images.

Description

Technical area

[0001] The present invention relates to an image processing apparatus and an image processing method. State of the art

[0002] To use an image processing device to detect a three-dimensional position in the outside world that appears in an image, it is necessary to calibrate the image processing device. Calibration is performed by calculating camera parameters and relative parameters. Camera parameters are parameters that indicate lens distortion, focal length, and the like of cameras used in the image processing device. Relative parameters are parameters that indicate relationships between relative positions and postures before and after the movement of two cameras or a moving camera.

[0003] In a case where a stereo camera, comprising two cameras, is used to capture an image, one of the cameras serves as a reference. The relationships between the relative position and orientation of the other camera with respect to the reference camera are calculated as relative parameters. Additionally, using the position of a right camera as a reference, the calculation of the relative three-dimensional position (axial position, x, y, z) of a left camera with respect to the reference and its relative orientation (axial angle, x, y, z) by the image processing device is referred to as calibration.

[0004] The technique disclosed in PTL 1 is known as a method for calibrating an image processing device. PTL 1 discloses "selecting an image in which feature points are sufficiently distributed at a distance according to the intended use of the two cameras, and calculating external parameters of the two cameras using the image coordinates of feature points and corresponding points in the image that are distributed at a distance according to the intended use of the two cameras". Citation listPatent literature

[0005] PTL 1: JP 2009-276233 A Summary of the inventionTechnical problem

[0006] For high-precision calibration of an image processing device, the distribution of feature points captured from an acquired image is important. If the feature point distribution of an acquired image is unsuitable for calibration, the calibration accuracy decreases. In the method disclosed in PTL 1, where an acquired image is determined based on a spacing distribution of feature points, and where only one acquired image, for which a high-precision calibration result can be expected, is extracted to calibrate an imaging device, only one acquired image for which a high-precision calibration result can be expected is extracted. However, in the technique disclosed in PTL 1, an acquired image for which the calculated number of feature points is less than a threshold value is unsuitable for calibration.Captured images that are not suitable for such a configuration are discarded, and there is the problem that the possibility of performing calibration is reduced.

[0007] The present invention has been conceived in view of such a situation, and an object of the present invention is to improve the accuracy of calibration of an image processing apparatus. Solution to the problem

[0008] An image processing device according to the present invention comprises: an area subdivision unit that subdivides each of a plurality of captured images taken by an in-vehicle camera into a plurality of subdivided areas; a feature point detection unit that detects as a feature point a section that exceeds a threshold for each of the subdivided areas; a threshold setting unit that sets the threshold for each of the subdivided areas in order to distribute the feature points detected in each of the subdivided areas evenly; a feature point matching unit that performs a matching by linking the feature points detected in the plurality of captured images between the plurality of captured images;and a relative parameter calculation unit that calculates a relative parameter based on the feature points linked between the plurality of captured images; Advantageous effects of the invention

[0009] According to the present invention, the image processing device is capable of more accurately recognizing a three-dimensional position in the outside world by improving the accuracy of calibration of the image processing device. Short description of the drawings [ Fig. 1] Fig. 1 is a block diagram showing an internal configuration example of an image processing apparatus according to an embodiment of the present invention. [ Fig. 2] Fig. 2 is a flowchart showing an example of the overall processing by the image processing apparatus according to the embodiment of the present invention. [ Fig. 3] Fig. 3 is a flowchart showing an example of feature point balance detection processing performed by a feature point balance detection unit according to the embodiment of the present invention. [ Fig. 4] Fig. Figure 4 is a diagram showing an example of an image that is divided into an equal grid shape according to the embodiment of the present invention. [ Fig. 5] Fig. 5 is a diagram showing an example of an image divided into a lattice shape by increasing the number of divisions according to the embodiment of the present invention. [ Fig. 6] Fig. Figure 6 is a diagram showing an example of a captured image subdivided according to the condition of a vehicle according to the embodiment of the present invention. [ Fig. 7] Fig. 7 is a diagram showing a state in which a captured image, which was captured before a time series A according to the embodiment of the present invention, is subdivided, [ Fig. 8] Fig. Figure 8 is a diagram showing a state in which a captured image, acquired according to a time series B according to the embodiment of the present invention, is subdivided. [ Fig. 9] Fig. Figure 9 is a diagram showing an example of feature points that were captured from a first captured image according to the embodiment of the present invention. [ Fig. 10] Fig. Figure 10 is a diagram showing an example of feature points that were captured from a second captured image according to the embodiment of the present invention. [ Fig. 11] Fig. Figure 11 is a diagram showing an example of a parallelized image to which a conventional technique is applied and in which a distribution of feature points is distorted. [ Fig. 12] Fig. 12 is an example of a parallelized image in which the distribution of feature points is equalized by applying the technique according to the embodiment of the present invention. [ Fig. 13] Fig. Figure 13 is a block diagram showing a hardware configuration example of a computer according to the embodiment of the present invention. Description of embodiments

[0010] Modes for carrying out the present invention will be described below with reference to the accompanying drawings. In the present description and the drawings, components having substantially the same function or configuration are denoted by the same reference numerals, and redundant descriptions thereof are omitted. The present invention is applicable, for example, to a computing device for vehicle control with which an advanced driver assistance system (ADAS) or an in-vehicle electronic control unit (ECU) for autonomous driving (AD) can communicate. [Embodiment]

[0011] Fig. Figure 1 is a block diagram showing an internal configuration example of an image processing device 100 according to an embodiment.

[0012] The image processing device 100 is one of the devices mounted in an image processing ECU (not shown) installed in a vehicle. Additionally, the image processing device 100 features a high-precision calibration function.

[0013] An in-vehicle camera 101 is an example of a camera mounted on the vehicle to enable the image processing device 100 to detect the vehicle's surroundings and to record the surroundings at a predetermined time. The in-vehicle camera 101 outputs a captured image 10, obtained by recording the vehicle's environment to detect three-dimensional distances in the outside world, to the feature point compensation detection unit 102 of the image processing device 100. The in-vehicle camera 101 can be two stereo cameras or a monocular camera.

[0014] The captured image 10, which is output by the vehicle's in-vehicle camera 101 to the feature point adjustment detection unit 102, can be two captured images 10 taken simultaneously by two cameras (stereo camera), or it can be two captured images 10 taken at different times by one camera (monocular camera). For example, in a case where a monocular camera is used as the vehicle's in-vehicle camera 101, one captured image 10 obtained by the monocular camera capturing an object while the vehicle is stationary, and another captured image 10 obtained by capturing the same object after the vehicle has moved slightly, are output.

[0015] The image processing device 100 performs processing to calibrate the relative parameters of the in-vehicle camera 101 based on two captured images 10 and to detect three-dimensional distances in the outside world from the captured images 10. Here, the calibration is performed by the image processing device 100 each time the in-vehicle camera 101 is activated. For example, the calibration is performed immediately after the vehicle's power source is turned on and the in-vehicle camera 101 is activated, or at predetermined time intervals while the vehicle is traveling. The image processing device 100 includes a feature point compensation detection unit 102, a feature point adjustment unit 103, a relative parameter calculation unit 104, a parallax image generation unit 105, and a three-dimensional distance detection unit 106.In the following, the processing by each functional unit is described with reference to . Fig. 2 described.

[0016] Fig. 2 is a flowchart showing an example of the overall processing by the image processing apparatus 100. Fig. 2 shows an example of an image processing method of the image processing apparatus 100.

[0017] First, the feature point compensation detection unit 102 acquires the captured image 10 from the in-vehicle camera 101 (S1). The received captured image 10 is stored in a storage device such as a RAM 53 and a non-volatile memory 55, which are Fig. Figure 13 shows the image processing device 100, which is described below. The image is read from the storage device and rewritten during processing by each of the functional units of the image processing device 100. The following explanation omits a description of the processing for inputting and outputting the captured image 10 to and from the storage device.

[0018] In step S1, the feature point adjustment detection unit 102 analyzes the captured image 10, which is received from the vehicle's internal camera 101, and detects the contour lines of objects (roads, sidewalks, other vehicles, signs, etc.) that appear in the captured image 10. The feature point adjustment detection unit 102 then performs the feature point adjustment acquisition processing, which is described in Fig. 3 (S2) to detect, as feature points, points having a feature such as a corner portion of a contour line or a portion where contour lines intersect.

[0019] The feature point adjustment detection unit 102 employs a unique procedure in feature point adjustment acquisition processing, as shown in step S2, and performs operations so that feature points are detectable for which the accuracy of the relative parameters calculated by the relative parameter calculation unit 104 is high. The operations by the feature point adjustment detection unit 102 include processing to split the captured image 10, processing to detect feature points and count the number of feature points in the areas divided by the splitting processing, and processing to set thresholds (individual thresholds) to be used in feature point acquisition.According to the operation of the feature point equalization detection unit 102, feature points are captured to make the number of feature points in each of the subdivided areas obtained from the captured image 10, which is divided by the division processing, uniform.

[0020] Here, as in Fig. 1, the feature point compensation detection unit 102 includes an area dividing unit 1021, a feature point detection unit 1022, and a threshold setting unit 1023.

[0021] The area division unit (area division unit 1021) divides each of a multitude of captured images 10 taken by the vehicle's in-vehicle camera (in-vehicle camera 101) into a multitude of subdivided areas.

[0022] The feature point detecting unit (feature point detecting unit 1022) detects, as feature points, portions exceeding a threshold value (individual threshold value) for each divided area.

[0023] The threshold setting unit (the threshold setting unit 1023) sets thresholds (individual thresholds) for each of the divided areas to evenly distribute the feature points detected in each of the divided areas.

[0024] Hereinafter, the feature point balance detection processing performed by each functional unit included in the feature point balance detection unit 102 will be described with reference to Fig. 3 and the following drawings are described.

[0025] Fig. Figure 3 is a flowchart showing an example of feature point adjustment acquisition processing performed by the feature point adjustment detection unit 102. Feature point adjustment acquisition processing is part of the image processing procedure performed by each functional unit of the image processing device 100 located in Fig. 1 is shown.

[0026] First, the area dividing unit 1021 divides the captured image 10 received from the in-vehicle camera 101 into a predetermined size and a predetermined number of divisions (S11). Here, an example of the processing performed by the area dividing unit 1021 and an example of the divided areas will be described with reference to Fig. 4 to Fig. 8 described.

[0027] For example, the area division unit 1021 sequentially imports two captured images 10 acquired by the in-vehicle camera 101. The area division unit (area division unit 1021) performs division by increasing the number of divisions of the captured images 10 and reducing the size of the divided areas so that at least one feature point is acquired for each divided area. Examples of dividing the captured image(s) 10 include, for example, the Fig. 4 and Fig. 5 examples shown.

[0028] Fig. 4 is a diagram showing an example of an image (captured image 10) divided into a uniform grid shape.

[0029] The captured image 10, which was taken by the Fig. 4 is divided by the area dividing unit 1021 into four equal grid-like divided areas of 2 rows x 2 columns. In this case, the number of divisions of the captured image 10 is "4." The size of each divided area is the same. By dividing the captured image 10 into four, the size of the divided areas is then 1 / 4 of the size of the original captured image 10. Even in a case where the captured image 10 is to be divided into four as described above, the feature points in the captured image 10 are expected to be evenly arranged in each divided area depending on the object.

[0030] Fig. 5 is a diagram showing an example of an image divided into a grid shape by increasing the number of divisions.

[0031] The captured image 10, which was taken by the Fig. 5 is divided into sixteen equal grid-like divided regions of 4 rows x 4 columns by the region dividing unit 1021. In this case, the number of divisions of the captured image 10 is 16. Furthermore, the size of each divided region is the same. By increasing the number of divisions, the size of the Fig. The 5 shown subdivided areas are 1 / 4 the size of the area shown in Fig. The 4 subdivided areas shown. By increasing the number of subdivisions, the area subdivision unit 1021 is able to arrange the feature points contained in the recorded image 10 evenly in each subdivided area.

[0032] The reason why the feature points in the captured image 10 are evenly arranged by increasing the number of divisions of the captured image 10 by the area division unit 1021 will now be described. For example, the uneven distribution of feature points present in an area cannot be subsequently corrected. Therefore, in a case where a sign and a vehicle are shown in the upper half of the captured image 10 before division, and a road is shown in the lower half, it is expected that more feature points are unevenly detected in the upper half of the captured image 10 than in the lower half.

[0033] Therefore, since the region dividing unit 1021 increases the number of divisions of the captured image 10 and divides the captured image 10 into small divided regions, even if there is an uneven distribution in the feature points in each of the divided regions, the feature points are evenly arranged in each of the divided regions throughout the entire captured image 10. For example, if 100 feature points are randomly arranged throughout the entire captured image 10, the positions where the feature points are arranged become uneven. If the entire captured image 10 is evenly divided into 100 equal parts, after 100 feature points are randomly arranged throughout the entire captured image 10, it is expected that divided regions containing one or a plurality of feature points and divided regions containing no feature points will be generated.Meanwhile, if the entire captured image 10 is evenly divided into one hundred equal parts and then a feature point is arranged in each divided area, the feature points can be evenly arranged throughout the entire captured image 10. Therefore, after the captured image 10 is evenly divided, the feature points can be detected in each divided area, and thus the image processing device 100 is able to perform calibration using the feature points detected throughout the entire captured image 10.

[0034] Fig. 6 is a diagram showing an example of a captured image 10 divided according to the state of a vehicle.

[0035] Considering that the in-vehicle camera 101 is mounted on the vehicle, a relatively close object is likely to appear large on the bottom of the captured image 10, and a distant object is likely to appear small on the top of the captured image 10. Furthermore, in a case where no other vehicle is traveling in front of the vehicle, the bottom of the captured image 10 has a large area in which a road with a uniform density appears.

[0036] Therefore, the area dividing unit (area dividing unit 1021) changes the positions and sizes of the areas into which the captured image 10 is to be divided according to the position where the in-vehicle camera (in-vehicle camera 101) is mounted. For example, taking into account the vehicle state and the characteristics of the captured image 10, the area dividing unit 1021 divides the captured image 10 by using large divided areas while reducing the number of divisions in a portion where a nearby object appears large. Meanwhile, the captured image 10 is divided by using a large number of divisions for a portion where a small distant object appears and using a small size for the divided areas.

[0037] In the Fig. 6, the distance from the vehicle to the object must be detected by the three-dimensional distance detection unit 106 (see Fig. 1) in the section of the captured image 10 corresponding to the vehicle's direction of travel, which is the long-distance section (the upper half of the captured image 10), are detected in detail. To detect feature points from the captured image 10, the area subdivision unit 1021 of the feature point compensation detection unit 102 determines the subdivision pattern of the captured image 10 (the number of subdivisions of subdivided areas and the subdivision size of each subdivided area) taking into account driving scenes, such as where and how large feature points appear on average, and how much and in which direction the feature points move within the section over time.Therefore, the region dividing unit (region dividing unit 1021) divides the captured image 10 of a portion predicted to have many feature points by a size smaller than that of the portion predicted to have few feature points. For example, in . Fig. 6 the area near the center of the distant section (the section where the roadway appears) is divided using a small size, and the area near the two ends of the distant section (the section where the sidewalk appears) is divided using a large size.

[0038] As described above, the area subdivision unit 1021 is able to subdivide the captured image 10 for areas (a distant roadway, a distant sidewalk, etc.) where the number of feature points is expected to be small, by using a larger size than for areas (a nearby roadway, a nearby sidewalk, etc.) where the number of feature points is expected to be large.

[0039] The region division unit 1021 changes the number of divisions and the division size according to the characteristics of objects appearing in the captured image 10. Thus, the number of feature points detected from a divided region in which a nearby object appears and the number of feature points detected from a divided region in which a distant object appears can be made equally close to each other. As a result, it is possible to improve the calibration accuracy achieved by the image processing device 100 in the depth direction of the captured image 10.

[0040] Next, an example will be described in which the area dividing unit 1021 performs another division based on images before and after a time series of a running vehicle (before a time series A and after a time series B). Fig. 7 and Fig. 8 are diagrams showing examples of captured images 10 taken by the in-vehicle camera 101 at different times.

[0041] Fig. 7 is a diagram showing a state in which the captured image 10 taken before the time series A is divided.

[0042] Fig. Fig. 8 is a diagram showing a state in which the captured image 10 taken after the time series B is divided. The captured image 10 taken in Fig. 8, a time t is recorded after the captured image 10, which is shown in Fig. 7. That is, the captured image 10, which is in Fig. 8 is an image taken by the in-vehicle camera 101 in a state where the vehicle has moved forward for time t.

[0043] The region dividing unit (region dividing unit 1021) divides the plurality of captured images 10 into different sizes based on a prediction result of feature points corresponding to three-dimensional positions in the outside world acquired from a plurality of captured images 10 captured by the in-vehicle camera (in-vehicle camera 101) while the vehicle is moving. For example, the region dividing unit 1021 predicts the location of feature points corresponding to the three-dimensional positions acquired from the captured image 10 captured in the pre-time series A and the captured image 10 captured in the post-time series B, respectively. The region dividing unit 1021 divides the two captured images 10 using different division sizes so that the predicted positions of the feature points are included.

[0044] If that in Fig. 7 shown captured image with the in Fig. 8 is compared to the recorded image 10, the height H1 of the upper area, which is divided into eight areas in the Fig. 7 shown image, the height H2 in the Fig. 8, and the upper area of ​​the captured image 10 is wider. In addition, the area on the left side of the image shown in Fig. 7 is divided by a width W11, which includes a road that is removed, and a width W12 of an area that includes a sidewalk that is removed. Meanwhile, the vehicle moves when the Fig. 8 is taken, moves forward and approaches the object. Therefore, the area on the left side of the image shown in Fig. 8 shown image is wider than the one in Fig. 7, as indicated by the width W21, which includes the distant roadway. Meanwhile, the width W22, which includes the distant sidewalk in the area on the left side of the Fig. 8 shown image, narrower than the one in Fig. 7 shown width W12.

[0045] It should be noted that in order to perform a match between divided regions of the acquired image 10 acquired in the pre-time series A and the acquired image 10 acquired in the post-time series B, divided regions of the acquired image 10 acquired in the pre-time series A and divided regions of the acquired image 10 acquired in the post-time series B must exist. Therefore, the number of divisions of the two acquired images 10 is the same.

[0046] Since the area subdivision unit 1021 changes the subdivision size of the recorded image 10 before and after the time series in this way, the feature point detection unit 1022 is able to reliably detect feature points in the area where a notable object is present in the direction of travel of the vehicle (for example, a roadway). Additionally, when subdivided areas are matched, the feature point matching unit 103, described below, is able to minimize malfunctions when matching erroneously subdivided areas.

[0047] With further reference to Fig. 3 the description continues.

[0048] After the processing of step S11, the feature point detection unit 1022 selects a divided region resulting from the division by the region division unit 1021 (S12). The feature point detection unit 1022 performs feature point detection in the divided regions using the individual threshold value for the divided region selected in step S12 (S13).

[0049] In feature point detection processing, detection thresholds are used to increase or decrease the number of feature points detected in the divided regions by the feature point detection unit 1022. Typically, a value is set as the detection threshold for feature points for a captured image 10. Meanwhile, according to the present embodiment, the feature point detection unit 1022 is capable of individually setting a detection threshold for each divided region. Therefore, the detection threshold set for each divided region is referred to as an "individual threshold." The individual threshold is a value set for each divided region by the threshold setting unit 1023 described below.

[0050] Furthermore, the individual thresholds set by the threshold setting unit 1023 have a range of adjustable values. Here, luminance thresholds are used as the individual thresholds. By using luminance thresholds, it is possible to perform binarization processing for each pixel such that white is displayed in areas with a certain luminance or more, and black is displayed in areas with a luminance less than the certain luminance. For example, increasing the luminance threshold increases the black area in the captured image 10, and decreasing the luminance threshold increases the white area in the captured image 10.When the luminance threshold is increased to the maximum value, the captured image 10 is completely filled with black, and when the luminance threshold is decreased to the minimum value, the captured image 10 is completely filled with white. Therefore, a feature point cannot be detected in either case. Therefore, the range of luminance thresholds effective for feature point detection is determined for each operation scene. The threshold setting unit 1023 increases or decreases the individual thresholds within this range.

[0051] After step S13, the feature point detection unit 1022 counts the number of feature points in the divided areas detected in step S13 (S14). The number of feature points of the divided areas counted by the feature point detection unit 1022 is temporarily stored in the RAM 53 (see Fig. 13) or similar.

[0052] After processing (S12 to S14) by the feature point acquisition unit 1022, processing by the threshold setting unit 1023 is initiated. The threshold setting unit 1023 sets a threshold to balance the feature points acquired by the feature point acquisition unit 1022. Therefore, the threshold setting unit 1023 compares the number of feature points with a target value for each subdivided area for which the number of feature points was counted in step S14 and determines whether the number of feature points is greater than the target value (S15).In steps S15 to S17, the threshold setting unit (threshold setting unit 1023) performs an adjustment to increase the thresholds set for subdivided ranges where the number of feature points is greater than the target value, and performs an adjustment to decrease the thresholds set for subdivided ranges where the number of feature points is less than the target value. For this reason, the individual thresholds are adjusted so that the number of feature points captured by the feature point capture unit 1022 is close to the target value. Here, the number of feature points is determined with reference to... Fig. 9 and Fig. 10 described.

[0053] Fig. 9 is a diagram showing an example of feature points acquired from the first acquired image 10.

[0054] Fig. 10 is a diagram showing an example of feature points acquired from the second acquired image 10.

[0055] The solid circles and dashed circles shown in Fig. 9 and Fig. The 10 shown represent feature points. However, a solid circle represents a feature point for which a feature point indicating the same part was recorded and compared in two recorded images 10, and a dashed circle represents a feature point that was not recorded in the same part from the two recorded images 10.

[0056] Typically, feature point matching is performed by comparing the feature amounts of a feature point in one captured image 10 (standard image) with the feature amounts of all feature points in the other captured image 10 (reference image) to find the feature point with the closest feature amount. The feature amount is a numerical value indicating the shape of a feature point and is obtained by converting the arrangement of pixels (luminance values) around the feature point into a numerical value using a specific procedure. Different shapes are assumed for the feature points; however, if the feature points are the same object, the feature points are expected to be the same.However, since the matching of the feature points is performed on the divided areas at the same position in the two captured images 10, the load can be made lighter than in the processing to match the feature points detected over the entire captured image 10, and thus the time until the matching is completed can be shortened.

[0057] The description returns to Fig. 3 back.

[0058] It is assumed that the upper left side (facing backwards) of the in Fig. The recorded image 10 shown in 9 is a subdivided area, which is created by subdivision using the in Fig. 6 is obtained. In this case, since many feature points have been detected in the divided area on the upper left side (toward the rear) of the captured image 10, the threshold setting unit 1023 determines that the number of feature points in this divided area is greater than the target value (NO in S15). Therefore, the threshold setting unit 1023 performs processing to increase the individual thresholds (S16). As a result, the detection threshold in the divided area reaches a maximum value. That is, the detection threshold is the maximum detection threshold within the adjustable range. However, by dividing the current individual threshold to the maximum value at predetermined intervals, the individual thresholds can be set at the intervals at which the division is performed in step S16.Therefore, the individual threshold can be set to a value smaller than the maximum value.

[0059] Meanwhile, it is assumed that the lower side (towards the front) of the Fig. The recorded image 10 shown in 9 is a subdivided area, which is created by subdivision using the in Fig. 6 is obtained. In this case, since almost no feature points are detected in the divided area on the lower side (front side) of the captured image 10, the threshold setting unit 1023 determines that the number of feature points in this divided area is equal to or less than the target value (NO in S15). Therefore, the threshold setting unit 1023 performs processing to lower the feature point detection threshold (S17). As a result, the detection threshold in the divided area reaches the minimum value. That is, the detection threshold is the minimum detection threshold within the adjustable range. However, by dividing the current individual threshold to the minimum value at predetermined intervals, the individual thresholds can be set at the intervals at which the division is performed in step S17.Therefore, the individual threshold can be set to a value greater than the minimum value.

[0060] After step S16 or S17, the feature point balancing detection unit 102 determines whether the processing of step S12 and subsequent steps has been performed on all subdivided areas subdivided by the area subdivision unit 1021 (S18). If it is determined that there is a subdivided area that has not been processed (NO in S18), the feature point balancing detection unit 102 re-executes the processing of step S12 on the unprocessed subdivided area. Here, the feature point acquisition unit (feature point acquisition unit 1022) captures the feature points using the threshold set by the threshold setting unit (threshold setting unit 1023). Therefore, the number of feature points captured by the feature point acquisition unit 1022 is adjusted to match the target value.In a case where it is determined that the processing has been performed on all the divided regions (YES in S18), the feature point compensation detection unit 102 ends the feature point compensation detection processing shown in FIG. Fig. 3 is shown.

[0061] It should be noted that in the feature point compensation detection processing described in Fig. 3, after the individual threshold value is set for a particular divided area by means of the processing of steps S16 and S17, the feature point detection unit 1022 performs the feature point detection processing on another divided area using the individual threshold value set in the previous processing. Such processing is performed in order to shorten the processing time for all divided areas in a case where the feature point equalization detection processing performed in Fig. 3 is implemented.

[0062] However, if the image processing device 100 has a spare processing capacity, it is more effective for the feature point detection unit 1022 to perform feature point detection again in the same divided area. In this case, based on the individual threshold set by the threshold setting unit 1023 for a particular divided area, the feature point detection unit 1022 uses the set individual threshold to detect the feature points in the same divided area again. After that, the individual threshold setting processing by the threshold setting unit 1023 is not performed, and the processing of step S12 can be performed on another divided area.

[0063] With further reference to Fig. 1 and Fig. 2 the description continues.

[0064] After processing by the feature point compensation detection unit 102, which is Fig. As shown in Fig. 3, the feature point matching unit (feature point matching unit 103) performs matching by linking the feature points detected in the plurality of captured images 10 between the plurality of captured images 10 (S3). At this time, the feature point matching unit 103 performs matching to link feature points detected as the same portion in the two captured images 10 output from the feature point matching detection unit 102. Therefore, the feature point matching unit 103 is able to match the feature points detected in the two captured images 10 before division, as in a conventional case.

[0065] Note that, in a case where divided regions are generated by dividing the two captured images 10, the feature point matching unit (feature point matching unit 103) performs matching by linking feature points for each divided region having the same position in the plurality of captured images 10. Therefore, the feature point matching unit 103 is capable of performing matching by linking feature points uniformly detected throughout the entire captured image 10, compared with conventional processing in which matching is performed by linking feature points by taking the entire captured image 10 as a target.

[0066] The feature point matching processing is as described above with reference to Fig. 9 and Fig. 10. For example, the feature point matching unit 103 uses the divided region divided by the region dividing unit 1021 and changes a feature point in the divided region in one captured image 10 to compare the feature point in the divided region corresponding to the other captured image 10. Since the range of the divided regions to be matched is limited by such processing, the feature point matching unit 103 is able to avoid erroneous matching with a feature point acquired in another divided region, and thus the time required for the matching operation can be shortened.

[0067] The Relative Parameter Calculation Unit (Relative Parameter Calculation Unit 104) calculates relative parameters based on the feature points linked among the multitude of captured images 10 (S4). Therefore, the Relative Parameter Calculation Unit 104 calculates relative parameters using the camera parameters and the matched feature points. The relative parameters are parameters that specify the relationships between the relative positions and attitudes of one camera relative to another camera before and after the movement of two in-vehicle cameras 101 or a moving in-vehicle camera 101. Furthermore, when calculating the relative parameters, the Relative Parameter Calculation Unit 104 performs a calculation to minimize the sum of reprojection errors of all feature points matched in the two captured images 10.

[0068] In this case, where the vehicle's internal camera (vehicle-internal camera 101) is a stereo camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses as references the axial position and axial angle of one of the two cameras forming the stereo camera when one camera captures a recorded image 10, and uses as relative parameters the relative axial position and relative axial angle of the other camera when the other camera captures the other recorded image 10. For example, using the position of a right camera as a reference, a relative three-dimensional position ((x, y, z) axial position) of a left camera with respect to the reference and a relative orientation ((x, y, z) axial angle) are used as the relative parameters.

[0069] It should be noted that in a case where the vehicle's in-vehicle camera 101 is a monocular camera, images 10 are used that are taken before and after the movement of the moving in-vehicle camera 101. However, the attitude of the monocular camera is hardly constant before and after the movement. For example, under an ideal condition where the vehicle is traveling in a straight line, the attitude of the in-vehicle camera 101 is constant. However, it is assumed that the attitude of the in-vehicle camera 101 changes because the angle of the in-vehicle camera 101 actually changes left and right due to a minute steering wheel movement, or because the angle of the in-vehicle camera 101 changes up and down as a result of the application of the brakes or accelerator pedal.

[0070] Therefore, in a case where the vehicle-internal camera (vehicle-internal camera 101) is a monocular camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses as references the axial position and axial angle of the monocular camera at the time of capturing a captured image 10 among a multitude of captured images 10 taken by the monocular camera while the vehicle is moving, and uses as relative parameters the relative axial position and relative axial angle of the monocular camera at the time of capturing the other captured image 10.For example, using the position of a monocular camera captured as a reference at a specific time, a relative three-dimensional position ((x, y, z) axial position) with respect to a reference of the monocular camera captured after a time t, and a relative orientation ((x, y, z) axial angle) are used as the relative parameters. Consequently, even in a case where a monocular camera is used as the vehicle's in-vehicle camera 101, the relative parameters are calculated correctly.

[0071] Next, the parallax image generation unit (parallax image generation unit 105) generates a parallax image of the plurality of captured images 10 based on the relative parameters (S5). Therefore, the parallax image generation unit 105 generates a parallelized image obtained by parallelizing the two captured images 10 based on the relative parameters. In the parallelized image, the epipolar line is horizontal and has the same Y coordinate. The parallax image generation unit 105 then performs stereo matching on the parallelized image to generate a parallax image.

[0072] Next, the three-dimensional distance detection unit (three-dimensional distance detection unit 106) detects three-dimensional distances in the external world based on the parallax image (S6). For example, the three-dimensional distance detection unit 106 calculates three-dimensional distances in the external world based on the parallax value of the parallax image generated by the parallax image generation unit 105, the focal length of the camera contained in the camera parameters, the pixel size of the light receiving element contained in the vehicle's internal camera 101, and the like, and detects objects in the external world. The processing by the image processing device 100 then ends.

[0073] As described above, even in a case where the in-vehicle camera 101 includes a monocular camera, the three-dimensional distance detection unit 106 is capable of calculating the three-dimensional distances in the outside world and detecting objects in the outside world. In this case, the in-vehicle camera 101 is required to capture an image of the outside world at a time before and after the vehicle moves in time series, and how the in-vehicle camera 101 moved between the two captured images 10 must be accurately known. Note that when the in-vehicle camera 101 includes two stereo cameras, the camera positions of the two captured images 10 captured by the stereo cameras are fixed and can be accurately known.Since the camera position before and after the movement of a monocular camera that moves along with the vehicle tends to be subject to large errors, the detection accuracy of three-dimensional distances in the outside world is reduced compared to the case of using a stereo camera. However, there is an advantageous effect in that three-dimensional distances can be detected using a single camera.

[0074] Although the functional units included in the image processing device 100 have been described above, the advantageous effects will be described in a case where the technique according to the present embodiment is applied and the feature point equalization detection unit 102 equalizes the feature points. Therefore, an example of a parallelized image generated by the parallax image generation unit 105 after the relative parameter calculation unit 104 calculates the relative parameters will be described with reference to Fig. 11 and Fig. 12 described.

[0075] Fig. Figure 11 is a diagram showing an example of a parallelized image to which a conventional technique is applied, resulting in a distorted distribution of feature points. Fig. 11, the horizontal axis represents the X-axis of the captured image 10 and the vertical axis represents the Y-axis of the captured image 10.

[0076] The black circles and dashed lines shown in the drawing indicate the feature points extracted from the captured image 10 (reference image) captured by one camera and the inclination of the collimated image. Furthermore, the white circles and solid lines shown in the drawing indicate the feature points and inclination of the collimated image extracted from the captured image 10 (reference image) captured by the other camera. Ideally, two collimated images are expected to have the same inclination after collimation. Here, the two captured images 10 are divided into regions 21 to 24 in 2 rows x 2 columns.

[0077] In the example of the parallelized image, which in Fig. As shown in Figure 11, weights are calculated for the feature points when calculating the relative parameters to minimize reprojection errors. Weights are values ​​integrated for errors of feature points acquired from a two-dimensional captured image 10 (the errors between the white circles and the black circles shown in the drawing). In areas with many feature points, the errors of many feature points are integrated even if the error of one feature point is small, and thus the weight of the feature point with respect to the evaluation value increases. Meanwhile, in an area where the number of feature points is small, errors of a small number of feature points are integrated, and thus the weight of the feature point with respect to the evaluation value becomes small even if the error of one feature point is large.

[0078] Based on such a relationship between the feature points and the weights, the weight in the relative parameter calculation increases for area 22, in which feature points are densely packed, which in Fig. 11, and the weight for the region 23 where feature points are sparse decreases. Therefore, in the relative parameter calculation, relative parameters optimized for regions with a large weight and where feature points are dense are calculated, and a large deviation occurs in the region 23 where the feature points are sparse. As a result, the captured image 10 (reference image) indicated by the solid line is not significantly tilted and parallelized with respect to the captured image 10 (standard image) indicated by the dashed line.

[0079] Fig. 12 is an example of a parallelized image in which the distribution of feature points is equalized by applying the technique according to the present embodiment.

[0080] Also in Fig. 12, the black circles, white circles and dashed and solid lines shown in the drawing are similar to those shown in Fig. 11 are shown.

[0081] In the example of the parallelized image, which in Fig. As shown in Figure 12, the feature points are evenly distributed in the areas 21 to 24 (entire image) of the two captured images 10. Therefore, the weights in the calculation of the relative parameters are also equalized in the two captured images 10. As a result, a calculation result with less distortion can be obtained. In addition, the captured image 10 (reference image) indicated by the solid line is collimated with the tilt suppressed with respect to the captured image 10 (standard image) indicated by the dashed line.

[0082] Next, a hardware configuration of the computer 50, which forms the image processing device 100, is described.

[0083] Fig. Figure 13 is a block diagram showing a hardware configuration example of a computer 50. The computer 50 is an example of hardware used as a computer capable of functioning as the image processing device 100 according to the present embodiment. As a result of the computer 50 (computer) executing a program, the image processing device 100 according to the present embodiment implements an image processing method carried out by the respective functional blocks shown in Figure 13. Fig. 1 and Fig. Three are shown, which interact with each other.

[0084] The computer 50 includes a central processing unit (CPU) 51, a read-only memory (ROM) 52, and a random-access memory (RAM) 53, each connected to a bus 54. The computer 50 also includes non-volatile memory 55 and a network interface 56.

[0085] The CPU 51 reads the program code from a ROM 52 to implement each function according to the present embodiment, loads the program code into a RAM 53, and executes the program code. Variables, parameters, and the like generated by the CPU 51 during processing are temporarily written to the RAM 53, and these variables, parameters, and the like are read by the CPU 51 as needed. However, a microprocessing unit (MPU) can be used instead of the CPU 51.

[0086] For example, a hard disk drive (HDD), a solid-state drive (SSD), a flexible disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory, or the like is used as the non-volatile memory 55. In addition to an operating system (OS) and various parameters, a program for executing so that the computer 50 functions is recorded in the non-volatile memory 55. The ROM 52 and the non-volatile memory 55 record programs, data, and the like necessary for the operation of the CPU 51 and are used as an example of a computer-readable non-volatile storage medium that stores a program executed by the computer 50.

[0087] For example, a network interface card (NIC) or the like is used as the network interface 56, and various data can be transmitted and received between devices via a local area network (LAN), a dedicated line, or the like connected to a port of the NIC.

[0088] In the image processing device 100 according to the embodiment described above, when the feature points are captured from a recorded image 10, the operation is performed such that the captured feature points have a distribution within the recorded image 10 that is suitable for calibration, and thus the calibration (calculation of the relative parameters) can be carried out more accurately. Therefore, the three-dimensional distance detection unit 106, which is included in the image processing device 100, is able to detect three-dimensional positions, represented by three-dimensional distances in the external world, more accurately than conventionally.

[0089] Here, the two captured images 10 used for matching are divided by the region division unit 1021 using the same size and number of divisions. The captured image 10 is divided into divided regions so that at least one feature point is detected. Therefore, the feature points are evenly arranged in each divided region, and thus the feature points are evenly arranged throughout the entire captured image 10. In addition, feature point matching is easily performed for each divided region at the same position in each captured image 10.

[0090] Further, the size of the divided region is changed according to the number of feature points predicted for each section of the captured image 10. The captured image 10 is divided so that the size of a section predicted to have a large number of feature points is small, and the size of a section predicted to have a small number of feature points is large. As a result, at least one feature point is detected even in a section predicted to have a small number of feature points.

[0091] Additionally, the individual threshold for detecting feature points can be changed for each divided region. Therefore, the number of detected feature points can be reduced by setting high individual thresholds for sections predicted to have a large number of feature points. Conversely, the number of detected feature points can be increased by setting low individual thresholds for sections predicted to have a small number of feature points. As a result, the number of feature points detected in each divided region is equalized.

[0092] In addition, since the arrangement of the feature points is uneven, even a captured image 10 that would conventionally have been discarded can be used for calibration by the image processing device 100. Therefore, the image processing device 100 has more opportunities to perform calibration and is capable of performing calibration more frequently and accurately than conventionally. [Modification]

[0093] It should be noted that if a fault occurs in one of the cameras included in the stereo camera, calibration can be performed using only the other camera (for example, the right camera) by using the captured image 10, taken before and after the vehicle's movement. In this case, the camera on one side can be calibrated by processing similar to the case where a monocular camera is used as the vehicle's in-vehicle camera 101.

[0094] In addition, by acquiring the two captured images 10 captured by the stereo camera at different times, the same divided regions of the two captured images 10 captured at different times can be aligned. For example, the feature point alignment detection unit 102 sets the individual thresholds after acquiring the feature points from the divided regions obtained by dividing two captured images 10 captured at an initial time point. The feature point alignment detection unit 102 then acquires the feature points by using the individual thresholds set at the initial time point for the divided regions obtained by dividing the two captured images 10 captured at the next time point.The three-dimensional distance detection unit 106 is able to detect three-dimensional distances in the outside world more accurately by using, during the subsequent detection of feature points, the individual threshold values ​​that were first set for the recorded images 10, which were taken twice in a short time, as described.

[0095] It should be noted that the present invention is not limited to or by the embodiments described above, and it is obvious that various other application examples and modifications can be taken without departing from the core of the present invention as described in the claims.

[0096] For example, the above-described embodiments describe system configurations in detail and specifically to describe the present invention in an easily understandable manner, and are not necessarily limited to those that include all of the described configurations. Additionally, it is also possible to add, delete, and replace other configurations with some of the configurations of the present embodiment.

[0097] Furthermore, control lines and information lines are indicated as necessary for the description and do not necessarily list all control lines and information lines on the product. In practice, it can be assumed that almost all configurations are interconnected. List of reference symbols 10 captured image 100 image processing device 101 in-vehicle camera 102 Feature point compensation detection unit 103 Feature point matching unit 104 Relative parameter calculation unit 105 Parallax image generation unit 106 three-dimensional distance detection unit 1021 area division unit 1022 Feature point detection unit 1023 Threshold setting unit QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP 2009-276233 A

[0005]

Claims

[1] Image processing device comprising: an area dividing unit that divides each of a plurality of captured images captured by an in-vehicle camera into a plurality of divided areas; a feature point detecting unit that detects, as a feature point, a portion exceeding a threshold value for each of the divided areas; a threshold setting unit that sets the threshold for each of the divided areas to evenly distribute the feature points detected in each of the divided areas; a feature point matching unit that performs matching by linking the feature points detected in the plurality of captured images between the plurality of captured images; and a relative parameter calculation unit that calculates a relative parameter based on the feature points linked between the plurality of captured images. [2] Image processing apparatus according to claim 1, wherein the threshold value setting unit performs an adjustment to increase the threshold values ​​set for the divided regions in which the number of feature points is greater than a target value and performs an adjustment to decrease the threshold values ​​set for the divided regions in which the number of feature points is smaller than the target value, and the feature point detecting unit detects the feature points using the threshold values ​​set by the threshold setting unit. [3] The image processing apparatus according to claim 2, wherein the region dividing unit performs division by increasing the number of divisions of the captured images and reducing the size of the divided regions so that at least one feature point is detected for each divided region. [4] The image processing apparatus according to claim 3, wherein the area dividing unit changes the positions and sizes of the areas into which the captured image is to be divided according to the position in which the in-vehicle camera is mounted. [5] The image processing apparatus according to claim 4, wherein the region dividing unit divides the captured image of a portion predicted to have many feature points by using a size smaller than that of a portion predicted to have few feature points. [6] The image processing apparatus according to claim 4, wherein the area dividing unit divides the plurality of captured images using different sizes based on a prediction result of feature points corresponding to three-dimensional positions in the outside world detected from a plurality of captured images captured by the in-vehicle camera while the vehicle is moving. [7] The image processing apparatus according to claim 5, wherein the feature point matching unit performs matching by linking the feature points in each divided area having the same position in the plurality of captured images. [8] The image processing apparatus according to claim 6, wherein, in a case where the in-vehicle camera is a stereo camera, the relative parameter calculation unit uses as references the axial position and the axial angle of one of two cameras constituting the stereo camera when one camera captures the captured image, and uses as the relative parameters the relative axial position and the relative axial angle of the other camera when the other camera captures the other captured image. [9] The image processing apparatus according to claim 6, wherein, in a case where the in-vehicle camera is a monocular camera, the relative parameter calculation unit uses, as references, the axial position and the axial angle of the monocular camera at the time of capturing the one captured image among the plurality of captured images captured by the monocular camera while the vehicle is moving, and uses, as the relative parameters, the relative axial position and the relative axial angle of the monocular camera at the time of capturing the other captured image. [10] An image processing apparatus according to claim 8, further comprising: a parallax image generating unit that generates a parallax image of the plurality of captured images based on the relative parameters; and a three-dimensional distance detection unit that detects three-dimensional distances in the outside world based on the parallax image. [11] Image processing method, comprising processing for: Dividing each of a plurality of captured images captured by an in-vehicle camera into a plurality of divided regions; detecting, as a feature point, a portion exceeding a threshold for each of the divided regions; Setting the threshold for each of the divided regions to evenly distribute the feature points detected in each of the divided regions; Linking the feature points detected in the plurality of captured images between the plurality of captured images; and Calculating a relative parameter based on the feature points linked between the plurality of captured images.

Images