Image processing device, image processing method, and computer program

Abstract

To provide an image processing apparatus that can obtain an accurate image deviation amount while coping with image deviation generated in a direction orthogonal to parallax.SOLUTION: An image processing apparatus acquires a first image and a second image having parallax in a first direction, acquires correlation information between an image of a fiducial region in the first image and an image of a reference region corresponding to the fiducial region in the second image, corrects the correlation information on the basis of a distance in a second direction orthogonal to the first direction, between the fiducial region and the reference region, and calculates an image deviation amount between the image of the fiducial region and the image of the reference region on the basis of the corrected correlation information.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, a computer program, and the like. [Background technology]

[0002] In digital cameras installed as information acquisition sensors for autonomous driving and industrial robots, a technology is known in which ranging pixels with ranging capabilities are arranged in some or all of the pixels of the image sensor, and the distance to the subject is detected using a phase difference method.

[0003] In this type of system, multiple photoelectric conversion units are arranged in the ranging pixel, and light beams that pass through different regions on the pupil of the photographing lens are guided to different photoelectric conversion units. Optical images (hereinafter referred to as "Image A" and "Image B," respectively) generated by light beams that pass through different pupil regions can be obtained using signals output by the photoelectric conversion units included in each ranging pixel, and multiple images can be obtained based on Image A and Image B. The pupil region corresponding to Image A and the pupil region corresponding to Image B are decentered in different directions along an axis known as the pupil division direction.

[0004] Furthermore, a relative positional shift occurs between the multiple acquired images (hereinafter referred to as "image A" and "image B") along the pupil division direction according to the amount of defocus. This positional shift is called image shift, and the distance to the subject can be calculated by converting the amount of image shift, which is the amount of image shift, into the amount of defocus using a predetermined conversion coefficient. Unlike conventional contrast methods, this method does not require moving the lens to measure the distance, making it possible to measure distances quickly and with high accuracy.

[0005] To calculate the amount of image misalignment, a region-based corresponding point search technique called template matching is generally used. In template matching, one of image A or image B is used as the base image, and the other image is used as the reference image. A reference region is set on the base image with a point of interest at its center, and a reference region is set on the reference image with a reference point corresponding to the point of interest at its center. The reference point is then moved sequentially to search for the point where the correlation between the reference region in image A and the reference region in image B is highest. The amount of image misalignment is calculated from the amount of positional misalignment between this point and the point of interest.

[0006] In template matching, when searching for a point with high correlation while sequentially moving the reference point, the search is performed along the direction in which image shift occurs. For example, if the sensor is positioned so that image shift occurs in the horizontal direction, the search is performed in the horizontal direction. However, the light beam passing through the above-mentioned photographing lens may cause image shift in an unintended direction (vertical direction in the above example) due to errors in the optical system. Furthermore, this error is not fixed at the time of manufacturing and may fluctuate from time to time due to, for example, heat.

[0007] If there is vertical image misalignment, the calculation of the image misalignment amount and therefore distance measurement may not be performed correctly depending on the image in the reference area used for template matching. For example, if the image contains an object consisting of diagonal lines, the vertical image misalignment will be interpreted as horizontal image misalignment and added to the measured distance. In Patent Document 1, a search is also performed by changing the reference point in the vertical direction, and if the amount of image shift is significantly different between when the reference point is changed and when it is not changed, the reliability is determined to be low and the distance measurement result is discarded. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-194069 Summary of the Invention [Problem to be solved by the invention]

[0009] However, although the method of Patent Document 1 can be applied to obtain the distance corresponding to an object, it cannot be applied to obtain the distance measurement result for each pixel. One method for correcting vertical image shift caused by errors in the optical system is to perform template matching searches in both the horizontal and vertical directions to find locations where correlation is high. However, even in this case, while the amount of image shift can be calculated correctly if the reference area contains an object that is likely to have high correlation, if it contains an object made up of diagonal lines, the location where correlation is high cannot be uniquely determined, making it impossible to calculate the amount of image shift correctly.

[0010] The present invention has been made to solve the above-mentioned problems, and aims to provide an image processing device that can obtain a highly accurate amount of image shift while dealing with image shift that occurs in a direction perpendicular to the parallax. [Means for solving the problem]

[0011] In order to solve the above problems, an image processing device according to one aspect of the present invention comprises: an image acquisition means for acquiring a first image and a second image having a parallax in a first direction; a correlation acquisition means for acquiring correlation information between an image of a standard area in the first image and an image of a reference area corresponding to the standard area in the second image; a correction means for correcting the correlation information based on a distance between the standard region and the reference region in a second direction perpendicular to the first direction; and an image shift amount calculation means for calculating an image shift amount between the image of the standard area and the image of the reference area based on the correlation information corrected by the correction means. death, the image shift amount calculation means determines the image shift amount for each of the plurality of reference positions, the reference region in the first image including a reference position, and a plurality of reference positions are set in the first image; the correlation acquisition means is configured such that the reference region in the second image includes a reference position, a plurality of the reference positions are set in the second image, and the correlation information is acquired for each reference position with respect to the standard position; the correlation acquisition means sets a plurality of reference positions for each of the first direction and the second direction; The correction means reduces the correlation information as the distance between the standard area and the reference area in the second direction becomes farther from a predetermined deviation amount reference value. do 、 It is characterized by: [Effects of the Invention]

[0012] According to the present invention, an image processing device can be realized that can obtain a highly accurate amount of image shift while dealing with image shift that occurs in a direction orthogonal to the parallax. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a functional block diagram showing the overall configuration of a distance measuring device 100 as an image processing device according to a first embodiment. [Figure 2] 10A and 10B are diagrams illustrating an example of an operation in which only horizontal deviation is taken into consideration in the correlation calculation unit. [Figure 3] 10A and 10B are diagrams illustrating an example of an operation in which horizontal and vertical deviations are taken into consideration in a correlation calculation unit. [Figure 4] 10A and 10B are diagrams for explaining examples of problems that arise when horizontal and vertical misalignments are taken into account in a correlation calculation unit. [Figure 5] 10A and 10B are diagrams for explaining another example of a problem that occurs when horizontal and vertical misalignments are taken into consideration in the correlation calculation unit. [Figure 6] 10A and 10B are diagrams illustrating an example of dissimilarity correction in the correlation calculation unit 108 according to the first embodiment. [Figure 7] 10A and 10B are diagrams illustrating another example of dissimilarity correction in the correlation calculation unit. [Figure 8] FIG. 8 is a diagram showing an example of a deviation amount reference value map 800 stored in the storage unit 112. [Figure 9] 3 is a flowchart showing a control flow in the first embodiment. [Figure 10] 10A and 10B are diagrams illustrating an example of dissimilarity correction in which a vertical shift is detected in a correlation calculation unit according to the second embodiment. [Figure 11] FIG. 10 is a diagram showing a control flow in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described using examples with reference to the drawings. However, the present invention is not limited to the following examples. In each drawing, the same members or elements are given the same reference numerals, and duplicate descriptions will be omitted or simplified. In the embodiments, an example will be described in which the image processing device is applied to a camera mounted on a moving body such as an automobile, but the image processing device in the embodiments also includes electronic devices with an imaging function, such as digital still cameras, digital movie cameras, smartphones with cameras, tablet computers with cameras, network cameras, drone cameras, and cameras mounted on robots. [Example]

[0015] FIG. 1 is a functional block diagram showing the overall configuration of a distance measuring device 100 as an image processing device according to a first embodiment. Some of the functional blocks shown in FIG. 1 are implemented by causing a computer (not shown) included in the image processing device to execute a computer program stored in a memory (not shown) as a storage medium. However, some or all of these functional blocks may be implemented by hardware. Examples of hardware that may be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in FIG. 1 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0016] In this embodiment, the distance measuring device 100 is mounted on a moving object such as an automobile, and the imaging unit is configured as a stereo camera. In order to triangulate the distance to the subject at each pixel of the captured image, the device has at least two imaging units, 101a and 101b, which are arranged in a predetermined direction (first direction) and spaced apart by a predetermined distance (baseline length). Hereinafter, the image captured by the imaging unit 101a will be referred to as the A image or first image, as a reference image, and the image captured by the imaging unit 101b will be referred to as the B image or second image. As described above, the first image and the second image have parallax in the first direction.

[0017] Reference numeral 102 denotes a distance image generation unit, which generates a distance map indicating the distance of each pixel based on the image received from the imaging unit 101. The distance image generation unit 102 is configured, for example, from an LSI, a CPU as a computer, memory for storing computer programs executed by the CPU, various I / Os, etc., but the configuration method is not particularly important. Note that the distance image generation unit 102 does not have to be mounted on a mobile object such as an automobile, and can be, for example, a PC terminal or tablet located away from the mobile object such as an automobile.

[0018] Reference numeral 103a denotes a lens that forms a subject image on the image sensor 104a. The image sensor 104a is an image sensor configured with a CMOS (complementary metal-oxide semiconductor) or a CCD (charge-coupled device). The subject image formed on the image sensor 104a via the lens 103a is converted into an electrical signal by the image sensor 104a. Reference numeral 105a denotes an image transmission unit that transmits the electrical signal acquired by the image sensor 104a to the distance image generation unit 102 as image data of image A.

[0019] 1 shows lens 103a, image sensor 104a, and image transmission unit 105a provided in imaging unit 101a, but imaging unit 101b also has corresponding lens 103b, image sensor 104b, and image transmission unit 105b. Image data of image B is generated by imaging unit 101b and transmitted from image transmission unit 105b to distance image generation unit 102.

[0020] Note that the imaging units 101a and 101b may share some of their configurations. For example, ranging pixels with ranging functions may be arranged in some or all of the pixels of the image sensor, and two photoelectric conversion units may be arranged in each ranging pixel, so that light beams passing through different regions on the lens pupil are guided to the respective photoelectric conversion units. In this case, the lenses and image transmission units may be shared.

[0021] The image receiving unit 106 receives the image data of the image A and the image B transmitted from the imaging unit 101. That is, the image receiving unit 106 functions as an image acquiring unit that acquires a first image and a second image having parallax in a first direction. The image correction unit 107 performs preprocessing necessary for generating a distance map on the image data sent from the image receiving unit 106. Examples of the preprocessing include shading correction for correcting brightness unevenness due to peripheral light loss caused by the lenses 103a and 103b, and filtering for emphasizing correlation.

[0022] The correlation calculation unit 108 calculates the correlation between image A and image B at each pixel position by template matching or the like within a predetermined search range, as will be described later. The image shift amount calculation unit 109 calculates the image shift amount at each pixel position by selecting the correlation distance with the highest correlation based on the correlation calculated by the correlation calculation unit 108.

[0023] If necessary, interpolation less than the search resolution may be performed using parabolic fitting, etc. Furthermore, if necessary, a known image shift amount in the parallax direction that occurs uniquely in the image processing device may be offset against the calculated image shift amount. Furthermore, the image shift amount caused by the refractive index of the optical system may be offset based on the contrast of color information contained in the image.

[0024] The distance calculation unit 110 calculates the distance to the subject at each pixel position using the image shift amount calculated by the image shift amount calculation unit 109 and the interval (baseline length) between the two lenses 103a and 103b. By performing the above distance calculations over the entire image, a distance map for the entire image can be generated.

[0025] The distance image generating unit 102 also includes an overall control unit 111 that controls each unit, and a memory unit 112 that stores the operation setting values of each unit and buffers intermediate data as needed. The overall control unit 111 also has a built-in CPU as a computer, and functions as a control means that controls the operation of each unit of the entire distance measuring device based on a computer program stored in the memory unit 112 that serves as a storage medium.

[0026] FIG. 2 shows an example of operation in the correlation calculation unit that considers only horizontal misalignment, and does not consider vertical misalignment, which will be described later. 200a is the reference position for which distance calculation is to be performed in image A. 200b is a corresponding reference position on image B that shows the same coordinates as 200a. 201a is an example of an object (subject) captured on image A, and 201b is an object that indicates the same subject as 201a captured on image B. 201a and 201b exist at different coordinate positions depending on the placement interval (baseline length) of the two lenses 103a and 103b and the distance to the object.

[0027] Here, the two lenses 103a and 103b are assumed to be arranged with a gap between them in the horizontal direction, and images are captured with a horizontal misalignment between 201a and 201b. Template matching, for example, is performed to calculate this amount of misalignment. Reference numeral 202 denotes a reference area for performing template matching. Reference numeral 203 denotes a horizontal search area for searching for an image in image B that has a high correlation with reference area 202. Since the misalignment occurs in the horizontal direction, the search area is a long area in the horizontal direction.

[0028] Reference numeral 204 denotes a plurality of reference positions on the B image side where template matching is performed. Reference numeral 205 denotes a plurality of reference regions made up of pixels surrounding each reference position 204. In this example, five search positions are searched for, which are shifted by -2, -1, ±0, +1, and +2. The correlation is calculated at each of these positions using, for example, a known method such as SSD (Sum of Squared Difference), which uses the sum of squares of the difference between pixel values of the standard area 202 and the reference area 205 as the dissimilarity.

[0029] In the method using SSD, the calculated value is the dissimilarity, and the part where the dissimilarity is smallest is the location where the correlation is highest. In this embodiment, the correlation is calculated specifically using the dissimilarity, and an example will be described in which the correlation is high when the dissimilarity is small. In the example of Figure 2, the image shift amount calculation unit 109 at the subsequent stage determines that the image of the reference area at the position where the dissimilarity is 10 and the search position is +1 is most similar (has high correlation) to the image of the standard area 202.

[0030] Then, the distance calculation unit 110 calculates the distance based on the information that there is a shift of +1. That is, the reference area in the first image includes a reference position, and multiple reference positions are set in the first image, and the image shift amount is determined for each of the multiple reference positions. Also, the reference area in the second image includes a reference position, and multiple reference positions are set in the second image, and correlation information is obtained for each of the multiple reference positions with respect to the reference position.

[0031] FIG. 3 is a diagram showing an example of the operation of the correlation calculation unit that takes into account horizontal and vertical misalignment. In FIG. 2, template matching was performed under the assumption that image A and image B have parallax only in the horizontal direction (first direction). However, in the example of FIG. 3, it is assumed that image A and image B are also misaligned in the vertical direction (second direction orthogonal to the first direction) due to errors in the optical system, and objects 201a and 201b are captured with misalignment in both the horizontal and vertical directions. 300 is a two-dimensional search area for searching for an image area in image B that is similar to the image of reference area 202. That is, in FIG. 3, multiple reference positions are set in both the first and second directions.

[0032] The meanings of other symbols are the same as in Figure 2. In Figure 3, the search is performed by shifting the search position horizontally from -2 to +2 and vertically from -1 to +1. In Figure 3, "search position: -2, -1" means a search position shifted by -2 horizontally and +1 vertically, and the dissimilarity is lowest at "search position: +1, -1." In other words, the position where the horizontal search position is +1 and the vertical search position is -1 has the lowest dissimilarity (highest correlation). The image shift amount calculation unit 109 determines that the image of the reference area at this position is most similar to the image of the standard area 202, and the distance calculation unit 110 calculates the distance using the value of the horizontal search position + 1 as the image shift amount.

[0033] In this way, the correlation calculation unit obtains correlation information (degree of difference) between the image of the standard area in the first image and the image of the reference area corresponding to the standard area in the second image. The operation of the correlation calculation unit 108 described above is effective when an appropriate object can be obtained by template matching. However, problems arise when an object consisting of a diagonal line is included, for example. An example of this is described with reference to FIGS. 4 and 5.

[0034] Fig. 4 is a diagram illustrating an example of a problem that occurs when horizontal and vertical misalignment is taken into account in the correlation calculation unit. The example shown in Fig. 4 is characterized in that the subject (object) includes a diagonal line that is unsuitable for template matching. Note that the example in Fig. 4 uses a two-dimensional search area 300 to deal with vertical misalignment. 400a is a diagonal line object captured on image A, and 400b is a diagonal line object that indicates the same thing as 400a captured on image B.

[0035] The amount of deviation that actually occurs between these diagonal line objects 400a and 400b is (horizontal, vertical) = (+1, ±0). However, in this example, if each dissimilarity is calculated using multiple reference areas 205, multiple search position candidates with the lowest dissimilarity will be found. In other words, although the desired amount of deviation is (horizontal, vertical) = (+1, ±0), because the object is made up of diagonal lines, the dissimilarity values will be similar even when the search positions are (±0, -1) and (+2, +1), and the correct amount of deviation cannot be determined.

[0036] Fig. 5 is a diagram for explaining another example of a problem that occurs when horizontal and vertical misalignment is taken into account in the correlation calculation unit. The example shown in Fig. 5 is characterized in that the object is made up of diagonal lines that are unsuitable for template matching, and has not only horizontal misalignment due to parallax but also vertical misalignment due to errors in the optical system. In the example of Figure 4, it was explained that multiple image shift amount candidates are generated by performing template matching search in the vertical direction as well, but in the example of Figure 5, it will be explained that another problem occurs even when template matching in the vertical direction is not performed.

[0037] The amount of misalignment that actually occurs in Figure 5 is both horizontal and vertical, and this will be explained using sample objects 500a and 500b and copy object 500c. 500a is a sample object captured in image A, and 500b is a sample object that points to the same thing as 500a captured in image B. 500c is a copy object that shows sample object 500a in image A at the same coordinates in image B. According to copy object 500c and sample object 500b, the amount of misalignment that actually occurs in image B is actual horizontal misalignment 501 (for example, +1) and actual vertical misalignment 502 (for example, -1).

[0038] Based on actual horizontal displacement 501 and actual vertical displacement 502, the position on image B that should correspond to reference area 202 is ideal corresponding area 503. However, when a search is performed in horizontal search area 203, the location determined to have the lowest degree of dissimilarity is the horizontal search position +2. The resulting calculated horizontal displacement 504 (+2 in this example) differs from actual horizontal displacement 501 (+1 in this example), making it impossible to calculate the correct distance.

[0039] As described above, if template matching is performed in the vertical direction to correct for vertical misalignment caused by the optical system, multiple candidates with low dissimilarity will be generated when a diagonal line object is present. On the other hand, if no search for vertical misalignment is performed at all, a calculated horizontal misalignment that differs from the actual horizontal misalignment may be calculated when a diagonal line object is present.

[0040] Therefore, in this embodiment, the error is suppressed by correcting the dissimilarity in template matching in the vertical direction. 6A and 6B are diagrams illustrating an example of dissimilarity correction in the correlation calculation unit 108 according to the first embodiment.

[0041] Fig. 6(A) is a diagram showing an example of a correction coefficient 600 by which the dissimilarity is multiplied. When performing a two-dimensional search such as that shown in Fig. 4, the dissimilarity calculated according to the vertical search position (second direction perpendicular to the first direction) is multiplied by a coefficient. 601 is a deviation reference value, and here, the deviation reference value 601 is set to the vertical search position ±0. The dissimilarity calculated using the deviation reference value 601 is scaled at 1, the dissimilarity calculated at the vertical search position ±5 is scaled at 1.5, and the intermediate values are linear.

[0042] That is, the farther the search position is from the deviation amount reference value 601, the greater the weight of the correction coefficient applied to the degree of difference, thereby increasing the degree of difference and thereby reducing the correlation. In this embodiment, the farther the distance in the second direction between the standard area and the reference area is from a predetermined standard position, the smaller the correlation information (the greater the degree of difference). Note that here, the deviation amount reference value 601 is set to the vertical search position ±0, and the magnification (correction coefficient) is set to 1, but this deviation amount reference value 601 can be set separately, as will be described later.

[0043] Figure 6(B) shows an example of dissimilarity before and after correction. 602 is the dissimilarity before correction. Here, an example is shown in which dissimilarity is calculated for a diagonal line object, as in Figure 4. In reality, dissimilarity for two dimensions is calculated using horizontal and vertical searches, but only the minimum value for the horizontal search is plotted.

[0044] In this case, as described above, for a diagonal line object, similar pre-correction dissimilarity values 602 are calculated at multiple vertical search positions before correction, and in the example of FIG. 6A, the position of -5 is determined to have the lowest dissimilarity. However, in the example of FIG. 6A, for a diagonal line object that produces similar dissimilarity values, it is desirable that the vertical search position of ±0 has the lowest dissimilarity. Therefore, the displacement amount at the vertical search position of ±0 is used as the deviation amount reference value 601. 603 is the post-correction dissimilarity, and by using the post-correction dissimilarity, the position of the vertical search position of ±0 is determined to have the lowest dissimilarity, and the dissimilarity value of the deviation amount reference value 601 is prioritized even for a diagonal line object.

[0045] 7(A) and (B) are diagrams showing another example of dissimilarity correction in the correlation calculation unit. In FIG. 6, the deviation reference value 601 was set to ±0, but in FIG. 7, the deviation reference value 601 is set to +2, and as shown in FIG. 7(A), the correction coefficient 600 has a magnification of 1 at the vertical search position of +2. FIG. 7(B), like FIG. 6(B), shows an example of calculating the dissimilarity for a diagonal line object such as that shown in FIG. 4. In FIG. 7(B), the deviation reference value 601 is set to +2, so that the post-correction dissimilarity 603 is lowest at the vertical search position of +2 compared to the pre-correction dissimilarity 602.

[0046] Here, we will provide additional information regarding the setting of the misalignment reference value 601 determined in Figures 6 and 7. The amount of vertical misalignment caused by the optical system is either static, determined when the device is assembled, or dynamic, fluctuating thereafter due to thermal fluctuations and the like during operation. The former, static misalignment amount, is preferably stored (held) in advance as a set value in the memory unit 112 of the distance image generation unit 102.

[0047] 8 is a diagram showing an example of a misalignment amount reference value map 800 stored in the storage unit 112, in which a static misalignment amount is held individually for each pixel as a misalignment amount reference value 601. The misalignment amount reference value map 800 stored in the storage unit 112 may be configured to be writable via an external I / F (not shown). Alternatively, the misalignment amount reference value map 800 may be stored in an external server or the like and acquired from the server.

[0048] Furthermore, it is desirable to store the static misalignment amount as pixel-by-pixel data as a characteristic of each pixel. That is, it is desirable to store information about a predetermined reference position for each pixel in the first image. This allows the static misalignment amount of the corresponding pixel position to be calculated when calculating the dissimilarity for each pixel.

[0049] Fig. 9 is a flowchart showing a control flow in the first embodiment, in which the correlation calculation unit 108, the image shift amount calculation unit 109, the distance calculation unit 110, etc. perform the processing shown in Fig. 9 in response to instructions from the overall control unit 111. Note that the CPU serving as a computer within the overall control unit 111 executes a computer program stored in the storage unit 112, thereby performing the operation of each step in the flowchart in Fig. 9.

[0050] For example, the correlation calculation unit 108 performs the processes of steps S901 to S902 and steps S904 to S906, the image shift amount calculation unit 109 performs step S908, and the distance calculation unit 110 performs step S909. Steps S900, S903, S907, and S910 are instruction steps by the overall control unit 111.

[0051] In step S900, the central control unit 111 starts loop processing for the x and y coordinates of the reference position 200a, and performs processing for each coordinate in sequence. Also, in step S901, the correlation calculation unit 108 reads data on the reference region 202 around the reference position 200a. In step S902, the correlation calculation unit 108 reads the deviation amount reference value 601 prepared for each pixel and stored in the storage unit 112.

[0052] In step S903, the central control unit 111 performs loop processing on the multiple reference positions 204, sequentially processing each area in the two-dimensional search area 300 shown in Fig. 4 etc. In step S904, the correlation calculation unit 108 reads data on the reference area 205 around the selected reference position 204.

[0053] In step S905 (correlation acquisition step), the correlation calculation unit 108 calculates the degree of dissimilarity using the data of the standard area 202 and the data of the reference area 205. That is, in step S905, correlation information is acquired between the image of the standard area in the first image and the image of the reference area corresponding to the standard area in the second image.

[0054] In step S906 (correction step), the correlation calculation unit 108 performs a correction operation on the dissimilarity calculated in step S905. That is, based on the deviation amount reference value 601 read in step S902, the correlation calculation unit 108 multiplies the dissimilarity by a correction coefficient such as those illustrated in Fig. 6 and Fig. 7. In this way, in step S906, the correlation information is corrected based on the distance between the standard area and the reference area in the second direction perpendicular to the first direction.

[0055] In step S907, the central control unit 111 repeats the loop until the processing shown in step S903 for the plurality of reference positions 204 is completed. After the loop in step S907 ends, in step S908 (image shift calculation step), the image shift amount calculation unit 109 compares the degrees of difference calculated for the plurality of reference regions 205 and calculates the image shift amount. That is, Based on the correlation information corrected in the correction step of step S906, the image shift amount between the image of the standard area and the image of the reference area is calculated. In step S909, the distance calculation unit 110 calculates the distance to the subject based on the amount of image shift calculated in step S908.

[0056] In step S910, the central control unit 111 determines whether the loop for the x and y coordinates of the reference position 200a shown in step S900 has ended. That is, the central control unit 111 repeats the processing of steps S900 to S910 until it determines in step S910 that the processing for all x and y coordinates has ended, and ends the flow of FIG.

[0057] As described above, in this embodiment, the distance to the subject is calculated by performing template matching with reference to multiple reference areas 205 in the horizontal and vertical directions, and the vertical deviation of the optical system is corrected by using a previously prepared deviation amount reference value 601. Therefore, accurate distance measurement is possible even if the subject includes diagonal lines. That is, even when there are multiple reference areas with similar dissimilarity in the vertical direction, such as in a subject that includes diagonal lines, the influence of unreliable dissimilarity can be reduced because the dissimilarity of the reference area is weighted more as the reference area becomes farther away from the deviation amount reference value 601. Therefore, vertical deviation caused by the characteristics of the optical system can be appropriately corrected. [Example]

[0058] Next, a second embodiment will be described with reference to FIGS. 1 is the same in the second embodiment. In the second embodiment, the deviation reference value 601 described in the first embodiment is dynamically updated.

[0059] Fig. 10 is a diagram showing an example of dissimilarity correction for detecting vertical misalignment in the correlation calculation unit according to the second embodiment. Fig. 6 shows an example of calculating dissimilarity for an object including a diagonal line, but Fig. 10 shows a case where an object including no diagonal line is included in the reference region 202. Fig. 10 also shows an example where a dissimilarity that shows a clear difference occurs at, for example, vertical search position +2.

[0060] As in the first embodiment, the static misalignment amount is stored (held) as a preset value in the storage unit 112 of the distance image generation unit 102. For example, the misalignment amount reference value 601 for a given pixel is ±0. However, because the type, position, and distance of an object included in an image change over time, there are times when it is easy to clearly obtain the dissimilarity and times when it is not. FIG. 10 shows an example of when a clear dissimilarity can be obtained. 1000 indicates the lowest dissimilarity before correction, and 1001 indicates the dissimilarity after correction.

[0061] Although the initial deviation amount reference value 601 is set to ±0, the deviation amount at the vertical search position +2 is detected as the smallest after the correction. This is because a more dynamic vertical deviation has occurred from the position of ±0, which is the initial deviation amount reference value 601. The example in FIG. 10 shows that even for a normal object, a dynamic vertical deviation has been detected at the position of +2. In this case, in the second embodiment, the image deviation amount 1001 at the vertical search position +2 is the smallest based on the corrected deviation amount, so this is adopted as the image deviation amount, and the deviation amount reference value 601 stored in the storage unit 112 for this pixel is updated from the initial ±0 to +2.

[0062] Fig. 11 is a diagram showing a part of a control flow instructed by the overall control unit 111 in this embodiment. Note that the operation of each step in the flowchart in Fig. 11 is performed by a CPU serving as a computer in the overall control unit 111 executing a computer program stored in the storage unit 112. Note that steps in Fig. 11 with the same reference numerals as in Fig. 9 perform the same processing as in Fig. 9, and therefore description thereof will be omitted.

[0063] 9 is that the flowchart in Fig. 11 has added step S1100 in response to dynamic updating of the misalignment amount reference value 601. In step S1100, the integrated control unit 111 uses the image misalignment amount calculated in step S908 to update the misalignment amount reference value 601 stored in the storage unit 112. That is, the information relating to the reference position is updated based on the reference position corresponding to the image misalignment amount calculated by the image misalignment amount calculation unit.

[0064] (Other Examples) For example, in the update step in step S1100, if a predetermined condition is met, the update may not be performed. In the second embodiment, it may be determined by, for example, image recognition whether the subject (object) included in the reference area 202 includes diagonal lines at a predetermined rate or more, and if the subject (object) does not include diagonal lines at a predetermined rate or more, updating may be performed in step S1100. If the subject (object) included in the reference area 202 includes diagonal lines at a predetermined rate or more, the reliability of the dissimilarity is low, so updating of the deviation amount reference value 601 stored in the storage unit 112 may not be performed.

[0065] Furthermore, if the dissimilarity is equal to or greater than a predetermined threshold (the correlation information is equal to or less than a predetermined value), the deviation reference value 601 stored in the storage unit 112 may not be updated. This is because if the dissimilarity is so large as to exceed the predetermined threshold, the reliability of the calculated dissimilarity may be low.

[0066] Furthermore, if the amount of change from the deviation reference value 601 before update stored in the storage unit 112 is outside a predetermined range, the deviation reference value 601 stored in the storage unit 112 may not be updated. This is because if the amount of change in the deviation reference value is outside the predetermined range, the reliability of the dissimilarity at that time is likely to be low.

[0067] Furthermore, for example, the deviation reference value 601 is a result of aging and does not change significantly in a short period of time. Therefore, when updating the deviation reference value 601, if a predetermined period has not elapsed since the previous update, the update may not be performed.

[0068] Furthermore, if the contrast of the image of the reference area is low, if there are few structures, or if the reliability of template matching is low, updating may not be performed for that pixel position. That is, if the contrast of the subject is low, if it is determined by image recognition that there are few structures, or if the reliability of template matching (reliability of correlation information) is lower than a predetermined value, updating may not be performed for that pixel position.

[0069] The reliability of template matching may be determined by analyzing the variance of the image, etc. Alternatively, the attributes of the image may be acquired by image recognition, and if, for example, a white line that could be a diagonal line object is detected, the template may not be updated. In this way, if at least one of the above predetermined conditions is satisfied, it is desirable not to update the deviation reference value 601 in step S1100.

[0070] As described above, the predetermined condition includes at least one of the following: the image of the reference region includes a predetermined percentage or more of diagonal lines, the correlation information is equal to or less than a predetermined value, and the amount of change from the information on the reference position before the update is outside a predetermined range. Alternatively, the predetermined period of time since the previous update has not elapsed, the contrast of the image of the reference region is equal to or less than a predetermined value, and the reliability of the correlation information is equal to or less than a predetermined value.

[0071] In the above embodiment, an example has been described in which the further away from a predetermined reference position the vertical direction (the second direction perpendicular to the first direction with parallax) the more weight is applied to the dissimilarity (the smaller the correlation information). However, it may also be possible to determine, for example, by image recognition or the like, whether or not the reference area contains a predetermined proportion of diagonal lines, and if the proportion is less than the predetermined proportion, not to apply weighting to the vertical direction (not to correct the correlation information) as described above. Furthermore, whether or not to correct the correlation information (degree of difference, etc.) in step S906 may be changed depending on the image attributes of the reference area and the coordinates of the reference position.

[0072] It should be noted that a parallax direction correction unit (not shown) may be further provided to correct the image shift amount calculated by the image shift amount calculation unit in the first direction. The parallax direction correction unit may correct the image shift amount based on, for example, chromatic aberration information of the lenses 103 a and 103 b included in the imaging unit. The parallax direction correction unit may also correct the image shift amount based on the distribution state of each color included in the image of the reference area or the contrast of each color.

[0073] Alternatively, an object whose image shift amount and shape are known may be imaged by the imaging unit, the image shift amount may be calculated by the image shift amount calculation unit, and difference information between the calculated image shift amount and the known image shift amount may be stored in advance in the difference storage unit.The image shift amount may then be corrected based on the difference information stored in advance in the difference storage unit.

[0074] Alternatively, an object whose distance and shape are known may be captured by the imaging unit, the distance may be calculated for each pixel by the distance calculation unit, and difference information between the calculated distance and the known distance may be stored in advance in the difference storage unit. Then, the distance may be corrected based on the difference information stored in advance in the difference storage unit. Note that the difference storage unit may be provided within or outside the image processing device.

[0075] By doing so, dynamic vertical misalignment caused by the optical system can be efficiently corrected, enabling accurate distance measurement. That is, when a normal object is present in the reference area 202 as a landmark, it is used to calculate the image misalignment amount and distance, and to detect dynamic vertical misalignment. Furthermore, when at least one of the above conditions is met, the detected vertical misalignment is used to update the misalignment amount reference value 601, which allows that vertical misalignment to be used from the next time onwards, improving calculation efficiency.

[0076] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the image processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0077] 100: Distance measuring device 101: Imaging unit 102: Distance image generation unit 103a: Lens 104a: image sensor 105a: Image transmission unit 106: Image receiving unit 107: Image correction unit 108: Correlation calculation unit 109: Image shift amount calculation unit 110: Distance calculation unit 111: General control unit 112: Storage part

Claims

1. an image acquisition means for acquiring a first image and a second image having a parallax in a first direction; a correlation acquisition means for acquiring correlation information between an image of a standard area in the first image and an image of a reference area corresponding to the standard area in the second image; a correction means for correcting the correlation information based on a distance between the standard region and the reference region in a second direction perpendicular to the first direction; an image shift amount calculation means for calculating an image shift amount between the image of the standard area and the image of the reference area based on the correlation information corrected by the correction means, the image shift amount calculation means determines the image shift amount for each of the plurality of reference positions, the reference region in the first image including a reference position, and a plurality of reference positions are set in the first image; the correlation acquisition means is configured such that the reference region in the second image includes a reference position, a plurality of the reference positions are set in the second image, and the correlation information is acquired for each reference position with respect to the standard position; the correlation acquisition means sets a plurality of reference positions for each of the first direction and the second direction; the correction means reduces the correlation information as the distance between the standard area and the reference area in the second direction becomes farther from a predetermined deviation amount standard value; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, further comprising storage means for storing information about the predetermined deviation reference value for each pixel in the first image.

3. 3. The image processing apparatus according to claim 2, further comprising an updating unit that updates the information relating to the deviation amount reference value based on the reference position corresponding to the image deviation amount calculated by the image deviation amount calculation unit.

4. 4. The image processing apparatus according to claim 3, wherein said updating means does not perform said updating when a predetermined condition is satisfied.

5. 5. The image processing device according to claim 4, wherein the predetermined conditions include at least one of the following: when the image of the reference area contains diagonal lines of a predetermined proportion or more; when the correlation information is equal to or less than a predetermined value; when the amount of change from the information regarding the reference position before the update is outside a predetermined range; when the elapsed time since the last update has not passed a predetermined period; when the contrast of the image of the reference area is equal to or less than a predetermined value; and when the reliability of the correlation information is equal to or less than a predetermined value.

6. An image processing device as described in claim 1, characterized in that whether or not the correction is performed by the correction means is changed depending on the coordinates of the reference position.

7. 2. The image processing apparatus according to claim 1, further comprising a parallax direction correcting means for correcting the image shift amount calculated by the image shift amount calculating means in the first direction.

8. a difference storage means for storing difference information between the image shift amount calculated by the image shift amount calculation means for a subject whose image shift amount is known and the known image shift amount; 2. The image processing apparatus according to claim 1, wherein the correction means corrects the amount of image shift based on the difference information stored in the difference storage means.

9. 2. The image processing apparatus according to claim 1, further comprising a distance calculation unit that calculates a distance to a subject based on the amount of image shift.

10. a difference storage means for storing difference information between the distance calculated by the distance calculation means for a subject whose distance or shape is known and the known distance, 10. The image processing apparatus according to claim 9, wherein the distance calculation means corrects the distance based on the difference information stored in the difference storage means.

11. an image acquisition step of acquiring a first image and a second image having a parallax in a first direction; a correlation acquisition step of acquiring correlation information between an image of a standard area in the first image and an image of a reference area corresponding to the standard area in the second image; a correction step of correcting the correlation information based on a distance between the standard region and the reference region in a second direction perpendicular to the first direction; an image shift amount calculation step of calculating an image shift amount between the image of the standard area and the image of the reference area based on the correlation information corrected by the correction step, the image shift amount calculation step includes: determining the image shift amount for each of the plurality of reference positions; and determining a reference area in the first image including a reference position, the plurality of reference positions being set in the first image. the correlation acquisition step includes: the reference region in the second image including a reference position; a plurality of the reference positions are set in the second image; and the correlation information is acquired for each reference position with respect to the standard position; the correlation acquisition step sets a plurality of reference positions for each of the first direction and the second direction; the correction step reduces the correlation information as the distance between the standard area and the reference area in the second direction becomes farther from a predetermined deviation amount standard value. An image processing method comprising:

12. 2. A computer program for controlling each unit of the image processing apparatus according to claim 1 by a computer.

Images