Stereo image processing device and stereo image processing method

Abstract

Provided is an image processing device capable of suppressing distance measurement errors caused by factors such as changes over time.　One aspect of the present invention provides a stereo image processing device that detects parallax from images captured by a first camera and a second camera, the stereo image processing device comprising: a celestial body position detecting unit that detects a prescribed celestial body from a first image captured by the first camera and detects the prescribed celestial body from a second image captured by the second camera; a comparing unit that compares the position of the prescribed celestial body in the first image with the position of the prescribed celestial body in the second image; a calibrating unit that generates calibration information for the first image, for performing calibration such that the position of the prescribed celestial body in the first image approaches the position of the prescribed celestial body in the second image, on the basis of a positional deviation of the prescribed celestial body in the first image with respect to the position of the prescribed celestial body in the second image; a conversion information recording unit that records the calibration information; and an image converting unit that uses the calibration information to calibrate the images captured by the first camera.

Description

Stereo image processing device and stereo image processing method

[0001] The present invention relates to a technique for processing stereo images.

[0002] Stereo camera technology is a 3D object recognition technology. Stereo camera technology uses the difference in how images are captured by two cameras placed at different positions to detect parallax based on trigonometry, and then uses this parallax to detect the depth and position of an object. Stereo camera technology makes it possible to accurately detect the position of an object. For this reason, stereo camera technology is used to detect and recognize 3D objects in vehicles, robots, etc.

[0003] The images from the two cameras in a stereo camera are converted into a predetermined projection (for example, the central projection method), and distance is measured by detecting the difference between the images. High-precision distance detection requires that the two images be aligned with a predetermined projection with high precision. Therefore, if the image detected by the camera changes due to factors such as aging, distance measurement errors occur. For example, with a fisheye lens compatible with a wide-angle field of view, image distortion is significant in the wide-angle field of view, so even a slight deviation results in a large error in the wide-angle field of view. Furthermore, if the orientation of one of the cameras shifts, the detected image also shifts, making it impossible to detect accurate parallax. These factors result in distance measurement errors.

[0004] In response to distortions in detected images, the following Patent Document 1 proposes a method for measuring a wide range of cloud base altitude distributions in a short time from images of clouds taken by a stereo camera, and for determining cloud movement speed and wind speed from time-delay images. The document describes a technique in which the altitude, longitude, and latitude of the installation positions of the two stereo cameras are known, and the image capture time, zenith position, azimuth angle, and zenith angle can be deciphered for the camera images. (1) The azimuth angle and zenith angle of a common object are read from each image of the cloud captured by each camera at the same time. (2) The altitude of the cloud is assumed. (3) The longitude and latitude of the cloud are determined from the installation positions, azimuth angle, and zenith angle of each camera for the captured images. (4) A new altitude is assumed and processing (3) is repeated until the longitude and latitude of the cloud are confirmed to match within an acceptable range for each camera image. The camera used here outputs the azimuth angle and zenith angle calibrated based on the coordinates of the star captured.

[0005] Furthermore, Patent Document 2 aims to address image misalignment by "providing a distance measurement device that can increase the degree of freedom in the mounting positions of multiple cameras and does not require calibration for distance measurement," and describes a technology that "includes an image acquisition means that acquires image data of two images of an object to be measured for distance using two cameras spaced a predetermined distance apart, an image synthesis means that superimposes the two pieces of image data so that images at infinity in the image data overlap each other, and a distance calculation means that determines the amount of misalignment between the images of the object to be measured when the images at infinity are superimposed on each other, and measures the distance to the object based on the amount of misalignment."

[0006] JP 2019-60754 A JP 2009-63319 A

[0007] The altimetry device described in Patent Document 1 derives distance from the positions of many stars with different zenith angles, plots the relationship with the zenith angle of each star, and fits a function to this to determine the relationship between distance and zenith angle, thereby calibrating the distortion of the detected image. Because this calibration method determines the relationship between distance and zenith angle, it is assumed that the distortion is rotationally symmetric with respect to the zenith.

[0008] On the other hand, actual changes over time include not only rotationally symmetric distortion but also asymmetric distortion. For example, if the lens's focal length is f and the lens's projection is an orthogonal projection (f sin θ), the projection of the detected image is converted to a perspective projection (f tan θ) to obtain parallax (affine transformation). Under normal conditions, conversion from f sin θ to f tan θ can be performed according to the corresponding pixel on the sensor surface. In this case, the conversion value corresponding to the pixel is processed as a two-dimensional table. However, for example, if the sensor shifts in a direction parallel to the sensor surface due to changes over time, the image of f sin θ on the sensor and the optimal position of the two-dimensional table will not match, resulting in image distortion that is asymmetric with respect to the zenith. For this reason, calibration methods such as those described in Patent Document 1 cannot correct distortion.

[0009] Furthermore, in the calibration method of Patent Document 1, distortions in the detected images of each camera are calibrated. Therefore, in reality, calibration residuals occur in each camera. In this case, the calibration residuals of the two detected images are superimposed, which poses a problem of increasing distance measurement errors.

[0010] Patent Document 2 claims that accurate distance measurement can be achieved by using a predetermined calibrated bright spot image of two image data to determine the amount of deviation D and calculating the difference in the amount of deviation D from the parallax. However, this calibration method poses a problem with calibration in the direction perpendicular to the direction of parallax calculation. A vertical deviation can lead to the detection of an incorrect parallax for the target object. For example, if there is a diagonal line in the detected image, a vertical deviation can lead to the line being detected as three-dimensional. Furthermore, when detecting clouds in Example 1, the cloud shapes differ vertically, resulting in an incorrect parallax being detected and a distance measurement error.

[0011] Furthermore, if a rotational deviation occurs in the camera or sensor relative to the lens optical axis, the detected image will rotate. This rotational deviation in the detected image results in parallax and vertical deviation, resulting in ranging errors. This cannot be corrected with Patent Document 2. Furthermore, distortion of the detected image cannot be corrected either. For this reason, for example, when the lens projection is orthogonal projection (f sin θ) or equidistant projection (f θ), if the sensor is deviated in a direction parallel to the sensor surface, the amount of deviation D will differ between a narrow-angle field of view (θ to 0 degrees) and a wide-angle field of view (θ: large). For this reason, a calibration method such as that in Patent Document 2 cannot perform complete calibration.

[0012] The present invention has been made in view of the above-mentioned problems, and aims to suppress distance measurement errors in a stereo camera caused by factors such as changes over time.

[0013] One aspect of the present invention is a stereo image processing device that detects parallax from images captured by a first camera and a second camera, comprising: a celestial body position detection unit that detects a predetermined celestial body from a first image captured by the first camera and detects the predetermined celestial body from a second image captured by the second camera; a comparison unit that compares the position of the predetermined celestial body in the first image with the position of the predetermined celestial body in the second image; a calibration unit that generates calibration information for the first image, based on a positional deviation of the predetermined celestial body in the first image relative to its position in the second image, to calibrate the position of the predetermined celestial body in the first image so that its position in the second image approaches that of the predetermined celestial body in the first image; a conversion information recording unit that records the calibration information; and an image conversion unit that calibrates the image captured by the first camera using the calibration information.

[0014] Another aspect of the present invention is a stereo image processing device that detects parallax from images captured by a first camera and a second camera, the stereo image processing device comprising: a celestial body position detection unit that detects a predetermined celestial body at multiple times from a first image captured by the first camera and detects the predetermined celestial body at multiple times from a second image captured by the second camera; a comparison unit that compares the position of the predetermined celestial body at multiple times in the first image with the position of the predetermined celestial body at multiple times in the second image; a calibration unit that generates calibration information based on the position of the predetermined celestial body at multiple times in the second image and the positional deviation of the predetermined celestial body at multiple times in the first image so that the position of the predetermined celestial body at multiple times in the second image and the position of the predetermined celestial body at multiple times in the first image become closer; a conversion information recording unit that records the calibration information; and an image conversion unit that uses the calibration information to calibrate so that the image captured by the first camera and the image captured by the second camera become closer.

[0015] Another aspect of the present invention is a stereo image processing method for detecting parallax from images captured by a first camera and a second camera, the stereo image processing method including: a celestial body position detection process for detecting a predetermined celestial body from a first image captured by the first camera, and detecting the predetermined celestial body from a second image captured by the second camera; a comparison process for comparing the position of the predetermined celestial body in the first image with the position of the predetermined celestial body in the second image; a calibration process for generating calibration information for the first image, which calibrates the position of the predetermined celestial body in the first image so that the position of the predetermined celestial body in the second image approaches the position of the predetermined celestial body in the first image based on a positional deviation of the predetermined celestial body in the first image from the position of the predetermined celestial body in the second image; a conversion information recording process for recording the calibration information; and an image conversion process for calibrating the image captured by the first camera using the calibration information.

[0016] According to the image processing device of the present invention, distance measurement errors of a stereo camera caused by factors such as changes over time can be suppressed.

[0017] FIG. 1 is a block diagram of a stereo image processing device according to Example 1. FIG. 2 is a block diagram of a stereo parallax image generating unit 200 according to Example 1. FIG. 3 is an image diagram illustrating an example of an image P3 and an image P4 according to Example 1. FIG. 4 is an image diagram illustrating processing by an image conversion unit 120 according to Example 1. FIG. 5 is a block diagram of a stereo image processing device according to Example 2. FIG. 6 is an image diagram illustrating an example of an image P3 and an image P4 according to Example 2. FIG. 7 is an image diagram illustrating processing by an image conversion unit 120 according to Example 2. FIG. 8 is a block diagram of a stereo image processing device according to Example 3. FIG. 9 is an image diagram illustrating a detected image according to Example 4. FIG. 10 is an image diagram illustrating another detected image according to Example 4. FIG. 11 is an image diagram illustrating a detected image according to Example 5. FIG. 12 is an image diagram illustrating a detected image according to Example 5. FIG. 13 is a perspective view illustrating the attitude of a stereo camera image processing device 10 according to Example 5. FIG. 14 is an image diagram illustrating another detected image according to Example 5. FIG. 15 is an image diagram illustrating another detected image according to Example 5.

[0018] The embodiments will be described in detail with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiments shown below. Those skilled in the art will easily understand that the specific configuration can be changed without departing from the concept or spirit of the present invention.

[0019] In the configurations of the embodiments described below, the same parts or parts having similar functions are denoted by the same reference numerals in different drawings, and redundant explanations may be omitted.

[0020] When there are multiple elements having the same or similar functions, they may be described using the same reference numeral with different subscripts. However, when there is no need to distinguish between multiple elements, the subscripts may be omitted.

[0021] The terms "first," "second," "third," and the like used in this specification are used to identify components and do not necessarily limit the number, order, or content of the components. Furthermore, numbers used to identify components are used in different contexts, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

[0022] In order to facilitate understanding of the invention, the position, size, shape, range, etc. of each component shown in the drawings etc. may not represent the actual position, size, shape, range, etc. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings etc.

[0023] All publications, patents, and patent applications cited herein are incorporated by reference in their entirety.

[0024] As used herein, elements referred to in the singular are intended to include the plural unless the context clearly indicates otherwise.

[0025] In this embodiment, as an example, a stereo image processing device that detects parallax from image 1 and image 2 captures images of multiple celestial objects, detects calibration information for calibrating image 2 from the difference in the position of the same celestial object among the multiple celestial objects in image 1 and image 2, and converts image 1 and image 2 based on the calibration information so that they approximately match. In this way, an image processing device that performs high-precision calibration using star images and a high-precision calibration method using star images are provided.

[0026] A configuration of a stereo camera image processing device 10 according to a first embodiment will be described with reference to FIG. 1A . The stereo camera image processing device 10 is used, for example, to detect the shape of a structure. It is also used to detect the distance to clouds or structures. It is also mounted on a vehicle such as an automobile and used to detect the distance from the vehicle to three-dimensional objects (other automobiles, buildings, pedestrians, etc.) around the vehicle. Hereinafter, a case where the stereo camera image processing device 10 is used to detect the shape of a structure will be described as an example, but the present invention is not limited to this.

[0027] 1A is a block diagram showing an example of the configuration of a stereo camera image processing device 10 according to Example 1. This stereo camera image processing device 10 is configured to be able to detect a three-dimensional object based on images obtained by a first camera (right camera) 50 and a second camera (left camera) 60. The first camera 50 and the second camera 60 form a stereo camera. This stereo camera image processing device 10 is configured to include, as an example, an image processing unit 100, a stereo parallax image generation unit 200, and a stereoscopic three-dimensional object detection unit 500.

[0028] Although not shown, the first camera 50 and the second camera 60 each include a lens and an image sensor. The first camera 50 and the second camera 60 each capture (take) an image of an object via the lens using the image sensor. The stereo camera image processing device 10 captures an image P1 (first image) from the first camera 50 and an image P2 (second image) from the second camera 60.

[0029] The image processing unit 100 is configured to include, for example, affine processing units 20a and 20b, an image conversion unit 120, luminance correction units 21a and 21b, pixel interpolation units 22a and 22b, and luminance information generation units 23a and 23b. The image processing unit 100 applies predetermined image processing to images P1 and P2 obtained by the first camera 50 and the second camera 60, and supplies the images to the stereo parallax image generation unit 200 and the stereoscopic three-dimensional object detection unit 500.

[0030] The affine processing unit 20a applies affine processing to the image P1 from the first camera 50. As a result of this affine processing, the affine processing unit 20a acquires an image P3 (third image). Similarly, the affine processing unit 20b applies affine processing to the image P2 from the second camera 60 to acquire an image P4 (fourth image).

[0031] The affine processing units 20a and 20b may also perform distortion transformation processing other than affine processing. In this embodiment, f sin θ, a projection method of a fisheye lens, is projectively transformed into a coordinate system of (f tan θx, f tan θy). Here, f is the focal length of the fisheye lens, θ is the angle of view incident on the fisheye lens, and θx and θy are the horizontal and vertical components of the angle of view incident on the fisheye lens. Furthermore, in this embodiment, assuming that the stereo camera image processing device 10 is installed inside a vehicle, the affine processing units 20a and 20b correct vertical pixel displacement (pixel shift) due to the influence of the windshield. In other words, the affine processing units 20a and 20b also function as pixel shift correction processing units that correct pixel shift.

[0032] The image conversion unit 120 converts the image P3 based on the two pieces of conversion information for the up / down direction and the left / right direction stored in the conversion information recording unit 34, and outputs the image P5. The brightness correction unit 21a corrects the brightness of each pixel of the image P3. For example, the brightness correction of each pixel of the image P5 is performed based on the gain of the first camera 50, the difference in gain between each pixel in the image P5, etc. Similarly, the brightness correction unit 21b corrects the brightness of each pixel of the image P4.

[0033] The pixel interpolation unit 22a performs demosaicing on the image P5, for example, converting the raw image into a color image. Similarly, the pixel interpolation unit 22b performs demosaicing on the image P4.

[0034] The luminance information generator 23a generates luminance information for the image P5. For example, the luminance information generator 23a converts information representing a color image into luminance information for generating a parallax image. Similarly, the luminance information generator 23b generates luminance information for the image P4.

[0035] The stereo parallax image generating unit 200 generates a stereo parallax image using the obtained images P5 and P4.

[0036] The configuration of the stereo parallax image generation unit 200 will be described with reference to Fig. 1B. The stereo parallax image generation unit 200 generates an image that expresses the magnitude of parallax, for example, by using different colors, based on the image P1 and the image P2. The greater the parallax, the closer the distance to the object. The stereo parallax image generation unit 200 may employ a known configuration.

[0037] As an example, the stereo parallax image generation unit 200 includes an exposure adjustment unit 210 and a sensitivity adjustment unit 220, and may be configured to be able to perform feedback control of the exposure, sensitivity, etc. of the first camera 50 and the second camera 60 to the first camera 50 and the second camera 60. The stereo parallax image generation unit 200 further includes a geometric correction unit 230 that performs geometric correction of the left and right images, a matching unit 240 that performs matching processing of the left and right images, a pixel shift amount calculation unit 260 that calculates a pixel shift amount, and a correction function derivation unit 270 that derives a correction function for performing the necessary correction. The pixel shift amount calculation unit 260 and the correction function derivation unit 270, together with the affine processing units 20a and 20b, constitute a pixel shift correction processing unit.

[0038] The stereoscopic object detection unit 500 outputs the shape of a structure according to the stereoscopic parallax images generated by the stereoscopic parallax image generation unit 200 .

[0039] In the stereo camera image processing device 10 described above, this embodiment is characterized in that the image conversion unit 120 is used to convert the images so that the positions of celestial bodies in the left and right images (images P5 and P4) become closer (ideally so that they become the same).

[0040] Next, the image conversion by the image conversion unit 120 will be described. First, images of the sky are captured at night to obtain images P3 and P4. Then, position information of celestial bodies (for example, stars and the moon) is extracted from images P3 and P4. Differences in this position information are detected by the comparison unit 32. Correspondence between stars in images P3 and P4 can be identified by the brightness and color of the stars using a known matching method. It is only necessary to be able to associate the stars, and the actual positions of the stars do not need to be known.

[0041] The information detected by the comparison unit 32 only provides information on the deviation of the positions of celestial bodies, and is therefore converted into information on the entire image by the variation interpolation unit 33. At this time, the interpolation between the stars may be linear interpolation, or an approximated surface may be used. Note that the degree of the approximated surface is not limited. The deviation information (calibration information) is then recorded in the conversion information recording unit 34, and the image conversion unit 120 converts image P3 into image P5 based on this information. At this time, the image conversion unit 120 converts the images so that the positions of the celestial bodies in image P5 and image P4 are as similar as possible. Next, the derivation of the conversion information will be described.

[0042] Figure 2 shows examples of images P3 and P4 from Figure 1. The dots indicate detected celestial bodies. The solid lines show the image when a grid-like chart is projected. This assumes that a virtual chart made up of straight lines, like the celestial bodies, is placed at infinity. The chart distortion shown in Figure 2 corresponds to image distortion. For ease of understanding, we will assume that there is image distortion only in the vertical direction. Also, while this figure shows large distortion, in reality we are discussing distortions smaller than one pixel. Without distortion, the chart would be a grid-like chart made up of equally spaced straight lines.

[0043] Because the distance to the celestial object is significantly greater than the distance (baseline length) between the first camera 50 and the second camera 60, the positions of the same celestial object in images P3 and P4 should match. However, in actual cameras, the camera-detected images change due to factors such as aging. For example, aging and temperature changes can distort the optical system, or the position and angle between the lens and sensor can shift. A high-precision stereo camera requires parallax detection accuracy of less than one pixel. For example, if the sensor pixel pitch is 1 μm, even a deviation of less than 1 μm cannot be tolerated. For this reason, it is difficult to achieve high-precision parallax detection accuracy using a distortion calibration method in which two errors are superimposed, such as that described in Patent Document 1. Furthermore, such minute changes in the detected image include not only the rotationally symmetric distortion calibrated in Example 1, but also asymmetric distortion. Therefore, in this example, one of the images is calibrated using the position information of the celestial object.

[0044] In this embodiment, one image is calibrated, but the distortion of the other image is not calibrated. In this case, the image detected by the stereo camera will be shifted by the amount of distortion. However, since the amount of shift is small (for example, one pixel), it does not affect distance measurement or image recognition. Taking the chart mentioned above as an example, this embodiment does not correct the chart to a linear grid, but performs processing to match the distortion of one chart to the distortion of the other chart.

[0045] FIG. 3 is a diagram illustrating the processing performed by the image conversion unit 120 in this embodiment. This diagram shows image P5, which is the result of converting image P3 in FIG. 1 using the image conversion unit 120. The dots in the diagram indicate detected celestial bodies. The solid lines indicate the image when a mesh-like chart is projected. The dotted lines indicate the solid lines in image P3 in FIG. 2.

[0046] The amount of conversion by the image conversion unit 120 is indicated by the difference between the dotted line and the solid line. This difference is converted into tables for the up / down direction and the left / right direction by the change amount interpolation unit 33. Here, the result is conversion by the image conversion unit 120 so that point P5 on image in Figure 3 becomes the same as point P4 on image in Figure 2. Note that the image conversion unit 120 calibrates the image based on differences in the positions of celestial bodies, so it can calibrate the position, rotation, and distortion of the image.

[0047] By performing the image calibration method described above, an image processing device can be realized that can suppress distance measurement errors due to factors such as changes over time. In this embodiment, the positions of the celestial objects are determined and then compared between the positions of the celestial objects in images P3 and P4. However, this is not limiting and a calculation method such as block matching used for parallax detection in stereo cameras may also be used. While block matching can be performed for each celestial object, block matching using multiple celestial objects can provide more accurate parallax information. Accordingly, it is desirable to optimize the image size for block matching.

[0048] As described above, the calibration information is recorded in the conversion information recording unit 34, and then the calibration information is used to calibrate one of the images P3, thereby obtaining a highly accurate stereo camera image. The calibration information is, for example, a table that records the amount of movement for each pixel. The frequency with which the calibration information is recorded is arbitrary, and the calibration information may be updated, for example, annually or daily, depending on the required accuracy.

[0049] 1 can be realized by a computer equipped with an input device, an output device, a processing device, and a storage device, by executing software stored in the storage device on the processing device. Alternatively, functions equivalent to those configured by software can be realized by hardware such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The conversion information recording unit 34 can be configured, for example, by a non-volatile semiconductor memory.

[0050] 4 is a block diagram of a stereo camera image processing device 10 according to a second embodiment of the present invention. This embodiment differs from the first embodiment in that the change amount interpolation unit 33 and the change amount interpolation unit in the first embodiment are replaced by a position and rotation information calculation unit 330. The image processing device of this embodiment can calibrate the influence of a relative attitude deviation of one camera with respect to the other camera, and the influence of a relative sensor rotation deviation of one camera with respect to the other camera. Here, the position and rotation information calculation unit 330, which is a feature of this embodiment, will be described.

[0051] Figure 5 shows examples of images P3 and P4. The dots indicate detected celestial bodies. The solid lines indicate the image when a mesh-like chart is captured. As described above, this embodiment assumes the effects of relative attitude deviation of one camera with respect to the other camera and relative sensor rotation deviation of one camera with respect to the other camera, so no image distortion occurs. The relative positions of the two cameras are usually fixed, but positional deviations may occur due to changes over time or mechanical shock.

[0052] The position information of the celestial bodies shown in Figure 5 is used to index the amount of deviation of image P3 from image P4. Then, using this index, the position and rotation information calculation unit 330 calibrates the position and rotation deviation of image P3 from image P4. Here, the position coordinates of images P3 and the nth celestial body in image P4 are (Xan, Yan) and (Xbn, Ybn), respectively. Furthermore, when the rotation of image P3 is α, the horizontal shift is dx, and the vertical shift is dy, the following equation is defined as the amount of deviation Ln.

[0053] ...(Formula 1)

[0054] Here, the sum L of Ln is calculated.

[0055] ...(Formula 2)

[0056] Here, the rotation α, horizontal shift dx, and vertical shift dy that minimize this sum L are found. This makes it possible to clarify the position and rotational deviation of image P3 relative to image P4. Next, this calibration information is recorded in the conversion information recording unit 34, and image P3 is calibrated by the image conversion unit 120 using this calibration information.

[0057] Figure 6 shows image P5 after calibration. The dots in the figure indicate detected celestial bodies. The solid lines show the image when a mesh-like chart is projected. The dotted lines show the solid lines in image P3 in Figure 5.

[0058] By performing calibration in the image conversion unit 120 based on the conditions of rotation α0, horizontal shift dx0, and vertical shift dy0 that minimize the sum L, the positions of the celestial bodies in images P5 and P4 match. This completes the calibration.

[0059] By performing the image calibration method described above, it is possible to realize an image processing device that can suppress distance measurement errors due to factors such as changes over time. In this embodiment, position and rotational deviations are detected simultaneously, but the order of detection is not limited to this. For example, rotation may be detected first, followed by horizontal and vertical shifts.

[0060] Furthermore, the sum L may be used to determine whether or not the camera itself needs to be calibrated. In this case, for example,

[0061] ...(Formula 3)

[0062] If the calculated value exceeds one pixel, it may be determined that calibration should be performed using a predetermined device such as a calibration chart.

[0063] 7 is a block diagram of a stereo camera image processing device 10 according to a third embodiment of the present invention. In the second embodiment, the stereo camera was calibrated by determining the positional and rotational deviation amounts of image P3 relative to image P4. This embodiment differs from the second embodiment in that it addresses the positional deviation of the sensor.

[0064] In stereo cameras, the system is generally transformed into an f tan θ projection system to detect parallax. If fisheye lenses are used as wide-angle lenses in the first camera 50 and the second camera 60, an affine transformation from f sin θ to f tan θ, for example, is required. Because this affine transformation requires a significant change in the image in the wide-angle area, if the sensor position is misaligned, the image after the affine transformation will deviate more from the ideal position as it approaches the wide-angle area (wide field of view). Therefore, if the image is corrected before the affine transformation, angle-dependent errors can be suppressed.

[0065] In this embodiment, the displacement at a wide viewing angle is used to calibrate the positional displacement of the sensor in the second image conversion unit 121. Here, the position coordinates of the celestial bodies present at the wide viewing angle θ of images P3 and P4 are (Xa(θ), Ya(θ)) and (Xb(θ), Yb(θ)), and Equation 4 is defined.

[0066] ...(Formula 4)

[0067] θ|| and θ⊥ indicate the field of view in the horizontal and vertical directions (wide field of view). Here, when the second image conversion unit 121 shifts the image P1 in the up / down and left / right directions, the values ​​of Dx and Dy change. Here, (X1, Y1) is derived so that Dx and Dy are zero. Then, as in the second embodiment, the rotation α1, left / right shift dx1, and up / down shift dy1 are calculated so that the sum L of Equation 2 is minimized.

[0068] Under the conditions of Example 2, the entire screen can be used, but under the conditions of Example 3, the influence of sensor positional deviation in a wide field of view is significant, so it is desirable to obtain the image with a narrow field of view. For example, when the lens projection is an orthogonal projection (f sin θ), a field of view angle of 30 degrees or less is desirable, so that the image height positions are approximately the same, from the perspective of converting from orthogonal projection to central projection. By performing calibration in the image conversion unit 122 based on the conditions of rotation α1, horizontal shift dx1, and vertical shift dy1, the positions of celestial bodies in image P5 and image P4 are matched. This completes the calibration.

[0069] As described above, by performing the image calibration method, it is possible to realize an image processing device that can suppress distance measurement errors due to factors such as changes over time. By performing rotation and shift correction using the second image conversion unit 121 before the affine processing unit 20a, it is possible to deal with positional deviation of the sensor, as compared to Example 2.

[0070] 8 is a diagram showing a detected image according to Example 4 of the present invention. When performing the calibrations of Examples 1 to 3, the more celestial bodies there are, the more accurately the distortion can be calibrated. Therefore, in this example, a method for increasing the number of celestial bodies to be detected will be described.

[0071] The multiple points in Figure 8 are the result of acquiring time-series data for the celestial body images in Figure 2. By acquiring data at different times in this way, the number of celestial body points can be significantly increased. This allows the positions of celestial bodies in the images from the left and right cameras to match with high accuracy.

[0072] 9 is a diagram illustrating another detected image according to Example 4. As shown in Fig. 9, the positions of celestial objects in the images captured by the left and right cameras can be aligned in successive images. In this case, the images captured by the left and right cameras are configured to bring the trajectories of the celestial objects closer to each other.

[0073] 10A to 10C are diagrams showing detected images according to Example 5 of the present invention. When performing the calibrations of Examples 1 to 3, the more celestial bodies there are, the more accurately distortion can be calibrated. Therefore, in this example, a method for substantially increasing the number of celestial bodies to be detected will be described.

[0074] 11 is a diagram illustrating the orientation of the stereo camera image processing device 10 according to Example 5. The first camera 50 and the second camera 60 are fixed to a base 110 at a predetermined distance, and the base 110 is equipped with an orientation control mechanism that mechanically tilts the camera about a predetermined axis. With this configuration, the optical axes of the first camera 50 and the second camera 60 can be tilted simultaneously.

[0075] The multiple points in Fig. 10 are obtained by changing the orientation of the stereo camera image processing device 10 and capturing multiple images as shown in Fig. 11. Figs. 10A, 10B, and 10C show image P4 taken by the left camera when the orientation of the stereo camera image processing device 10 is changed in the left-right direction, the up-down direction, and the diagonal direction, respectively. In this case, each image can be acquired by tilting the base 110 in the xz plane (Fig. 10A), in the zy plane (Fig. 10B), or in both the xz plane and the zy plane (Fig. 10C).

[0076] In this case, images containing substantially more celestial bodies than in Example 1 and the like can be used for calibration. Also, it becomes easier to detect distortions occurring in a specific direction. In this way, images corresponding to the distortions can be obtained. This allows the positions of celestial bodies in the images from the left and right cameras to match with high precision.

[0077] 12A to 12C are diagrams showing other examples of detected images according to the fifth embodiment of the present invention. As shown in FIGS. 12A to 12C, the positions of celestial bodies in the images captured by the left and right cameras can be aligned in consecutive images. Note that FIGS. 12A to 12C show an image P4 captured by the left camera when the orientation of the stereo camera image processing device 10 is changed in the left-right, up-down, and diagonal directions.

[0078] As described above, the system includes a base 110 to which the first camera 50 and the second camera 60 are fixed, an attitude control mechanism that mechanically tilts the base 110 about a predetermined axis, and the optical axis directions of the first camera 50 and the second camera 60 can be changed. The system also includes a celestial object position detection unit that detects a predetermined celestial object in multiple optical axis directions from a first image captured by the first camera 50 and detects the predetermined celestial object in multiple optical axis directions from a second image captured by the second camera 60, a comparison unit that compares the position of the predetermined celestial object in the multiple optical axis directions in the first image with the position of the predetermined celestial object in the multiple optical axis directions in the second image, and a calibration unit that generates calibration information for the second image based on the positional deviation of the predetermined celestial object in the multiple optical axis directions in the second image relative to the position of the predetermined celestial object in the multiple optical axis directions in the first image. This allows the horizontal and vertical positions of celestial objects in the images captured by the left and right cameras to be matched with high precision.

[0079] The above embodiments can be combined in any way. For example, the distortion calibration of embodiment 1 can be combined with the rotation / shift calibration of embodiment 2. In this case, distortion calibration and rotation / shift calibration are performed alternately, and the accuracy of the calibration is evaluated by comparing the positions of the same celestial object in image P4 and image P5. By repeating calibration based on the evaluation results, the accuracy of the calibration information can be improved by addressing both distortion and positional deviation. Furthermore, the second image conversion unit 121 of embodiment 3 can also be added to these.

[0080] The present invention is not limited to the above-described embodiments, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0081] According to the above-described embodiment, it is possible to efficiently and accurately detect and recognize three-dimensional objects, thereby enabling the generation of highly accurate digital twins, thereby consuming less energy, reducing carbon emissions, preventing global warming, and contributing to the realization of a sustainable society.

[0082] 10: Stereo camera image processing device 50: First camera 60: Second camera 31a: Celestial body position detection unit 31b: Celestial body position detection unit 32: Comparison unit 33: Variation amount interpolation unit 34: Conversion information recording unit 120: Image conversion unit

Claims

1. A stereo image processing device that detects parallax from images captured by a first camera and a second camera, comprising: a celestial object position detection unit that detects a specified celestial object from a first image captured by the first camera, and detects the specified celestial object from a second image captured by the second camera; a comparison unit that compares the position of the specified celestial object in the first image with the position of the specified celestial object in the second image; a calibration unit that generates calibration information for the first image, which calibrates the position of the specified celestial object in the first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image based on a positional deviation of the specified celestial object in the first image relative to the position of the specified celestial object in the second image; a conversion information recording unit that records the calibration information; and an image conversion unit that calibrates the image captured by the first camera using the calibration information.

2. The stereo image processing device according to claim 1, wherein the calibration unit is a change amount interpolation unit that generates the calibration information to calibrate the distortion of the entire first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image.

3. The stereo image processing device according to claim 1, wherein the calibration unit is a position and rotation information calculation unit that generates the calibration information for rotating and shifting the entire first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image.

4. The stereo image processing device according to claim 1, comprising: a first affine processing unit that applies affine processing to an image captured by the first camera to generate the first image; and a second affine processing unit that applies affine processing to an image captured by the second camera to generate the second image.

5. The stereo image processing device according to claim 4, further comprising a stereo parallax image generating unit that generates a stereo parallax image using an image obtained by performing affine processing on an image captured by the second camera and an image obtained by calibrating an image obtained by performing affine processing on an image captured by the first camera in the image conversion unit.

6. The stereo image processing device of claim 4, wherein the calibration unit is a position / rotation information calculation unit that generates the calibration information for rotating and shifting the entire first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image; a first image conversion unit that uses the calibration information to calibrate the image captured by the first camera after affine processing has been performed in the first affine processing unit; and a second image conversion unit that uses the calibration information to calibrate the image captured by the first camera before affine processing has been performed in the first affine processing unit.

7. A stereo image processing device as described in claim 1, comprising: an astronomical object position detection unit that detects the specified celestial object at multiple times from a first image captured by the first camera and detects the specified celestial object at multiple times from a second image captured by the second camera; an comparison unit that compares the position of the specified celestial object at multiple times in the first image with the position of the specified celestial object at multiple times in the second image; and an calibration unit that generates calibration information for the first image based on the positional deviation of the specified celestial object at multiple times in the first image relative to the position of the specified celestial object at multiple times in the second image.

8. A stereo image processing device as described in claim 1, comprising: a base to which the first camera and the second camera are fixed, an attitude control mechanism for mechanically tilting the base about a predetermined axis, and enabling the optical axis directions of the first camera and the second camera to be changed; an astronomical object position detection unit for detecting the specified celestial object in a plurality of optical axis directions from a first image captured by the first camera and detecting the specified celestial object in a plurality of optical axis directions from a second image captured by the second camera; a comparison unit for comparing the position of the specified celestial object in the plurality of optical axis directions in the first image with the position of the specified celestial object in the plurality of optical axis directions in the second image; and a calibration unit for generating calibration information for the first image based on the positional deviation of the specified celestial object in the plurality of optical axis directions in the first image relative to the position of the specified celestial object in the plurality of optical axis directions in the second image.

9. A stereo image processing device that detects parallax from images captured by a first camera and a second camera, comprising: a celestial object position detection unit that detects a predetermined celestial object at multiple times from a first image captured by the first camera and detects the predetermined celestial object at multiple times from a second image captured by the second camera; a comparison unit that compares the position of the predetermined celestial object at multiple times in the first image with the position of the predetermined celestial object at multiple times in the second image; a calibration unit that generates calibration information based on the position of the predetermined celestial object at multiple times in the second image and the positional deviation of the predetermined celestial object at multiple times in the first image, so that the position of the predetermined celestial object at multiple times in the second image and the position of the predetermined celestial object at multiple times in the first image become closer; a conversion information recording unit that records the calibration information; and an image conversion unit that uses the calibration information to calibrate so that the image captured by the first camera and the image captured by the second camera become closer.

10. A stereo image processing method for detecting parallax from images captured by a first camera and a second camera, comprising: a celestial object position detection process for detecting a specified celestial object from a first image captured by the first camera, and detecting the specified celestial object from a second image captured by the second camera; a comparison process for comparing the position of the specified celestial object in the first image with the position of the specified celestial object in the second image; a calibration process for generating calibration information for the first image, which calibrates the position of the specified celestial object in the first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image based on the positional deviation of the specified celestial object in the first image relative to the position of the specified celestial object in the second image; a conversion information recording process for recording the calibration information; and an image conversion process for calibrating the image captured by the first camera using the calibration information.

11. The stereo image processing method according to claim 10, wherein the calibration process is a change amount interpolation process that generates the calibration information that calibrates the distortion of the entire first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image.

12. The stereo image processing method according to claim 10, wherein the calibration process is a position and rotation information calculation process that generates the calibration information for rotating and shifting the entire first image so that the position of the specified celestial object in the first image approaches the position of the specified celestial object in the second image.

13. A stereo image processing method as described in claim 12, comprising the steps of: a first affine processing for performing affine processing on an image captured by the first camera to generate the first image; a second affine processing for performing affine processing on an image captured by the second camera to generate the second image; a first image conversion processing for calibrating the image captured by the first camera after the first affine processing using the calibration information; and a second image conversion processing for calibrating the image captured by the first camera before the first affine processing using the calibration information.

14. A stereo image processing method as described in claim 10, comprising the steps of: detecting the specified celestial object at multiple times from a first image captured by the first camera, and detecting the specified celestial object at multiple times from a second image captured by the second camera; comparing the position of the specified celestial object at multiple times in the first image with the position of the specified celestial object at multiple times in the second image; and generating calibration information for the first image based on the positional deviations of the specified celestial object at multiple times in the first image relative to the position of the specified celestial object at multiple times in the second image.

15. A stereo image processing method as claimed in claim 10, further comprising the steps of: using a base to which the first camera and the second camera are fixed, performing an attitude control process of mechanically tilting the base about a predetermined axis to change the optical axis directions of the first camera and the second camera; detecting the specified celestial object in a plurality of optical axis directions from a first image captured by the first camera, and detecting the specified celestial object in a plurality of optical axis directions from a second image captured by the second camera; comparing the position of the specified celestial object in the plurality of optical axis directions in the first image with the position of the specified celestial object in the plurality of optical axis directions in the second image; and generating calibration information for the first image based on the positional deviation of the specified celestial object in the plurality of optical axis directions in the first image relative to the position of the specified celestial object in the plurality of optical axis directions in the second image.

Images