Image processing device and image calibration method

Abstract

To enable calibration of wide-angle cameras without the need for large charts. [Solution] An image processing apparatus comprising: an image processing unit 20a, 20b that acquires two images captured by two cameras 10a, 10b whose imaging areas partially overlap and performs processing including pixel shift correction on the images; a stereo parallax image generation unit 30 that generates a parallax image based on the overlapping imaging areas of the two images; a road surface cross-section shape estimation unit 50 that generates a first distance image including distance information to a three-dimensional object for the overlapping imaging areas based on the parallax image; and an image calibration unit 90 that acquires the distance to a three-dimensional object in the monocular viewing area captured in only one image using moving parallax obtained by movement and generates a second distance image, and generates calibration information for calibrating the amount of pixel shift correction based on the first distance image and the second distance image.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image calibration method, and more particularly to an image processing apparatus suitable for being mounted on a moving body to recognize the surrounding situation of the moving body and an image calibration method thereof.

Background Art

[0002] Conventionally, a stereo camera is known as an apparatus for three-dimensionally recognizing an object. The stereo camera utilizes the difference in the way an image is captured by a plurality of cameras arranged at different positions, obtains the parallax between the cameras based on triangulation, and detects the depth and position of an object using the parallax, and can accurately detect the position of the measurement target object.

[0003] In recent years, stereo cameras have been mounted on vehicles such as automobiles and applied to a technology (in-vehicle sensing technology) for detecting the positions of obstacles and the like around the vehicle. In in-vehicle sensing technology, in order to support many use cases, there is a demand for a wider angle of view that can detect a wider range of obstacles and the like, and a longer range that can detect obstacles farther away. As a technology for meeting such demands, for example, Patent Document 1 discloses a stereo image processing apparatus in which the central position of a sensor is shifted and arranged in a direction away from or approaching each other with respect to the optical axis of a lens to widen the visual field range and achieve a wider angle of view.

[0004] Furthermore, in order to avoid the effects of dirt and other factors, in-vehicle stereo cameras are generally mounted inside the vehicle, and subjects such as obstacles are captured through the windshield. Conventionally, stereo cameras have undergone calibration work called aiming during vehicle manufacturing or inspection to correct for optical distortion and positional deviations caused by the influence of the windshield. As a method for calibrating stereo cameras that capture subjects through a transparent material such as a windshield, for example, the technology disclosed in Patent Document 2 is known. Patent Document 2 discloses obtaining a correction parameter that calibrates both absolute position deviations and relative position deviations by correcting one of the absolute position correction parameters of two cameras calculated by simulation using an image obtained by photographing a calibration chart. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2019-32409 [Patent Document 2] Japanese Patent Publication No. 2019-132855 [Overview of the project] [Problems that the invention aims to solve]

[0006] In the technology disclosed in Patent Document 2, as described above, correction parameters are obtained using a calibration chart. However, calibrating a wide-angle image processing device, such as the one described in Patent Document 1, requires a chart large enough to cover a wide field of view, making the chart unnecessarily large. Such large charts can create problems, for example, requiring large-scale modifications to the manufacturing line where calibration work is performed at a vehicle manufacturer. Furthermore, since stereo camera calibration is also performed when the windshield is replaced, there is a risk that dealerships and other repair shops may face challenges such as securing storage space for the charts.

[0007] This invention has been made in view of the above-mentioned problems, and provides an image processing device and an image calibration method that can calibrate a stereo camera with a wide maximum field of view with high accuracy without using a large chart. [Means for solving the problem]

[0008] In one preferred embodiment, the image processing apparatus according to the present invention includes: an image processing unit that acquires images captured by a first camera and a second camera mounted on a moving object; a stereo parallax image generation unit that acquires parallax information relating to the parallax between the first imaging area captured by the first camera and the first imaging area captured by the second camera for a first imaging area that is captured in both images captured by the first camera and the second camera; a distance image generation unit that acquires the distance to a three-dimensional object in the first imaging area based on the parallax information and generates a first distance image including first distance information relating to said distance; and a hybrid image including the first imaging area and the second imaging area from the first distance image and an image of the second imaging area that is captured by either the first camera or the second camera but not by the other camera, and acquires the distance to a three-dimensional object captured in the second imaging area based on the first distance information. The system includes a BRID 3D object detection unit, a moving parallax image generation unit that generates a moving parallax image including moving parallax information relating to the parallax between the first image and the second image, using either the first image or the second image as a reference image, based on a first image captured at a first time including a second imaging area and a second image captured at a second time different from the first time including a second imaging area, and using either the first image or the second image as a reference image, a movement amount acquisition unit that acquires the amount of movement of a moving object between the first time and the second time, a distance conversion unit that generates a second distance image including second distance information relating to the distance to a 3D object captured in the second imaging area of ​​the reference image, based on the moving parallax image and the amount of movement, and a calibration unit that generates calibration information used to calibrate correction information for correcting distortion of an image captured by a first camera or a second camera that captured an image including a second imaging area, based on a first distance image and a second distance image generated based on an image captured at the same time as the reference image.

[0009] Furthermore, in another preferred embodiment, the image processing apparatus according to the present invention includes: an image processing unit that acquires an image from a camera mounted on a moving body; a distance calculation unit that acquires the distance to a three-dimensional object based on the position of the three-dimensional object in the image; a moving parallax image generation unit that generates a moving parallax image including moving parallax information relating to the parallax between the first image and the second image, using either the first image or the second image as a reference image, based on a first image captured at a first time and a second image captured at a second time different from the first time; a movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time; a distance conversion unit that generates a distance image including distance information relating to the distance to a three-dimensional object in the reference image based on the moving parallax image and the amount of movement; and a calibration unit that generates calibration information used to calibrate correction information for correcting distortion of an image captured by a camera, based on the position of the three-dimensional object on the reference image and distance information.

[0010] Furthermore, in one preferred embodiment, the image calibration method according to the present invention acquires parallax information relating to the parallax between the image captured by the first camera and the image captured by the second camera for a first imaging region that is captured in both of two images captured by a first camera and a second camera mounted on a moving object, acquires the distance to a three-dimensional object in the first imaging region based on the parallax information, and generates a first distance image including first distance information relating to said distance, and a first image captured at a first time that includes a second imaging region captured by either the first camera or the second camera but not by the other camera, and a second image captured at a different time that includes the second imaging region. Based on the second image captured at the first time, a moving parallax image is generated, using either the first or second image as a reference image, and including moving parallax information relating to the parallax between the first and second images. The amount of movement of the moving object between the first and second time points is obtained. Based on the moving parallax image and the amount of movement, a second distance image is generated, including second distance information relating to the distance to the three-dimensional object captured in the second imaging area of ​​the reference image. Based on the first and second distance images generated based on the reference image, calibration information is generated, which is used to calibrate correction information for correcting distortion of images captured by the first or second camera that captured the image including the second imaging area. [Effects of the Invention]

[0011] According to the present invention, it is possible to calibrate cameras with a wide maximum field of view with high accuracy without using large charts. Other novel features of the present invention and the technical problems solved thereby will become apparent from the description and drawings herein. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic block diagram showing the logical configuration in a first embodiment of the image processing apparatus according to the present invention. [Figure 2] This is a schematic diagram showing the imaging area captured by the camera. [Figure 3]It is a schematic block diagram showing an example of the configuration of an image processing unit. [Figure 4] It is a schematic diagram showing the relationship between the front glass and the incident light to the camera in a horizontal plane as viewed from above the vehicle. [Figure 5] It is a schematic diagram showing the relationship between the front glass and the incident light to the camera in a vertical plane as viewed from the side of the vehicle. [Figure 6] It is a schematic diagram showing the change in the position of the image formed on the image sensor depending on the presence or absence of the front glass. [Figure 7] It is a schematic diagram showing the change in the pixel shift amount in the vertical direction. [Figure 8] It is a schematic block diagram showing an example of the configuration of an image calibration unit. [Figure 9] It is a schematic diagram for explaining the distance error between the first distance image and the second distance image. [Figure 10] It is a schematic diagram for explaining the acquisition of the distance to an object using a moving parallax image. [Figure 11] It is a schematic diagram showing the pixel shift amount in the horizontal angular direction of a stereo parallax image based on two images obtained by moving a camera. [Figure 12] It is a schematic diagram showing the magnitude of the parallax error in the horizontal angular direction of a stereo parallax image based on two images obtained by moving a camera. [Figure 13] It is a graph showing an example of the relationship between the angular width and the difference in the angular width to an object in two images. [Figure 14] [[ID=...]] It is a schematic diagram showing the movement of a camera in the alignment direction. [Figure 15] It is a schematic diagram showing the movement of a camera in a direction different from the alignment direction. [Figure 16] It is a schematic block diagram showing the logical configuration of an image processing apparatus in the second embodiment. [Figure 17] It is a schematic block diagram showing the logical configuration of an image processing apparatus in the third embodiment. [Figure 18] It is a schematic diagram showing the imaging region in the third embodiment. [[ID=...]] [Modes for carrying out the invention]

[0013] Hereinafter, representative embodiments of the present invention will be described with reference to the drawings. Note that the embodiments and drawings described below are illustrative examples for illustrating the present invention, and have been omitted or simplified as appropriate for clarity of explanation. Furthermore, please note that the positions, sizes, shapes, and extents of the components shown in the drawings may not necessarily accurately represent them, in order to facilitate understanding of the invention.

[0014] Figure 1 is a schematic block diagram showing the logical configuration of a first embodiment of the image processing apparatus according to the present invention.

[0015] The image processing device 1 is used, for example, mounted on a vehicle such as an automobile, to detect the distance to other vehicles, buildings, pedestrians, and other three-dimensional objects in the vicinity of the vehicle. In this embodiment, an image processing device mounted on a vehicle is described as an example, but it should be noted that the image processing device of this embodiment can also be applied to other moving objects and other applications.

[0016] Image processing device 1 is connected to cameras 10a and 10b, which constitute a stereo camera, to form a stereo camera system. The image processing device 1 may be housed together with cameras 10a and 10b in a single housing, or they may be housed separately in different housings. In the latter case, the image processing device 1 is connected to cameras 10a and 10b, for example, via an in-vehicle network (not shown), controls these cameras via the in-vehicle network, and acquires images captured by these cameras. In the following description, when it is not necessary to distinguish between cameras 10a and 10b individually, the alphabetical subscripts attached to their reference numbers are omitted, and they are simply referred to as camera 10.

[0017] Camera 10 is equipped with a lens and an image sensor (not shown), and the image sensor converts the light guided through the lens to the image sensor into an electrical signal, which is then acquired as image information. The image processing device 1 is configured to detect three-dimensional objects in the surroundings based on the image acquired by camera 10 and to issue alarms, etc.

[0018] Figure 2 is a schematic diagram showing the area captured by the camera 10. In Figure 2, A shows the area around the vehicle 100 on which the image processing device 1 is mounted, as seen from above. B shows the image captured by the camera 10.

[0019] In this embodiment, camera 10a is mounted on the right side of the vehicle 100 to capture the area in front of the vehicle 100. Similarly, camera 10b is mounted on the left side of the vehicle to capture the area in front of the vehicle 100. Here, "right side" and "left side" refer to the relative positions of the two cameras, and do not mean their absolute positions on the vehicle 100. Cameras 10a and 10b are positioned on the vehicle 100 such that a portion of the area they each capture (area 110) overlaps, forming a so-called stereo camera. Hereafter, the overlapping area 110 will be referred to as the stereo viewing area. In this embodiment, the image captured by camera 10a includes the stereo viewing area 110 and an area 111 to the front left of the vehicle that is not captured by camera 10b. Similarly, the image captured by camera 10b includes the stereo viewing area 110 and an area 112 to the front right of the vehicle that is not captured by camera 10a. The region imaged by only one of camera 10a or camera 10b is referred to here as the monocular region.

[0020] The image processing device 1 can measure the distance to a three-dimensional object, for example, the distance to a pedestrian 120 as a three-dimensional object, based on the parallax between two images captured by cameras 10a and 10b in the stereo viewing area 110. On the other hand, in the monocular viewing areas 111 and 112, the distance to a three-dimensional object outside the stereo viewing area 110, for example, the pedestrian 120, is measured by utilizing the distance measurement results based on the parallax in the stereo viewing area 110.

[0021] The distances measured in monocular areas 111 and 112 are estimated using the distance measurement results in stereo viewing area 110, and therefore their accuracy is inferior to the distance measurement results in stereo viewing area 110. However, by shifting the imaging areas of camera 10a and camera 10b, it becomes possible to detect three-dimensional objects over a wider range (wider angle) compared to when the imaging areas of camera 10a and camera 10b completely overlap and the entire area is imaged as a stereo viewing area.

[0022] Returning to Figure 1, the image processing device 1 includes an image processing unit 20a, an image processing unit 20b, a stereo disparity image generation unit 30, a monocular image generation unit 40, a road surface cross-section shape estimation unit 50, a stereoscopic object detection unit 60, a hybrid object detection unit 70, an alarm control unit 80, and an image calibration unit 90.

[0023] The image processing units 20a and 20b each function as image acquisition units that acquire images captured from the corresponding cameras 10a and 10b, and perform predetermined processing on the acquired images. The specific processing performed by the image processing units 20a and 20b will be described later. In the following description, as with the camera 10, if it is not necessary to distinguish between the image processing units 20a and 20b individually, the alphabetical subscript will be omitted and they will be referred to as the image processing unit 20.

[0024] The stereo disparity image generation unit 30 performs a matching process using the two images processed by the image processing units 20a and 20b, and generates a stereo disparity image that includes disparity information regarding the disparity between the two images for a three-dimensional object present in the stereo viewing region 110. The matching process may be performed on the entire image including the monocular viewing regions 111 and 112, or it may be performed only on the stereo viewing region 110 after extracting the stereo viewing region 110 from the two images. In the latter case, the amount of computation can be reduced and the processing time can be shortened compared to the former. Furthermore, the stereo disparity image generation unit 30 may be configured to, for example, analyze the brightness of the image based on the image acquired from the image processing unit 20a, and then provide feedback control to the exposure amount and sensitivity of the camera 10 so that the next image captured is captured with appropriate brightness and the appearance of the same subject captured by the image processing unit 20a and the image processing unit 20b is as similar as possible.

[0025] The monocular image generation unit 40 extracts images of monocular regions 111 and 112 from the two images processed by the image processing unit 20 and generates a monocular image. At this time, the monocular image generation unit 40 applies a projection transformation to each image so that the unit length along the same horizontal line of each monocular image represents an equal distance from one another.

[0026] The road surface cross-section shape estimation unit 50 estimates the distance to a three-dimensional object from which parallax information has been acquired based on the parallax information of the stereo parallax image, generates a first distance image including first distance information relating to the estimated distance, and estimates the cross-sectional shape of the road surface that the vehicle 100 is expected to travel on.

[0027] The stereoscopic object detection unit 60 detects three-dimensional objects in the stereoscopic viewing area 110 based on the stereo disparity image generated by the stereo disparity image generation unit 30, and identifies what type of three-dimensional object it is, such as a pedestrian, bicycle, vehicle, or building. If the identified three-dimensional object is a moving object such as a pedestrian, bicycle, or vehicle, the stereoscopic object detection unit 60 also estimates the position of the moving object and tracks it to estimate its direction of movement and speed.

[0028] The hybrid 3D object detection unit 70 generates a hybrid image from the first distance image generated by the road surface cross-section shape estimation unit 50 and the left and right monocular images generated by the monocular image generation unit 40, and detects 3D objects based on the hybrid image. The hybrid image is generated by placing monocular images on both sides of the stereoscopic viewing area of ​​the distance image and combining them, as shown as image B in Figure 2.

[0029] The hybrid 3D object detection unit 70 detects 3D objects present in the monocular view portion of the hybrid image and obtains the distance to those 3D objects based on first distance information included in the stereoscopic view portion. For example, if the 3D object (object to be measured) detected in the monocular view portion is a person, the distance to the person is estimated by detecting the position of the person's feet in the vertical direction of the monocular view image and obtaining first distance information for the vertical image position corresponding to the foot position in the stereoscopic view portion of the image. Similarly, if the object to be measured is a vehicle, the distance to the vehicle is estimated by detecting the contact surface between the vehicle and the ground. The hybrid 3D object detection unit 70 identifies the type of the detected 3D object in the same way as the stereoscopic 3D object detection unit 60, and estimates the position of the 3D object identified as a moving object, as well as its direction of movement and speed of movement.

[0030] The alarm control unit 80 utilizes the detection results of three-dimensional objects from the stereoscopic object detection unit 60 and the hybrid three-dimensional object detection unit 70 to perform driver assistance control to support vehicle operation by the driver, or preventive safety control such as issuing alarms and emergency braking in emergencies. For example, if the detected three-dimensional object is identified as a vehicle, the detection result will be used to perform follow control to the preceding vehicle or emergency braking control. If the detected three-dimensional object is identified as a pedestrian or bicycle, emergency braking control or alarm activation control will be performed.

[0031] The image calibration unit 90 performs calibration of correction information used to correct image distortion caused by the influence of the vehicle's windshield, which will be described later. Based on the monocular image generated by the monocular image generation unit 40 and the estimation results from the road surface cross-section shape estimation unit 50, the image calibration unit 90 detects the amount of image shift on the monocular image due to the influence of distortion and estimates a pixel shift correction amount to correct it.

[0032] In this embodiment, the pixel shift correction amount estimated by the image calibration unit 90 is supplied to each image processing unit 20 and used to calibrate the correction information in the image processing unit 20 to eliminate the effects of distortion during image processing.

[0033] Physically, the image processing apparatus 1 of this embodiment is configured to include a computing device (not shown), memory, and an interface for the computing device to exchange data with the camera 10 and other control devices. The computing device executes a control program stored in memory, thereby realizing the functions of each of the above-mentioned parts, excluding the camera 10. The computing device can be a so-called CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit). Some or all of the functions of each of the above-mentioned parts may be realized by hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0034] The image processing unit 20 and the image calibration unit 90 will be described in detail below with reference to the drawings. Note that the specific processing performed by the stereo disparity image generation unit 30, monocular image generation unit 40, road surface cross-section shape estimation unit 50, stereoscopic object detection unit 60, and hybrid object detection unit 70 is publicly known, for example, from Japanese Patent Application Publication No. 2019-106026, and a detailed explanation thereof will be omitted here.

[0035] Figure 3 is a schematic block diagram showing an example of the configuration of the image processing unit 20a. Note that the image processing unit 20a and the image processing unit 20b have similar configurations, and the following description applies to both. Therefore, please note that in Figure 3 and the following description, the alphabetical subscripts added to the reference numbers to distinguish between the two have been omitted.

[0036] When the image processing unit 20 acquires an image captured by the camera 10 using the image acquisition unit 210, the affine processing unit 220 performs an affine transformation on the acquired image using parameters stored in the affine table storage unit 230. The coordinate transformation performed by the affine transformation may be linear or nonlinear. In this embodiment, the camera 10 uses a fisheye lens with a wide maximum field of view, and the affine transformation transforms the orthogonal projection of the fisheye lens, fsinθ, to a coordinate system of (ftanθx, ftanθy). Here, f is the focal length of the fisheye lens, θ is the angle of incidence of light rays to the lens (field of view), and θx and θy are the horizontal and vertical components of the angle of incidence θ in the Cartesian coordinate system. In addition, in this embodiment, vertical pixel displacement (called pixel shift) due to the influence of the windshield is corrected during the affine transformation in the affine processing unit 220. In other words, the affine processing unit 220 also functions as a pixel shift correction processing unit that corrects pixel shifts associated with image distortion caused by the influence of the windshield. In addition to the processing described above, other distortion transformation processing may be performed in the affine processing unit 220.

[0037] The affine table storage unit 230 stores the affine table used by the affine processing unit 220. The affine table holds parameters used when adjusting the position and angle of the image captured by the camera 10 through affine processing. The parameters include parameters that are calibrated by the pixel shift correction amount estimated by the image calibration unit 90 and become correction information used to correct the pixel shift. In practice, the affine table is stored, for example, in a memory area (not shown) provided by the image processing device 1.

[0038] The brightness correction unit 240 corrects the brightness of each pixel in the image converted by the affine processing unit 220. The brightness correction unit 240 corrects the brightness of each pixel in the image based on, for example, the gain of the camera 10 that captured the image and the differences in sensitivity of each pixel in the image, so that a brightness suitable for processing in the subsequent processing unit is obtained.

[0039] The pixel interpolation unit 250 performs demosaicing on the image after brightness correction, converting the RAW image into a color image.

[0040] The luminance information generation unit 260 acquires luminance information for each pixel based on the color information of each pixel in the image converted to a color image. This luminance information is used, for example, when generating a disparity image in the stereo disparity image generation unit 30. The image processing unit 20 passes the images processed by the above-described units to the stereo disparity image generation unit 30 and the monocular image generation unit 40.

[0041] Before providing a detailed explanation of the image calibration unit 90, the following section will describe the effect of the windshield on the captured image.

[0042] As described above, in this embodiment, the hybrid 3D object detection unit 70 estimates the distance to objects such as the feet of a person or the ground contact points of a vehicle located in the monocular viewing areas 111 and 112. Therefore, if the detection position shifts vertically in the image due to the influence of the windshield, an error will occur in the measured distance.

[0043] Figure 4 is a schematic diagram showing the relationship between the windshield and the incident light on the camera in a horizontal plane as viewed from above the vehicle.

[0044] Light rays entering camera 10 are refracted according to Snell's law as they pass through the windshield 200. If the inclination of the incident and exit surfaces of the light rays on the windshield 200 is equal, the incident and exit angles of the light rays are equal, and the inclination of the light rays is equal before and after passing through the windshield 200. If there is a difference in the inclination of the incident and exit surfaces of the light rays on the windshield 200, the greater the difference, the greater the difference between the incident and exit angles of the light rays, and the greater the inclination of the light rays before and after passing through the windshield 200.

[0045] Generally, the windshield 200 has a curved shape, and if the angle of incidence is minimal, the inclination of both sides of the windshield 200 through which the light ray passes is equal. As the angle of incidence increases, the difference between the inclination of the surface of the windshield 200 from which the light ray enters and the inclination of the surface of the windshield from which the light ray exits becomes larger. The camera 10 that images the front of the vehicle is often installed near the center of the windshield 200 in the horizontal direction. Therefore, a light ray entering the camera 10 from the front of the vehicle passes through a point where the inclination of the outer and inner surfaces of the windshield 200 is almost the same, and the inclination of the light ray does not change much before and after passing through the windshield 200. For example, the angle of the light ray R00 entering the camera 10a from the front of the vehicle does not change much before and after passing through the windshield 200. On the other hand, a light ray from the diagonal front of the vehicle, such as the light ray R01, enters the windshield 200 at an angle, so the direction of the light ray changes as it passes through the windshield 200. Therefore, objects located to the left or right of the front of the vehicle will appear to be in a horizontally shifted position from their actual location.

[0046] Figure 5 is a schematic diagram showing the relationship between the windshield and the incident light on the camera in a vertical plane viewed from the side of the vehicle.

[0047] Generally, the windshield 200 of a vehicle is mounted at an angle to the vertical. Therefore, light rays from the front of the vehicle enter the windshield 200 at an overall angle. For example, as shown in Figure 5, when viewed in the vertical direction, the direction of the light rays R10 passing near the optical axis of camera 10a and R11 at the lower field of view changes significantly before and after passing through the windshield 200. As a result, when viewing an object through the windshield 200, the position of the object appears to be shifted downwards in the vertical direction compared to its actual position.

[0048] Figure 6 is a schematic diagram showing the change in the position of the image on the image sensor of camera 10a with and without the windshield. Here, we will explain using camera 10a as an example, but the same applies to camera 10b. Note that the image actually captured on the image sensor 300 will be inverted vertically and horizontally compared to the diagram, but for convenience, we will explain it here assuming that it will be the same as what a person sees. In addition, we will assume that there is no distortion in the optical system of camera 10a other than that caused by the windshield 200.

[0049] Now, consider four objects whose images are formed in the dashed regions 330, 331, 332, and 333 on the image sensor 300 when the windshield 200 is absent. Region 330 is the region near the optical axis 310 of the camera 10a, region 331 is the wide-angle region on the left along the horizontal line 320 passing through the optical axis 310, region 332 is the wide-angle region on the lower side along the vertical line 321 passing through the optical axis 310, and region 333 is the wide-angle region on the left along the horizontal line 322 passing through region 332. When the objects whose images are formed in these regions when the windshield 200 is absent are imaged with the windshield 200 present, as described above, the path of the light rays changes when they pass through the windshield 200, so the regions in which the images of the objects are formed change to the solid regions 340, 341, 342, and 343, respectively. The arrows in each region in the figure schematically show the direction and magnitude of the change in the position of the object's image.

[0050] Generally, in a camera, images are captured by focusing light rays incident on the lens onto an image sensor 300. If the path of the light rays is shifted, the position of the image on the image sensor also changes. This shift in the light rays' path from their original path and subsequent image formation on the image sensor is called pixel shift. The pixel shift caused by the presence of the windshield varies in amount depending on the position on the image sensor and appears as distortion in the captured image.

[0051] Figure 7 is a schematic diagram showing the change in the amount of vertical pixel shift that occurs when light rays pass through the windshield 200.

[0052] As described above, in this embodiment, the distance estimation of monocular viewing regions 111 and 112 performed by the hybrid 3D object detection unit 70 is carried out by detecting the ground contact position of the object in the vertical direction (up and down direction on the image) of the monocular viewing image, such as a person's feet. For this reason, the amount of pixel shift in the vertical direction of the image will affect the distance estimation. Furthermore, the distance estimation is performed using the distance measured in the stereo viewing region 110 near the central field of view in the horizontal direction. Consequently, if the amount of pixel shift in the monocular viewing image changes in relation to the amount of pixel shift in the stereo viewing region 110 near the central field of view, an error will occur in the estimated distance.

[0053] In Figure 7, the vertical axis of the graph shows the change in the amount of pixel shift in the vertical direction, with the amount of pixel shift at zero horizontal field of view as the reference. The range of the horizontal field of view indicated by arrow 401 shows the change in the amount of pixel shift in the stereo viewing region 110, and the range of the horizontal field of view indicated by arrow 402 shows the change in the amount of pixel shift in the monocular viewing regions 111 and 112. As shown in the figure, the change in the amount of pixel shift increases as the horizontal field of view increases, and it is understood that the rate of change is larger in the range of the horizontal field of view that is the monocular viewing region compared to the stereo viewing region. The image distortion caused by this change in the amount of pixel shift can cause errors in the estimation of distance in the monocular viewing regions 111 and 112 and can be a factor that hinders the normal operation of the image processing device 1. Therefore, in this embodiment, the amount of pixel shift is corrected in the affine processing unit 220.

[0054] The amount of pixel shift depends on the shape of the windshield 200, its tilt when mounted, and the mounting position of the camera. In particular, the amount of pixel shift increases when the radius of curvature of the windshield 200 is small or when the windshield 200 is mounted with a large tilt relative to the vertical. Furthermore, the radius of curvature and tilt of the windshield 200 vary from vehicle to vehicle in which the image processing device 1 is mounted, and the amount of pixel shift changes accordingly. Therefore, the amount of pixel shift correction needs to be calibrated for each vehicle. In this embodiment, the image calibration unit 90 acquires calibration information that is used in the affine processing unit 220 to obtain an appropriate correction amount.

[0055] Figure 8 is a schematic block diagram showing an example of the configuration of the image calibration unit 90. The image calibration unit 90 includes, for example, an image recording unit 91, a moving parallax image generation unit 92, a distance conversion unit 93, a movement amount acquisition unit 94, and a pixel shift correction amount calibration unit 95.

[0056] The image recording unit 91 records at least two images taken at different times for each of the cameras 10a and 10b. The two images are taken by the same camera 10 at positions shifted in the direction of the vehicle's movement. These images are recorded, for example, using a memory area (not shown) provided by the image processing device 1.

[0057] The moving parallax image generation unit 92 takes two images captured by the same camera 10 and recorded by the image recording unit 91, transforms them so that they are equivalent to images viewed from a direction perpendicular to the direction of vehicle movement, and performs matching processing in the same way as the stereo parallax image generation unit 30 to generate a stereo parallax image. The stereo parallax image generated here is an image that includes parallax information (moving parallax information) obtained by the matching processing, and hereafter this image will be referred to as the moving parallax image. The moving parallax image is a parallax image that includes parallax when viewed from a direction perpendicular to the direction of vehicle movement.

[0058] The distance conversion unit 93 generates a second distance image in the direction of motion based on the moving disparity image and the distance between the imaging points of the two images obtained by the movement amount acquisition unit 94. The specific method for generating the distance image based on the moving disparity image will be described later.

[0059] The movement amount acquisition unit 94 acquires the distance between two points where images used to generate the moving parallax image are captured as the movement amount of the camera 10. The distance between two points where images are captured can be acquired, for example, using distance information measured in the stereo viewing area. If the vehicle has other distance measuring devices that have the function of acquiring distance information, such as LiDAR (Light Detection And Ranging), millimeter-wave radar, or other camera devices, information on movement such as moving speed and moving distance may be obtained from these devices via the in-vehicle network. Alternatively, although less accurate, it is also possible to acquire the distance between two imaging points based on information such as the vehicle's speed and tire rotation speed obtained via the in-vehicle network, and information on the time when the images used to generate the moving parallax image were captured.

[0060] The pixel shift correction amount calibration unit 95 estimates the pixel shift amount based on the second distance information contained in the second distance image and the first distance information contained in the first distance image generated by the road surface cross-section shape estimation unit 50 based on the image used to obtain the second distance image.

[0061] Figure 9 is a schematic diagram illustrating the distance error between the first and second distance images.

[0062] Figure 9 shows an example where a plane parallel to the alignment direction of cameras 10a and 10b is imaged on the road surface at a predetermined distance from the vehicle. Arrow 500 indicates the horizontal field of view range corresponding to the stereo viewing area 110. Arrows 501 and 502 indicate the horizontal field of view ranges corresponding to the monocular viewing areas 111 and 112, respectively. The dashed line 510 shows the distance error of the first distance image, and the solid line 520 shows the distance error of the second distance image. As explained with reference to Figure 7, the first distance image has a vertical pixel shift, resulting in a distance error. On the other hand, the distance error of the second distance image is very small, as will be described later. Therefore, by calibrating the parameters used in the affine processing unit 220 using a pixel shift amount that matches the second distance image of the first distance image, and correcting the pixel shift of the image used to generate the first distance image in the affine processing unit 220, the first distance image can be made correct. This makes it possible to improve the accuracy of the distance estimated by the hybrid 3D object detection unit 70. Specifically, the pixel shift amount can be obtained by using feature points on the second depth image, such as a person's feet, the contact surface of a vehicle's tires, or a structure like a building, comparing these feature points with any point on the first depth image that has distance information indicating the same distance as the feature points, and calculating the difference in the vertical position (pixel coordinate position) between them on the image.

[0063] The above processing is performed using images captured by camera 10a and camera 10b, respectively, and the pixel shift correction amount is estimated for camera 10a and camera 10b. The pixel shift correction amount obtained in this way is sent as calibration information to image processing unit 20a and image processing unit 20b, respectively, and is used to calibrate the parameters stored in the affine table storage unit 230 of each image processing unit 20a and image processing unit 20b.

[0064] Furthermore, the amount of pixel shift can be obtained using features such as those on a person's feet, the contact patch of a vehicle's tires, or structures like buildings. In addition, if parameter calibration is performed at a car manufacturer's factory or a dealer's service center, patterns such as charts displayed on the road surface or charts standing upright on the ground may be used. Even in this case, since the estimation of the amount of pixel shift utilizes the movement of the camera position, the size of the chart used does not need to be large.

[0065] Figure 10 is a schematic diagram illustrating the acquisition of distance to an object using moving parallax images performed by the distance conversion unit 93. Here, we will explain the case where the distance to an object 600 captured in two images taken by camera 10a at times t1 and t2 as the vehicle moves is acquired. For simplicity, we will use an object 600 on the right side of the vehicle as an example, but the distance to an object on the left side of the vehicle can be determined in the same way. Similarly, the distance to an object captured by camera 10b can also be determined in the same way.

[0066] Now, let's consider camera 10a at time t1 as a virtual camera 610, and camera 10a at time t2 as a virtual camera 611. In this case, if we consider images captured by cameras 610 and 611 in a direction perpendicular to the direction of movement, these two images are composed of cameras 610 and 611, and the distance traveled is B. d This can be seen as an image captured by a stereo camera with a baseline length of . Therefore, the distance Ld from camera 611 to object 600 in a direction perpendicular to the direction of movement is,

[0067]

number

[0068] It can be obtained as follows. Here, d d is the parallax between the two images, and f is the focal length of cameras 610 and 611, i.e., the focal length of camera 10a. In reality, since the images captured by cameras 610 and 611 are images in the direction of movement, the captured images need to be transformed so that they are equivalent to images in the direction perpendicular to the direction of movement.

[0069] Then, if α1 is the angle formed by the line connecting camera 611 and object 600 with respect to the direction of vehicle movement, then the distance L from camera 611 to object 600 in the direction of movement is:

[0070]

number

[0071] This can be obtained as follows. Here, since the projection of the camera 10's lens is known, α1 can be determined from the position of the object 600 on the image. In this embodiment, the position of the object in the horizontal direction on the image is the position fsinα1 from the optical axis position on the image sensor of the camera 611, so α1 can be obtained based on the focal length and the distance from the optical axis position on the image to the object 600. Although α1 will have an error due to the effect of the windshield, it is a small amount and does not pose a problem.

[0072] In this embodiment, the moving parallax image generation unit 92 converts the image captured by the camera 10 to be equivalent to an image perpendicular to the direction of movement, thereby generating a moving parallax image. Based on this moving parallax image, d dThis can be obtained. The distance conversion unit 93 can use the moving parallax image to obtain the distance L based on equations 1 and 2 and generate a distance image corresponding to the moving parallax image. The distance conversion unit 93 converts the distance image thus generated to become an image in the direction of movement and generates a second distance image that includes distance information in the direction of movement (second distance information). Here, the distance is obtained using the image captured at time t2 as the reference image, but the distance from camera 610 to object 600 may be obtained using the image captured at time t1 as the reference image. Since the second distance image is generated based on the reference image, the first distance image that is compared with the second distance image by the distance conversion unit 93 also uses a distance image generated based on the reference image (an image captured at the same time as the reference image). The distance information contained in the second distance image generated in this way can be treated as correct for the reasons explained below.

[0073] Figure 11 is a schematic diagram showing the amount of pixel shift in the horizontal field of view of a stereo disparity image based on two images obtained by moving camera 10.

[0074] The graph in Figure 11 shows the relationship between the horizontal field of view position (horizontal axis) of the image captured by camera 10 and the amount of pixel shift in the horizontal field of view direction at that horizontal field of view position (vertical axis). The dashed line 700 shows the amount of pixel shift in the horizontal field of view of image g1 (corresponding to the image captured by camera 610 in Figure 10) captured before movement, and the dotted line 710 shows the amount of pixel shift in the horizontal field of view of image g2 (corresponding to the image captured by camera 611 in Figure 10) captured after movement. This amount of pixel shift is a factor that causes distance errors in the second distance information included in the second distance image generated by the distance conversion unit 93. The parallax error between the two images can be shown as the difference between the amount of pixel shift of image g1 and the amount of pixel shift of image g2 at the same horizontal field of view. The characteristics of the pixel shift amounts of image g1 and image g2 (dashed line 700 and dotted line 710) are shifted in the horizontal direction. This discrepancy corresponds to the change in the horizontal field of view to the object due to the movement of camera 10 (the difference between angles α1 and α0 in Figure 9). Since the change in the horizontal field of view is very small compared to the maximum horizontal field of view of the stereo camera composed of cameras 10a and 10b, the parallax error is small across the entire horizontal field of view, as shown by the solid line 620 in Figure 12.

[0075] Figure 13 is a graph showing an example of the relationship between the field of view and the difference in the field of view to the object in two images. Here, in Figure 10, B d Let L be 1m. d The relationship between the angle of view (angle α1) and the difference in the angle of view (angle α1 - angle α0) to the object in the horizontal direction when the distance is changed to 30, 50, and 70m is shown. From Figure 13, L d It can be seen that using a larger object reduces the difference in horizontal field of view. Furthermore, the difference decreases as the horizontal field of view increases. As shown in Figure 11, the amount of pixel shift is greater in the wide-angle region than in the center of the field of view, but in the wide-angle region the difference in horizontal field of view between the two images is small, so as shown in Figure 12, there is almost no parallax error caused by the difference in field of view.

[0076] From the above, the distance indicated by the depth image obtained based on the moving parallax image is approximately correct. Therefore, by calibrating the parameters stored in the affine table with the pixel shift correction amount acquired by the image calibration unit 90, it becomes possible to measure distances with high accuracy over a wide field of view.

[0077] In this explanation, we have used a stereo camera that images the area in front of the vehicle as an example, and therefore described the direction of movement of camera 10 as being perpendicular to the direction in which the two cameras 10 are aligned. However, the direction of camera movement is not limited to this. For example, in the case of a stereo camera that images the area to the side of the vehicle using camera 10, the movement may be in the direction in which the cameras 10 are aligned, as shown in Figure 14. Alternatively, the movement may be in a direction different from the direction in which the cameras 10 are aligned, as shown in Figure 15. In these cases as well, the same effects as described above can be obtained.

[0078] As described above, according to this embodiment, the amount of pixel shift in the monocular view image captured by the stereo camera, which has a wide field of view achieved by shifting the imaging areas of the two cameras constituting the monocular view stereo camera, can be calibrated using the camera's moving parallax, making it possible to perform highly accurate measurements over a wide field of view range. Since the calibration of the pixel shift amount is performed using the camera's moving parallax, it can be done without using a large chart that covers a wide field of view range for calibration.

[0079] In this embodiment, the distance the camera moves as the vehicle travels is used as the baseline length for obtaining a moving parallax image. However, in places such as an automobile manufacturer's production site or a dealer's service center, a moving parallax image may be obtained by fixing the vehicle with a jig and moving it by a predetermined amount. In this case, the amount of vehicle movement measured externally may be provided to the movement amount acquisition unit 94 from an external device that can be connected to the in-vehicle network.

[0080] The image processing device 1 of this embodiment can be applied, for example, to an automatic emergency braking (AEB), which is one of the safety functions for preventing or mitigating traffic accidents. In AEB, objects around the vehicle are detected, the distance to those objects is measured, their movement is tracked to estimate their speed, and control such as warnings and braking is performed. In particular, for moving objects, it is preferable to be able to detect their presence over a wider range and measure their distance and speed. The image processing device of this embodiment can accurately measure the distance of objects in a wide-angle range and can be used for more appropriate control such as warnings and braking.

[0081] Figure 16 is a schematic block diagram showing the logical configuration of the image processing device in the second embodiment. The image processing device 2 in this embodiment also has basically the same configuration as the image processing device 1 in the first embodiment. Therefore, in Figure 16, the same reference numerals as in Figure 1 are used for parts that have the same functions and configurations as in the first embodiment, and redundant explanations are omitted except where particularly necessary.

[0082] In the first embodiment, the image processing apparatus 1 performed pixel shift correction in the image processing unit 20, but in this embodiment, the image processing apparatus 2 performs pixel shift correction in the hybrid 3D object detection unit 71.

[0083] The image processing unit 21 of this embodiment basically has the same configuration as the image processing unit 20 shown in Figure 3. However, the affine table stored in the affine table storage unit 230 does not include parameters used to correct pixel shift. Consequently, the affine transformation process performed by the affine processing unit 220 does not correct pixel shift. In other words, the affine processing unit 220 of the image processing unit 21 does not function as a pixel shift correction processing unit that corrects pixel shift.

[0084] The hybrid 3D object detection unit 71, similar to the hybrid 3D object detection unit 70 in the first embodiment, generates a hybrid image by combining a first distance image generated by the road surface cross-section shape estimation unit 50 with left and right monocular images generated by the monocular image generation unit 40. The hybrid 3D object detection unit 71 stores the pixel shift amount as correction information used to correct the pixel shift in the monocular region described above. When the hybrid 3D object detection unit 71 acquires the distance to a 3D object in the monocular region based on the hybrid image, it uses the stored pixel shift amount to correct the pixel shift of the monocular image and calculates the distance. The pixel shift amount held by the hybrid 3D object detection unit 71 is calibrated by the pixel shift amount estimated by the image calibration unit 90. As a result, the hybrid 3D object detection unit 71 can obtain highly accurate distance information.

[0085] In the first embodiment, the pixel shift is directly corrected by the affine transformation of the image captured by the camera 10, whereas in this embodiment, as described above, the correction is performed during distance calculation to obtain the distance to a three-dimensional object based on the hybrid image. Alternatively, instead of correcting the pixel shift during distance calculation, the system may be configured to correct the pixel shift of the left and right monocular images when generating the hybrid image.

[0086] In this embodiment as well, by shifting the imaging areas of the two cameras constituting a monocular stereo camera to widen the field of view, the amount of pixel shift in the monocular image captured by the stereo camera is calibrated using the camera's moving parallax, making it possible to perform highly accurate measurements over a wide field of view.

[0087] Figure 17 is a schematic block diagram showing the logical configuration of the image processing apparatus in the third embodiment. The image processing apparatus 3 in this embodiment also basically has the same configuration as the image processing apparatus in the first and second embodiments. For this reason, in Figure 17, the same reference numerals as in Figure 1 or Figure 16 are used for parts that have the same functions and configurations as in the first or second embodiment, and redundant explanations are omitted unless particularly necessary.

[0088] In the first and second embodiments, cameras 10a and 10b with the same field of view are used to form a stereoscopic and monocular viewing area by shifting their fields of view (imaging area). However, in this embodiment, the field of view of the right camera and the left camera are different, with a wide-angle camera 11 on the right and a narrow-angle camera 12 on the left having a narrower field of view than the wide-angle camera 11. Note that the relative positions of the wide-angle camera 11 and the narrow-angle camera 12 may be reversed.

[0089] Figure 18 is a schematic diagram showing the imaging area in this embodiment.

[0090] The wide-angle camera 11 and the narrow-angle camera 12 are installed such that the field of view captured by the wide-angle camera 11 encompasses the field of view captured by the narrow-angle camera 12. In this embodiment, the area 113 that can be captured by both the wide-angle camera 11 and the narrow-angle camera 12 is used as the stereoscopic viewing area, and the area 114 that is outside the field of view captured by the narrow-angle camera 12 and is captured only by the wide-angle camera 11 is used as the monocular viewing area.

[0091] The wide-angle camera image processing unit 22 is configured in the same way as the image processing unit 20 in the first embodiment. That is, the affine table stored in the affine table storage unit of the wide-angle camera image processing unit 22 includes parameters for correcting pixel shift in the parameters used for affine processing, and an image with corrected pixel shift is generated when affine transformation is performed in the affine processing unit. The parameters used for correcting pixel shift are calibrated based on the amount of pixel shift estimated by the image calibration unit 90, as in the first embodiment.

[0092] On the other hand, the narrow-angle camera image processing unit 23 is configured similarly to the image processing unit 21 in the second embodiment, and the affine processing unit of the narrow-angle camera image processing unit 23 does not function as a pixel shift correction processing unit. Basically, the imaging area of ​​the image captured by the narrow-angle camera 12 does not include the area outside the field of view of the image captured by the wide-angle camera 11, so the image processed by the narrow-angle camera image processing unit 23 does not include the monocular view area used for distance measurement. For this reason, pixel shift correction is unnecessary in the narrow-angle camera image processing unit 23. The image processed by the narrow-angle camera image processing unit 23 is sent only to the stereo disparity image generation unit 30 and not to the monocular view image generation unit 40.

[0093] The monocular image generation unit 40 receives only images processed by the wide-angle camera image processing unit 22 and processes only images captured by the wide-angle camera 11. However, the image processing itself is the same as in the first and second embodiments. The same applies to the hybrid 3D object detection unit 70.

[0094] Furthermore, the image calibration unit 90 will only estimate the pixel shift correction amount for images captured by the wide-angle camera 11, but the process of estimating the pixel shift correction amount using moving parallax is no different from that of the first and second embodiments. In this embodiment, the pixel shift amount estimated by the image calibration unit 90 is only for images captured by the wide-angle camera 11, and is sent only to the wide-angle camera image processing unit 22, where it is used to calibrate the parameters held by the wide-angle camera image processing unit 22.

[0095] This embodiment also allows for the calibration of the parameters used to correct the pixel shift of images in the monocular field of view captured by a wide-angle camera, using the amount of pixel shift estimated using the camera's moving parallax, thereby enabling highly accurate measurements over a wide field of view.

[0096] Although the present invention has been described above using representative embodiments as examples, the present invention is not limited to these embodiments. For example, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, for each embodiment, it is possible to delete or replace components, or to add other configurations.

[0097] For example, in the third embodiment, pixel shift correction is performed in the wide-angle image processing unit, but this can also be done in the hybrid 3D object detection unit, similar to the second embodiment. In addition, in each of the embodiments described above, a moving parallax image is generated based on the monocular image processed by the monocular image generation unit 40, but for example, the same processing as the monocular image generation unit 40 may be performed in the brightness information generation unit 260 of the image processing unit 20.

[0098] In the embodiments described above, the correction of pixel shift due to the influence of the windshield was explained as an example, but it goes without saying that image distortion due to other factors can also be corrected. For example, image distortion due to misalignment of the lens and sensor, and image distortion due to changes in magnification due to lens temperature can also be corrected in the same way. When the position of the lens and sensor shifts due to changes over time and temperature, the correction in the affine processing unit becomes inadequate, and image distortion occurs. Also, when the lens temperature changes, the magnification of the image changes, and image distortion occurs. Pixel shift caused by these factors can also be corrected based on the amount of pixel shift estimated by the image calibration unit. The image processing device may continuously perform calibration by the image calibration unit after being started up. Furthermore, if the image processing device is configured to acquire environmental information such as the temperature of the camera and lens, and the humidity inside and outside the vehicle, it may be configured to detect when the environment has changed by a predetermined amount based on the environmental information and perform calibration by the image calibration unit.

[0099] Furthermore, in the embodiments described above, distance information in the monocular view region is obtained based on distance information in the stereo view region, but distance information obtained through stereo viewing is not necessarily required. For example, the distance to an object may be obtained based on its vertical position in the monocular view image, or based on the size of people or cars in the image. In these cases as well, pixel shift correction and calibration of the correction amount can be performed based on distance information obtained from the moving parallax image and the position and size of objects in the image. Therefore, the present invention is not limited to stereo cameras and can also be applied to monocular cameras. In addition, the direction of pixel shift is not limited to the vertical direction, but may also be applied to horizontal pixel shift.

[0100] Thus, the present invention can be implemented in various ways without departing from the spirit of the invention as described in the claims. [Explanation of symbols]

[0101] 1, 2, 3…Image processing device, 10a, 10b…Camera, 11…Wide-angle camera, 12…Narrow-angle camera, 20a, 20b, 21a, 21b…Image processing unit, 22…Wide-angle camera image processing unit, 23…Narrow-angle camera image processing unit, 30…Stereo disparity image generation unit, 40…Monocular image generation unit, 50…Road surface cross-section shape estimation unit, 60…Stereo 3D object detection unit, 70, 71…Hybrid 3D object detection unit, 80…Alarm control unit, 90…Image calibration unit, 91…Image recording unit, 92…Stereo disparity image generation unit, 93…Distance conversion unit, 94…Movement amount acquisition unit, 95…Pixel shift correction amount calibration unit

Claims

1. An image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, An image processing unit that acquires images captured by a first camera and a second camera mounted on the mobile body, A stereo disparity image generation unit acquires disparity information relating to the disparity between the first imaging region captured by the first camera and the first imaging region captured by the second camera, with respect to the first imaging region captured in both images taken by the first camera and the second camera, respectively. A distance image generation unit that obtains the distance to a three-dimensional object in the first imaging area based on the parallax information and generates a first distance image that includes first distance information relating to that distance, A hybrid three-dimensional object detection unit generates a hybrid image including the first imaging region and the second imaging region from the first distance image and an image of a second imaging region captured by either the first camera or the second camera but not by the other camera, and acquires the distance to the three-dimensional object captured in the second imaging region based on the first distance information. A moving parallax image generation unit generates a moving parallax image that includes moving parallax information relating to the parallax between the first image and the second image, based on a first image captured at a first time including the second imaging region and a second image captured at a second time different from the first time including the second imaging region, using either the first image or the second image as a reference image. A movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time, A distance conversion unit generates a second distance image which includes second distance information relating to the distance to a three-dimensional object captured in the second imaging area of ​​the reference image based on the moving parallax image and the amount of movement, A calibration unit generates calibration information used to calibrate correction information for correcting distortion of an image captured by the first camera or the second camera that captured an image including the second imaging region, based on the first distance image generated based on the reference image and the second distance image, An image processing device having

2. The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit acquires the distance to the three-dimensional object based on the position of the three-dimensional object on the hybrid image.

3. The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit acquires the distance to the three-dimensional object based on the size of the object in the image.

4. The image processing apparatus according to claim 1, wherein the movement amount acquisition unit acquires information regarding the movement of the moving body measured by another distance measuring device mounted on the moving body, and acquires the movement amount based on the information regarding the movement.

5. The image processing apparatus according to claim 1, wherein the movement amount acquisition unit acquires information regarding the movement of the moving body measured outside the moving body, and acquires the movement amount based on the information regarding the movement.

6. The image processing apparatus according to claim 1, wherein the processing by the moving disparity image generation unit, the distance conversion unit, and the calibration unit is configured to be performed in accordance with changes in the environment inside and outside the moving body.

7. The image processing apparatus according to claim 6, wherein the change in the environment is a change in humidity.

8. The image processing apparatus according to claim 6, wherein the change in the environment is a change in the temperature of the first camera or the second camera.

9. The image processing apparatus according to claim 1, wherein the moving disparity image generation unit converts the first image and the second image from images in the direction of movement of the moving body to images in a direction perpendicular to the direction of movement to generate the moving disparity image, and the moving disparity information is the disparity in the perpendicular direction.

10. The image processing apparatus according to claim 9, wherein the distance conversion unit obtains a distance in the perpendicular direction, which is the distance to the three-dimensional object in the perpendicular direction, based on the moving parallax information, and obtains a distance to the three-dimensional object in the moving direction based on the said distance in the perpendicular direction.

11. The image processing apparatus according to claim 10, wherein the calibration unit generates the calibration information based on the difference between the position of the first distance information, which corresponds to the second distance information, on the first distance image and the position of the three-dimensional object on the second distance image.

12. The image processing apparatus according to claim 1, wherein the image processing apparatus holds parameters including correction information that is calibrated by the calibration information, performs image conversion processing using the parameters, and corrects the distortion of the image.

13. The image processing apparatus according to claim 1, wherein the hybrid three-dimensional object detection unit is configured with the calibration information, holds the correction information for correcting image distortion in the second imaging region included in the hybrid image, and uses the correction information to obtain the distance to the three-dimensional object captured in the second imaging region.

14. An image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, An image processing unit that acquires images from a camera mounted on the aforementioned mobile body, A distance calculation unit that obtains the distance to a three-dimensional object based on the position of the three-dimensional object in the image, A moving parallax image generation unit generates a moving parallax image that includes moving parallax information relating to the parallax between the first image and the second image, based on a first image captured at a first time and a second image captured at a second time different from the first time, using either the first image or the second image as a reference image. A movement amount acquisition unit that acquires the amount of movement of the moving body between the first time and the second time, A distance conversion unit generates a distance image that includes distance information relating to the distance to a three-dimensional object depicted in the reference image, based on the moving parallax image and the amount of movement. A calibration unit generates calibration information used to calibrate correction information for correcting distortion in an image captured by the camera, based on the position of the three-dimensional object on the reference image and the distance information. An image processing device having

15. An image calibration method for calibrating an image processing device mounted on a mobile body for recognizing the surrounding conditions of the mobile body, Images captured by the first camera and the second camera mounted on the mobile body are acquired. For the first imaging region captured in both of the two images captured by the first camera and the second camera, disparity information relating to the disparity between the image captured by the first camera and the image captured by the second camera is acquired. Based on the parallax information, the distance to the three-dimensional object within the first imaging area is obtained, and a first distance image is generated that includes first distance information relating to that distance. Based on a first image captured at a first time, which includes a second imaging region captured by either the first camera or the second camera but not captured by the other camera, and a second image captured at a second time different from the first time, which includes the second imaging region, a moving parallax image is generated, which includes moving parallax information relating to the parallax between the first image and the second image, with either the first image or the second image as the reference image. The amount of movement of the moving body between the first time and the second time is obtained, Based on the moving disparity image and the amount of movement, a second distance image is generated which includes second distance information relating to the distance to the three-dimensional object captured in the second imaging area of ​​the reference image. Based on the first distance image and the second distance image generated based on the aforementioned reference image, calibration information is generated that is used to calibrate correction information for correcting distortion of images captured by the first or second camera. Image calibration methods.

Images