Image processing device and image processing method

Abstract

To provide an image processing device capable of correcting images output from two cameras in accordance with change in a stereo camera (two cameras).SOLUTION: An image processing device 1 includes an image comparison unit 30 and a distortion conversion processing unit 50. The image comparison unit compares a first image detected by a first image sensor 12 and a second image detected by a second image sensor 22 input from a stereo camera including a first camera 10 and a second camera 20 arranged side by side in a first direction and calculates a correction processing coefficient for correcting the second image on the basis of difference between the first image and the second image in a second direction different from the first direction. The distortion conversion processing unit corrects the second image by the correction processing coefficient calculated by the image comparison unit 30.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Stereo camera technology has traditionally been used as a three-dimensional object recognition technology. Stereo camera technology uses the difference in the appearance of images captured by two cameras (called "stereo cameras") placed at different positions to detect parallax based on trigonometry, and then uses this parallax to detect the depth and position of an object. Stereo camera technology makes it possible to accurately detect the position of an object. For this reason, stereo camera technology is applied to in-vehicle cameras, cameras mounted on robots, and other devices to detect and recognize three-dimensional objects. The images from the two cameras of a stereo camera are transformed into a predetermined projection using, for example, the central projection method, and the distance to an object can be measured by detecting the difference between the transformed images.

[0003] High-precision distance detection requires highly accurate alignment of two images with a predetermined projection. Therefore, if the image captured by the camera changes due to factors such as temperature changes or aging of the camera or lens, distance measurement errors will occur. For example, with plastic lenses, which have the advantages of low cost and asphericity, the detected image changes with temperature, causing distance measurement errors as the plastic lens changes temperature. Furthermore, even with glass lenses, fisheye lenses with a wide-angle field of view experience significant image changes over a wide angle of view, resulting in distance measurement errors.

[0004] To address the issue of distance measurement errors caused by changes in lens temperature, the following technologies are known, as disclosed in Patent Documents 1 and 2. For example, Patent Document 1 states that "for camera information that changes depending on the zoom and focus states of each imaging means, the camera information storage means holds multiple pieces of camera information corresponding to those states, and the stereo image processing means acquires the zoom and focus states of each imaging means, acquires the corresponding camera information from the camera information storage means, and uses it for stereo image processing."

[0005] Furthermore, Patent Document 2 states that "based on the temperature detected by the temperature sensor, the distortion correction processing unit reads out from the correction coefficient storage unit the correction coefficient that minimizes the index value (amount of deviation) indicating the degree of distortion of the captured image at that detected temperature, and performs distortion correction processing using the read correction coefficient according to the above-mentioned distortion correction formula." [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-241491 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-147281 Summary of the Invention [Problem to be solved by the invention]

[0007] Patent Document 1 describes how temperature-related image shifts caused by lens expansion and contraction and structural deformation can be corrected by acquiring temperature information from the first and second imaging units and acquiring corresponding correction information from a camera information storage unit along with the camera information. However, the process described in Patent Document 1 requires temperature-dependent correction information to be stored in a storage device, which increases the storage capacity of the storage device and poses a cost issue. Furthermore, when correcting distortion in an image captured by a camera, the correction information is read from the storage device, so storage devices compatible with both cameras must be secured in advance. In particular, 3D measurement devices are increasingly using higher pixel count cameras to achieve high resolution and a wide field of view. This means that the amount of information, such as correction information, is constantly increasing, necessitating larger storage capacities for storage devices.

[0008] Furthermore, the image processing device described in Patent Document 2 uses a polynomial to reduce the amount of data stored in the storage device, which addresses the problem of increased storage capacity that was identified in Patent Document 1. Therefore, even though the storage capacity of the storage device can be reduced, the image processing device described in Patent Document 2 has the same problem as the device described in Patent Document 1 in that it must secure storage devices corresponding to the two cameras.

[0009] Furthermore, the technologies described in Patent Documents 1 and 2 have serious issues with correction accuracy. The temperatures detected by the temperature detection unit described in Patent Document 1 and the temperature sensor described in Patent Document 2 do not directly indicate the temperature of the lens. For example, if the lens is exposed to sunlight or heat from the camera processing circuit is transferred to the lens, the temperature inside the lens may differ from the temperature outside. In this case, the temperatures of the two cameras may differ, which may result in errors when correcting images detected by the two cameras. Furthermore, the temperature characteristics of actual lenses vary, and the same temperature change may not necessarily result in the same characteristics. This makes it difficult to detect changes in images from temperature changes. Furthermore, if the temperatures of the two cameras change, the sensitivity of the sensors built into the cameras also changes, resulting in different images detected by each sensor.

[0010] The present invention has been made in view of the above circumstances, and has an object to correct images output from two cameras in accordance with changes in the stereo camera (two cameras). [Means for solving the problem]

[0011] The image processing device according to the present invention compares a first image detected by a first image sensor of the first camera with a second image detected by a second image sensor of the second camera, the first image being input from a stereo camera having a first camera and a second camera arranged side by side in a first direction, and detects an image in a second direction different from the first direction. A difference between the displacement amount of the image in the region where the first image extends in the second direction and the displacement amount of the image in the region where the second image extends in the second direction is calculated. Difference between the first and second images The difference between the first and second images is The second image based on Eliminate the difference in the amount of displacement of the region extending in the second direction. correction is the magnification for The image processing device includes an image comparison unit that calculates a correction processing coefficient, and an image processing unit that corrects the second image using the correction processing coefficient calculated by the image comparison unit. [Effects of the Invention]

[0012] According to the present invention, images output from the stereo camera (two cameras) can be corrected in accordance with changes in the two cameras. Problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing an example of the internal configuration of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 3 is a diagram illustrating a correction algorithm according to the first embodiment of the present invention. [Figure 3] FIG. 3 is a diagram showing a change in an image on a sensor with a change in temperature according to the first embodiment of the present invention. [Figure 4] 3A to 3C are diagrams illustrating a method for correcting the difference in changes between two lenses according to the first embodiment of the present invention. [Figure 5]4 is a graph showing k dependency of the magnification error signal MES according to the first embodiment of the present invention. [Figure 6] 3A and 3B are diagrams illustrating the relationship between the positions of lenses and sensors according to the first embodiment of the present invention. [Figure 7] 10A and 10B are diagrams illustrating the effect of correcting the image height displacement amount and the image height displacement amount difference when the second image sensor according to the first embodiment of the present invention moves in the Y direction. [Figure 8] 10A and 10B are diagrams illustrating the effect of correcting the image height displacement amount and the image height displacement amount difference when the second image sensor according to the first embodiment of the present invention is rotated around the Y axis as the rotation axis. [Figure 9] 10A and 10B are diagrams illustrating the effect of correcting the image height displacement amount and the image height displacement amount difference when the image height displacement amount changes nonlinearly with respect to the vertical angle of view according to the first embodiment of the present invention. [Figure 10] 3A and 3B are diagrams showing an example of displacement of image height in the vertical direction at each position in the horizontal direction of the first lens and the second lens according to the first embodiment of the present invention. [Figure 11] 3A to 3C are diagrams showing examples of various regions detected by two sensors and used to calculate an MES according to the first embodiment of the present invention. [Figure 12] 1 is a block diagram showing an example of the hardware configuration of a computer according to a first embodiment of the present invention. [Figure 13] 4 is a flowchart illustrating an example of processing performed by the image processing device according to the first embodiment of the present invention. [Figure 14] FIG. 10 is a block diagram showing an example of the internal configuration of an image processing apparatus according to a second embodiment of the present invention. [Figure 15] FIG. 10 is a block diagram showing an example of the internal configuration of an image processing device according to a third embodiment of the present invention. [Figure 16] FIG. 10 is a block diagram showing an example of the internal configuration of an image processing device according to a fourth embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functions or configurations are designated by the same reference numerals, and redundant description will be omitted. The present invention is applicable to, for example, a computing device for vehicle control capable of communicating with an on-board ECU (Electronic Control Unit) for an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD).

[0015] [First embodiment] FIG. 1 is a block diagram showing an example of the internal configuration of an image processing device 1 according to the first embodiment of the present invention.

[0016] Image processing device 1 performs predetermined processing on images input from a stereo camera having a first camera (first camera 10) and a second camera (second camera 20) arranged side by side in a first direction. In this stereo camera, first camera 10 and second camera 20 are arranged side by side in a horizontal direction (first direction).

[0017] The first camera 10 has a first lens 11 and a first image sensor 12. The first camera 10 detects an image of an object with the first image sensor 12, on which image light is focused via the first lens 11. The first image detected by the first image sensor (first image sensor 12) of the first camera (first camera 10) is subjected to predetermined processing by each subsequent functional unit.

[0018] The second camera 20 includes a second lens 21 and a second image sensor 22. The second camera 20 detects an image of an object with the second image sensor 22, on which image light is focused via the second lens 21. The second image detected by the second image sensor (second image sensor 22) of the second camera (second camera 20) is subjected to predetermined processing by each subsequent functional unit.

[0019] In the following description, when there is no need to distinguish between the first lens 11 and the second lens 21, they will be collectively referred to as "lenses." Similarly, when there is no need to distinguish between the first image sensor 12 and the second image sensor 22, they will be collectively referred to as "sensors." Furthermore, the first camera 10 and the second camera 20 may be collectively referred to as "two cameras." Furthermore, the first image detected by the first image sensor 12 may be referred to as "the first image detected by the first camera 10." Similarly, the second image detected by the second image sensor 22 may be referred to as "the second image detected by the second camera 20."

[0020] The first image detected by the first camera 10 is input to the distortion conversion processing unit 50A, which converts the distortion of the first image. The second image detected by the second camera 20 is input to the distortion conversion processor 50B. The distortion conversion processor 50B then converts the distortion of the second image. When the distortion conversion processors 50A and 50B are not distinguished, they are referred to as the distortion conversion processor 50. The image processing unit according to the first embodiment is a distortion conversion processor (distortion conversion processor 50) that converts the distortion of the first image and the distortion of the second image. For example, if the lens projection method is orthogonal projection (f sin θ), the distortion conversion processors 50A and 50B convert the distortion of the first image and the second image by performing a projection transformation from orthogonal projection to central projection (f tan θ) on the first image and the second image, respectively. However, the effects of the present invention can be obtained even if the images after projection transformation are other than central projection (f tan θ).

[0021] For example, the magnification of the second image changes with respect to the first image due to a change in the relative temperature of the second lens 21 of the second camera 20 with respect to the temperature of the first lens 11 of the first camera 10. Even if the temperatures of the first lens 11 of the first camera 10 and the second lens 21 of the second camera 20 are the same, the magnification of the second image changes with respect to the first image due to differences in the sensitivity of the first lens 11 of the first camera 10 and the second lens 21 of the second camera 20 to changes in magnification with respect to temperature. To accommodate this change in magnification, the image processing unit (distortion conversion processing unit 50B) corrects the second image using a correction processing coefficient k calculated by the image comparison unit (image comparison unit 30). Correcting the second image with the correction processing coefficient k is a process for correcting the change in magnification of the second image.

[0022] An image comparison unit 30 is provided downstream of the two cameras. A first image is input to the image comparison unit 30 from the first camera 10, and a second image is input to the image comparison unit 30 from the second camera 20. The image comparison unit 30 compares the first image with the second image, and calculates a correction processing coefficient k for correcting the second image based on the difference between the first image and the second image in a second direction different from the first direction. In this embodiment, the substantially horizontal direction in which the two cameras are aligned is defined as the first direction, and the substantially vertical direction relative to the first direction and the second direction are defined as the second direction. However, as will be described later, the second direction is not limited to the substantially vertical direction, and may be a direction tilted up to approximately 45 degrees relative to the horizontal direction (see (5) and (6) in FIG. 11, which will be described later).

[0023] The image comparison unit 30 compares the first and second images in a direction substantially perpendicular to the alignment direction of the two cameras and outputs the comparison result to the distortion conversion processing unit 50B. In this embodiment, the image size of the second image can be corrected by multiplying the entire second image by a predetermined magnification represented by the correction processing coefficient k. Therefore, a difference in image size of a predetermined region of the second image relative to a predetermined region of the first image is equivalent to a difference in magnification of the second image relative to the first image. Therefore, to calculate the correction processing coefficient k, the image comparison unit 30 extracts regions of predetermined sizes from each of the first and second images. The size of the extracted regions may be the size of a portion of an image of an object that is commonly captured in both images, as shown in FIG. 3 (described later), or the size of the entire image for each of the two images, as shown in (1) and (2) of FIG. 11 (described later).

[0024] Here, it is assumed that the areas extracted from the first image and the second image are partial areas extracted from the first image and the second image (for example, areas R1 and R2 shown in FIG. 3, which will be described later). Then, the image comparison unit 30 calculates an image magnification by multiplying the second image by a predetermined magnification so that the image sizes of the first image and the second image match. In the following description, this image magnification is referred to as the "correction processing coefficient k." Then, the distortion conversion processing unit 50B corrects the change in magnification of the second image using the correction processing coefficient k input from the image comparison unit 30. A detailed description of the correction processing coefficient k will be given later.

[0025] A brightness correction unit 51, an image interpolation unit 52, and a brightness information generation unit 53 are provided downstream of the distortion conversion processing unit 50. The brightness correction unit 51, the image interpolation unit 52, and the brightness information generation unit 53 are provided for a path that performs predetermined processing on the image after distortion conversion output from the distortion conversion processing unit 50A, and a path that performs predetermined processing on the image after distortion conversion and magnification change correction output from the distortion conversion processing unit 50B, respectively.

[0026] As the brightness correction process, the brightness correction unit 51 performs a process of correcting, for example, the difference in gain between the two cameras and the difference in gain between the pixels in the first camera 10 and the pixels in the second camera 20. This correction makes the brightness of the same object captured in the first image and the second image the same. The image interpolation unit 52 performs, as the image interpolation process, for example, demosaicing process for converting a RAW image into a color image.

[0027] The luminance information generating unit 53 converts, for example, the first and second color images into luminance images as a process for generating luminance information to enable generation of parallax images by the downstream parallax image generating unit 60. When converting into luminance images, luminance information is generated for each of the first and second images.

[0028] By performing the above processing on the first image and the second image, a first processed image and a second processed image are obtained and output from the luminance information generation unit 53. In the image processing device 1, both the first processed image and the second processed image are the same as the luminance image, but since predetermined processing is performed on each image, they are called "processed images."

[0029] The parallax detection unit (parallax image generation unit 60) detects the parallax of an object appearing in the first image and the second image, based on a first image (i.e., a first processed image) in which at least the distortion has been converted and luminance information has been generated, and a second image (i.e., a second processed image) in which at least the distortion has been converted, luminance information has been generated, and magnification has been converted. Here, the object appearing in the first image and the second image, for which the parallax image generation unit 60 detects the parallax, appears in the first processed image processed based on the first image and the second processed image processed based on the second image. Therefore, the parallax image generation unit 60 generates parallax images for detecting the parallax of the object, based on the first processed image and the second processed image.

[0030] The distance calculation unit 70 calculates the distance to the object appearing in the first image and the second image based on the parallax (the above-described parallax image) detected by the parallax image generation unit 60. Then, the distance calculation unit 70 outputs the distance to the object to a vehicle control unit (not shown) or the like. A vehicle control unit (not shown) performs control such as steering the vehicle to change the direction of travel or stopping the vehicle in front of the object based on the calculation result of the distance to the object.

[0031] This image processing device 1 is characterized in that the image comparison unit 30 compares images captured by two cameras, and the distortion conversion processing unit 50B uses the comparison results to perform correction processing on the second image. In contrast, the techniques described in the above-mentioned prior art documents 1 and 2 differ from the technique of this embodiment in their approach to correcting distance measurement errors. In the techniques described in prior art documents 1 and 2, distortions are corrected for both the first and second images so that the images output from the two cameras are under a predetermined temperature condition in response to temperature changes in the two lenses. However, the techniques described in prior art documents 1 and 2 have problems, such as low image correction accuracy because the temperature measured by the thermometer is not the lens temperature, and inability to correct images taking into account variations in image displacement amounts due to variations between lenses in the amount of image displacement relative to temperature. In contrast, the image processing device 1 of this embodiment can correct images using the actual image misalignment detected by the two cameras, thereby improving image correction accuracy.

[0032] Next, the manner in which an image captured by a camera changes with temperature changes and the correction algorithm according to this embodiment will be described. Fig. 2 is a diagram illustrating a correction algorithm according to this embodiment. The upper part of Fig. 2 shows how the first image sensor 12 detects an image of the object 100, represented by a black dot, via the first lens 11, and how the second image sensor 22 detects the image of the object 100 via the second lens 21. Here, it is assumed that the first camera 10 is installed on the right side of the vehicle, and the second camera 20 is installed on the left side, relative to the forward direction of the vehicle. Furthermore, the object 100 may be, for example, a moving vehicle, a person, or a structure such as a traffic light or a pole installed on the road surface.

[0033] Also, the lower part of Fig. 2 shows (1) an example of an image on the sensor and (2) an example of an image after correction processing. Fig. 2 (1) shows an example of an image of the object 100 detected on the first image sensor 12 and the second image sensor 22. Fig. 2 (2) shows an example of an image corrected by applying the correction algorithm according to this embodiment to the image shown in Fig. 2 (1).

[0034] First, we will explain the mechanism by which distance measurement errors occur when the lens temperature of the first lens 11 changes. The dashed-dotted lines show light ray 110 from object 100 that enters first camera 10 (first lens 11) and light ray 120 from object 100 that enters second camera 20 (second lens 21) when measured under specified temperature conditions. When measured under specified temperature conditions, the distance D0 to the object is detected from the difference (parallax) between the light ray incident positions on first image sensor 12 and second image sensor 22. This is the same distance as the actual distance.

[0035] Here, the dashed line indicates a light ray 111 of the object 100 that is incident on the first camera 10 when the temperature of the first lens 11 changes. When the temperature of the first lens 11 changes, the relationship between the image height and the angle of view of the first lens 11 shifts due to the influence of changes in the refractive index of the first lens 11 and changes in the lens surface spacing. In this case, the first lens 11 converts the light ray 110 into the light ray 111, and the position of the image on the first image sensor 12 is detected as shifted in the direction of arrow 19. If the distance measurement process for the object 100 is the same as the process before the temperature of the first lens 11 changes, the light ray from the object 100 will be erroneously detected as being in the angle of view of the virtual light ray 112 indicated by the two-dot chain line.

[0036] The dashed line also indicates a light ray 121 of the object 100 that is incident on the second camera 20 when the temperature of the second lens 21 remains unchanged. The second lens 21 converts the light ray 120 into a light ray 121, and the position of the image on the second image sensor 22 is detected as being shifted in the direction of arrow 29. The intersection of the light ray 120 and the virtual light ray 112 is erroneously detected as the position of the object 100. Based on this position, a distance D1 is calculated from the first lens 11 and the second lens 21 as base points, and the distance D1 is output from the image processing device 1. Here, as shown in FIG. 2, the correct distance D0 to the object 100 does not match the erroneously calculated distance D1. This is the mechanism by which a distance measurement error occurs due to a change in the temperature of the first lens 11.

[0037] Next, the correction algorithm according to this embodiment will be described. 2(1) shows that when the temperature of the first lens 11 changes, the position of the image of the object 100 is detected as shifted in the direction of arrow 19 on the first image sensor 12. On the other hand, on the second image sensor 22, the position of the image of the object 100 is shifted and not detected.

[0038] By applying the correction algorithm according to this embodiment, correction information (a correction processing coefficient k, which will be described later) is generated based on two images detected in the image on the sensor shown in (1) of Fig. 2. In this embodiment, the magnification of the image height (image size) of the object 100 detected by the second image sensor 22 is converted based on the correction information.

[0039] 2(2), the image of object 100 on second image sensor 22 is corrected by being shifted in the direction of arrow 29, which is the same direction as arrow 19. When the image of object 100 on second image sensor 22 is corrected, it can be estimated that light ray 120 has changed to light ray 121 and virtual ray 122.

[0040] Therefore, the intersection of virtual ray 112 and virtual ray 122 is detected as the position of object 100. The intersection of virtual ray 112 and virtual ray 122 is a position shifted in the direction of arrow 91 from the position of the intersection of rays 110 and 120. Distance D2 is then calculated based on the position of the intersection of virtual ray 112 and virtual ray 122. At this time, distance D0 and distance D2 are almost the same distance. Any ranging error that occurs in distance D0 and distance D2 does not pose a practical problem.

[0041] In this way, it has been shown that changes in the image due to changes in lens temperature reduce distance measurement errors by matching the displacement of the first and second images. The lens characteristics described above can be interpreted as a change in lens magnification, as the wider the angle of the lens, the greater the change in image height that occurs. In the following explanation, changes in the image (image height) due to lens temperature will be referred to as lens magnification change.

[0042] Although this process causes the left and right detection positions of the object 100 to shift in the directions of the arrows 91, the change in the angle of view is small, and only amounts to a change in the angle of view to the second decimal place. Therefore, there is no problem in mounting and applying the image processing device 1 according to this embodiment to, for example, a vehicle or a robot.

[0043] Next, a method for obtaining correction information used in the correction algorithm according to this embodiment will be described. Figure 3 shows changes in the image on the sensor due to temperature changes. The right side of Figure 3 shows an example of changes in the image detected by the first image sensor 12, and the left side of Figure 3 shows an example of changes in the image detected by the second image sensor 22.

[0044] The first image sensor 12 and the second image sensor 22 each show the position of the object 100 on the sensor as a black spot 1000, which was detected before the temperature of the first lens 11 changed. The dashed circle C0 indicates the same image height position as the black spot 1000. This circle C0 indicates that the position of the black spot 1000 relative to the horizontal center is the same regardless of whether the first lens 11 and the second lens 21 are at the same temperature or whether the first image sensor 12 and the second image sensor 22 have the same sensitivity. Note that the sensitivity of the image sensor changes as the temperature of the camera changes. Therefore, the second image is corrected not only in response to changes in the lens temperature but also in response to changes in the sensitivity of the image sensor.

[0045] The position on the sensor of the object 100 detected by the first image sensor 12 after the temperature of the first lens 11 has changed, and the position on the sensor of the object 100 detected by the second image sensor 22 after the temperature of the second lens 21 has changed, are shown as black dots 1001. Furthermore, solid-line circles C1 indicate the same image height positions as the black dots 1001. In other words, the circle C1 indicates that the position of the black dot 1001 with respect to the horizontal center will differ when the temperatures of the first lens 11 and the second lens 21 are different, or when the sensitivities of the first image sensor 12 and the second image sensor 22 are different.

[0046] Furthermore, the displacement amounts from sunspot 1000 to sunspot 1001 due to temperature changes of the two lenses are shown as displacement amounts dx1 and dx2 on the first image sensor 12 and the second image sensor 22, respectively. Because the changes in magnification of the two lenses are different, the displacement amounts dx1 and dx2 shown on the first image sensor 12 and the second image sensor 22 are different. In other words, a difference occurs when the displacement amount dx2 is subtracted from the displacement amount dx1.

[0047] 1 determines the relationship between the displacement amount dx1 and the displacement amount dx2. In this case, the image comparison unit (image comparison unit 30) determines the difference between the displacement amount of the image of region R1 where the first image extends in the second direction and the displacement amount of the image of region R2 where the second image extends in the second direction as the difference between the first image and the second image, and the image processing unit (distortion conversion processing unit 50) calculates a correction processing coefficient k, which is a magnification for performing correction to eliminate the difference in the displacement amount of the region of the second image extending in the second direction.

[0048] The processing by the image comparison unit 30 can also be restated as follows, based on the image sizes of the partial regions of the first and second images: That is, the image comparison unit (image comparison unit 30) determines the difference in image size between the partial region R1 of the first image extending in the second direction and the partial region R2 of the second image extending in the second direction as the difference between the first and second images, and the image processing unit (distortion conversion processing unit 50) calculates a correction processing coefficient k, which is a magnification for performing correction to eliminate the difference in image size of the partial region of the second image.

[0049] The distortion conversion processing unit 50B then performs image processing according to the relationship between the displacement amounts dx1 and dx2, thereby correcting the difference in the changes made by the two lenses. However, the horizontal position of the image changes depending on the distance to the object. For this reason, the image comparison unit 30 cannot accurately determine the relationship between the displacement amounts dx1 and dx2 unless it uses information known in advance as the distance to the object 100. As a result, it is difficult for the image processing device 1 to correct the difference in the changes made by the two lenses based on information detected, for example, on a road, using only the horizontal displacement amounts dx1 and dx2.

[0050] To address this issue, the image comparison unit 30 according to this embodiment determines the difference between the circle C0 and the circle C1 in the vertical direction, rather than the horizontal direction, as the displacement amounts dy1 and dy2, respectively. The image comparison unit 30 then uses the vertical displacement amounts dy1 and dy2 to determine the relationship between the horizontal displacement amounts dx1 and dx2. Because lenses typically have rotationally symmetrical characteristics with respect to the optical axis, the displacement amounts dx1 and dy1 are the same. The displacement amounts dx2 and dy2 are also the same. Therefore, the image comparison unit 30 can correct the difference between the first and second images even if the horizontal displacement amounts dx1 and dx2 are changed to the vertical displacement amounts dy1 and dy2.

[0051] (When the image height displacement changes linearly) Next, a method for correcting the difference in the changes of the two lenses from the relationship between the displacement amount dy1 and the displacement amount dy2 will be described. Fig. 4 is a diagram illustrating a method for correcting the difference in change between two lenses according to this embodiment. Here, the amount of displacement of regions R1 and R2 shown in the first image sensor 12 and the second image sensor 22 in Fig. 3 will be described. Note that Fig. 4 illustrates a case where the amount of image height displacement changes linearly.

[0052] FIG. 4(1) is a graph showing the vertical angle of view dependency of the amount of image height displacement of the camera due to the temperature change Δt of the first lens 11 relative to the temperature of the second lens 12. The horizontal axis of this graph represents the vertical angle of view, and the vertical axis represents the amount of image height displacement. In the graph, the characteristic S11 of the first camera 10 and the characteristic S12 of the second camera 20 are shown as straight lines. However, it is difficult to directly detect the characteristics S11 and S12 shown in FIG. 4(1).

[0053] Therefore, the image comparison unit 30 according to this embodiment uses a graph shown in FIG. 4(2) that shows the vertical angle of view dependency of the difference in image height displacement amount of the camera associated with the temperature change Δt of the first lens 11 relative to the temperature of the second lens 12. The vertical axis of this graph shows the difference in image height displacement amount of the two cameras. The difference in image height displacement amount can be determined, for example, by the image comparison unit 30 detecting the difference in the position of the object 100.

[0054] The image comparison unit 30 then vertically shifts the second image relative to the first image, for example, as in a parallax matching process, and detects the amount of shift at which the two images most closely match. Note that the image comparison unit 30 can also determine the positions of feature points by comparing the first image with the second image.

[0055] In the graph shown in (2) of FIG. 4, black dots represent not only the specific object 100 but also objects detected using a predetermined image. Since the image height of any structure reflected in the first and second images can be detected, the number of black dots is not limited to five and may be any number. In this way, the difference in displacement amount is represented by black dots indicating the difference between the position of the object reflected in the first image and the position of the object reflected in the second image. Therefore, the image comparison unit (image comparison unit 30) detects the difference in displacement amount based on the difference between the position of the object reflected in the first image and the position of the object reflected in the second image. Then, the image comparison unit 30 calculates a correction processing coefficient k for changing the image magnification of the second image from the first image and the second image used in (1) and (2) of FIG. 4.

[0056] FIG. 4(3) shows the vertical angle of view dependency of the image height displacement amount when the second image is set at the optimal magnification. FIG. 4(3) is a graph showing the vertical angle of view dependency of the camera's image height displacement amount due to the temperature change Δt of the first lens 11 relative to the temperature of the second lens 12, and illustrates how the characteristic S12 of the second camera 20 changes to the characteristic SC1 when the second image is set at the optimal magnification. When set at the optimal magnification, the characteristic S11 of the first camera 10 and the characteristic SC1 of the second camera 20 become the same in terms of the image height displacement amount. In other words, when the characteristic S12 shown in FIG. 4(1) is multiplied by the correction processing coefficient k, the slope of the characteristic SC1, which is the increased slope of the characteristic S12, becomes the same as the slope of the characteristic S11 of the first camera 10.

[0057] (4) in Fig. 4 shows the vertical angle of view dependency of the image height displacement difference when the second image is set to the optimum magnification. Since the characteristic S11 of the first camera 10 and the characteristic SC1 of the second camera 20 are the same, as shown in (4) in Fig. 4, the image height displacement is constant (almost zero) regardless of the vertical angle of view.

[0058] In this embodiment, a magnification error signal MES (Magnification Error Signal) is defined so that the image comparison unit 30 can determine the correction processing coefficient k that results in the optimal magnification. In the following description, the magnification error signal MES may be abbreviated as "MES." The MES is expressed by the following equation (1). S(+θ) and S(-θ) in equation (1) represent the difference in the amount of image height displacement at the vertical angle of view θ. MES=S(+θ)-S(-θ) …(1)

[0059] 5 is a graph showing the k dependency of the magnification error signal MES. The horizontal axis of this graph represents the correction processing coefficient k (magnification), and the vertical axis represents the magnification error signal MES. The magnification error signal MES changes as the correction processing coefficient k changes.

[0060] Figure 3 shows that S(+θ) is expressed as dy1-k×dy2, and S(-θ) is expressed as -(dy1-k×dy2). Therefore, MES can be expressed as S(+θ)-S(-θ)=2(dy1-k×dy2). The graph shown in Figure 4(2) indicates that S(+θ) takes a positive value and S(-θ) takes a negative value. For example, in the case of the graph shown in Figure 4(2), MES > 0. And, if the graph shown in Figure 4(2) slopes upward to the left, MES < 0.

[0061] On the other hand, when the correction processing coefficient k is at the optimum value, in the graph shown in Figure 4 (4), S(+θ) and S(-θ) are both zero, and therefore, from equation (1), S(+θ)-S(-θ)=0, i.e., when MES is zero, the relationship is dy1-k×dy2=0, and therefore the correction processing coefficient k can be expressed as k=dy1 / dy2.

[0062] As shown in Fig. 5, when the correction processing coefficient k is larger than the optimal value, the MES becomes negative. Therefore, the image comparison unit 30 changes the correction processing coefficient k to calculate the MES and finds the correction processing coefficient k (optimal value) at which the MES becomes zero. The image comparison unit 30 outputs the correction processing coefficient k found as the optimal value to the distortion conversion processing unit 50B (see Fig. 1).

[0063] The image processing unit (distortion conversion processing unit 50) corrects the second image, which differs from the first image due to changes in temperature between the first camera (first camera 10) and the second camera (second camera 20). Furthermore, changes in temperature between the first camera (first camera 10) and the second camera (second camera 20) cause a difference in sensitivity between the first image sensor (first image sensor 12) of the first camera (first camera 10) and the second image sensor (second image sensor 22) of the second camera (second camera 20). Therefore, the image processing unit (distortion conversion processing unit 50) can correct the second image, which differs from the first image due to the difference in sensitivity between the first image sensor (first image sensor 12) and the second image sensor (second image sensor 22).

[0064] In this way, the distortion conversion processing unit 50B can correct the change in magnification of the second image due to the change in lens temperature or sensor sensitivity by using the correction processing coefficient k. As described above, since the lens has rotationally symmetric characteristics with respect to the optical axis, the result of correcting the change in magnification in the vertical direction is also valid in the horizontal direction.

[0065] Note that a change in the magnification of the second image can also occur due to a change in the distance between the lens and the sensor. The correction algorithm according to this embodiment can correct for this change in magnification, and therefore can also correct for changes in the image that accompany changes in the distance between the lens and the sensor. For example, since a change in the distance between the lens and the sensor directly changes the magnification, correcting this change in magnification can correct changes in the image that accompany changes in the distance.

[0066] Up to this point, the image comparison unit 30 has been described as using the vertical angle of view ±θ to calculate the MES. However, the image comparison unit 30 may, for example, detect a large number (six or more) of black points shown in (2) of Fig. 4, and then calculate the correction processing coefficient k that makes the MES zero using the image height displacement amount difference value interpolated at the black points.

[0067] Next, we will explain that the correction algorithm according to this embodiment is effective even when other disturbances are present in the first camera 10 and the second camera 20. Up to this point, we have explained that the displacement amount dx1 and the displacement amount dy1 are the same because the two lenses have rotationally symmetrical characteristics about the optical axis. However, if component misalignment occurs between the first camera 10 and the second camera 20, the displacement amount dx may not match the displacement amount dy. Furthermore, the detection magnification of the first camera (first camera 10) and the detection magnification of the second camera (second camera 20) may change over time. Even in this case, the image processing unit (distortion conversion processing unit 50) can correct the second image, which differs from the first image. Here, an example in which the detection magnification of the first camera (first camera 10) and the detection magnification of the second camera (second camera 20) change over time will be described with reference to FIG.

[0068] Fig. 6 is a diagram showing the relative positions of the lenses and sensors. The right side of Fig. 6 shows the arrangement of the first lens 11 and the first image sensor 12, and the left side of Fig. 6 shows the arrangement of the second lens 21 and the second image sensor 22. The first image sensor 12 and the second image sensor 22 are arranged side by side in the X-axis direction.

[0069] Here, for example, if the second image sensor 22 is displaced in the Y-axis direction (arrow 41), if the second image sensor 22 rotates around the Y-axis (arrow 42), or if the optical axis of the second image sensor 22 is displaced in the Y-direction (arrow 43), the vertical image height displacement amount detected by the second image sensor 22 will be displaced, and the vertical image height displacement amount difference will also be displaced. The following explains how the correction algorithm according to this embodiment is effective against these three factors that cause a deviation in the image height displacement amount difference. Note that the correction algorithm according to this embodiment is effective even when the detection magnifications of the first camera 10 and the second camera 20 change due to a deviation from the original mounting positions of the first camera 10 and the second camera 20 on the vehicle.

[0070] (When the second image sensor moves in the Y direction) Fig. 7 is a diagram showing the effect of correcting the image height displacement amount and the image height displacement amount difference when the second image sensor 22 shown in Fig. 6 moves in the Y direction (arrow 41 in Fig. 6). Note that (1), (2), (3), and (4) in Fig. 7 all show the same content as (1), (2), (3), and (4) in Fig. 4.

[0071] The characteristic S11 of the first camera 10 shown in the graph in FIG. 7(1) is the same as the characteristic S11 shown in FIG. 4. In contrast, the characteristic S22 of the second camera 20 shown in the graph in FIG. 7(1) differs from the example shown in FIG. 4 in that the image height displacement amount in the vertical direction is shifted downward. As a result, the image height displacement amount difference shown in the graph in FIG. 7(2) is shifted upward. The MES at this time is positive, just as when calculated with reference to the graph in FIG. 4(2).

[0072] Therefore, when the image comparison unit 30 performs a correction to set the MES to zero, the image height displacement amount and the image height displacement amount difference change, as shown in the graphs (3) and (4) of Fig. 7. The slopes of the characteristic S11 of the first camera 10 shown in (3) of Fig. 7 and the characteristic SC2 of the second camera 20 match, so that the image height displacement amount difference becomes constant regardless of the vertical angle of view, as shown in (4) of Fig. 7. Here, as shown in (4) of Fig. 7, an offset occurs in the image height displacement amount difference, but this does not affect parallax detection.

[0073] On the other hand, the vertical offset between the first and second images indicates a positional deviation between the two images, which may result in different detected objects. This can be corrected by offsetting the first or second image in the vertical direction using distortion conversion processing unit 50A or distortion conversion processing unit 50B, for example.

[0074] (When the second image sensor rotates) Fig. 8 is a diagram showing the effect of correcting the image height displacement amount and the image height displacement amount difference when the second image sensor 22 rotates around the Y axis (arrow 42 in Fig. 6). Note that (1) to (4) in Fig. 8 all show the same content as (1) to (4) in Fig. 4.

[0075] When the second image sensor 22 is not rotated, the image light incident along the optical axis is received by the second image sensor 22 at a position symmetrical with respect to the optical axis of the second lens 21. However, when the surface of the second image sensor 22 that receives the image light rotates at an angle with respect to the optical axis of the second lens 21, the distance over which the image light incident from the second lens 21 is received by the second image sensor 22 differs, even if the position is symmetrical with respect to the optical axis of the second lens 21, and therefore the image height of the second image sensor 22 changes.

[0076] The characteristic S11 of the first camera 10 shown in the graph of FIG. 8(1) is similar to the characteristic S11 shown in FIG. 4. However, the characteristic S32 of the second camera 20 shown in the graph of FIG. 8(1) is a combination of the characteristic S12 due to the magnification change (see FIG. 4(1)) and the characteristic due to the rotation of the second image sensor 22 about the Y axis as the rotation axis. The characteristic due to the rotation of the second image sensor 22 is nonlinear and approximately symmetrical with respect to the vertical angle of view. For this reason, the image height displacement difference shown in the graph of FIG. 8(2) has a nonlinear shape, and the MES is positive.

[0077] Therefore, when the image comparison unit 30 performs a correction to set the MES to zero, the characteristic S12 due to the change in magnification can be suppressed, as shown in the graphs (3) and (4) of FIG. 8. (3) of FIG. 8 shows the characteristic SC1 of the second camera 20 when the optimal magnification is set and the characteristic SC3 of the second camera 20 processed by the distortion conversion processing unit 50B. As described with reference to FIG. 4, the characteristic S11 of the first camera 10 and the characteristic SC1 of the second camera 20 are the same characteristic. On the other hand, the characteristic S11 of the first camera 10 and the characteristic SC3 of the second camera 20 processed by the distortion conversion processing unit 50B do not match. Therefore, only the characteristic due to the rotational deviation in the X-axis direction of the second image sensor 22 remains, but this characteristic does not affect the parallax detection by the parallax image generation unit 60.

[0078] In addition, by having the distortion conversion processing unit 50A or the distortion conversion processing unit 50B correct the first image or the second image based on the vertical image height displacement component, it is also possible to correct the MES when the second image sensor 22 rotates around the Y axis as the rotation axis.

[0079] Next, the correction effect when the optical axis of the second lens 21 is shifted in the Y direction (arrow 43 in FIG. 6) will be described. The optical axis misalignment of the second lens 21 can be expressed as a combination of the Y-axis direction misalignment and the X-axis direction rotational misalignment of the second image sensor 22. Therefore, the distortion conversion processing unit 50B can correct the change in magnification of the second lens 21 due to the temperature change.

[0080] In this embodiment, the image height displacement described with reference to Figures 4, 7, and 8 has been described as varying linearly with respect to the vertical angle of view. However, even if the image height varies nonlinearly with respect to the vertical angle of view, there is a correction effect for the image height displacement amount and the image height displacement amount difference. Here, the correction effect when the image height varies nonlinearly with respect to the vertical angle of view will be described with reference to Figure 9.

[0081] (When the image height displacement changes nonlinearly) FIG. 9 is a diagram showing the effect of correcting the image height displacement amount and the image height displacement amount difference when the image height displacement amount changes nonlinearly with respect to the vertical angle of view. The graph in FIG. 9(1) shows an example of the characteristic S41 of the first camera 10 and the characteristic S42 of the second camera 20 when the image height displacement amount changes nonlinearly with respect to the vertical angle of view.

[0082] Even when the image height displacement amount changes nonlinearly with respect to the vertical angle of view, the image height displacement amount is symmetrical with respect to the vertical angle of view of 0°. Therefore, the image comparison unit 30 detects a predetermined image height with respect to the vertical angle of view, determines the image height displacement amount difference, and calculates the MES. The image height displacement amount difference is shown in the graph (2) of FIG. 9. This graph shows a nonlinear change.

[0083] 9(3) shows how the characteristic S42 of the second camera 20 changes to the characteristic SC4 when the second camera 20 is set to the optimum magnification. As a result, the image height displacement amount becomes the same as the characteristic S11 of the first camera 10 and the characteristic SC4 of the second camera 20.

[0084] Graph (4) in Fig. 9 shows a graph similar to (4) in Fig. 4. By determining the correction processing coefficient k that makes the MES zero, the image comparison unit 30 makes the image height displacement amount difference zero regardless of the vertical angle of view, and therefore can correct the change in magnification due to the temperature of the second lens 21.

[0085] Furthermore, in this embodiment, information indicating that the horizontal angle of view is approximately 0 deg is used as shown in regions R1 and R2 in FIG. 3, but other regions may also be used. FIG. 10 is a diagram showing an example of the displacement of the image height in the vertical direction at each position of the first lens 11 and the second lens 21 in the horizontal direction.

[0086] The right side of Fig. 10 shows an example of the displacement of image height in the vertical direction caused by the first lens 11, and the left side of Fig. 10 shows an example of the displacement of image height in the vertical direction caused by the second lens 21. The vertical center in the figure represents the position of 0° in the vertical angle of view, and the horizontal center represents the position of 0° in the horizontal angle of view. Fig. 10 shows that the displacement of image height in the vertical direction caused by the two lenses is small near the vertical center of each lens, regardless of the distance from the horizontal center.

[0087] On the other hand, it is shown that the displacement of the vertical image height increases as the distance from the vertical center of each lens increases in the positive and negative directions. This is because the displacement of the vertical image height depends only on the vertical angle of view, not the horizontal angle of view. Therefore, when the image comparison unit 30 calculates the MES, it acquires information on the vertical image height for various regions of the two lenses, taking into account the characteristics of the displacement of the image height at each position by the two lenses.

[0088] Taking into consideration the displacement of the image height in the vertical direction at each position in the horizontal direction of the two lenses shown in FIG. 10, the region used for calculating the MES will be described with reference to FIG. FIG. 11 is a diagram showing examples of various regions detected by two sensors and used to calculate the MES.

[0089] (1), (3), (5), and (7) of Fig. 11 show examples of areas detected by the second image sensor 22, and (2), (4), (6), and (8) of Fig. 11 show examples of areas detected by the first image sensor 12. In Fig. 11 (1) to (8), plus and minus signs are added to the vertical and horizontal centers of each sensor, as in Fig. 10. Note that the vertical and horizontal centers of each sensor are omitted to avoid cluttering the drawings.

[0090] 11(1) and 11(2) show that information on the entire surface of each sensor (areas R11 and R12) is used. Although areas R11 and R12 are included in the entire surface of each sensor, if the same object is included in the vertical direction relative to the horizontal direction, the image comparison unit 30 compares the images using changes in the vertical direction, which is the same as the process for calculating the MES and correction processing coefficient k shown in FIGS. 4 and 5.

[0091] (3) and (4) in Figure 11 indicate that information on a horizontal angle of view other than 0° for each sensor is used. For example, information on a position (area R21, R22) that is shifted in the negative direction (left edge) relative to the horizontal center of 0° is used. Although areas R21 and R22 are shifted to one side in the horizontal direction, there is no difference between the process for calculating the MES and the correction processing coefficient k shown in Figures 4 and 5 when areas R21 and R22 contain the same object.

[0092] 11 (5) and (6) show that the diagonal information (areas R31 and R32) from each sensor is used. Although the areas R31 and R32 are diagonal to the horizontal direction, if the same object is included in the areas R31 and R32, the image comparison unit 30 compares the images using changes in the vertical direction, which is the same as the process for calculating the MES and the correction processing coefficient k shown in FIGS. 4 and 5.

[0093] 11 (7) and (8) show that information (regions R41 and R42) of positions symmetrical with respect to the vertical angle of view of 0° of each sensor is used. When regions R41 and R42 contain the same object, the image comparison unit 30 compares the images using changes in the vertical direction, which is the same as the process for calculating the MES and correction processing coefficient k shown in FIGS. 4 and 5. Therefore, the image comparison unit (image comparison unit 30) detects the difference in the amount of displacement using multiple regions of the first image that are symmetrical with respect to the optical axis of the first lens (first lens 11) of the first camera (first camera 10) and multiple regions of the second image that are symmetrical with respect to the optical axis of the second lens (second lens 21) of the second camera (second camera 20). In this way, the correction processing coefficient k is calculated from the regions R41 and R42 that include only objects near the vertical angle of view ±θ, and the magnification correction processing is performed, thereby making it possible to reduce the processing load of the image processing device 1.

[0094] Note that the image height displacement amount is larger at wider vertical angles of view θ, making it easier to detect the image height displacement amount difference. Furthermore, in order to reduce errors, the MES may be calculated using the following equation (2). ±θ1 and ±θ2 in the following equation (2) may be, for example, any of the black points shown in (2) of Figure 4. MES=[S(+θ1)-S(-θ1)]+[S(+θ2)-S(-θ2)]+… (2)

[0095] <Example of computer hardware configuration> Next, the hardware configuration of the computer 200 that constitutes each device of the image processing device 1 will be described. 12 is a block diagram showing an example of the hardware configuration of the calculator 200. The calculator 200 is an example of hardware used as a computer that can operate as the image processing device 1 according to this embodiment. The image processing device 1 according to this embodiment realizes an image processing method in which the functional blocks shown in FIG. 1 cooperate with each other by the calculator 200 (computer) executing a program.

[0096] The computer 200 includes a CPU (Central Processing Unit) 210, a ROM (Read Only Memory) 220, and a RAM (Random Access Memory) 230, each connected to a bus 240. The computer 200 further includes a non-volatile storage 250 and a network interface 260.

[0097] The CPU 210 reads out program code of software that realizes each function according to the present embodiment from the ROM 220, loads it into the RAM 230, and executes it. Variables, parameters, etc. generated during the calculation processing of the CPU 210 are temporarily written to the RAM 230, and these variables, parameters, etc. are read out by the CPU 210 as appropriate. However, an MPU (Micro Processing Unit) may be used instead of the CPU 210, or the CPU 210 may be used in combination with a GPU (Graphics Processing Unit). The CPU 210 executes the software to realize each function of the image comparison unit 30, the distortion conversion processing unit 50, the luminance correction unit 51, the image interpolation unit 52, the luminance information generation unit 53, the parallax image generation unit 60, and the distance calculation unit 70 shown in FIG. 1 .

[0098] The nonvolatile storage 250 may be, for example, a hard disk drive (HDD), a solid state drive (SSD), or a nonvolatile memory. In addition to an operating system (OS) and various parameters, programs for operating the computer 200 are recorded in the nonvolatile storage 250. The ROM 220 and the nonvolatile storage 250 record programs and data necessary for the CPU 210 to operate, and are used as an example of a computer-readable non-transitory storage medium that stores programs executed by the computer 200. The nonvolatile storage 250 records, for example, the image correction coefficient k calculated by the image comparison unit 30, the first image and the second image, the first processed image and the second processed image, a parallax image, distance information to the object, and the like.

[0099] For example, a network interface card (NIC) or the like is used for network interface 260, and various data can be transmitted and received between devices via a local area network (LAN) connected to a terminal of the NIC, a dedicated line, etc. Information about the distance to the object calculated by distance calculation unit 70 shown in Fig. 1 is transmitted by network interface 260 to an electronic control unit (ECU) or vehicle control device (not shown), and is used to control the vehicle.

[0100] <Example of processing by image processing device 1> 13 is a flowchart showing an example of processing performed by the image processing device 1. The image processing device 1 performs the image processing method according to this embodiment through processing performed by each functional unit.

[0101] First, image comparison unit 30 compares a first image input from first camera 10 with a second image input from second camera 20 (S1), and calculates correction processing coefficient k. Next, distortion conversion processing unit 50A converts the distortion of the first image, and distortion conversion processing unit 50B converts the distortion of the second image (S2). Note that the processes of steps S1 and S2 may be performed in parallel.

[0102] Next, the distortion conversion processing unit 50B converts the magnification of the second image whose distortion has been converted based on the correction processing coefficient k (S3). Note that the processes of steps S2 and S3 may be reversed or may be performed simultaneously.

[0103] Next, the brightness correction unit 51 corrects the brightness of the first image whose distortion has been converted, and corrects the brightness of the second image whose distortion has been converted and whose magnification has been converted (S4). Next, the image interpolation unit 52 interpolates the first image whose distortion has been converted, and interpolates the second image whose distortion has been converted and whose magnification has been converted (S5).

[0104] Next, the luminance information generating unit 53 generates luminance information of the first image whose distortion has been converted, and generates luminance information of the second image whose distortion has been converted and whose magnification has been converted (S6). Next, the parallax image generating unit 60 generates parallax images based on the luminance information of the first image whose distortion has been converted and the luminance information of the second image whose distortion has been converted and whose magnification has been converted (S7). Next, the distance calculation unit 70 calculates the distance to the object based on the parallax images (S8).

[0105] The image processing of steps S1 to S8 is repeated at predetermined time intervals. Then, a vehicle control unit (not shown) controls the vehicle based on the distance to the object calculated by the distance calculation unit 70. Note that the image processing device 1 and the vehicle control unit may be integrated into one electronic control device.

[0106] In the image processing device 1 according to the first embodiment described above, it is possible to correct the second image by calculating a correction processing coefficient k that suppresses the image height displacement difference that is particularly noticeable at the edges of the angle of view (±60 degrees from the center) of a camera having a lens with a wide horizontal angle of view (approximately 120 degrees) compared to the horizontal angle of view (approximately 60 degrees) of a conventional lens.

[0107] The image comparison unit 30 of the image processing device 1 calculates the MES so that the difference in image height displacement in the vertical direction (e.g., zero) between the image heights detected from the first image and the second image at the vertical angle of view ±θ is constant. This MES is a value dependent on the correction processing coefficient k, and the MES is zero when the correction processing coefficient k is at an optimal value. The distortion conversion processing unit 50B not only converts the distortion of the second image captured by the second camera 20 but also corrects the change in magnification of the second image using the correction processing coefficient k that results in zero MES. In this way, the image comparison unit 30 detects the difference in magnification of the images detected by the image sensors of each camera based on the difference in image height displacement in directions other than the horizontal direction, rather than the amount of displacement in the horizontal direction. The distortion conversion processing unit 50B then performs a correction to match the magnification of the image detected by the image sensor of one camera to the image detected by the image sensor of the other camera.

[0108] Therefore, whereas conventionally, it was necessary to secure a large-capacity storage device for storing image correction information, the image processing device 1 according to this embodiment can correct images output from two cameras in accordance with changes in the stereo camera (two cameras). Furthermore, the image processing device 1 does not need to secure a large-capacity storage device for storing image correction information.

[0109] Furthermore, each processing unit downstream of the distortion conversion processing unit 50 can handle the second image detected by the second image sensor 22 of the second camera 20 and corrected for the change in magnification in the same way as the first image captured by the first camera 10. This reduces errors in the parallax images generated by the parallax image generation unit 60, and the distance to the object calculated by the distance calculation unit 70 can be measured accurately.

[0110] Furthermore, even if the second image sensor 22 is shifted in the Y-axis direction, the second image sensor 22 rotates around the Y-axis as the rotation axis, the optical axis of the second image sensor 22 is shifted in the Y direction, or the image height changes nonlinearly with respect to the vertical angle of view, by matching the characteristics of the second camera 20 to the characteristics of the first camera 10, the image height displacement difference becomes constant regardless of the vertical angle of view. This reduces errors in the parallax images generated by the parallax image generator 60, and the distance to the object calculated by the distance calculator 70 is also accurately measured.

[0111] Various embodiments of the image processing device are conceivable. Below, configuration examples of image processing devices according to second to fourth embodiments of the present invention will be described. In the image processing devices according to each embodiment, the image used as the processing target is different, but similar to the image processing device 1 according to the first embodiment, this image is used to calculate a correction processing coefficient k that makes the MES zero, and one image (the image captured by the second camera 20, or a processed image obtained by performing a predetermined process on this image) is corrected using this correction processing coefficient k.

[0112] [Second embodiment] 14 is a block diagram showing an example of the internal configuration of an image processing device 1A according to a second embodiment of the present invention. The image processing device 1A according to the second embodiment is similar to the image processing device 1 according to the first embodiment in that it calculates a correction processing coefficient k, which is a result of comparing a first image with a second image.

[0113] Image comparison unit 30 is provided after first camera 10 and second camera 20, the same as in image processing device 1 shown in Fig. 1. Image comparison unit 30 outputs correction processing coefficient k to image magnification conversion unit 150.

[0114] The image magnification conversion unit 150 is disposed after the image comparison unit 30 and the luminance information generation unit 53B. The image processing unit according to the second embodiment is an image magnification conversion unit (image magnification conversion unit 150) that converts the magnification of the second image, whose distortion has been converted, using a correction processing coefficient k. The image magnification conversion unit 150 converts the magnification of the second processed image, which is input from the luminance information generation unit 53B, using the correction processing coefficient k input from the image comparison unit 30. The image magnification conversion unit 150 outputs the second processed image, whose magnification has been converted, to the parallax image generation unit 60.

[0115] The parallax detection unit (parallax image generation unit 60) detects the parallax of an object appearing in the first image and the second image, based on a first image (i.e., a first processed image) in which at least the distortion has been converted and the luminance information has been generated, and a second image (i.e., a second processed image) in which at least the distortion has been converted, the luminance information has been generated, and the magnification has been converted. Here, the parallax image generation unit 60 generates parallax images for detecting the parallax of the object, based on the first processed image input from the luminance information generation unit 53A and the second processed image in which the magnification has been converted and which is input from the image magnification conversion unit 150. The distance calculation unit 70 calculates the distance to the object based on the parallax images generated by the parallax image generation unit 60.

[0116] The configuration of the image processing device 1A according to the second embodiment described above can also correct the change in magnification of the second image due to the temperature change of the two lenses. Here, the image comparison unit 30 calculates a correction processing coefficient k, which is the result of comparing the first image and the second image, and the image magnification conversion unit 150 converts the magnification of the second processed image using the correction processing coefficient k. As a result, the parallax for each object generated by the parallax image generation unit 60 can be accurately determined, and the distance to the object can also be accurately calculated by the distance calculation unit 70.

[0117] [Third embodiment] 15 is a block diagram showing an example of the internal configuration of an image processing device 1B according to a third embodiment of the present invention. In the image processing device 1B according to the third embodiment, a correction processing coefficient k is calculated using the first processed image and the second processed image for which luminance information has been generated.

[0118] An image processing device 1B according to the third embodiment includes an image comparison unit 31 and an image magnification conversion unit 150 in addition to the functional units included in the image processing device 1. Here, the luminance information generation unit 53 includes a luminance information generation unit 53A that generates luminance information from an image captured by the first camera 10, and a luminance information generation unit 53B that generates luminance information from an image captured by the second camera 20. In the image processing device 1B, the processing of the distortion conversion processing unit 50 on the first image and the second image is performed in the same manner as in the example shown in FIG. 1, and therefore the reference numerals of the distortion conversion processing units 50A and 50B shown in FIG. 1 are omitted.

[0119] Image comparison unit 31 is provided in place of image comparison unit 30 shown in FIG. 1 and is arranged subsequent to luminance information generation unit 53. Image comparison unit 31 calculates a correction processing coefficient k based on a first image (i.e., first processed image) in which at least distortion has been converted and luminance information has been generated, and a second image (i.e., second processed image) in which at least distortion has been converted and luminance information has been generated. For this purpose, image comparison unit 31 receives the first processed image and the second processed image in which luminance information has been generated from luminance information generation units 53A and 53B, respectively. The first processed image is an image captured by first camera 10 and processed by luminance information generation unit 53A. The second processed image is an image captured by second camera 20 and processed by luminance information generation unit 53B. The correction processing coefficient k calculated by image comparison unit 31 by comparing the first processed image and the second processed image is output to image magnification conversion unit 150.

[0120] The image magnification conversion unit 150 is arranged after the image comparison unit 31 and the luminance information generation unit 53B. The image processing unit according to the third embodiment is an image magnification conversion unit (image magnification conversion unit 150) that converts the magnification of the second image using a correction processing coefficient k. The image magnification conversion unit 150 receives the correction processing coefficient k from the image comparison unit 31 and the second processed image from the luminance information generation unit 53B. The image magnification conversion unit 150 converts the magnification of the second processed image using the correction processing coefficient k. The image magnification conversion unit 150 outputs the second processed image whose magnification has been converted to the parallax image generation unit 60.

[0121] The parallax image generation unit 60 generates a parallax image for detecting the parallax of an object based on the first processed image input from the luminance information generation unit 53A and the second processed image with its magnification converted input from the image magnification conversion unit 150. The distance calculation unit 70 calculates the distance to the object based on the parallax images generated by the parallax image generation unit 60.

[0122] Here, for example, consider the case where the lens projection is a central projection (f tan θ). In this case, the projection is not changed by the distortion conversion processing unit 50A and the distortion conversion processing unit 50B. The first image output by the first camera 10 and the first processed image processed by the luminance information generation unit 53A are the same. Similarly, the second image output by the second camera 20 and the second processed image processed by the luminance information generation unit 53B are the same. Therefore, the state of each image described in the third embodiment is the same as the state of each image described in the first embodiment, and it can be seen that the second processed image can also be corrected by the correction algorithm according to the third embodiment.

[0123] Next, other projections will be considered. For example, consider the case where the lens projection method is orthogonal projection (f sin θ). In this case, a projective transformation to a central projection (f tan θ) is performed on the first image detected by the first camera 10 and the second image detected by the second camera 20. For example, if the image height displacement amount between the first image and the second image is expressed by the characteristic shown in (1) of FIG. 4, the image height displacement amount between the first processed image and the second processed image will be expressed by the characteristic shown in (1) of FIG. 9. Therefore, it can be seen that the second processed image can be corrected using the correction algorithm according to the third embodiment.

[0124] Although the third embodiment describes central projection and orthogonal projection, similar effects can be obtained with stereographic projection, equisolid angle projection, or equidistant projection, which have distortion characteristics between central projection and orthogonal projection. The image magnification conversion unit 150 according to this embodiment may have the same function as the distortion conversion processing unit 50 and convert distortion for the second processed image. The distortion conversion processing unit 50B may have the same function as the image magnification conversion unit 150 and convert the magnification of the second image.

[0125] In the image processing device 1B according to the third embodiment described above, a correction processing coefficient k is calculated based on the comparison result between the first processed image and the second processed image output from the luminance information generation units 53A and 53B, and the image magnification conversion unit 150 converts the magnification of the second processed image. In the image processing device 1 according to the first embodiment, after the image comparison unit 30 calculates the correction processing coefficient k, the distortion conversion processing unit 50B corrects the distortion caused by the second lens 21 and the change in magnification of the second image. In contrast, the image processing device 1B according to the third embodiment corrects only the magnification of the second processed image, after the distortion of the second image has already been converted. Therefore, in the image processing device 1B, even if the first camera 10, the second camera 20, and the functional units 50 to 53 are integrated, for example, there is no need to change the processing of the distortion conversion processing unit 50B.

[0126] [Fourth embodiment] 16 is a block diagram showing an example of the internal configuration of an image processing device 1C according to a fourth embodiment of the present invention. In the third embodiment, the image comparison unit 31 compared the first processed image and the second processed image, and the image magnification conversion unit 150 converted the image magnification of the second processed image based on the comparison result. In contrast, the image processing device 1C according to the fourth embodiment does not require the image magnification conversion unit 150, and is configured so that the comparison result of the image comparison unit 31 is output to the distortion conversion processing unit 50B.

[0127] The image processing device 1C according to the fourth embodiment has a configuration in which the image magnification conversion unit 150 is removed from the functional units included in the image processing device 1B according to the third embodiment.

[0128] The image comparison unit 31 is provided after the luminance information generation units 53A and 53B, which are the same as those in the image processing device 1B shown in Fig. 15. The image comparison unit (image comparison unit 31) calculates a correction processing coefficient k based on a first image (i.e., a first processed image) in which at least the distortion has been converted and the luminance information has been generated, and a second image (i.e., a second processed image) in which at least the distortion has been converted and the luminance information has been generated. The image comparison unit 31 then calculates the correction processing coefficient k based on the first processed image and the second processed image output from the distortion conversion processing unit 50B and the image magnification conversion unit 150, and outputs this correction processing coefficient k to the distortion conversion processing unit 50B.

[0129] The image processing unit according to the fourth embodiment is a distortion conversion processing unit (distortion conversion processing unit 50B) that converts distortion in a first image, converts distortion in a second image, and converts the magnification of the second image using a correction processing coefficient k. This distortion conversion processing unit 50B converts distortion in the second image detected by the second camera 20, and corrects the change in magnification of the second image based on the correction processing coefficient k input from the image comparison unit 31. The subsequent processing is the same as that of the image processing device 1 according to the first embodiment.

[0130] For example, the disparity detection unit (disparity image generation unit 60) detects the disparity between an object appearing in a first image and a second image, based on a first image (i.e., a first processed image) in which at least the distortion has been converted and the luminance information has been generated, and a second image (i.e., a second processed image) in which at least the distortion has been converted, the luminance information has been generated, and the magnification has been converted. Here, the disparity image generation unit 60 generates disparity images for detecting the disparity of an object, using the first processed image input from the luminance information generation unit 53A and the second processed image input from the luminance information generation unit 53B, the magnification of which has been converted by the distortion conversion processing unit 50B. The distance calculation unit 70 calculates the distance to the object based on the parallax images generated by the parallax image generation unit 60.

[0131] The configuration of the image processing device 1C according to the fourth embodiment described above can also correct the change in magnification of the second image due to the temperature change of the two lenses. At this time, the correction processing coefficient k of the distortion conversion processor 50B is changed based on the comparison result between the first and second images by the image comparator 31, and the difference in image height between the first and second images is eliminated. Therefore, the parallax for each object generated by the parallax image generator 60 can be accurately calculated, and the distance to the object can also be accurately calculated by the distance calculator 70.

[0132] In the image processing device 1C according to the fourth embodiment, the image comparison unit 31 calculates the correction processing coefficient k based on the first processed image and the second processed image, thereby correcting the change in magnification of the second image output from the distortion conversion processing unit 50B. When the second processed image with the corrected change in magnification is input to the image comparison unit 31 as the second processed image after subsequent processing, the image comparison unit 31 calculates a correction processing coefficient k that does not correct the change in magnification of the second image. Therefore, once the image comparison unit 31 calculates the correction processing coefficient k, the processing of the image comparison unit 31 is stopped for a certain period of time. Then, after the certain period of time has passed, the processing of the image comparison unit 31 is resumed.

[0133] It should be noted that the present invention is not limited to the above-described embodiment, and it goes without saying that various other applications and modifications are possible without departing from the gist of the present invention as set forth in the claims. For example, the above-described embodiments have described in detail and specifically the configuration of an image processing device in order to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0134] Furthermore, in the above embodiment, the process of correcting the change in magnification due to the change in lens temperature has been described, but the same effect can be obtained even if the magnification of the two cameras changes over time. Furthermore, in the above-described embodiment, the process of converting both the horizontal and vertical magnifications of the image has been described. However, even if the image processing device according to the modified example converts only the magnification in the horizontal direction in which parallax is detected, the same effect as that of the process according to the above-described embodiment can be obtained. In this case, the image processing unit (distortion conversion processing unit 50) corrects the second image in at least the first direction. In other words, even if the magnification in the direction in which parallax is detected is converted, the same effect as that of the process according to the above-described embodiment can be obtained. Furthermore, the image processing device according to each of the above-described embodiments is capable of correcting images in real time. In addition, the image processing device according to each of the above-described embodiments may correct images based on, for example, temperature information or time information, or may monitor whether the MES has shifted by a predetermined amount and correct the image when the MES has shifted by the predetermined amount. [Explanation of symbols]

[0135] REFERENCE SIGNS LIST 1...image processing device, 10...first camera, 11...first lens, 12...first image sensor, 20...second camera, 21...second lens, 22...second image sensor, 30...image comparison unit, 50A, 50B...distortion conversion processing unit, 51...luminance correction unit, 52...image interpolation unit, 53...luminance information generation unit, 60...parallax image generation unit, 70...distance calculation unit, 100...object

Claims

1. an image comparison unit that compares a first image detected by a first image sensor of a first camera with a second image detected by a second image sensor of a second camera, the first image being input from a stereo camera having a first camera and a second camera arranged side by side in a first direction, determines a difference between an amount of displacement of an image of a region of the first image extending in the second direction in a second direction different from the first direction, and an amount of displacement of an image of a region of the second image extending in the second direction as a difference between the first image and the second image, and calculates a correction processing coefficient that is a magnification for performing correction to eliminate the difference in the amount of displacement of the region of the second image extending in the second direction based on the difference between the first image and the second image; an image processing unit that corrects the second image using the correction processing coefficient calculated by the image comparison unit; Image processing device.

2. The image processing unit corrects the second image that differs from the first image due to a change over time in the detection magnification of the first camera and the detection magnification of the second camera. The image processing device according to claim 1 .

3. The image processing unit corrects the second image that differs from the first image due to a change in temperature between the first camera and the second camera. The image processing device according to claim 1 .

4. The image processing unit corrects the second image that differs from the first image due to a difference in sensitivity between a first image sensor of the first camera and a second image sensor of the second camera, which difference is caused by a change in temperature between the first camera and the second camera. The image processing device according to claim 3 .

5. The image comparison unit detects the difference in the amount of displacement based on a difference between the position of the object captured in the first image and the position of the object captured in the second image. The image processing device according to claim 1 .

6. The image comparison unit detects the difference in the amount of displacement using a plurality of regions of the first image that are symmetrical with respect to an optical axis of a first lens of the first camera and a plurality of regions of the second image that are symmetrical with respect to an optical axis of a second lens of the second camera. The image processing device according to claim 1 .

7. The image processing unit corrects the second image in at least the first direction. The image processing device according to claim 1 .

8. the image processing unit is a distortion conversion processing unit that converts distortion of the first image and distortion of the second image, a parallax detection unit that detects parallax between an object shown in the first image and the second image based on the first image for which at least distortion has been converted and luminance information has been generated, and the second image for which at least distortion has been converted, luminance information has been generated, and magnification has been converted. The image processing device according to claim 1 .

9. the image processing unit is an image magnification conversion unit that converts the magnification of the second image, the distortion of which has been converted, using the correction processing coefficient, a parallax detection unit that detects parallax between an object shown in the first image and the second image based on the first image for which at least distortion has been converted and luminance information has been generated, and the second image for which at least distortion has been converted, luminance information has been generated, and magnification has been converted. The image processing device according to claim 1 .

10. the image processing unit is an image magnification conversion unit that converts a magnification of the second image using the correction processing coefficient, the image comparison unit calculates the correction processing coefficient based on the first image, in which at least distortion has been converted and luminance information has been generated, and the second image, in which at least distortion has been converted and luminance information has been generated; a parallax detection unit that detects parallax between an object shown in the first image and the second image based on the first image for which distortion has been converted and luminance information has been generated, and the second image for which distortion has been converted, luminance information has been generated, and magnification has been converted. The image processing device according to claim 1 .

11. the image processing unit is a distortion conversion processing unit that converts distortion of the first image, converts distortion of the second image, and converts a magnification of the second image using the correction processing coefficient, the image comparison unit calculates the correction processing coefficient based on the first image, in which at least distortion has been converted and luminance information has been generated, and the second image, in which at least distortion has been converted and luminance information has been generated; a parallax detection unit that detects parallax between an object shown in the first image and the second image based on the first image for which at least distortion has been converted and luminance information has been generated, and the second image for which at least distortion has been converted, luminance information has been generated, and magnification has been converted. The image processing device according to claim 1 .

12. a step of comparing a first image detected by a first image sensor of a first camera and a second camera detected by a second image sensor of a second camera, the first image being input from a stereo camera having a first camera and a second camera arranged side by side in a first direction, and determining a difference between an amount of displacement of an image of a region of the first image extending in the second direction in a second direction different from the first direction and an amount of displacement of an image of a region of the second image extending in the second direction as a difference between the first image and the second image, and calculating a correction processing coefficient, which is a magnification for performing a correction to eliminate the difference in the amount of displacement of the region of the second image extending in the second direction, based on the difference between the first image and the second image; and correcting the second image using the calculated correction processing coefficient. Image processing methods.

Images