Image processing apparatus and image processing method

Abstract

The present application aims to generate a parallelized image that saves memory most while maintaining the resolution of the original image as much as possible, in consideration of parallax accuracy and the size of the image. An image processing apparatus of the present application includes: an image acquisition unit (101) that acquires a first captured image (P1) and a second captured image (P2); a parallelization processing unit (104) that generates a first parallelized image and a second parallelized image by parallelizing the first captured image and the second captured image in such a manner that the vertical coordinates of the same image photographed in the first captured image and the second captured image coincide at least at an arbitrary point within the first captured image and the second captured image; a parallax image generation unit (106) that generates parallax from a plurality of parallelized images; and a stereoscopic object recognition unit (107) that recognizes an object based on the parallax. The parallelization processing unit (104) parallelizes the first captured image and the second captured image in such a manner that the resolution per unit pixel of the baseline direction increases in a prescribed angle-of-view range in a plane including the vertical axis.

Description

Technical Field

[0001] This invention relates to an image processing apparatus and an image processing method. Background Technology

[0002] Stereo cameras use multiple cameras, such as two, to photograph three-dimensional objects. The image processing unit of a stereo camera calculates the parallax of the stereo object projected into the images captured by each camera and measures the depth of the stereo object using the principle of triangulation, thus enabling three-dimensional recognition of the stereo object. Therefore, stereo cameras are widely used in a wide range of applications, including collision avoidance systems in vehicles, robot sensors and surveillance cameras, and security. For example, stereo cameras mounted on automobiles can detect the positions of surrounding cars, pedestrians, bicycles, motorcycles, and other three-dimensional objects, and use this information to perform automatic emergency braking and other automatic vehicle controls.

[0003] Furthermore, to cope with the diverse applications, stereo cameras are required to measure distances to stereoscopic objects with wider angles of view and at greater distances. Especially in situations involving measuring distances to stereoscopic objects with wide angles of view, extreme wide-angle cameras employing fisheye lenses are frequently used. Fisheye lenses, by introducing significant distortion into the image, can even capture a wide-angle subject within a single image.

[0004] In stereo cameras, stereo matching, which calculates the parallax (relative position) of the same image projected onto the left and right cameras, is the most computationally expensive process, especially in applications like automotive, where real-time performance is crucial and computational costs must be minimized. Therefore, before stereo matching, the left and right images are parallelized in the stereo camera. Parallelization limits the exploration of the same image to a single parallel line, significantly reducing computational costs. Typically, the parallelized image is transformed into a perspective projection image (distortion-free image). However, the challenge lies in the fact that the perspective projection image of a wide-angle image captured by a fisheye lens becomes larger.

[0005] Patent Document 1 describes a system that includes "a mapper that applies a mapping to the first wide-angle image in order to generate a corrected image with a corrected projection," and that "the mapper provides: a vertical mapping function that is consistent with a nonlinear vertical mapping of a corrected vertical projection from the perspective projection to the corrected projection; and a horizontal mapping function that is consistent with a nonlinear horizontal mapping of a mapping from the first projection to the perspective projection and a subsequent corrected horizontal projection from the perspective projection to the corrected projection."

[0006] Prior technology documents

[0007] Patent documents

[0008] Patent Document 1: Japanese Patent Publication No. 2022-506104 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] As mentioned above, a method is proposed to replace perspective projection by applying a nonlinear mapping function to the images captured by the left and right cameras, thereby generating a more compressed parallelized image than before. However, the technology described in Patent Document 1 does not specify a particular compression function, only limiting it to at least some compression of the perspective projection. While large compression can compress the image accordingly, if the compression degrades the image resolution, it leads to a deterioration in the accuracy of disparity calculation. Conversely, even if enlargement is performed, it is ultimately only through interpolation, and therefore cannot fundamentally exceed the resolution of the original image. In summary, compression that degrades the resolution of the original image also leads to a deterioration in disparity accuracy, while enlargement merely increases memory capacity meaninglessly.

[0011] In order to balance parallax accuracy and image size, it has become a challenge to maintain the resolution of the original image while generating a parallelized image that saves the most memory.

[0012] The present invention was made in view of the above points, and its object is to provide an image processing apparatus that can maintain resolution during the transformation from images captured by multiple cameras to parallelized images, thereby balancing memory saving and ranging accuracy.

[0013] Methods for solving problems

[0014] The image processing apparatus of the present invention, which solves the above-mentioned problems, comprises:

[0015] An image acquisition unit acquires a first image and a second image captured by each of multiple camera units;

[0016] The parallelization processing unit takes a direction perpendicular to the baseline direction defined by the plurality of camera units and the optical axis direction of the plurality of camera units as the vertical axis, and parallelizes the first captured image and the second captured image in such a way that the vertical coordinates of the same image captured in the first captured image and the second captured image are consistent at least at any point within the first captured image and the second captured image, thereby generating a first parallelized image and a second parallelized image.

[0017] A disparity image generation unit generates a disparity based on a plurality of said parallelized images; and

[0018] The object recognition unit identifies objects based on the parallax.

[0019] The parallelization processing unit parallelizes the first captured image and the second captured image in such a way that the resolution per unit pixel in the baseline direction within a defined viewing angle range in a plane including the vertical axis is increased relative to the first captured image and the second captured image.

[0020] The effects of the invention

[0021] According to the present invention configured as described above, by maintaining resolution during the transformation from images captured by multiple cameras to parallelized images, both memory saving and ranging accuracy can be achieved. Further features related to the present invention will become clear from the description and accompanying drawings. Furthermore, issues, configurations, and effects other than those described above will become clear from the following description of embodiments. Attached Figure Description

[0022] Figure 1 This is a functional block diagram illustrating the configuration of the stereo camera image processing apparatus according to the embodiments.

[0023] Figure 2 This is a flowchart illustrating the stereo calibration of the stereo camera image processing apparatus according to the first embodiment.

[0024] Figure 3 This is a schematic diagram illustrating the relationship between light rays and image coordinates in a fisheye camera image according to the first embodiment.

[0025] Figure 4 This is a three-dimensional model diagram of triangulation in the stereo camera of the first embodiment.

[0026] Figure 5 This is a schematic diagram showing the relationship between the light rays and image coordinates of the parallelized image in the first embodiment.

[0027] Figure 6 This is a two-dimensional pattern diagram of triangulation in the stereo camera of the first embodiment.

[0028] Figure 7 This is an example of resolution-maintaining parallelized image generation in Embodiment 1.

[0029] Figure 8 This is a block diagram illustrating the configuration of the stereo camera image processing apparatus according to the first embodiment.

[0030] Figure 9 This is an example of the second implementation focusing on the vertical viewing angle range.

[0031] Figure 10 This is an example of resolution-preserving parallelized image generation in the second embodiment.

[0032] Figure 11This is an example of the focus on the horizontal and vertical viewing angle range in the second implementation.

[0033] Figure 12 This is an example of resolution-preserving parallelized image generation in the second embodiment.

[0034] Figure 13 This is an example of resolution-preserving parallelized image generation in the second embodiment.

[0035] Figure 14 This is an example of resolution-preserving parallelized image generation in the second embodiment.

[0036] Figure 15 This is an example of resolution-preserving parallelized image generation in the second embodiment.

[0037] Figure 16 This is a schematic diagram showing the relationship between light rays and image coordinates in the parallelized image of the third embodiment.

[0038] Figure 17 This is a two-dimensional pattern diagram of triangulation in the stereo camera of the third embodiment. Detailed Implementation

[0039] Hereinafter, embodiments for carrying out the present invention will be described with reference to the accompanying drawings. In this specification and drawings, constituent elements having substantially the same function or structure are given the same reference numerals, thus omitting repetitive descriptions. The present invention can be applied, for example, to a computing device for vehicle control that can communicate with an onboard ECU (Electronic Control Unit) for Advanced Driver Assistance Systems (ADAS) or Autonomous Driving (AD).

[0040] <First Embodiment>

[0041] [Example of the configuration of the image processing apparatus 10]

[0042] First, the configuration of the image processing apparatus 10 according to the first embodiment of the present invention will be described. Hereinafter, an example configuration of the image processing apparatus 10 mounted on this vehicle will be described. However, the application of the present invention is not limited to this; the image processing apparatus 10 can also be mounted on a robot's sensor or a monitoring camera, etc. Furthermore, in the following embodiments, the example of using the image processing apparatus of the present invention in a stereo camera having two imaging units will be described; however, the configuration of the present invention is not limited to stereo cameras and can also be applied to multi-camera systems having three or more cameras.

[0043] The image processing apparatus 10 of this embodiment is provided on a vehicle-mounted stereo camera 100. In this embodiment, the stereo camera 100 includes a first camera unit (left camera unit) 12 and a second camera unit (right camera unit) 13 as a plurality of camera units 11 (see reference). Figure 8 (1) The left camera unit 12 and the right camera unit 13 are both single-lens cameras equipped with lenses and image sensors, which capture images of three-dimensional objects in front of the vehicle. The lens is a fisheye lens, and the captured image is a circular fisheye image. Hereinafter, without distinguishing between the left camera unit 12 and the right camera unit 13, the left camera unit 12 and the right camera unit 13 will be collectively referred to as "left and right camera units 11".

[0044] Figure 1 This is a functional block diagram illustrating an example configuration of the image processing apparatus 10 according to this embodiment. The image processing apparatus 10 is configured as a computer system including an arithmetic unit, a storage unit, and input / output devices. The image processing apparatus 10 reads and executes a software program stored in the storage unit via the arithmetic unit, thereby implementing the following functions. For example... Figure 1 As shown, the image processing apparatus 10 includes an image acquisition unit 101, an image preprocessing unit 103, a parallelization processing unit 104, an image interpolation unit 105, a parallax image generation unit 106, a stereoscopic object recognition unit 107, a vehicle control unit 108, and a parallelization image transformation table storage unit 109.

[0045] The image acquisition unit 101, the image preprocessing unit 103, the parallelization processing unit 104, the image interpolation unit 105, the parallax image generation unit 106, the stereoscopic object recognition unit 107, and the vehicle control unit 108 are connected in this order, and the parallelization image transformation table storage unit 109 is connected to the parallelization processing unit 104.

[0046] Image acquisition unit 101 acquires image P1 captured by left camera unit 12 and image P2 captured by right camera unit 13 (image acquisition step). Image P1 and image P2 become the first and second captured images of an optical image formed by an optical system having the image height characteristics of lenses with left camera unit 12 and right camera unit 13. As a preprocessing step for images P1 and P2 input from left and right camera units 11, image preprocessing unit 103 performs gain adjustment of the brightness of images P1 and P2 and performs shadow correction to correct the brightness unevenness of images P1 and P2 respectively. Image preprocessing unit 103 outputs the shadow-corrected images P1 and P2 to parallelization processing unit 104.

[0047] The parallel image transformation table storage unit 109 stores the parallel image transformation table processed by the parallelization processing unit 104. The parallel image transformation tables for each of the left and right camera units 11 are generated and stored during prior stereo calibration. The parallel image transformation table storage unit 109 also stores transformation information (transformation information storage unit) for radial distortion of the lenses mounted in the first and second camera units 13.

[0048] The parallelization processing unit 104 parallelizes multiple captured images based on the transformation information stored in the parallelization image transformation table storage unit 109, in a manner where the parallelization magnification of the vertical and horizontal axes of the parallelized image is determined by the optical distortion of the lens, which follows the radial distortion of the lens. Specifically, the parallelization processing unit 104 performs image transformation processing (parallelization processing) on ​​images P1 and P2 after brightness and grayscale transformation in a manner where the coordinates of the same point in the left and right images are always equal on the vertical axis of the image coordinate system (so that the epipolar lines on both sides are parallel to the horizontal axis of the image coordinate system). That is, parallelization is performed so that the difference in the vertical coordinates of the image coordinate system of the same image projected into images P1 and P2 is set to be approximately fixed across the entire image.

[0049] In this parallelization process, the direction perpendicular to the baseline direction defined by the left and right camera units 11 and the optical axis direction of the left and right camera units 11 is taken as the vertical axis. The vertical coordinates of the same image captured in image P1 (first image) and image P2 (second image) are consistent at least at any point in image P1 and image P2. The images P1 and P2 are then parallelized by image processing to generate a first parallelized image and a second parallelized image (parallelization process step).

[0050] The parallelization processing unit 104 parallelizes images P1 and P2 by increasing the resolution per unit pixel in the baseline direction within a defined viewing angle range in a plane including the vertical axis, relative to images P1 and P2, thereby generating a first parallelized image and a second parallelized image. Specifically, the parallelization processing unit 104 parallelizes images P1 and P2 with a resolution of pixels per unit viewing angle by using a parallelization function designed such that, within a defined vertical and horizontal viewing angle range in the parallelized image, the resolution of any viewing angle is above the resolution along the image center axis and below the resolution along the epipolar line of the widest angle. This function represents the image height per viewing angle in the horizontal direction after parallelization.

[0051] The parallelized images P1 and P2 (the first parallelized image and the second parallelized image) are output from the parallelization processing unit 104 to the image interpolation unit 105. The parallelization process described above can also sequentially perform coordinate transformations of each image coordinate in the image transformation, but since recalculation is inefficient, an image transformation table is prepared to preserve the original image coordinates for each pixel of the parallelized image, and the parallelized image is generated based on this table. The generation of the parallelized image transformation table is performed during the prior calibration of the stereo camera.

[0052] The image interpolation unit 105 performs demosaic processing on the parallelized images P1 and P2. Demosaic processing may include, for example, converting the Bayer image into a color image or a grayscale image. Furthermore, the image interpolation unit 105 outputs the demosaiced images P1 and P2 to the parallax image generation unit 106.

[0053] The disparity image generation unit 106 uses one of the first parallelized image and the second parallelized image as a reference image and the other as a reference image. For an image projected onto a certain vertical coordinate in the reference image, it searches along a roughly horizontal direction within the reference image to find the same image projected onto a vertical coordinate in the reference image that has a certain difference from the vertical coordinate, thereby generating disparity (disparity generation step). The disparity image generation unit 106 is an example of a disparity calculation unit. It uses the image captured by the first camera (image P1) and the image captured by the second camera (image P2), which have undergone image correction processing by the image interpolation unit 105, to calculate disparity and generate a disparity image. Here, the disparity image generation unit 106 performs stereo matching processing on the images P1 and P2 input from the image interpolation unit 105 to calculate disparity. The images P1 and P2 input from the image interpolation unit 105 are the images P1 and P2 after image preprocessing by the image preprocessing unit 103 and after various processing by the parallelization processing unit 104 and the image interpolation unit 105.

[0054] Stereo matching processing refers to the process of searching for the region with the highest similarity to a pixel of interest in image P2 within the same area of ​​image P1 that corresponds to a region of a specific range centered on that pixel. Matching methods include, for example, SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference). In this embodiment, the disparity image generation unit 106 uses the horizontal difference between corresponding pixels in image P2 and image P1 as the disparity of that pixel, and calculates the disparity for all pixels in image P2 to generate a disparity image. The disparity image generation unit 106 outputs the generated disparity image to the stereo recognition unit 107 (object recognition unit).

[0055] The stereoscopic object recognition unit 107 detects stereoscopic objects using parallax images, identifies the category of the stereoscopic object (e.g., pedestrian, bicycle, car, motorcycle, light spot, etc.) through image recognition, and recognizes the image containing a defined area of ​​the identified stereoscopic object as the image of the stereoscopic object (object recognition step). Additionally, the stereoscopic object recognition unit 107 measures the distance to the identified stereoscopic object. The stereoscopic object recognition unit 107 outputs information related to the image of the identified stereoscopic object, the category of the stereoscopic object, and the measured distance to the vehicle control unit 108.

[0056] The vehicle control unit 108 implements vehicle control such as acceleration, deceleration, and steering based on the image of the three-dimensional object identified by the three-dimensional object recognition unit 107, the type of the three-dimensional object, distance-related information, and the vehicle's speed, in order to perform emergency braking or follow-the-leader actions.

[0057] [3D calibration processing]

[0058] Next, the stereo calibration in the image processing device 10 will be explained. Figure 2 This is a flowchart illustrating the stereo calibration process of the image processing apparatus 10 in this embodiment.

[0059] First, a calibration chart is captured (step S21). Next, using the captured calibration chart, the camera's intrinsic and extrinsic parameters are determined (step S22). Here, the intrinsic parameters include the origin of the image plane, the focal lengths in the vertical and horizontal directions of the image, and the radiation distortion parameter, which represents the radiation distortion of the image point on the subject image relative to the origin. The radiation distortion parameter is the coefficient of each term in a Taylor expansion of the image height characteristic function following the projective equation of a fisheye camera, centered at the incident angle, and is expressed as k in the following formula. 2n+1 .

[0060]

[0061] In addition, the external parameters are the rotation and translation matrices that transform the camera coordinate system from one camera to the camera coordinate system of the other camera. As a representative method for obtaining these parameters, there are also implementations in OpenCV, such as "Zhengyou Zhang. A flexible new technique for camera calibration. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 22(11):1330-1334,2000." However, as long as the internal and external parameters can be accurately estimated, the method is not limited.

[0062] Next, based on the obtained internal and external parameters, a resolution-maintaining parallel image transformation table is generated to maintain the resolution of the sensor image (step S23). The generated parallel image transformation table is stored in the parallel image transformation table storage unit 109. The generation of this resolution-maintaining parallel image transformation table will be described in detail below.

[0063] [Resolution-Preserving Parallelized Image Transformation Table Generation]

[0064] Figure 3 It is a graph showing the relationship between the incident ray and its position on the image coordinate system. For example... Figure 3 As shown in (1), the camera's coordinate system is set to (X, Y, Z), with Z being the direction of the camera's optical axis. Furthermore, for the subject P, when the incident light rays to the camera use a sphere with a radius of focal length f as a virtual image plane, the point of contact between the sphere and the light ray is set to p. When this sphere is unfolded in a manner that results in a typical fisheye image, as shown... Figure 3 As shown in (2), the point corresponding to p is called p'.

[0065] When a fisheye lens mounted in a camera is designed for orthographic projection, the distance between the image and the image center, i.e., the image height characteristic, is expressed as follows: Let the distance from the origin (image center) be r.

[0066]

[0067] Angle θ is the angle between the incident light ray on the fisheye lens and the optical axis of the fisheye lens.

[0068] In actual lenses, due to deviations from the design, the radiation distortion parameters of the aforementioned internal parameters are used. However, for simplicity, this will be explained as a lens where Equation (2) is used as the image height function. The coordinates of p' are represented using the image height function fsinθ, which follows the aforementioned θ, and φ in the figure.

[0069] Figure 4 This diagram represents two stereo cameras arranged side-by-side with the aforementioned fisheye cameras. The coordinate systems of the stereo cameras, after being parallelized, are set as (X...). r ,Y r Z r ), (X l ,Y l Z l The coordinate systems before parallelization are set as (X'). r ,Y' r ,Z' r ), (X' l ,Y' l ,Z' lAdditionally, for the subject P, when the incident light rays to the camera are represented by a sphere with radius f as the virtual image plane, the point of contact between the sphere and the light ray is set as p. l ,p r Parallelization requires transforming the coordinate system so that the camera's optical axis is perpendicular to the line connecting the center of the left and right cameras (called the baseline). The transformed optical axis is called the parallelized optical axis. The direction of the parallelized optical axis has degrees of freedom, but it is defined as the front direction relative to the position where the housing is mounted.

[0070] In parallelization, besides setting the coordinate system, defining the image coordinates is also crucial. (Line E on the sphere) l E r It is the epipolar line. When the sphere is unfolded into an image plane, the epipolar line needs to be parallel to the horizontal axis of the image coordinate system.

[0071] Figure 5 This is a diagram illustrating one method for obtaining an image coordinate system that achieves such parallelization. For example... Figure 5 As shown in (1), the direction of the light rays about the subject P is determined by the pitch angle φ′ and the origin θ containing the subject P and the left and right sides. l ,0 r The yaw angle θ′ in the plane is represented as follows: Figure 5 As shown in (2), the image coordinate system is set as the horizontal axis fsinθ′ and the vertical axis fsinφ′, thus achieving parallelization. The method of obtaining the horizontal and vertical axes of the image coordinate system has degrees of freedom. The horizontal axis can be set as a function x(θ′) of any θ′, and the vertical axis can be set as a function y(φ′) of any φ′. Here, the function x(θ′) is called the parallelization function.

[0072] Figure 6 It means to include Figure 4 The subject p l And the origin 0 on the left and right l ,0 r The planar image. Sinθ′ is obtained by stereo matching of the left and right parallelized images. l , Sinθ′ r and as the following formula

[0073]

[0074] By determining the distance D from the baseline length B, the three-dimensional position of the subject P can be measured.

[0075] Although it is mentioned that there are degrees of freedom in obtaining the coordinate system of the parallelized image, high-precision stereo matching is required to accurately calculate the distance D in formula (3). If the horizontal resolution deteriorates during the transformation to the parallelized image, the accuracy of the matching position will also deteriorate. Therefore, the design of this parallelization function requires special attention.

[0076] Figure 7 This diagram illustrates the process of transforming the image to a parallelized form in this embodiment. Figure 7 (1) is the image before parallelization. Figure 7 (2) is an intermediate image between before and after parallelization. Figure 7 (3) is the image after parallelization. Figure 7 Image 91 of (1) is a diagram showing the coordinate system of a fisheye lens with a 180° viewing angle before parallelization, along with the epipolar lines. A lens does not necessarily have a 180° viewing angle; it can have any viewing angle, but it is set to 180° for ease of understanding. P n P s These are the two ends of the central axis of the image (polar line E0), which are the so-called North and South Poles in latitude and longitude coordinates.

[0077] The parallelization processing unit 104 parallelizes the image 91 before parallelization by increasing the resolution per unit pixel in the baseline direction within a defined viewing angle range in a plane including the vertical axis, thereby generating a parallelized image. Here, parallelization is performed using a parallelization function that represents the image height per viewing angle in the horizontal direction after parallelization, where the resolution of the first imaging unit 12 and the second imaging unit 13 is above the resolution along the image center axis and below the resolution along the epipolar line of the widest angle.

[0078] exist Figure 7 (1) to Figure 7 In (3), the epipolar line of the unparallelized image 91 is set to E, the epipolar line of the intermediate image 92 is set to E′, and the epipolar line of the parallelized image 93 is set to E′′. First, the epipolar line of the unparallelized image 93 is set to E′′. Figure 7 The image 91 before parallelization of (1) shows all the polar lines E of the fisheye lens coordinate system image, as shown in (1). Figure 7 As shown in the intermediate image 92 of (2), the image is unfolded in a way that maintains the distortion while becoming horizontal. That is, the image is unfolded so that all epipolar lines become parallel straight lines while maintaining resolution (length). And, as Figure 7 As shown in the parallelized image 91 of (3), in this embodiment, according to... Figure 7 The vertical viewing angle reflected in the middle image 92 of (2) is the widest angle polar line E′3, which makes the image parallel.

[0079] As can be seen from the epipolar line E′ of the intermediate image 92, in the intermediate image 92, the epipolar line with a larger vertical coordinate, for example, the epipolar line E′3 has a higher resolution relative to the epipolar line E′0. In this case, if the image is parallelized using the parallelization function fsinθ′, it is the same as parallelizing according to the shortest epipolar line E′0. Therefore, the epipolar line E′′3 of the parallelized image 93 is compressed compared to the epipolar line E′3 of the intermediate image 92, and the resolution is degraded compared to the original image 91 before parallelization.

[0080] In contrast, in this embodiment, parallelization is performed using a parallelization function that represents the image height per viewpoint in the horizontal direction after parallelization. Specifically, as in Figure 7 As with the parallelized image 93 of (3), in order to fix the resolution of the entire image, the epipolar line E′3, which is the widest vertical viewing angle reflected in the image, is used as the baseline for image parallelization. That is, the parallelization function is used to parallelize the image as follows.

[0081]

[0082] Therefore, the resolution of any viewpoint within the image becomes higher than the resolution along the image's central axis and lower than the resolution along the widest epipolar line. Thus, image parallelism can be achieved without degrading the resolution of all epipolar lines E. Furthermore, further enlargement would waste image memory capacity, so it can be said that an upper limit to enlargement has been shown. However, if the epipolar line E′3 of the intermediate image 92 is larger than the camera lens's viewpoint, the epipolar line E′3 of the intermediate image 92 is not included in the intermediate image 92, but even in that case, the effect of parallelism achieved without degrading the resolution can still be obtained.

[0083] In this embodiment, the image height characteristic of the fisheye lens is set to r = fsinθ, but in reality, the image height characteristic of the lens includes deviations from the design. Furthermore, the projection method is not limited to orthographic projection fsinθ; for example, it can also be a general equidistant projection fθ, a solid angle projection 2fsin(θ / 2), or an equal solid angle projection 2ftan(θ / 2), and other projections can also achieve the effects of this invention. Therefore, the image height characteristic can also be generalized to r = r(θ). Furthermore, the epipolar line in the generalized case lies on the fisheye coordinates of the image 91 before parallelization, and φ is treated as a constant, which can be expressed as the following formula:

[0084]

[0085] Here, x(θ′), which is horizontally expanded from the polar line of the vertical viewing angle φ′ as shown in the middle image 92, can be written as the following formula:

[0086]

[0087] It can be calculated using the relationship cosθ=cosθ′cosφ′.

[0088] [Hardware Configuration Example of Image Processing Device 10]

[0089] Next, the hardware configuration of the image processing apparatus 10 of this embodiment will be described. Figure 8 This is a block diagram illustrating an example of the hardware configuration of the image processing apparatus 10 in this embodiment.

[0090] like Figure 8 As shown, the image processing apparatus 10 includes a CPU (Central Processing Unit) 81, a ROM (Read-Only Memory) 82, a RAM (Random Access Memory) 83, a storage device 84, an input / output interface 85, and a bus 86. The bus 86 is a signal path that electrically connects the various components to enable the input and output of information data between the various components.

[0091] CPU 81 controls the operation of various units within the image processing apparatus 10. For example, CPU 81 controls the processing of the image acquisition unit 101, the image preprocessing unit 103, and the image interpolation unit 105. Additionally, CPU 81 controls the processing of the parallelization processing unit 104, the image interpolation unit 105, the parallax image generation unit 106, the stereoscopic object recognition unit 107, and the vehicle control unit 108. Furthermore, a GPU (Graphics Processing Unit) may be used instead of CPU 81, or both CPU 81 and GPU (Graphics Processing Unit) may be used in combination.

[0092] ROM 82 is composed of a storage medium such as non-volatile memory, and stores programs, data, etc., executed and referenced by CPU 81. RAM 83 is composed of a storage medium such as volatile memory, and temporarily stores information (data) required for various processes performed by CPU 81.

[0093] Storage device 84 is composed of a computer-readable, non-transitory storage medium that stores software programs executed by CPU 81, such as a storage device like an HDD (Hard Disk Drive). Storage device 84 stores programs used by CPU 81 to control various components, programs for the operating system (OS), controllers, etc., and data. Furthermore, a portion of the programs and data stored in storage device 84 may also be stored in ROM 82. Additionally, the computer-readable, non-transitory storage medium storing programs executed by CPU 81 is not limited to an HDD; for example, it may be an SSD (Solid State Drive), CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, or other storage media.

[0094] The input / output interface 85, controlled by the CPU 81, transmits and receives information data with the external environment of the image processing device 10. Through the coordinated operation of these hardware components and software programs, the image processing device 10 enables... Figure 1 The various constituent parts described in the text are materialized.

[0095] [Effect]

[0096] As described above, in the image processing apparatus 10 of this embodiment, a parallel image transformation table is generated to perform parallelization while maintaining resolution, and stored in the parallel image transformation table storage unit 109. This parallel image transformation table enables the generation of parallel images that balance ranging accuracy and memory efficiency.

[0097] Furthermore, the present invention is not limited to the above-described embodiments. Various other applications and modifications may be made as long as they do not depart from the spirit of the invention as described in the claims.

[0098] For example, the above-described embodiments are provided to facilitate understanding of the present invention by detailing and specifying the configuration of the image processing apparatus, but are not limited to having all the described configurations. Furthermore, a portion of the configuration of the embodiments described herein can be replaced with the configuration of other embodiments, and it is further possible to add the configuration of other embodiments to the configuration of a certain embodiment. Additionally, other configurations can be added, deleted, or replaced for a portion of the configuration of each embodiment.

[0099] Furthermore, while the designation of control lines or information lines is considered necessary in the description, it is not limited to representing all control lines or information lines on the product. In fact, almost all components can be considered interconnected.

[0100] <Second Implementation>

[0101] In the first embodiment, regarding the method for selecting the baseline, parallelization was performed by using the epipolar line, which represents the widest vertical viewing angle reflected in the image, to keep the resolution constant across the entire image. Alternatively, the range of the vertical viewing angle required for accurate distance measurement can be limited. That is, image parallelization may not be performed across the entire image, but rather by keeping the resolution constant within a defined range of vertical viewing angles.

[0102] Figure 9 This is a diagram showing the positional relationship between the bumper and the vehicle-mounted camera.

[0103] For example, on road surface 121, there are speed bumps 120 designed to discourage speed. If vehicle V attempts to pass over a speed bump at high speed, it will experience a significant impact, potentially affecting passenger comfort, driving operation, and the overall performance of vehicle V. Therefore, precise distance measurement of the bumps 120 is required for occupant warnings or braking control of vehicle V.

[0104] Let the configuration height of the stereo camera 100 above the ground be h, and the distance along the optical axis from the stereo camera 100 to the Bump 120 be defined as the ranging distance L1. The vertical viewing angle α is tan ... -1 (h / L1). For example, if the vehicle V is a common vehicle such as a passenger car, then when h=1.6m and L=5m, α=approximately 20°.

[0105] Figure 10 This diagram illustrates the process of transforming the image to a parallelized form in this embodiment. Figure 10 (1) is the image before parallelization. Figure 10 (2) is an intermediate image between before and after parallelization. Figure 10 (3) is the image after parallelization.

[0106] exist Figure 10 In the example shown, the polar line E passing through the vertical angle α will be... α Parallelization is performed as a parallelization function. Parallelization in this way will not degrade the resolution within the field of view of the Bump120, where accurate detection of the object being detected is desired.

[0107] The parallelization processing unit 104 parallelizes the image 140 in the vertical direction such that, relative to the image 140 before parallelization, the resolution per unit pixel in the baseline direction is increased within a predetermined viewing angle range r(α) in a plane including the vertical axis, and the resolution per unit pixel in the baseline direction is not increased in a range outside the predetermined viewing angle range r(α), thereby generating a parallelized image 142. That is, in the region below the vertical viewing angle α (within the viewing angle range r(α)) for the direction perpendicular to the direction in which the multiple cameras are arranged, the first parallelized image and the second parallelized image have more pixels per unit viewing angle in the direction in which the multiple cameras are arranged compared to the first captured image and the second captured image; and in the region larger than the vertical viewing angle α (outside the viewing angle range r(α)) for the direction perpendicular to the direction in which the multiple cameras are arranged, the first parallelized image and the second parallelized image have fewer pixels per unit viewing angle in the direction in which the multiple cameras are arranged compared to the first captured image and the second captured image. Figure 10 (1) to Figure 10 In (3), the epipolar line of the unparallelized image 140 is set as E. α Set the epipolar line of the intermediate image 141 to E′. α The epipolar line of the parallelized image 142 is set as E′′. α .

[0108] Figure 10 The image 140 before parallelization of (1) shows all the polar lines E of the fisheye lens coordinate system image, as shown in (1). Figure 10 As shown in the intermediate image 141 of (2), the image is unfolded in a way that makes it horizontal while maintaining the distortion. That is, the image is unfolded so that all epipolar lines become parallel straight lines while maintaining the resolution (length). And, as Figure 10 As shown in the parallelized image 142 of (3), the epipolar line E′′ passing through the vertical viewing angle α will be... α The image is parallelized using the baseline of the parallelization function. As a result, the resolution per unit pixel in the baseline direction in a specified viewing angle range r(α) in a plane including the vertical axis is improved relative to the image 140 before parallelization, and the image is parallelized without degrading the resolution in the field of view of the Bump 120, which is intended to accurately detect the object being detected.

[0109] Furthermore, the parallelization processing unit 104 can also set the vertical viewing angle according to the type of object. Moreover, the conditions for the vertical viewing angle can be applied to different regions by dividing the area from the horizontal viewing angle. Figure 11 This diagram represents a bump 120 extending along the width of road surface 121 and a pedestrian 122 attempting to cross road surface 121. Figure 12It is to compare the image 140 of the fisheye lens's coordinate system before parallelization with the polar line E. α E β The diagram is shown together. When it is assumed that these Bump120 and pedestrian122 are in the nearest vicinity of this vehicle V, the vertical angle becomes within the angle range r(α) as long as it is within the horizontal angle γ1, and the vertical angle can become within the angle range r(β) as long as it is within the horizontal angle γ2.

[0110] At this time, as Figure 12 In that case, the polar line E corresponding to these vertical perspectives can also be selected for each region of the horizontal perspective. α and E β The function connecting these parallelization functions is then used as a new parallelization function for parallelization. That is, the polar line E′′ passing through the vertical viewpoint α can also be parallelized within the range of the horizontal viewpoint γ1. α Image parallelization is performed using the baseline of the parallelization function, and within the range of the horizontal viewing angle γ2, the epipolar line E′′ passing through the vertical viewing angle β will be used. β The image is parallelized using the baseline of the parallelization function. Furthermore, these vertical perspectives α and β can also be differentiated based on the vehicle's speed V. That is, the vertical perspectives α and β can be changed according to the speed of the moving object. Figure 12 The image 140 is shown before parallelization, with the epipolar line E selected at the horizontal viewing angle γ1. α In the horizontal perspective γ2, select the polar line E. β Under these conditions, in other words, the parallelization processing unit 104 can parallelize the captured image based on a parallelization function that outputs the image height of the multiple cameras in different orientations depending on the viewing angle range of the multiple cameras.

[0111] Furthermore, as an example of a defined upper limit for the vertical angle of view, this can also occur when the camera's angle of view is less than 180°. In lenses where the polar angle of view differs from 180°, the epipolar line that becomes the upper limit sometimes does not simply become an epipolar line.

[0112] Figure 13 This is a parallelized diagram representing the case of a wide-angle lens with a field of view of less than 180°.

[0113] Figure 13 Image 183 of (1) is a diagram showing the coordinate system of the wide-angle lens before parallelization, along with the epipolar lines. Figure 13 (2) is an intermediate image between before and after parallelization. Figure 13 (3) is the image after parallelization.

[0114] If the camera's maximum vertical angle of view is α, then with Figure 10 Similarly, determine the polar line E.α In the range outside the maximum vertical viewing angle α, parallelization is performed without increasing the resolution per pixel in the baseline direction. At this point, the final image size changes according to the camera's maximum horizontal viewing angle. For example, as... Figure 13 As with image 183 before parallelization (1), in the case of a wide-angle lens with a maximum horizontal field of view smaller than that of image 180, such as Figure 13 The intermediate image 181 of (2) and Figure 13 Like the parallelized image 182 of (3), the horizontal dimension also becomes smaller.

[0115] Furthermore, when following the vertical polar line, resolution cannot be guaranteed for areas protruding diagonally from the camera's field of view. Therefore, as... Figure 14 In that way, for each horizontal viewing angle γ1, γ2, the maximum epipolar line can be determined and parallelized without increasing the resolution.

[0116] Furthermore, so far, appropriate epipolar lines have been selected, and parallelization has been performed according to a parallelization function that follows those epipolar lines. However, from the point of view of maintaining resolution, it is not necessary to use a parallelization function that follows epipolar lines, as long as the resolution, defined at least by the number of pixels per unit viewpoint, is higher than that in any horizontal viewpoint. Figure 7 If the epipolar line E0 is larger than the epipolar line E3, then the image resolution will not be degraded across the entire image, nor will it be unnecessarily enlarged. If we set the parallelization function following the epipolar line E0 as x0(θ′) and the parallelization function following the epipolar line E3 as x3(θ′), then we can obtain x(θ′) as shown in the following equation by using any θ′.

[0117]

[0118] For example, if formulas (2) and (4) used in the first embodiment are used, then...

[0119]

[0120] If it becomes the above formula, we can obtain the desired x(θ′).

[0121] Under the aforementioned resolution conditions, a baseline can also be selected to replace the polar lines. Figure 15 This indicates the appropriate baseline L for being such an example.

[0122] <Third Implementation>

[0123] The configuration, or most of the processing, in this embodiment is the same as in the first embodiment. The difference lies in the relationship between the parallelized incident light rays and the incident position on the image coordinates, as follows: Figure 16 As shown, thus replacing Figure 5 Specifically, the angle from the X-axis when the light ray projects onto the ZX plane is set as θ′.

[0124] in addition, Figure 17 It indicates that in Figure 6 The subject P is projected onto the ZX plane. At this time, D, calculated in the same way as formula (3), is not the distance from the baseline, but always represents the depth. In this embodiment, the depth can be directly obtained from the distance calculated from the parallax, so the projection method of the sensor used for front sensing becomes a suitable configuration.

[0125] While the embodiments of the present invention have been described in detail above, the present invention is not limited to the above-described embodiments. Various design changes can be made without departing from the spirit of the invention as set forth in the claims. For example, the embodiments described in detail above are for the purpose of making the present invention easy to understand, but are not limited to having all the described configurations. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can also be added to the configuration of a certain embodiment. Furthermore, other configurations can be added, deleted, or replaced for a part of the configuration of each embodiment.

[0126] Symbol Explanation

[0127] 10…Image processing device, 11…Camera unit (multiple camera units), 12…First camera unit (left camera unit), 13…Second camera unit (right camera unit), 100…Stereo camera, 101…Image acquisition unit, 103…Image preprocessing unit, 104…Parallelization processing unit, 105…Image interpolation unit, 106…Parallelism image generation unit, 107…Stereoscopic object recognition unit (object recognition unit), 108…Vehicle control unit, 109…Parallelization image transformation table storage unit (transformation information storage unit).

Claims

1. An image processing apparatus, characterized in that, have: An image acquisition unit acquires a first image and a second image captured by each of multiple camera units; The parallelization processing unit takes a direction perpendicular to the baseline direction defined by the plurality of camera units and the optical axis direction of the plurality of camera units as the vertical axis, and parallelizes the first captured image and the second captured image in such a way that the vertical coordinates of the same image captured in the first captured image and the second captured image are consistent at least at any point within the first captured image and the second captured image, thereby generating a first parallelized image and a second parallelized image. The disparity image generation unit generates disparity based on a plurality of the parallelized images; as well as The object recognition unit identifies objects based on the parallax. The parallelization processing unit parallelizes the first captured image and the second captured image in such a way that the resolution per unit pixel in the baseline direction within a defined viewing angle range in a plane including the vertical axis is increased relative to the first captured image and the second captured image.

2. The image processing apparatus according to claim 1, characterized in that, When a predetermined vertical viewing angle α is set for a direction perpendicular to the direction in which the plurality of cameras are arranged, In a direction perpendicular to the direction in which the plurality of cameras are arranged, in the region below the vertical viewing angle α, the first parallelized image and the second parallelized image have more pixels per unit viewing angle than the first captured image and the second captured image, at least in the direction in which the plurality of cameras are arranged. In a direction perpendicular to the direction in which the plurality of cameras are arranged, in a region larger than the vertical viewing angle α, the first parallelized image and the second parallelized image have fewer pixels per unit viewing angle relative to the first captured image and the second captured image, at least in the direction in which the plurality of cameras are arranged.

3. The image processing apparatus according to claim 1, characterized in that, The parallelization processing unit performs the parallelization using the parallelization function expressed by the following formula (4). Where f is the focal length, and θ′ is the yaw angle in the plane containing the direction in which the subject and the plurality of cameras are arranged.

4. The image processing apparatus according to claim 1, characterized in that, The parallelization processing unit parallelizes the captured image based on a parallelization function that outputs the image height of the multiple cameras in the direction in which they are arranged, depending on the viewing angle range of the multiple cameras.

5. The image processing apparatus according to claim 2, characterized in that, When the configuration height of the plurality of cameras above the ground is set to h, and the corresponding ranging distance is set to L1, Let the vertical angle α be tan -1 (h / L1) or higher.

6. The image processing apparatus according to claim 1, characterized in that, The epipolar lines of the plurality of cameras are parallel to the direction in which the plurality of cameras are arranged in the first parallelized image and the second parallelized image.

7. The image processing apparatus according to claim 1, characterized in that, It has a transformation information storage unit that stores transformation information for the radial distortion of the lenses mounted on the plurality of camera units. In the parallelization processing unit, based on the transformation information, the first captured image and the second captured image are parallelized in such a way that the parallelization magnification of the vertical and horizontal axes of the first parallelized image and the second parallelized image is determined by the optical distortion of the lens following the radial distortion of the lens.

8. The image processing apparatus according to claim 1, characterized in that, The parallelization processing unit expands the first and second captured images by converting the epipolar lines of the first and second captured images into parallel straight lines while maintaining their lengths. An epipolar line is selected as the baseline to make the resolution of the entire captured image fixed, and the first captured image and the second captured image are parallelized according to the baseline.

9. The image processing apparatus according to claim 8, characterized in that, The parallelization processing unit selects the epipolar line based on a predetermined vertical viewing angle reflected in the captured image.

10. The image processing apparatus according to claim 9, characterized in that, The parallelization processing unit selects the vertical viewing angle as the widest polar line.

11. The image processing apparatus according to claim 9, characterized in that, The parallelization processing unit sets the vertical viewing angle according to the category of the object.

12. The image processing apparatus according to claim 9, characterized in that, The parallelization processing unit changes the vertical viewing angle according to the speed of the mounted mobile body.

13. The image processing apparatus according to claim 9, characterized in that, The parallelization processing unit sets the vertical angle for each region of the horizontal angle of the captured image.

14. An image processing method, characterized in that, Include: The image acquisition step involves acquiring a first image and a second image captured by each of the multiple camera units. The parallelization process involves taking a direction perpendicular to the baseline direction defined by the plurality of cameras and the optical axis direction of the plurality of cameras as the vertical axis, and parallelizing the first captured image and the second captured image in such a way that the vertical coordinates of the same image captured in the first captured image and the second captured image are consistent at least at any point within the first captured image and the second captured image, thereby generating a first parallelized image and a second parallelized image. The disparity generation step generates disparity based on multiple parallelized images; as well as The object recognition step involves identifying objects based on the parallax. In the parallelization process, the first captured image and the second captured image are parallelized in such a way that the resolution per unit pixel in the baseline direction within a defined viewing angle range in a plane including the vertical axis is increased relative to the first captured image and the second captured image, thereby generating the first parallelized image and the second parallelized image.

Images