A distortion correction method and device, vehicle-mounted equipment and vehicle

Abstract

The disclosure relates to a distortion correction method and device, vehicle-mounted equipment and vehicle, and particularly relates to the technical field of computer vision. The method comprises: obtaining a first horizontal field of view angle and a first vertical field of view angle of each pixel in a pixel coordinate system of a to-be-processed image, and a second horizontal field of view angle and a second vertical field of view angle of a plurality of coordinate points on a hemispherical model in a camera coordinate system; determining a target coordinate point from the plurality of coordinate points in the camera coordinate system; determining a pixel value corresponding to the target coordinate point from the to-be-processed image based on the target coordinate point; determining projection image data according to the target coordinate point, the pixel value corresponding to the target coordinate point and projection parameters; and performing rendering based on the projection image data to obtain a target image after distortion correction. The embodiment of the disclosure is used to solve the problem of low efficiency of existing distortion correction.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and in particular to a distortion correction method, device, vehicle-mounted equipment, and vehicle. Background Technology

[0002] To eliminate blind spots and meet driving requirements, in-vehicle rearview systems are equipped with fisheye cameras with a large field of view. However, images captured by fisheye cameras suffer from severe distortion, which does not conform to users' visual habits. Existing technology first calibrates the fisheye camera to estimate its internal and external parameters; then, it establishes an imaging model of the fisheye camera based on these parameters; finally, it corrects the image using this model to obtain an image that meets human visual perception. However, calibration is required whenever the fisheye camera's position changes, which is labor-intensive and affects the efficiency of distortion correction. Summary of the Invention

[0003] To solve, or at least partially solve, the above-mentioned technical problems, this disclosure provides a distortion correction method, apparatus, vehicle-mounted equipment, and vehicle, which can reduce the cumbersome calibration process and improve the efficiency of distortion correction.

[0004] To achieve the above objectives, the technical solutions provided by the embodiments of this disclosure are as follows:

[0005] Firstly, a distortion correction method is provided, including:

[0006] The first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed are obtained, as well as the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system. The image to be processed is an image captured by a camera.

[0007] In the camera coordinate system, the target coordinate point is determined from multiple coordinate points. The target coordinate point is the coordinate point where the second horizontal field of view and the first horizontal field of view are the same, and the second vertical field of view and the second vertical field of view are the same.

[0008] Based on the target coordinate point, determine the pixel value corresponding to the target coordinate point from the image to be processed;

[0009] Based on the target coordinate point, the corresponding pixel value, and the projection parameters, the projected image data is determined. The projection parameters are the projection parameters of the camera and the imaging plane.

[0010] Rendering is performed based on the projected image data to obtain the target image after distortion correction.

[0011] As an optional implementation of this disclosure, obtaining the first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed includes:

[0012] Based on the camera's distortion table, determine the first horizontal field of view and the first vertical field of view for each pixel of the image to be processed in the pixel coordinate system.

[0013] As an optional implementation of this disclosure, the first horizontal field of view and the first vertical field of view of each pixel of the image to be processed in the pixel coordinate system are determined according to the camera's distortion table, including:

[0014] Based on the image coordinates of each pixel in the image to be processed and the transformation relationship between the image coordinate system and the pixel coordinate system, the pixel coordinates of each pixel in the image to be processed are determined.

[0015] Based on the distortion table and the pixel coordinates of each pixel in the image to be processed, determine the first horizontal field of view and the first vertical field of view for each pixel.

[0016] As an optional implementation of this disclosure, before determining the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters, the method further includes:

[0017] Obtain projection parameters;

[0018] in

[0019] Projection parameters include: the camera's field of view, the camera's projection scale, and the camera's projection distance.

[0020] As an optional implementation of this disclosure, after rendering based on the projected image data to obtain the distortion-corrected target image frame, the method further includes:

[0021] Save the camera's rotation direction and the camera's field of view;

[0022] Based on the camera's rotation direction and field of view, process the remaining frames in the video to be corrected after the image to be processed.

[0023] As an optional implementation of this disclosure, the method is applied to a vehicle. After rendering based on the projected image data to obtain a distortion-corrected target image, the method further includes: identifying the target image to obtain road condition information, which includes at least one of the following: obstacle location, obstacle size, obstacle type, pothole location, pothole size, and pothole depth.

[0024] Control vehicle movement based on road condition information.

[0025] Secondly, a distortion correction device is provided, comprising:

[0026] The acquisition module is used to acquire the first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed, and the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system, wherein the image to be processed is an image captured by a camera; the coordinate determination module is used to determine the target coordinate point from multiple coordinate points in the camera coordinate system, wherein the target coordinate point is the coordinate point whose second horizontal field of view and the first horizontal field of view are the same, and whose second vertical field of view and the second vertical field of view are the same.

[0027] The pixel value determination module is used to determine the pixel value corresponding to the target coordinate point from the image to be processed based on the target coordinate point;

[0028] The projection module is used to determine the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters. The projection parameters are the projection parameters between the camera and the imaging plane.

[0029] The rendering module is used to render based on the projected image data to obtain the target image after distortion correction.

[0030] As an optional implementation of this disclosure, the acquisition module is specifically used for:

[0031] Based on the camera's distortion table, determine the first horizontal field of view and the first vertical field of view for each pixel of the image to be processed in the pixel coordinate system.

[0032] As an optional implementation of this disclosure, the coordinate determination module is specifically used to: determine the pixel coordinates of each pixel in the image to be processed based on the image coordinates of each pixel in the image to be processed and the transformation relationship between the image coordinate system and the pixel coordinate system;

[0033] Based on the distortion table and the pixel coordinates of each pixel in the image to be processed, determine the first horizontal field of view and the first vertical field of view for each pixel.

[0034] As an optional implementation of this disclosure, the projection module is further configured to: acquire projection parameters; wherein the projection parameters include: the camera's field of view, the camera's projection ratio, and the camera's projection distance.

[0035] As an optional implementation of this disclosure, the rendering module is further configured to: save the camera's rotation direction and the camera's field of view.

[0036] Based on the camera's rotation direction and field of view, process the remaining frames in the video to be corrected after the image to be processed.

[0037] As an optional implementation of this disclosure, the rendering module is further configured to: identify the target image to obtain road condition information, the road condition information including at least one of the following: obstacle location, obstacle size, obstacle type, pothole location, pothole size, and pothole depth;

[0038] Control vehicle movement based on road condition information.

[0039] Thirdly, an in-vehicle device is provided, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements a distortion correction method as described in the first aspect or any of its alternative embodiments.

[0040] Fourthly, a vehicle is provided, comprising: a distortion correction device according to the second aspect or any alternative embodiment thereof, or an on-board device as described in the third aspect.

[0041] Fourthly, a computer-readable storage medium is provided, comprising: storing a computer program on the computer-readable storage medium, wherein when the computer program is executed by a processor, it implements a distortion correction method as described in the first aspect or any of its optional embodiments.

[0042] Fifthly, a computer program product is provided, comprising: when the computer program product is run on a computer, causing the computer to implement the distortion correction method as described in the first aspect or any of its optional embodiments.

[0043] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0044] The technical solution provided in this disclosure firstly determines the target coordinate points of each pixel in the image to be processed in the camera coordinate system based on the field of view angle of each pixel in the pixel coordinate system and the field of view angles of multiple coordinate points on the virtual hemispherical model in the camera coordinate system. The points where the second horizontal field of view angle is the same as the first horizontal field of view angle, and the second vertical field of view angle is the same as the second vertical field of view angle, are then determined to identify the corresponding coordinate points of each pixel in the image to be processed on the virtual hemispherical model. Next, the pixel value corresponding to the target coordinate point is determined, thus mapping the image to be processed onto the virtual hemispherical model. Further, based on the target coordinate point, the pixel value of the target coordinate point, and projection parameters, projected image data is obtained. Finally, rendering is performed based on the projected image data to obtain the distortion-corrected target image. By using the same horizontal and vertical field of view angles as a mapping relationship, each pixel in the pixel coordinate system is mapped to the camera coordinate system, eliminating the need for camera calibration. This avoids the tedious work of camera calibration, reduces workload, and improves the efficiency of distortion correction. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0046] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0047] Figure 1A This diagram illustrates the relationship between the camera coordinate system, image coordinate system, and world coordinate system.

[0048] Figure 1B This is a schematic diagram illustrating the relationship between the image coordinate system and the pixel coordinate system;

[0049] Figure 2A This is a schematic diagram of barrel distortion and pincushion distortion in the prior art;

[0050] Figure 2B This is a schematic diagram illustrating the principle of tangential distortion generation in existing technologies;

[0051] Figure 3 This is a schematic diagram illustrating an implementation scenario of a distortion correction method according to an embodiment of this disclosure;

[0052] Figure 4 This is a schematic flowchart of a distortion correction method according to an embodiment of the present disclosure;

[0053] Figure 5 This is a schematic diagram of the principle of optical distortion.

[0054] Figure 6 This is a schematic diagram of the projection parameters;

[0055] Figure 7 This is a structural diagram of a distortion correction device according to an embodiment of the present disclosure;

[0056] Figure 8 This is a structural diagram of a vehicle-mounted device according to an embodiment of this disclosure. Detailed Implementation

[0057] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0058] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0059] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the key terms used in the description of the embodiments or the prior art will be introduced below:

[0060] Figure 1A This is a schematic diagram illustrating the relationship between the camera coordinate system, image coordinate system, and world coordinate system, as shown below. Figure 1A As shown.

[0061] The camera coordinate system, also known as the optical center coordinate system, has the optical center of the camera as the origin. The X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively, and the optical axis of the camera is the Z-axis. Its coordinate values ​​are represented by (Xc, Yc, Zc).

[0062] Image coordinate system: The origin is the center of the image plane of the charge-coupled device (CCD). The X-axis and Y-axis are parallel to the two vertical sides of the image plane, and their coordinate values ​​are represented by (x, y). The image coordinate system uses physical units (e.g., millimeters) to represent the position of pixels in the image.

[0063] World coordinate system: This is the absolute coordinate system of the objective three-dimensional world, also known as the objective coordinate system. Because a digital camera is placed in three-dimensional space, the world coordinate system is needed as a reference coordinate system to describe the position of the digital camera, and also to describe the position of any other object placed in this three-dimensional environment. Its coordinate values ​​are represented by (Xw, Yw, Zw).

[0064] Figure 1B This is a schematic diagram illustrating the relationship between the image coordinate system and the pixel coordinate system, such as... Figure 1B As shown. Pixel coordinate system: The origin is the upper left corner of the CCD image plane, and the X and Y axes are parallel to the X and Y axes of the image coordinate system, respectively. The coordinate values ​​are represented by (u, v).

[0065] Images captured by a digital camera are first generated as standard electrical signals, and then converted into digital images through analog-to-digital conversion. Each image is stored as an M × N array, where the value of each element in the M x N image represents the grayscale level of an image point. Each such element is called a pixel, and the pixel coordinate system is the image coordinate system with pixels as the unit.

[0066] Causes of distortion: Distortion is a geometric distortion of image formation. It is a distorted image phenomenon caused by the different magnification of different areas of the focal plane. The degree of distortion increases from the center of the image to the edge. The smaller the focal length, the larger the field of view, and the more severe the distortion.

[0067] Camera lens distortion can be divided into two categories: radial distortion and tangential distortion. Radial distortion is caused by the inherent characteristics of the lens itself (convex lens), and it occurs because light rays bend more far from the center of the lens than near the center. Radial distortion is distributed along the radius of the lens and mainly includes barrel distortion and pincushion distortion, such as... Figure 2A As shown, Figure 2A This diagram illustrates barrel distortion and pincushion distortion in existing technologies. Tangential distortion occurs because the lens itself is not parallel to the camera sensor plane (imaging plane) or image plane. This is often caused by installation misalignment when the lens is mounted on the lens module, such as... Figure 2B As shown, Figure 2B This is a schematic diagram illustrating the principle of tangential distortion generation in existing technologies.

[0068] Currently, to meet users' actual driving needs, existing in-vehicle rearview systems use fisheye cameras to capture images of the area behind the vehicle. While fisheye cameras have a large field of view, the images they capture are distorted and do not conform to users' visual habits. Existing technology first calibrates the fisheye camera to estimate its internal and external parameters; then, it establishes an imaging model of the fisheye camera based on these parameters; finally, it uses this imaging model to correct individual images to obtain images that meet human visual perception. However, calibration is required whenever the fisheye camera's position changes, which is a labor-intensive process and affects the efficiency of distortion correction.

[0069] In addition, existing technologies mainly use the central processing unit for image correction. However, when processing images captured by fisheye cameras, calculations are performed pixel by pixel, involving a large number of floating-point operations. This results in a large computational load, making it difficult to achieve real-time performance at 1080p resolution. Furthermore, the processor's clock speed is limited, making it difficult to guarantee the real-time performance of distortion correction.

[0070] To address the aforementioned problems, embodiments of this disclosure provide a distortion correction method, apparatus, vehicle-mounted device, and vehicle. The method first determines the target coordinate point in the camera coordinate system for each pixel of the image to be processed, based on the field of view angle of each pixel in the pixel coordinate system and the field of view angles of multiple coordinate points on a virtual hemispherical model in the camera coordinate system. The target coordinate point is determined by having the same second horizontal field of view angle as the first horizontal field of view angle and the same second vertical field of view angle as the second vertical field of view angle. This determines the corresponding coordinate point for each pixel of the image to be processed on the virtual hemispherical model. Then, the pixel value corresponding to the target coordinate point is determined, thus mapping the image to be processed onto the virtual hemispherical model. Further, projection image data is obtained based on the pixel coordinates of the target coordinate point, the pixel value of the target coordinate point, camera parameters, and projection parameters. Finally, the projection image data is rendered to obtain the distortion-corrected target image. By using the same horizontal and vertical field of view as a mapping relationship, each pixel in the pixel coordinate system is mapped to the camera coordinate system, eliminating the need for camera calibration. This avoids the tedious work of camera calibration, reduces workload, and improves the efficiency of distortion correction.

[0071] In some embodiments, this disclosure utilizes the Graphics Processing Unit (GPU) for parallel computing by calling the Open Graphics Library (OpenGL) interface. Since the GPU has a large-scale parallel throughput architecture with multiple concurrent threads, the distortion correction method provided in this disclosure overcomes the limitations of the processor's clock speed, achieving high-performance computing. OpenGL, also known as the Open Graphics Library, is a cross-language, cross-platform Application Programming Interface (API) for rendering 2D and 3D vector graphics.

[0072] like Figure 3 As shown, Figure 3 This is a schematic diagram illustrating an implementation scenario of the distortion correction method described in this embodiment. The vehicle 301 is equipped with a fisheye camera 302 and an electronic rearview mirror 303. The electronic rearview mirror 303 includes an image processing module 303a and a display module 303b. The image to be processed captured by the fisheye camera 302 is distorted and needs to be corrected by the image processing module 303a of the electronic rearview mirror 303 before being displayed by the display module 303b. This reduces blind spots behind the vehicle and meets the driving needs of the user.

[0073] This disclosure first determines the target coordinate point based on the field of view of each pixel in the pixel coordinate system of the image to be processed, and the field of view of multiple coordinate points on the virtual hemispherical model in the camera coordinate system. The target coordinate point is the same as the second horizontal field of view and the first horizontal field of view, and the same as the second vertical field of view and the second vertical field of view. Then, the pixel value corresponding to the target coordinate point is determined to map the image to be processed onto the virtual hemispherical model. Further, based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters, the projected image data is obtained. Finally, the projected image data is rendered to obtain the distortion-corrected target image.

[0074] The distortion correction method provided in this disclosure can be implemented through a distortion correction device or an in-vehicle device, which may include devices using fisheye cameras such as electronic rearview mirrors and panoramic surround view systems. This disclosure does not impose any limitations on this.

[0075] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a distortion correction method according to an embodiment of the present disclosure. The method includes:

[0076] S401. Obtain the first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed, and the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system.

[0077] In some embodiments, when performing image distortion correction, the target image to be distorted is first acquired. This acquisition method can be to acquire images in real time from image acquisition devices such as fisheye cameras. The fisheye camera can be deployed in vehicle-mounted equipment (such as electronic rearview mirrors) to acquire images of the vicinity of the vehicle in real time.

[0078] The images to be processed are taken with a fisheye camera and contain distortions that need to be corrected.

[0079] For the image to be processed, the first thing to obtain is the position coordinates of each pixel in the image coordinate system. For example, the coordinates of pixel A in the image are (X, Y). It can be understood that the image coordinate system is a coordinate system established with the center of the image as the origin.

[0080] Based on the position coordinates of each pixel in the image to be processed, the horizontal and vertical field of view angles of each pixel in the image coordinate system can be determined. Taking pixel A(X, Y) as an example, the component of pixel A on the x-axis of the image coordinate system is X, denoted as projection point Ax; the component of pixel A on the y-axis is Y, denoted as projection point Ay. Based on the relationship between the camera coordinate system Xc-O-Yc and the image coordinate system XOY, the angle Ax-O-O1 on the O-O1-X plane is determined as the horizontal field of view angle of pixel A in the image coordinate system, where Ax is positive and represents the right field of view angle, and Ax is negative and represents the left field of view angle. The angle Ay-O-O1 on the O-O1-Y plane is determined as the vertical field of view angle of pixel A in the image coordinate system, where Ay is positive and represents the lower field of view angle, and Ay is negative and represents the upper field of view angle.

[0081] Furthermore, based on the transformation relationship between the image coordinate system and the pixel coordinate system, and the distortion table, the horizontal and vertical field of view angles of each pixel in the image to be processed in the pixel coordinate system are determined. The position coordinates of pixel B in the pixel coordinate system are (Xb, Yb). Since the origin of the pixel coordinate system is the top-left vertex of the image to be processed, the position coordinates of the midpoint of the image to be processed in the pixel coordinate system are marked as (u0, v0). Therefore, the position coordinates of pixel B in the pixel coordinate system are (Xb-u0, Yb-v0). The distance l from pixel B to the midpoint (u0, v0) of the image to be processed is:

[0082]

[0083] In the formula, l corresponds to the true image height in the distortion table.

[0084] Figure 5 This is a schematic diagram of the principle of optical distortion, such as Figure 5 As shown, pixel A represents the actual pixel location in the image, and A' represents the distorted pixel location in the image to be processed. According to the distortion rate D in the distortion table:

[0085]

[0086] In the formula, It is the height of the true image in the distortion table. The reference image height is used. It should be noted that distortion is defined as the difference between the true image height and the reference image height. However, in practical applications, distortion is often expressed as a percentage of the ratio of distortion to the reference image height, which is called relative distortion, or distortion rate D.

[0087] The distortion table is configured by the manufacturer of the fisheye camera at the factory. For example, Table 1 is a distortion table for a certain fisheye camera. The table includes the angle, the height of the real image, the height of the reference image, and the distortion rate.

[0088]

[0089] Table 1

[0090] Therefore, based on the distortion rate D, the reference image height L can be obtained:

[0091]

[0092] Therefore, the theoretical pixel coordinates (xx, yy) are:

[0093]

[0094]

[0095] Furthermore, the horizontal field of view Vx and the vertical field of view Vy of pixel point B in the pixel coordinate system are determined.

[0096]

[0097]

[0098] In the formula, It represents the pixel focal length.

[0099]

[0100] The focal length is represented by the fisheye camera coordinates in mm, and 0.003 represents the pixel size of the fisheye camera's optical sensor. This is merely an illustrative example and not a specific limitation; therefore, the pixel focal length can be calculated based on the actual pixel size. .

[0101] In some embodiments, the image to be processed is any frame in the video to be corrected. The camera focal length corresponding to the current image to be processed is obtained; the camera focal length corresponding to the current image to be processed is compared with the camera focal length corresponding to the previous frame of the image to be processed to determine whether the camera has zoomed based on the comparison result. Embodiments of this disclosure can also obtain information such as the camera focal length corresponding to the current target image. By comparing the camera focal length corresponding to the current target image with the camera focal length corresponding to the previous frame of the target image, if they are the same, it indicates that the camera has not zoomed; if they are different, it indicates that the camera has zoomed. The above process is mainly to determine at what camera focal length each frame of the image was captured or whether the camera is in a real-time dynamic zoom situation, thereby enriching the application scenarios of the embodiments of this disclosure.

[0102] Furthermore, when the theoretical pixel x-coordinate xx is positive, the first horizontal field of view of pixel B in the image to be processed in the pixel coordinate system is the right field of view; when the theoretical pixel x-coordinate xx is negative, the first horizontal field of view of pixel B in the image to be processed in the pixel coordinate system is the left field of view. When the theoretical pixel y-coordinate yy is positive, the first vertical field of view of pixel B in the image to be processed in the pixel coordinate system is the lower field of view; when the theoretical pixel y-coordinate yy is negative, the first vertical field of view of pixel B in the image to be processed in the pixel coordinate system is the upper field of view.

[0103] In the above embodiment, after determining the horizontal and vertical field of view of each pixel in the image to be processed in the image coordinate system, the first horizontal and first vertical field of view of each pixel in the image to be processed in the pixel coordinate system are determined according to the transformation relationship between the image coordinate system and the pixel coordinate system and the distortion table of the fisheye camera.

[0104] Then, this disclosure calls OpenGL to establish a camera coordinate system based on the virtual camera in OpenGL. The plane facing the camera is the XOY plane, the right side is the positive x-axis, the bottom is the positive y-axis, and the vertical plane is the z-axis. A right-handed Cartesian three-dimensional coordinate system is established and determined as the camera coordinate system.

[0105] In the camera coordinate system, establish a hemispherical model. First, construct a sphere with radius r centered at the camera origin Q. Divide the hemisphere (taking the portion where z>0) into triangular facets. The coordinates of the vertex P of each triangular facet are (xr, yr, zr). The orthographic projection of P onto the XOZ(xr, 0, zr) plane is p. The angle poz between the vector Qp and the z-axis is:

[0106]

[0107] Let xr be the second horizontal field of view. If xr is positive, then the second horizontal field of view is the right field of view; if xr is negative, then the second horizontal field of view is the left field of view.

[0108] Let h be the orthographic projection of P(xr, yr, zr) onto the YOZ plane, and let the angle hoz be between the vector Qh and the z-axis.

[0109]

[0110] Let yr be the second vertical field of view, where yr is positive and is the lower field of view, and yr is negative and is the upper field of view.

[0111] S402. In the camera coordinate system, determine the target coordinate point from multiple coordinate points. The target coordinate point is the coordinate point where the second horizontal field of view and the first horizontal field of view are the same, and the second vertical field of view and the second vertical field of view are the same.

[0112] In some embodiments, after obtaining the second horizontal field of view and the second vertical field of view, the coordinate point where the first horizontal field of view and the second horizontal field of view are the same, and the first vertical field of view and the second vertical field of view are the same, is determined as the target coordinate point.

[0113] The above embodiments, based on the transformation relationship between the image coordinate system and the camera coordinate system, combined with the distortion table, can find the position of each pixel in the image to be processed after distortion correction without prior calibration of the fisheye camera, thus eliminating the workload required for camera calibration and further improving the efficiency of distortion correction.

[0114] S403. Based on the target coordinate point, determine the pixel value corresponding to the target coordinate point from the image to be processed.

[0115] In some embodiments, after determining the target coordinate point of each pixel of the image to be processed in the pixel coordinate system in the camera coordinate system, the pixel value corresponding to the target coordinate point is determined from the image to be processed, so as to apply the image to be processed as a texture image onto the hemispherical model.

[0116] S404. Determine the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters.

[0117] The projection parameters refer to the projection parameters between the camera and the imaging plane, including: the camera's viewing angle, the camera's projection ratio, and the camera's projection distance. This embodiment uses the OpenGL function `gluPerspective(fovy, aspect, zNear, zFar)` as the projection function, where `fovy` is the camera's viewing angle, `aspect` is the aspect ratio of the object on the imaging plane, `zNear` is the distance from the camera to the nearest imaging plane of the object, and `zFar` is the distance from the camera to the farthest imaging plane of the object. Figure 6 As shown, Figure 6 This is a schematic diagram of the projection parameters.

[0118] In some embodiments, user interface controls are created to facilitate user adjustment of the camera's position for viewing the vehicle's surroundings. The user interface controls can be configured to include three parameters: the angle 'a' between the line of sight and the x-axis, the angle 'b' between the line of sight and the y-axis, and the camera's field of view (fovy). As the user adjusts the camera's position using these three user interface controls, the image processor can acquire the camera's rotation direction and field of view in real time, thereby determining the data included in the projection parameters and the data included in the camera parameters.

[0119] The camera parameters include the camera's position coordinates, the camera's line of sight, and the camera's rotation direction. In some embodiments of this disclosure, the camera parameters are matrices corresponding to the OpenGL function gluLookAt(eye, at, up), where eye is the camera's position and also the origin of the coordinate system; at is the camera's line of sight, with the angle between the line of sight and the x-axis denoted as 'a' and the angle between the line of sight and the y-axis denoted as 'b'; and up is the camera's rotation direction. For example, to ensure that the image of the object captured by the camera is positive, the camera's rotation direction is set to (0, 1, 0) so that the top of the camera faces positively.

[0120] To better explain the meaning of camera parameters, you can imagine the camera as your head, the camera's position coordinates as the head's position coordinates, the camera's line of sight as the direction your eyes are looking at the object, and the camera's rotation direction as the direction your head is facing.

[0121] In practical applications, users can adjust camera parameters and obtain the corresponding camera parameters. For example, if a fisheye camera is installed on the side and rear of a vehicle, the user can adjust the fisheye camera to see the ground conditions, changing the original eye-level view to a top-down view.

[0122] After obtaining the above projection parameters and camera parameters, this embodiment of the present disclosure uses OpenGL and an image processor to process the texture image on the hemispherical model to obtain the projected image in the camera coordinate system.

[0123] The above embodiments adjust the position of the fisheye camera according to the user interface controls, thereby determining different camera rotation directions and viewing angles according to different scenarios, improving the scene adaptability of distortion correction, and enhancing the distortion correction effect according to different scenarios.

[0124] S405. Render based on the projected image data to obtain the target image after distortion correction.

[0125] In some embodiments, OpenGL is used to render the projected image, which can then be visualized and displayed on the screen, allowing the user to see the distortion-corrected target image.

[0126] In some embodiments, since the fisheye camera captures the video to be corrected, and the image to be processed is any frame of the video to be corrected, after the image to be processed is distorted to obtain the target image, the rotation direction of the fisheye camera and the camera's field of view are saved, thereby processing the remaining frames of the video to be corrected captured by the fisheye camera. This enables rapid processing of the video data captured by the fisheye camera after distortion correction of a certain frame, reducing the amount of computation required for distortion correction, and making video correction no longer limited by the processor's clock speed.

[0127] In some embodiments, the distortion correction method described herein is applied to a vehicle, specifically using a fisheye camera, such as a panoramic system. The fisheye cameras are installed at the front, rear, left, and right of the vehicle, and the number can be four or eight. Users can understand the surrounding conditions of the vehicle through the panoramic system. After rendering the projected image to obtain the distortion-corrected target image, the target image is identified to obtain road condition information, which includes at least one of the following: obstacle location, obstacle size, obstacle type, pothole location, pothole size, and pothole depth. Further, based on the road condition information, the vehicle's movement is controlled, or a prompt message is generated to alert the user to the presence of obstacles or potholes around the vehicle.

[0128] In some embodiments, the distortion correction method described in this disclosure is applied to a vehicle, specifically to the vehicle's electronic rearview mirror, with a fisheye camera mounted on the side and rear of the vehicle. After rendering the projected image to obtain the distortion-corrected target image, and upon detecting a reversing signal from the vehicle, object recognition is performed on the target image to obtain road condition information on the side and rear of the vehicle. Then, based on the road condition information, the vehicle's driving is controlled, or a prompt message is generated to alert the user that there is an obstacle or pothole behind the vehicle.

[0129] In summary, this disclosure provides a distortion correction method. First, based on the field of view of each pixel in the pixel coordinate system and the field of view of multiple coordinate points on a virtual hemispherical model in the camera coordinate system, the method identifies target coordinate points in the camera coordinate system for each pixel based on the second horizontal field of view being the same as the first horizontal field of view, and the second vertical field of view being the same as the second vertical field of view. This determines the corresponding coordinate point for each pixel in the virtual hemispherical model. Then, the pixel value corresponding to the target coordinate point is determined, mapping the image to be processed onto the virtual hemispherical model. Further, projection image data is obtained based on the target coordinate point, its pixel value, and projection parameters. Finally, rendering is performed based on the projection image data to obtain the distortion-corrected target image. By mapping each pixel in the pixel coordinate system to the camera coordinate system through the same horizontal and vertical field of view, camera calibration is unnecessary, thus avoiding the tedious work of camera calibration, reducing workload, and improving the efficiency of distortion correction.

[0130] like Figure 7 As shown, Figure 7 This is a structural diagram of a distortion correction device according to an embodiment of the present disclosure. The device includes:

[0131] The acquisition module 701 is used to acquire the first horizontal field of view and the first vertical field of view of each pixel of the image to be processed in the pixel coordinate system, and the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system; wherein, the image to be processed is an image captured by a fisheye camera;

[0132] The coordinate determination module 702 determines the target coordinate point from multiple coordinate points in the camera coordinate system. The target coordinate point is the coordinate point where the second horizontal field of view and the first horizontal field of view are the same, and the second vertical field of view and the second vertical field of view are the same.

[0133] The pixel value determination module 703 is used to determine the pixel value corresponding to the target coordinate point from the image to be processed based on the target coordinate point;

[0134] The projection module 704 is used to determine the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters. The projection parameters are the projection parameters of the camera and the imaging plane.

[0135] The rendering module 705 is used to render based on the projected image data to obtain the target image after distortion correction.

[0136] As an optional implementation of this disclosure, the acquisition module 701 is specifically used for:

[0137] Based on the distortion table of the fisheye camera, determine the first horizontal field of view and the first vertical field of view of each pixel in the image to be processed in the pixel coordinate system.

[0138] As an optional implementation of this disclosure, the coordinate determination module 702 is specifically used to: determine the pixel coordinates of each pixel in the image to be processed based on the image coordinates of each pixel in the image to be processed and the transformation relationship between the image coordinate system and the pixel coordinate system;

[0139] Based on the distortion table and the pixel coordinates of each pixel in the image to be processed, determine the first horizontal field of view and the first vertical field of view for each pixel.

[0140] As an optional implementation of this disclosure, the projection module 704 is further configured to: acquire projection parameters;

[0141] in,

[0142] Projection parameters include: the camera's field of view, the camera's projection scale, and the camera's projection distance.

[0143] As an optional implementation of this disclosure, the rendering module 705 is further configured to: save the camera's rotation direction and the camera's field of view.

[0144] Based on the camera's rotation direction and field of view, process the remaining frames in the video to be corrected after the image to be processed.

[0145] As an optional implementation of this disclosure, the rendering module 705 is further configured to: identify the target image to obtain road condition information, the road condition information including at least one of the following: obstacle location, obstacle size, obstacle type, pothole location, pothole size, and pothole depth;

[0146] Control vehicle movement based on road condition information.

[0147] In summary, this disclosure provides a distortion correction device. Based on the field of view of each pixel in the pixel coordinate system of the image to be processed, and the field of view of multiple coordinate points on a virtual hemispherical model in the camera coordinate system, the device identifies the coordinate points among these points where the second horizontal field of view is the same as the first horizontal field of view, and the second vertical field of view is the same as the second vertical field of view, as target coordinate points for each pixel in the camera coordinate system. This determines the corresponding coordinate point for each pixel in the image to be processed on the virtual hemispherical model. Then, the pixel value corresponding to the target coordinate point is determined, thus mapping the image to be processed onto the virtual hemispherical model. Further, based on the target coordinate point, the pixel value of the target coordinate point, and projection parameters, projected image data is obtained. Finally, rendering is performed based on the projected image data to obtain the distortion-corrected target image. By ensuring that the horizontal and vertical field of view are the same, each pixel in the pixel coordinate system is mapped to the camera coordinate system, eliminating the need for camera calibration and thus avoiding the tedious work of camera calibration, reducing workload, and improving the efficiency of distortion correction.

[0148] like Figure 8 As shown, Figure 8 This is a structural diagram of a vehicle-mounted device according to an embodiment of this disclosure. Figure 8 This disclosure provides an in-vehicle device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the various processes of the distortion correction method described in the above-described method embodiments. Furthermore, it achieves the same technical effects, and to avoid repetition, it will not be described again here.

[0149] This disclosure provides a vehicle that includes: the distortion correction device described in the above embodiments, or the vehicle-mounted equipment described in the above embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0150] This disclosure provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the distortion correction method described in the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0151] The computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0152] This disclosure provides a computer program product that stores a computer program. When the computer program is executed by a processor, it implements the various processes of the distortion correction method described in the above method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0153] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0154] In this disclosure, the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0155] In this disclosure, memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0156] In this disclosure, computer-readable media includes both permanent and non-permanent, removable and non-removable storage media. Storage media can store information using any method or technology; the information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0158] The above are merely specific embodiments of this disclosure, enabling those skilled in the art to understand or implement this disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to these embodiments, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distortion correction method, characterized in that, include: The first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed are obtained, as well as the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system. The image to be processed is an image captured by a camera. In the camera coordinate system, a target coordinate point is determined from the plurality of coordinate points. The target coordinate point is a coordinate point where the second horizontal field of view is the same as the first horizontal field of view, and the second vertical field of view is the same as the second vertical field of view. Based on the target coordinate point, determine the pixel value corresponding to the target coordinate point from the image to be processed; Based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters, the projected image data is determined, wherein the projection parameters are the projection parameters of the camera and the imaging plane; The projected image data is rendered to obtain the target image after distortion correction.

2. The method according to claim 1, characterized in that, The step of obtaining the first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed includes: Based on the camera's distortion table, the first horizontal field of view and the first vertical field of view of each pixel of the image to be processed are determined in the pixel coordinate system.

3. The method according to claim 2, characterized in that, The step of determining the first horizontal field of view and the first vertical field of view of each pixel of the image to be processed in the pixel coordinate system according to the camera's distortion table includes: Based on the image coordinates of each pixel in the image to be processed and the transformation relationship between the image coordinate system and the pixel coordinate system, the pixel coordinates of each pixel in the image to be processed are determined; Based on the distortion table and the pixel coordinates of each pixel in the image to be processed, the first horizontal field of view and the first vertical field of view of each pixel are determined.

4. The method according to claim 1, characterized in that, Before determining the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters, the method further includes: Obtain projection parameters; The projection parameters include: the camera's field of view, the camera's projection ratio, and the camera's projection distance.

5. The method according to claim 4, characterized in that, The image to be processed is any frame in the video to be corrected. After rendering based on the projected image data to obtain the target image frame after distortion correction, the process further includes: Save the camera's rotation direction and the camera's field of view; Based on the camera's rotation direction and the camera's field of view, process the remaining frames in the video to be corrected after the image to be processed.

6. The method according to claim 1, characterized in that, The method is applied to a vehicle. After rendering based on the projected image data to obtain a distortion-corrected target image, the method further includes: identifying the target image to obtain road condition information, the road condition information including at least one of the following: obstacle location, obstacle size, obstacle type, pothole location, pothole size, and pothole depth. Based on the road condition information, the vehicle's movement is controlled.

7. A distortion correction device, characterized in that, include: The acquisition module is used to acquire the first horizontal field of view and the first vertical field of view of each pixel in the pixel coordinate system of the image to be processed, and the second horizontal field of view and the second vertical field of view of multiple coordinate points on the hemispherical model in the camera coordinate system, wherein the image to be processed is an image captured by a camera; A coordinate determination module is used to determine a target coordinate point from the plurality of coordinate points in the camera coordinate system. The target coordinate point is a coordinate point where the second horizontal field of view and the first horizontal field of view are the same, and the second vertical field of view and the second vertical field of view are the same. A pixel value determination module is used to determine the pixel value corresponding to the target coordinate point from the image to be processed based on the target coordinate point; The projection module is used to determine the projected image data based on the target coordinate point, the pixel value corresponding to the target coordinate point, and the projection parameters, wherein the projection parameters are the projection parameters of the camera and the imaging plane. The rendering module is used to render based on the projected image data to obtain the target image after distortion correction.

8. A vehicle-mounted device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the distortion correction method as described in any one of claims 1 to 6.

9. A vehicle, characterized in that, include: The distortion correction device as described in claim 7, or the vehicle-mounted device as described in claim 8.

10. A computer-readable storage medium, characterized in that, include: A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the distortion correction method as described in any one of claims 1 to 6.

11. A computer program product, characterized in that, include: When the computer program product is run on a computer, the computer implements the distortion correction method according to any one of claims 1 to 6.

Images