Image correction method, storage medium, correction system, head-up display device, and vehicle

Abstract

The application provides an image correction method, comprising: obtaining a pre-distortion image according to a test image to be projected and an actual image obtained by projection, the actual image being the test image after projection and distortion, and the pre-distortion image being used for compensating the distortion; obtaining a plurality of first inner corner point coordinates of a plurality of inner corner points corresponding to the actual image and a plurality of second inner corner point coordinates of the plurality of inner corner points corresponding to the pre-distortion image; performing circle fitting on the second inner corner point coordinates to obtain a center coordinate of the pre-distortion image; establishing a third coordinate system with the center coordinate as an origin of the coordinate system, and converting the coordinates of each second inner corner point into third inner corner point coordinates; solving a distortion coefficient according to the first inner corner point coordinates and the third inner corner point coordinates, and correcting an image output according to the distortion coefficient. The application also provides a storage medium, a correction system, a head-up display device and a vehicle.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to an image correction method, a storage medium, a correction system, a head-up display device, and a vehicle using the head-up display device. Background Technology

[0002] Head-up display (HUD) devices are categorized into combiner head-up displays (CHUD), windshield head-up displays (WHUD), and augmented reality head-up displays (AR-HUD). AR-HUD and WHUD require projecting an image onto a windshield. Since windshields are typically curved, the image projected onto the curved windshield is distorted when reflected to the human eye, resulting in a distorted image as perceived by the viewer.

[0003] Existing techniques use the Brown-Conrady model or a piecewise affine matrix to correct projected image distortion. However, the accuracy of correction using the Brown-Conrady model is low, while correction using a piecewise affine matrix is ​​slow. Summary of the Invention

[0004] The first aspect of this application provides an image correction method, comprising:

[0005] A pre-distorted image is obtained based on the test image to be projected and the actual image obtained by projection. The actual image is the test image that has been distorted after projection. The pre-distorted image is used to compensate for the distortion. The test image, the actual image, and the pre-distorted image all include multiple grids, and the multiple grids intersect to form multiple interior corner points.

[0006] Obtain the coordinates of multiple first interior corner points corresponding to the multiple interior corner points in the actual image and the coordinates of multiple second interior corner points corresponding to the multiple interior corner points in the pre-distorted image;

[0007] The coordinates of the second inner corner point are fitted with a circle to calculate the center coordinates of the pre-distorted image;

[0008] Establish a third coordinate system using the coordinates of the center of the circle as the origin of the coordinate system, and convert the coordinates of each of the second interior corner points into the coordinates of the third interior corner point;

[0009] The distortion coefficients are calculated based on the coordinates of the first and third interior corners, and the image is output after correction based on the distortion coefficients.

[0010] The image correction method provided in this application first obtains a pre-distorted image by combining a test image and an actual image, then establishes a coordinate system based on the center coordinates to obtain the coordinates of the third interior corner, and finally solves for the distortion coefficient based on the coordinates of the first and third interior corners. This method can effectively correct distorted images. When the image correction method is used to correct a device to be corrected, the device to be corrected can adjust the image output according to the distortion coefficient, which can effectively correct the distortion of the device to be corrected, improve the accuracy of distortion correction, and speed up the operation.

[0011] In one embodiment, the formula for calculating the center coordinates of the pre-distorted image by performing circle fitting on the coordinates of the second interior corner point is as follows:

[0012] (x-Cx) 2 +(y-Cy) 2 =R 2 #(1)

[0013] In the above formula (1), x is the abscissa of the second interior corner point, y is the ordinate of the second interior corner point, Cx is the abscissa of the center coordinate, Cy is the ordinate of the center coordinate, and R is the radius of the center of the fitted circle. The center coordinates (Cx, Cy) can be obtained through the fitted circle.

[0014] In one embodiment, the step of solving the distortion coefficients based on the coordinates of the first interior corner and the coordinates of the third interior corner includes:

[0015] Establish a mapping relationship between the coordinates of each first interior corner point and the coordinates of each third interior corner point;

[0016] The distortion coefficients are determined based on the mapping relationship.

[0017] In one embodiment, the formula for establishing the mapping relationship between the coordinates of each first interior corner point and the coordinates of each third interior corner point is as follows:

[0018] u d =k1ur 2 +k2ur 4 +2p1uv+p2(r 2 +2u 2 )+a1u+a2u 3 #(2)

[0019] v d =k1vr 2 +k2vr 4+p1(r 2 +2v 2 )+2p2uv+3a1v+a3v 3 #(3)

[0020] In equations (2) and (3) above, u is the abscissa of each of the first interior corner points, v is the ordinate of each of the first interior corner points, and u d v is the x-coordinate of each of the aforementioned third interior corner points. d Let be the ordinate of each of the third interior corner points, k1k2 be the radial distortion coefficient, p1p2 be the tangential distortion coefficient, a1a2a3 be the distortion influence coefficient, and r be the distance from the origin of each of the third interior corner points.

[0021] In one embodiment, the step of obtaining a pre-distorted image based on the test image to be projected and the actual image obtained by projection includes:

[0022] The pre-distorted image is obtained by combining the test image and the actual image using the piecewise affine matrix method.

[0023] A second aspect of this application provides a storage medium storing a program that can be executed by a processor using the image correction method described in any of the above embodiments. The storage medium provided in this application can be used to execute the image correction method described in any of the above embodiments, and therefore also possesses the same beneficial effects, which will not be elaborated further here.

[0024] A third aspect of this application provides a calibration system, comprising:

[0025] Receiver, the receiver being used to receive the actual image output from the device to be calibrated;

[0026] A controller is configured to control the device to be corrected to output a test image and receive the actual image output from the receiver. The controller performs the image correction method as described in any of the above embodiments to obtain the distortion coefficient based on the actual image and the test image.

[0027] The correction system provided in this application obtains the distortion coefficients by executing the image correction method described in any of the above embodiments. The distortion coefficients can be used to correct the distortion of the device to be corrected. The image output can be adjusted more conveniently according to the distortion coefficients. The distortion correction of the device to be corrected can be performed more effectively, which is conducive to improving the accuracy of distortion correction, speeding up the operation, and thus improving the performance of the device to be corrected.

[0028] In one embodiment, the controller transmits the distortion coefficients to the device to be corrected, and the device to be corrected outputs the corrected image based on the distortion coefficients.

[0029] A fourth aspect of this application provides a head-up display device, comprising:

[0030] The image generation unit is used to emit image light;

[0031] An optical display system for receiving the image light and projecting the image light into a human eye;

[0032] The processing module obtains the distortion coefficients by executing the image correction method as described in any of the above embodiments, and adjusts the image light emitted from the image generation unit according to the distortion coefficients.

[0033] The head-up display device provided in this application embodiment can easily adjust the image output according to the distortion coefficient by correcting distortion based on the distortion coefficient, which is beneficial to speeding up the operation, improving the performance of the head-up display device, and extending the life of the head-up display device.

[0034] The fifth aspect of this application provides a vehicle, comprising:

[0035] The subject; and

[0036] Head-up display device as described in any of the above embodiments.

[0037] The vehicle provided in this application includes the head-up display device as described in any of the above embodiments, and therefore has the same beneficial effects, which will not be elaborated here. Attached Figure Description

[0038] Figure 1 This is a schematic flowchart of an image correction method according to an embodiment of this application.

[0039] Figure 2 This is a standard chessboard diagram and a schematic diagram of the interior corner points according to an embodiment of this application.

[0040] Figure 3 This is a schematic diagram showing the coordinates of the second interior corner point according to an embodiment of this application.

[0041] Figure 4 This is a schematic diagram showing the coordinates of the third interior corner point according to an embodiment of this application.

[0042] Figure 5 This is a schematic diagram of a correction system according to an embodiment of this application.

[0043] Figure 6 This is a schematic diagram of a head-up display device according to an embodiment of this application.

[0044] Figure 7 This is a schematic diagram of a vehicle module according to an embodiment of this application.

[0045] Explanation of key component symbols:

[0046] Steps S100, S200, S300, S400

[0047] S500, S501, S502,

[0048] Test image 200

[0049] Grid W

[0050] Interior corner point D

[0051] Calibration System 300

[0052] Receiver 301

[0053] Controller 303

[0054] Device to be calibrated 306

[0055] Human eye E

[0056] Actual image T

[0057] Head-up display device 500

[0058] Image generation unit 501

[0059] Optical Display System 502

[0060] Processing module 503

[0061] Vehicle 700

[0062] Ontology 701

[0063] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0064] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0065] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0066] To further illustrate the technical means and effects adopted by this application in achieving its intended purpose, the following detailed description of this application is provided in conjunction with the accompanying drawings and preferred embodiments.

[0067] Please see Figure 1 The image correction method provided in this application includes the following steps S100 to S500:

[0068] Step S100: Obtain a pre-distorted image based on the test image to be projected and the actual image obtained by projection.

[0069] Step S200: Obtain the coordinates of multiple first interior corner points of the actual image and the coordinates of multiple second interior corner points of the pre-distorted image;

[0070] Step S300: Perform circle fitting on the coordinates of the second inner corner point to calculate the center coordinates of the pre-distorted image;

[0071] Step S400: Establish a third coordinate system with the center coordinates as the origin, and convert the coordinates of each second interior corner point to the coordinates of the third interior corner point;

[0072] Step S500: Solve for the distortion coefficients based on the coordinates of the first and third interior corners, and output the corrected image based on the distortion coefficients.

[0073] The image correction method provided in this application first obtains a pre-distorted image by combining a test image and an actual image. Then, it establishes a coordinate system based on the center coordinates to obtain the coordinates of the third interior corner. Finally, it calculates the distortion coefficients based on the coordinates of the first and third interior corners, thus effectively correcting distorted images. When this image correction method is used to correct a device, the device can adjust its image output according to the distortion coefficients, effectively correcting distortion in the device, improving the accuracy of distortion correction, and accelerating the operating speed.

[0074] Please see Figure 2 In this embodiment, the test image 200 in step S100 is a standard chessboard pattern. The test image 200 includes multiple grids W, and the multiple grids W intersect to form multiple interior corner points D. Therefore, since the actual image is the test image 200 that has been distorted after projection, both the actual image and the pre-distorted image include multiple grids, and the multiple grids intersect to form multiple interior corner points. In other embodiments, the test image may also be an image that includes information about other reference objects, and this application does not impose any restrictions.

[0075] Step S100, which involves obtaining a pre-distorted image based on the test image to be projected and the actual image obtained through projection, is as follows: The pre-distorted image is obtained by combining the actual image and the test image using a piecewise affine matrix method. Specifically, all image data of the test image is first denoted as matrix A, and all image data of the actual image is denoted as matrix B. The matrix C represented by all image data of the pre-distorted image is calculated according to the following formula (1):

[0076] A=B*k+b#(1)

[0077] In equation (1), k and b represent the coefficient matrices of the transformation relationship from matrix A to matrix B, reflecting the mapping relationship between the test image and the actual image. By obtaining the coefficient matrices k and b, and then obtaining the inverse transformation coefficient matrix, matrix C is derived. The transformation relationship between matrix C and matrix A is the opposite of that between matrix B and matrix A. All the data in matrix C represents the pre-distorted image data. The pre-distorted image has the opposite deformation direction to the actual image and is used for preliminary distortion compensation.

[0078] In step S200, the coordinates of multiple second interior corner points and multiple first interior corner points correspond to multiple interior corner points formed by the intersection of multiple grids in the pre-distorted image and the test image, respectively. Since the distortion direction of the pre-distorted image is opposite to that of the actual image, the pre-distorted image is used to compensate for distortion. The pre-distorted image needs to be rotated to align its deformation direction with that of the actual image, thereby facilitating subsequent calculations.

[0079] Specifically, since the actual image is distorted, the shape of the pre-distorted image can be approximated as a circle, so circle fitting can be performed. The formula for calculating the center coordinates of the pre-distorted image by circle fitting the coordinates of the second inner corner point in step S300 is as follows:

[0080] (x-Cx) 2 +(y-Cy) 2 =R 2 #(2)

[0081] In equation (2) above, x is the abscissa of the second interior corner point, y is the ordinate of the second interior corner point, Cx is the abscissa of the center of the circle, Cy is the ordinate of the center of the circle, and R is the radius of the center of the circle after fitting. The center coordinates (Cx, Cy) can be obtained by fitting the circle.

[0082] Please refer to the following: Figure 3 and Figure 4Specifically, in step S400, the coordinates of the circle center after fitting are (-131, -7171). Other values ​​may also be used in other embodiments, and this application does not impose any restrictions. A third coordinate system is established using the circle center coordinates as the origin of the coordinate system for the second interior corner points. Before the coordinate system transformation, the x-coordinate is X and the y-coordinate is Y. The third coordinate system is then established with the circle center coordinates as the origin of the coordinate system for the second interior corner points, and the x-coordinate of the third coordinate system is X' and the y-coordinate is Y'. Specifically, after using the circle center coordinates as the origin of the third coordinate system, the distance between each third interior corner point and the origin is greater than the distance between each second interior corner point and the origin.

[0083] Specifically, step S500, which involves calculating the distortion coefficients based on the coordinates of the first and third interior corners, includes:

[0084] S501: Establish the mapping relationship between the coordinates of each first interior corner point and the coordinates of each third interior corner point;

[0085] S502: Calculate the distortion coefficients based on the mapping relationship.

[0086] Specifically, in step S501, the Brown-Conrady distortion model is used to establish the mapping relationship between the coordinates of each first interior corner point and the coordinates of each third interior corner point. First, the coordinates of each first interior corner point are labeled (u, v), and the coordinates of each third interior corner point are labeled (u, v). d ,v d The mapping relationship between the coordinates of each first interior corner point and the coordinates of each third interior corner point is established by the following formula (3):

[0087] u d =k1ur 2 +k2ur 4 +2p1uv+p2(r 2 +2u 2 )+a1u+a2u 3 #(3)

[0088] v d =k1vr 2 +k2vr 4 +p1(r 2 +2v 2 )+2p2uv+3a1v+a3v 3 #(4)

[0089] In equations (3) and (4) above, u is the abscissa of each first interior corner point, v is the ordinate of each first interior corner point, and u d v is the x-coordinate of each third interior corner point. dLet be the ordinate of each third interior corner point, and let be the distortion coefficients, including radial distortion coefficients k1k2, tangential distortion coefficients p1p2, and distortion influence coefficients a1a2a3. Let r be the distance from the origin of each third interior corner point.

[0090] r is obtained by equation (5):

[0091] r 2 =u d 2 +v d 2 #(5)

[0092] The head-up display (HUD) outputs corrected image data based on the radial and tangential distortion coefficients. The image correction method operates at a speed ranging from 5 to 50 ms. The corrected inner corner coordinates and the pre-distorted image are evaluated. The root mean square error is calculated by substituting the inner corner coordinates of the corrected image and the third inner corner coordinates of the pre-distorted image into equations (6) and (7). Please refer to equations (6) and (7).

[0093]

[0094]

[0095] In the formula, (x, y) represents the coordinates of the interior corners of the corrected image generated by the image correction method, and (x', y') represents the coordinates of the third interior corner of the pre-distorted image. In this embodiment, the root mean square error (RMSEx) between the corrected image and the pre-distorted image is calculated to be 5 pixels; RMSEy = 3 pixels. In other embodiments, the RMSEx can also take other values, which are not limited in this application. Based on the RMSEx values, the image correction method provided in this application can achieve a relatively excellent correction effect. By first combining the test image and the actual image to obtain the pre-distorted image, establishing a coordinate system based on the center coordinates to obtain the coordinates of the third interior corner, and then solving for the distortion coefficients based on the coordinates of the first and third interior corners, the distorted image can be corrected relatively effectively. When the image correction method is used to correct the device to be corrected, the device to be corrected can adjust the image output according to the distortion coefficients, which can effectively correct the distortion of the device to be corrected, improve the accuracy of distortion correction, and speed up the operation.

[0096] The storage medium provided in this application embodiment stores a program that can be executed by a processor using the image correction method described in any of the above embodiments. The types of storage media include, but are not limited to, phase-change memory, static random access memory, dynamic random access memory, other types of random access memory, read-only memory, electrically erasable programmable read-only memory, or other optical and magnetic storage. Any storage medium storing a training method that can be executed by a processor using any of the above embodiments is within the scope of protection of this application.

[0097] The storage medium provided in this application embodiment can be used to execute the image correction method as described in any of the above embodiments. When the correction method is used on a head-up display device, the head-up display device can adjust the output according to the distortion coefficient when generating an image, which can more conveniently correct the distortion of the head-up display device, improve the accuracy of distortion correction, and speed up the operation.

[0098] Please see Figure 5 The correction system 300 provided in this application embodiment includes a receiver 301 and a controller 303. The receiver 301 is electrically connected to a processor. The receiver 301 is used to receive the actual image T output from the device to be corrected 306. In this embodiment, the receiver 301 is a camera. The receiver 301 is disposed on the receiving side of the human eye E. In other embodiments, the receiver 301 can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The controller 303 is used to receive the actual image T output from the receiver 301. The receiver 301 converts the optical signal into an electrical signal, transmits the electrical signal to the processor, the processor converts the electrical signal into a digital image signal for further processing, and performs the image correction method as described in any of the above embodiments to obtain the distortion coefficients based on the actual image T. The controller 303 is electrically connected to the device to be corrected 306. The controller 303 transmits the distortion coefficients to the device to be corrected 306, and the device to be corrected 306 outputs the corrected image based on the distortion coefficients. In this embodiment, the device to be calibrated 306 is a head-up display device. In other embodiments, the device to be calibrated 306 may also be a near-eye display device, a regular liquid crystal display, or a projector. This application does not impose any restrictions.

[0099] The correction system 300 provided in this application embodiment obtains distortion coefficients by executing the image correction method as described in any of the above embodiments. The distortion coefficients can be used to correct the distortion of the device 306 to be corrected. The image output can be adjusted more conveniently according to the distortion coefficients. The distortion correction of the device 306 to be corrected can be performed more effectively, which is conducive to improving the distortion correction accuracy and speeding up the operation, thereby improving the performance of the device 306 to be corrected.

[0100] Please see Figure 6 The head-up display device 500 provided in this application includes an image generation unit 501, an optical display system 502, and a processing module 503. The image generation unit 501 emits image light. The optical display system 502 receives the image light and projects it into the human eye E. The processing module 503 obtains the distortion coefficient by executing the image correction method as described in any of the above embodiments, and adjusts the image light emitted from the image generation unit 501 according to the distortion coefficient. In this embodiment, the head-up display device 500 is an augmented reality head-up display device. In other embodiments, the head-up display device 500 may also be a windshield-type head-up display device or a combined head-up display device; this application does not impose any limitations. The head-up display device 500 provided in this application, by correcting distortion according to the distortion coefficient, can conveniently adjust the image output according to the distortion coefficient, which is beneficial for accelerating operation, improving the usability of the head-up display device 500, and extending the lifespan of the head-up display device 500.

[0101] Please see Figure 7 The vehicle 700 provided in this application embodiment includes a body 701 and a head-up display device 500 as described in any of the above embodiments. The vehicle provided in this embodiment can be any one of an electric vehicle, a gasoline vehicle, a diesel vehicle, and a hybrid vehicle, and this application is not limited thereto. The body 701 may also include a positioning system (not shown) and a power system (not shown). The power system and the positioning system are electrically connected to the head-up display device 500. The power system provides the head-up display device 500 with vehicle 700 speed information, and the positioning system provides the head-up display device 500 with vehicle 700 position information. The head-up display device 500 projects the speed information and position information provided by the positioning system onto the human eye E.

[0102] The vehicle 700 provided in this application embodiment, by adopting the head-up display device 500 of any of the above embodiments, can adjust the output of the head-up display device 500 according to the distortion coefficient, and can more conveniently perform distortion correction on the head-up display device 500, which is beneficial to improving driving safety and increasing the life of the vehicle 700.

[0103] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. An image correction method, characterized in that, include: Based on the test image to be projected and the actual image obtained by projection, a pre-distorted image is obtained by combining the test image and the actual image using a piecewise affine matrix method. The actual image is the test image that has been distorted after projection, and the pre-distorted image is used to compensate for the distortion. The test image, the actual image, and the pre-distorted image all include multiple grids, and the multiple grids intersect to form multiple interior corner points. Obtain the coordinates of multiple first interior corner points corresponding to the multiple interior corner points in the actual image and the coordinates of multiple second interior corner points corresponding to the multiple interior corner points in the pre-distorted image; The coordinates of the second inner corner point are fitted with a circle to calculate the center coordinates of the pre-distorted image; Establish a third coordinate system using the coordinates of the center of the circle as the origin of the coordinate system, and convert the coordinates of each of the second interior corner points into the coordinates of the third interior corner point; Based on the coordinates of the first interior corner and the third interior corner, a mapping relationship is established between the coordinates of each first interior corner and the coordinates of each third interior corner; the formula for the mapping relationship is: In equations (1) and (2) above, u is the abscissa of each of the first interior corner points, and v is the ordinate of each of the first interior corner points. Let x be the x-coordinate of each of the aforementioned third interior corner points. Let y = the coordinates of each of the third interior corner points. , The tangential distortion coefficient is... The distortion influence coefficient is r, where r is the distance from the origin of each of the aforementioned third interior corner points. The distortion coefficients are solved based on the mapping relationship, and the image output is corrected based on the distortion coefficients.

2. The image correction method as described in claim 1, characterized in that, The formula for calculating the center coordinates of the pre-distorted image by performing circle fitting on the coordinates of the second inner corner point is as follows: In equation (3) above, x is the abscissa of the second interior angle point, and y is the ordinate of the second interior angle point. The x-coordinate of the center of the circle is given. Let R be the ordinate of the circle's center coordinates, and R be the radius of the fitted circle's center. The circle's center coordinates can be obtained through the circle fitting process. ).

3. A storage medium, characterized in that, The storage medium stores a program that can be executed by a processor to implement the image correction method as described in any one of claims 1-2.

4. A calibration system, characterized in that, include: Receiver, the receiver being used to receive the actual image output from the device to be calibrated; A controller is configured to control the device to be corrected to output a test image and receive the actual image output from the receiver. The controller performs the image correction method as described in any one of claims 1-2 to obtain the distortion coefficient based on the actual image and the test image.

5. The calibration system as described in claim 4, characterized in that, The controller transmits the distortion coefficients to the device to be corrected, and the device to be corrected outputs the corrected image based on the distortion coefficients.

6. A head-up display device, characterized in that, include: The image generation unit is used to emit image light; An optical display system for receiving the image light and projecting the image light into a human eye; The processing module obtains the distortion coefficient by executing the image correction method as described in any one of claims 1-2, and adjusts the image light emitted from the image generation unit according to the distortion coefficient.

7. A vehicle, characterized in that, include: ontology; as well as The head-up display device as described in claim 6.

Images