Image distortion correction method, electronic equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an image distortion correction method, electronic equipment and a storage medium. The image distortion correction method comprises the following steps: acquiring a shot image obtained by shooting a virtual image formed on a windshield by a calibration image; based on the calibration image and the shot image, mapping the sampling calibration point in the shot image to a screen coordinate system of the head-up display to obtain a distortion calibration point corresponding to the reference calibration point; for each control point, determining N reference calibration points meeting a preset condition, and determining a distortion matrix of the control points based on the N reference calibration points and N distortion calibration points corresponding to the N reference calibration points; and performing coordinate transformation on the original coordinates of the control points based on the distortion matrix of the control points to obtain distortion coordinates of the control points so as to realize image distortion correction. The method provided by the invention not only is simple to operate, but also efficiently and accurately realizes distortion correction of the projection image of the head-up display.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to an image distortion correction method, an electronic device, and a storage medium. Background Art

[0002] A HUD (Head Up Display) is a display device that projects information onto a windshield or other transparent medium, and is widely used in fields such as automobiles and aviation. Due to the curved surface characteristics of the windshield, the projected image will be distorted on the windshield, affecting the user's viewing experience. Traditional distortion correction methods usually rely on complex mathematical models and a large amount of computing resources, and it is difficult to meet the requirements of real-time performance and accuracy. Therefore, there is an urgent need for an efficient and accurate image distortion correction method. Summary of the Invention

[0003] Embodiments of this application provide an image distortion correction method, an electronic device, and a storage medium. By establishing a mapping relationship between a calibration image and a captured image, and combining the distortion matrix of control points, precise correction of the projected image of the head-up display is achieved.

[0004] In a first aspect, embodiments of this application provide a head-up display distortion correction method, and the method includes:

[0005] Obtain a captured image obtained by photographing a virtual image formed by a calibration image on a windshield, where the calibration image includes a plurality of reference calibration points, and the captured image includes sampled calibration points corresponding to the reference calibration points;

[0006] Based on the calibration image and the captured image, map the sampled calibration points in the captured image to the head-up display screen coordinate system to obtain distortion calibration points corresponding to the reference calibration points;

[0007] For each control point in the head-up display screen coordinate system, determine N reference calibration points that meet a preset condition, and based on the N reference calibration points and the N distortion calibration points corresponding to the N reference calibration points, determine the distortion matrix of the control point, where N is an integer greater than or equal to 4;

[0008] Based on the distortion matrices of each control point, perform coordinate transformation on the original coordinates of each control point to obtain the distortion coordinates of each control point, so that the head-up display corrects the projected image according to the distortion coordinates of each control point.

[0009] Optionally, the mapping the sampled calibration points in the captured image to the head-up display screen coordinate system based on the calibration image and the captured image to obtain distortion calibration points corresponding to the reference calibration points includes:

[0010] Based on the calibration image and the captured image, determine a mapping relationship that maps the sampled calibration points in the captured image to the coordinate system of the head-up display screen, and the mapping relationship satisfies a preset constraint condition;

[0011] Based on the mapping relationship and the sampled calibration points in the captured image, obtain the distorted calibration points corresponding to the reference calibration points.

[0012] Optionally, the determining, based on the calibration image and the captured image, a mapping relationship that maps the sampled calibration points in the captured image to the coordinate system of the head-up display screen includes:

[0013] Preset a mapping function including unknown parameters;

[0014] Using the singular value decomposition method, based on the first coordinates of each reference calibration point in the calibration image, the second coordinates of each sampled calibration point in the captured image, and the constraint condition, determine the parameter values of the mapping function.

[0015] Optionally, the constraint condition is that the cumulative distance between the distorted calibration points and the corresponding reference calibration points is minimized.

[0016] Optionally, the mapping, based on the calibration image and the captured image, the sampled calibration points in the captured image to the coordinate system of the head-up display screen to obtain the distorted calibration points corresponding to the reference calibration points includes:

[0017] Based on the calibration image and the captured image, map the sampled calibration points in the captured image to the coordinate system of the head-up display screen through scaling and / or translation to obtain the distorted calibration points corresponding to the reference calibration points.

[0018] Optionally, the determining, based on the N reference calibration points and the N distorted calibration points corresponding to the N reference calibration points, the distortion matrix of the control points includes:

[0019] Based on the coordinates of the N reference calibration points and the coordinates of the N distorted calibration points, calculate the homography matrix between the N reference calibration points and the N distorted calibration points;

[0020] Use the homography matrix as the distortion matrix of the control points.

[0021] Optionally, the obtaining the captured image after photographing the virtual image formed by the calibration image on the windshield includes:

[0022] Obtain the captured image after photographing the virtual image formed by the calibration image on the windshield at a preset eye box position.

[0023] Optionally, the N reference calibration points that meet the preset conditions are the N reference calibration points closest to the control point.

[0024] In a second aspect, an embodiment of the present application provides a head-up display, including: at least one processor and a memory communicatively connected to the at least one processor, where the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any one of the above.

[0025] In a third aspect, an embodiment of the present application provides a computer storage medium, where the computer storage medium stores instructions or programs, and when the instructions or programs are executed by at least one processor, the at least one processor is enabled to execute the method described in any one of the above.

[0026] In an embodiment of the present application, a method for correcting distortion of a head-up display is provided. First, a captured image obtained by photographing a virtual image formed by a calibration image on a windshield is acquired; then, based on the calibration image and the captured image, the sampled calibration points in the captured image are mapped to the coordinate system of the head-up display screen to obtain distortion calibration points corresponding to the reference calibration points; then, for each control point in the coordinate system of the head-up display screen, N reference calibration points that meet the preset conditions are determined, and based on the N reference calibration points and the N distortion calibration points corresponding to the N reference calibration points, a distortion matrix of the control point is determined; finally, coordinate transformation is performed on the original coordinates of the control point based on the distortion matrix of the control point to obtain the distorted coordinates of the control point, so that the head-up display corrects the distortion of the projected image according to the distorted coordinates of each control point. The method of the present application not only has simple operations, but also efficiently and accurately realizes the distortion correction of the projected image of the head-up display. Description of the Drawings

[0027] Figure 1 Exemplarily shows a schematic diagram of HUD projection display;

[0028] Figure 2 Exemplarily shows a schematic flowchart of an image distortion correction method;

[0029] Figure 3 Exemplarily shows a schematic diagram of the relative positions of the Hud, windshield, and camera when shooting a virtual image;

[0030] Figure 4 Exemplarily shows a schematic diagram of a calibration image;

[0031] Figure 5 Exemplarily shows a schematic diagram of a captured image;

[0032] Figure 6An exemplary comparison diagram of a captured image mapped to the coordinate system of the head-up display screen and a calibrated image is shown;

[0033] Figure 7 An exemplary schematic diagram of the hardware structure of an electronic device is shown. Detailed implementation manners

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0035] Please refer to Figure 1 , Figure 1 which shows a schematic diagram of HUD projection display. As Figure 1 shown, the HUD device includes an optical engine 1, a first reflector 2, a second reflector 3, etc. When the HUD device is working, the optical engine 1 projects a projection image containing information such as vehicle speed, fuel consumption, and navigation onto the first reflector 2. The projection image is projected onto the windshield 4 after passing through the first reflector 2 and the second reflector 3, and a virtual image 5 is formed. The human eye 6 can see the virtual image 5 through the windshield 4. Due to the curved surface characteristics of the windshield, the virtual image 5 will be distorted relative to the original projection image, affecting the user's viewing experience. Therefore, the present application provides an efficient and accurate image distortion correction method to correct the distortion of the image displayed by the HUD device, so as to improve the display performance of the HUD and improve the display effect of the HUD screen.

[0036] The image distortion correction method provided by the present application is applied to an electronic device, which may be, for example, a vehicle-mounted processor, a cloud server, etc. Please refer to Figures 2 - 6 , an embodiment of the present application provides an image distortion correction method. As Figure 2 shown, the method includes:

[0037] Step S201, obtaining a captured image obtained by photographing the virtual image formed by the calibrated image on the windshield.

[0038] Generally, the calibrated image is composed of reference calibration points arranged neatly horizontally and vertically. Each reference calibration point may be a solid square with a side length of n pixels, or a solid rectangle with a length of l pixels and a width of w pixels, or a solid dot with a radius of r pixels, etc. As Figure 4 shown in the calibrated image, the red dots are reference calibration points, and the blue box represents the design area of the user interface.

[0039] AsFigure 3 As shown, the HUD device projects a calibrated image onto the windshield to form a corresponding virtual image. The virtual image formed on the windshield and the calibration points therein can be obtained by scanning at a preset position through a camera or other acquisition devices, resulting in a captured image. For the sake of distinction, the calibration points in the captured image are called sampled calibration points. The sampled calibration points are no longer arranged neatly horizontally and vertically like the reference calibration points, but have undergone a certain distortion. As Figure 5 shown, the sampled calibration points (blue) in the captured image are distorted as a whole relative to the Figure 4 reference calibration points (red) in

[0040] In one embodiment, the acquisition device is an industrial camera, which is connected to an electronic device that executes the image distortion correction method through a wired network or a wireless network, and sends the captured image to the electronic device.

[0041] In one embodiment, the preset position is a preset eyebox position. The eyebox position refers to a conical area between the optical system of the HUD device and the eyeball, and it is also the area where the display content meets the optimal optical design indicators. Within the eyebox range, the eyes can see the complete FOV (i.e., all images) projected by the HUD. When the human eye exceeds this range, problems such as severe image distortion, color rendering errors, or even no display content may occur.

[0042] It can be understood that both the virtual image and the calibrated image are located in the head-up display screen coordinate system and have the same size. The captured image is obtained by shooting through a camera device and is located in the camera pixel coordinate system. Its size depends on the resolution of the camera and may be different from the size of the calibrated image.

[0043] Step S202: Based on the calibrated image and the captured image, map the sampled calibration points in the captured image to the head-up display screen coordinate system to obtain distorted calibration points corresponding to the reference calibration points.

[0044] In one embodiment, when mapping the sampled calibration points in the captured image to the head-up display screen coordinate system based on the calibrated image and the captured image, the distorted calibration points obtained by mapping and the reference calibration points need to meet a preset constraint condition. This constraint condition makes the position deviation between the mapped distorted calibration points and the corresponding reference calibration points as small as possible. Preferably, this constraint condition is that the cumulative value of the distances between the distorted calibration points and the corresponding reference calibration points is the smallest.

[0045] Based on this, the specific implementation method of step S202 is as follows: First, based on the calibrated image and the captured image, determine the mapping relationship for mapping the sampled calibration points in the captured image to the head-up display screen coordinate system, and make this mapping relationship meet the preset constraint condition; then, based on the mapping relationship and the sampled calibration points in the captured image, obtain the distorted calibration points corresponding to the reference calibration points.

[0046] Specifically, a mapping function including unknown parameters can be preset. Preferably, the mapping function is a linear function, that is, the sampling calibration points in the captured image are mapped to the coordinate system of the head-up display screen only through scaling and / or translation. Based on this mapping function, the following mapping relationship can be constructed between the sampling calibration points and the corresponding distorted calibration points:

[0047]

[0048] where (x i , y i ) are the coordinates of the sampling calibration point in the captured image, and (X i , Y i ) are the coordinates of the distorted calibration point in the coordinate system of the head-up display screen.

[0049] The constraint condition satisfied by this mapping relationship is that the value of the following formula (2) is minimized.

[0050]

[0051] where (X si , Y si ) are the coordinates of the reference calibration point corresponding to the distorted calibration point (X i , Y i ) in the calibration image.

[0052] It can be seen from formula (1) and formula (2) that in the process of solving the mapping relationship, the corresponding relationship between the sampling calibration points and the reference calibration points needs to be determined. Specifically, first, the first coordinates (X si , Y si ) of each reference calibration point in the calibration image and the second coordinates (x i , y i ) of each sampling calibration point in the captured image are determined by methods such as clustering; then, according to the arrangement of each reference calibration point and the arrangement of each sampling calibration point, the corresponding relationship between the sampling calibration points and the reference calibration points is determined. After determining the corresponding relationship between the sampling calibration points and the reference calibration points, according to the corresponding relationship between the sampling calibration points and the reference calibration points, the first coordinates of each reference calibration point, the second coordinates of each sampling calibration point, formula (1) and formula (2), the parameter values of the mapping function can be solved by the singular value decomposition method, that is, the values of a, b, c, and d in formula (1).

[0053] After the parameters of the mapping function are determined, based on the second coordinates of each sampling calibration point, the coordinates of the distorted calibration points can be obtained. As Figure 6 shown, the mapped distorted calibration points (blue dots) correspond one-to-one with the reference calibration points (red dots), and some distorted calibration points coincide or intersect with the corresponding reference calibration points, while some distorted calibration points are separated from the corresponding reference calibration points.

[0054] Step S203: For each control point in the coordinate system of the head-up display screen, determine N reference calibration points that meet the preset conditions, and based on the N reference calibration points and the N distorted calibration points corresponding to the N reference calibration points, determine the distortion matrix of the control point, where N is an integer greater than or equal to 4.

[0055] In one embodiment, the control point can be a pixel point set in the image that can be used for image distortion correction, such as multiple randomly calibrated pixel points, or multiple preset pixel points. The control point can also be a non-pixel point set in the image that can be used for image distortion correction, such as multiple randomly calibrated non-pixel points, or multiple preset non-pixel points. The control point can also include pixel points and non-pixel points. The more the number of control points, the better the effect of image distortion correction.

[0056] The distortion chip corrects the image by moving the coordinates of the control points evenly distributed on the HUD screen. The number and distribution positions of the control points may not be consistent with the number and distribution positions of the reference calibration points in the calibrated image. Therefore, it is necessary to obtain the distortion data of the control points based on the distortion data of the reference calibration points. Specifically, for each control point, find the four (or more than four) reference calibration points closest to the control point. According to the coordinates of the four reference calibration points and the coordinates of the four distorted calibration points corresponding to the four reference calibration points, calculate the homography matrix of the four reference calibration points and the four distorted calibration points, and use the homography matrix as the distortion matrix of the control point.

[0057] Step S204: Based on the distortion matrices of the respective control points, perform coordinate transformation on the original coordinates of the respective control points to obtain the distorted coordinates of the respective control points, so that the head-up display corrects the projection image according to the distorted coordinates of the respective control points.

[0058] In one embodiment, input the distorted coordinates of the respective control points into the distortion chip located inside the head-up display. The distortion chip corrects the projection image according to the original coordinates and the distorted coordinates of the respective control points.

[0059] The image distortion correction method provided by the present application first obtains a captured image obtained by photographing a virtual image formed by a calibration image on a windshield; then, based on the calibration image and the captured image, maps the sampled calibration points in the captured image to the coordinate system of the head-up display screen to obtain distortion calibration points corresponding to the reference calibration points; then, for each control point in the coordinate system of the head-up display screen, determines N reference calibration points that meet the preset conditions, and based on the N reference calibration points and the N distortion calibration points corresponding to the N reference calibration points, determines the distortion matrix of the control point; finally, performs coordinate transformation on the original coordinates of the control point based on the distortion matrix of the control point to obtain the distorted coordinates of the control point, so that the head-up display corrects the distortion of the projected image according to the distorted coordinates of each control point. The method of the present application, by solving the mapping matrix and performing block processing on the distortion data, not only makes the distortion values of the control points obtained by the distortion chip closer to the actual distortion effect, but also makes the movement of the control points obtained by the distortion chip minimal, facilitating chip processing, and is a method with simple operation that can efficiently and accurately correct the distortion of the projected image of the head-up display.

[0060] According to an embodiment of the present application, an electronic device is provided, as Figure 7 shown, which is a schematic hardware structure diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 includes a processor 10, a memory 20, and a communication interface 30. The processor 10, the memory 20, and the communication interface 30 are connected by lines. In Figure 7 the embodiment shown, the processor 10, the memory 20, and the communication interface 30 are communicatively connected to each other through a bus.

[0061] The memory 20 is used to store software programs, computer-executable program instructions, etc. The memory 20 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created according to the use of the electronic device, etc.

[0062] The memory 20 may be a read-only memory (ROM), or may be other types of static storage devices that can store static information and instructions, may also be a random access memory (RAM), or may be other types of dynamic storage devices that can store information and instructions, and may also be an electrically erasable programmable read-only memory (EEPROM). The specific type is not limited here.

[0063] Exemplarily, the aforementioned memory 20 may be a double data rate synchronous dynamic random access memory (DDR SDRAM for short, DDR). The memory 20 may exist independently, but is connected to the processor 10. Optionally, the memory 20 may also be integrated with the processor 10, for example, integrated within one or more chips.

[0064] In some embodiments, the memory 20 may optionally include memories remotely disposed relative to the processor 10, and these remote memories may be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, intranet, local area network, mobile communication network, and combinations thereof.

[0065] The processor 10 connects various parts of the entire electronic device 100 using various interfaces and lines. By running or executing software programs stored in the memory 20 and invoking data stored in the memory 20, the processor 10 performs various functions of the electronic device and processes data, for example, implementing the method described in any embodiment of the present application.

[0066] The processor 10 may be a field programmable gate array (FPGA), a digital signal processor (DSP), a central processing unit (CPU), etc.

[0067] The processor 10 may be a single-core processor or a multi-core processor. For example, the processor 10 may be composed of multiple FPGAs or multiple DSPs. In addition, the processor 10 may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions). The processor 10 may be a separate semiconductor chip or may be integrated with other circuits into a semiconductor chip. For example, it may form a system on a chip (SoC) with other circuits (such as codec circuits, hardware acceleration circuits, or various bus and interface circuits), or may also be integrated as a built-in processor of an application specific integrated circuit (ASIC) in the ASIC. The ASIC integrated with the processor may be separately packaged or may also be packaged together with other circuits.

[0068] The communication interface 30 may use a transceiver device such as a transceiver to implement communication between the electronic device and other devices or communication networks.

[0069] The embodiments of the present application also provide a computer storage medium. The computer storage medium stores instructions or programs, and when the instructions or programs are executed by one or more processors, for example Figure 7 one of the processors 10 in

[0070] it enables the above one or more processors to execute the image distortion correction method in any of the above method embodiments.

[0071] As described above, these are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application. Therefore, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image distortion correction method, characterized in that, The method includes: Obtaining a captured image obtained by photographing a virtual image formed by a calibration image on a windshield, the calibration image including a plurality of reference calibration points, and the captured image including sampled calibration points corresponding to the reference calibration points; Based on the calibration image and the captured image, mapping the sampled calibration points in the captured image to a head-up display screen coordinate system to obtain distorted calibration points corresponding to the reference calibration points; For each control point in the head-up display screen coordinate system, determining N reference calibration points that meet a preset condition, and based on the N reference calibration points and N distorted calibration points corresponding to the N reference calibration points, determining a distortion matrix of the control point, where N is an integer greater than or equal to 4; Based on the distortion matrices of each control point, respectively performing coordinate transformation on the original coordinates of each control point to obtain distorted coordinates of each control point, so that the head-up display corrects the distortion of the projection image according to the distorted coordinates of each control point.

2. The method according to claim 1, wherein The step of mapping the sampled calibration points in the captured image to a head-up display screen coordinate system based on the calibration image and the captured image to obtain distorted calibration points corresponding to the reference calibration points includes: Based on the calibration image and the captured image, determining a mapping relationship for mapping the sampled calibration points in the captured image to the head-up display screen coordinate system, and the mapping relationship meets a preset constraint condition; Based on the mapping relationship and the sampled calibration points in the captured image, obtaining distorted calibration points corresponding to the reference calibration points.

3. The method according to claim 2, wherein The step of determining a mapping relationship for mapping the sampled calibration points in the captured image to a head-up display screen coordinate system based on the calibration image and the captured image includes: Presetting a mapping function including unknown parameters; Using the singular value decomposition method, based on the first coordinates of each reference calibration point in the calibration image, the second coordinates of each sampled calibration point in the captured image, and the constraint condition, determining the parameter values of the mapping function.

4. The method according to claim 2, wherein The constraint condition is that the cumulative value of the distances between the distorted calibration points and the corresponding reference calibration points is the smallest.

5. The method according to claim 1, wherein The step of mapping the sampled calibration points in the captured image to a head-up display screen coordinate system based on the calibration image and the captured image to obtain distorted calibration points corresponding to the reference calibration points includes: Based on the calibration image and the captured image, mapping the sampled calibration points in the captured image to the head-up display screen coordinate system by scaling and / or translation to obtain distorted calibration points corresponding to the reference calibration points.

6. The method according to any one of claims 1 to 5, characterized in that The step of determining the distortion matrix of the control point based on the N reference calibration points and the N distorted calibration points corresponding to the N reference calibration points includes: Based on the coordinates of the N reference calibration points and the coordinates of the N distorted calibration points, calculating a homography matrix between the N reference calibration points and the N distorted calibration points; Taking the homography matrix as the distortion matrix of the control point.

7. The method according to any one of claims 1 to 5, characterized in that, The step of obtaining a captured image obtained by photographing a virtual image formed by a calibration image on a windshield includes: Obtain a captured image obtained by photographing a virtual image formed on a windshield of a calibrated image at a preset eye box position.

8. The method according to any one of claims 1 to 5, characterized in that The N reference calibration points that meet the preset conditions are the N reference calibration points closest to the control point.

9. An electronic device, characterized in that, It includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor. When the instructions are executed by the at least one processor, the at least one processor is enabled to execute the image distortion correction method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions or programs. When the instructions or programs are executed by at least one processor, the at least one processor is caused to execute the image distortion correction method according to any one of claims 1 to 8.

Images