Method, apparatus, electronic device and product for image correction

Abstract

The present disclosure relates to a method, apparatus, electronic device and product for image correction, applied to a vehicle-mounted head-up display system. The method comprises obtaining a reference image, wherein the reference image is used to determine a correction parameter for the vehicle-mounted head-up display system. The method further comprises expanding an initial edge of the reference image. The method further comprises determining, by the vehicle-mounted head-up display system, an initial virtual image based on the expanded reference image. In addition, the method further comprises determining the correction parameter based on the initial edge of the reference image and the initial virtual image, to correct a target image displayed by the vehicle-mounted head-up display system. In this way, the position of the image edge is effectively corrected when the image is corrected, and the user's viewing experience is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and more particularly, to a method, an apparatus, an electronic device and a product for image correction. BACKGROUND

[0002] With the development of computer technology and transportation technology, more and more in-cabin applications or devices are proposed to enrich the experience of drivers or passengers. For example, a vehicle head-up display system can project display navigation signals, vehicle speed, power, and the like on the front windshield of the vehicle, so that the driver can view the relevant information without shifting his gaze when focusing on driving, ensuring driving safety and improving driving experience. SUMMARY

[0003] In a first aspect of embodiments of the present disclosure, a method for image correction is provided, applied to a vehicle head-up display system. The method includes obtaining a reference image, wherein the reference image is used to determine a correction parameter for the vehicle head-up display system. The method further includes expanding an initial edge of the reference image. The method further includes determining an initial virtual image by the vehicle head-up display system based on the expanded reference image. In addition, the method further includes determining the correction parameter based on the initial virtual image within the initial edge range, to correct a target image displayed by the vehicle head-up display system.

[0004] In a second aspect of embodiments of the present disclosure, an apparatus for image correction is provided, applied to a vehicle head-up display system. The apparatus includes an image acquisition module configured to obtain a reference image, wherein the reference image is used to determine a correction parameter for the vehicle head-up display system. The apparatus further includes an edge expansion module configured to expand an initial edge of the reference image. The apparatus further includes a virtual image determination module configured to determine an initial virtual image by the vehicle head-up display system based on the expanded reference image. In addition, the apparatus further includes a parameter determination module configured to determine the correction parameter based on the initial virtual image within the initial edge range, to correct a target image displayed by the vehicle head-up display system.

[0005] In a third aspect of embodiments of the present disclosure, an electronic device is provided. The electronic device includes one or more processors; and a storage device storing one or more programs, when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the method provided by the first aspect of the present disclosure.

[0006] In a fourth aspect of embodiments of the present disclosure, a computer program product is provided. The computer program product is tangibly stored on a non-transitory computer-readable medium and includes machine executable instructions that, when executed, cause a machine to implement the method provided by the first aspect of the present disclosure.

[0007] It is to be understood that the description of the summary section is not intended to identify key or essential features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be apparent from review of the disclosure, which is described in its entirety herein. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, aspects and advantages of various embodiments of the present disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, in which like reference numbers represent like elements throughout. In the drawings:

[0009] Figure 1 A schematic diagram showing an example environment in which various embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic diagram showing a method for image correction according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing a flow of determining correction parameters and displaying a corrected image according to some embodiments of the present disclosure;

[0012] Figure 4A A schematic diagram showing an example of edge extension of a reference image according to some embodiments of the present disclosure;

[0013] Figure 4B A schematic diagram showing an example of determining a mapping relationship according to some embodiments of the present disclosure;

[0014] Figure 4C A schematic diagram showing an example of storing a correction matrix for a reference image according to some embodiments of the present disclosure;

[0015] Figure 5 A block diagram showing an apparatus for image correction according to some embodiments of the present disclosure;

[0016] Figure 6 A block diagram showing an apparatus capable of implementing various embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of embodiments of the disclosure, the term "includes" and its conjugates are to be interpreted as open-ended terms that mean "includes, but is not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The term "one embodiment" or "the embodiment" are to be interpreted as "at least one embodiment." The terms "first," "second," etc. can refer to different or the same objects. Other explicit or implicit definitions can also be included below.

[0019] As described above, a head-up display system (which can be referred to simply as "head-up display system") is installed in the cockpit of some vehicles, such as cars. The head-up display system generally includes an image generation unit and an optical assembly. The image displayed by the image generation unit is reflected by a series of flat or curved mirrors in the optical assembly and projected onto the front windshield of the vehicle. When the driver's head is within the eyebox range at the driving position, the driver can see the virtual image of the projected image on the front windshield.

[0020] Since the front windshield is generally inclined and has a certain curvature, the shape, curvature and inclination angle of the front windshield of different brands or models are different, and there is also a certain tolerance in the front windshield of each vehicle of the same brand or model. Therefore, if the image displayed by the head-up display system is not corrected, the image projected on the front windshield may be distorted or shifted, resulting in display errors and affecting the viewing experience.

[0021] To solve the above problems, some related technologies calibrate the mapping between the screen of the image generation unit and the front windshield, fit a polynomial equation that satisfies the positional relationship between the two, and then refresh the image coordinates in the image generation unit according to the polynomial equation for pre-distortion projection. However, this method does not take into account the influence of the free-form mirrors in the optical assembly of the head-up display system on the image of the image generation unit in the horizontal and vertical directions. Due to the differences between the free-form mirrors, the shift and distortion of the image of the image generation unit in the horizontal and vertical directions caused by each set of optical assembly are different. If only the positional relationship between the screen of the image generation unit and the front windshield is considered, the imaging distortion caused by the reflection of the image light by the free-form mirrors cannot be solved, especially the position of the image edge points cannot be effectively corrected.

[0022] Therefore, the present disclosure proposes a method for image correction applied to a vehicle-mounted head-up display system. In the method, a processor obtains a reference image, wherein the reference image is used to determine a correction parameter for the vehicle-mounted head-up display system. The initial edge of the reference image is expanded, and based on the expanded reference image, an initial virtual image is determined by the vehicle-mounted head-up display system. Based on the initial edge of the reference image and the initial virtual image, the correction parameter is determined to correct a target image displayed by the vehicle-mounted head-up display system.

[0023] In this way, the processor expands the initial edge of the acquired reference image, and then determines an initial virtual image through the vehicle-mounted head-up display system. After determining the correction parameter according to the initial edge of the reference image and the initial virtual image, the target image displayed through the vehicle-mounted head-up display system can be corrected by using the correction parameter, and the imaging distortion caused by the reflection of image light by the vehicle-mounted head-up display system can be solved, and the viewing experience of the user can be improved.

[0024] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented is shown. As shown, the environment 100 includes a vehicle 112 and a processor 102 integrated on the vehicle 112. According to embodiments of the present disclosure, the vehicle 112 refers to any type of motorized or non-motorized carrier capable of carrying people and / or objects and being movable. As shown in Figure 1 , the vehicle 112 is illustrated as a sedan. It should be understood that although the vehicle 112 is illustrated as a sedan in Figure 1 , this is only exemplary and far from limiting, and examples thereof can also include a passenger car, a truck, a motorcycle, an electric vehicle, and the like. In some embodiments of the present disclosure, the processor 102 can be an embedded intelligent platform integrated in the vehicle 112, which can realize functions such as vehicle state monitoring, vehicle audio output, human-computer interaction, and Internet of Vehicles services through the cooperation of hardware (such as a display screen, a sensor) and software (an operating system, an application program). It can be understood that in other examples, the vehicle in the environment 100 can also be other vehicles, for example, an aircraft, an air vehicle, and the like. Figure 1

[0025] In the example environment 100, various types of sensors can be deployed inside and outside the vehicle 112 for sensing the internal environment of the vehicle, the surrounding environment, and the vehicle speed, and the like. Among them, the sensors can include but are not limited to visual sensors (such as cameras, video cameras, and the like), millimeter wave radars, laser radars, infrared sensors, global positioning systems, ultrasonic sensors, collision sensors, inertial measurement units, and other sensors.

[0026] As shown in Figure 1 ​As shown, in the environment 100, the vehicle 112 is also equipped with a head-up display system 106, which can project images onto the front windshield 110 of the vehicle 112. The head-up display system 106 includes an image production unit for displaying images. The image production unit includes a liquid crystal display screen composed of a plurality of thin film transistors, and the head-up display system 106 can receive images sent by the processor 102 and display them through the image production unit. The head-up display system 106 also includes an optical assembly for projecting images. The optical assembly can include at least one optical element such as a freeform mirror, a plane mirror, a concave mirror, etc., and the images displayed by the image production unit are imaged through the optical assembly and projected onto the front windshield 110. In the example environment 100, an image receiving unit 104 is also included, which can receive a virtual image 108 of the images projected onto the front windshield 110 within the eyebox range of the head-up display system 106. It can be understood that in the environment 100, the image receiving unit 104 is shown as the eyes and head of a user or driver, and in other examples, the image receiving unit 104 can also be a camera, an image collector, or an image sensor, etc.

[0027] In some examples, as in Figure 1In the illustrated environment 100, the processor 102 can send an image (e.g., a picture, text, video, etc.) to be projected to the head-up display system 106, which projects the received image onto the front windshield 110. In the environment 100, the image receiving unit 104 can be a camera, and the calibration parameters of the head-up display system 106 for correcting a target image to be projected on the front windshield 110 by the head-up display system 106 on the vehicle 112 can be determined before the vehicle is shipped. In determining the calibration parameters, the camera can capture a virtual image 108 of a projection of the head-up display system 106 on the front windshield 110 (hereinafter referred to as a "reference image") and send the captured virtual image 108 to the processor 102. After receiving the virtual image 108 sent by the image receiving unit 104, the processor 102 can adjust or correct the image and send the adjusted or corrected image to the head-up display system 106, which projects the image again. When the projected image again meets the relevant requirements, it means that the correction is completed, and the calibration parameters can be determined. In correcting the image, to ensure that the edge position of the image is effectively corrected, the initial edge of the reference image is expanded before the processor 102 sends the reference image to the head-up display system 106. At the same time, the processor 102 simulates the reflection imaging of the reference image after the edge expansion to obtain a reference virtual image. After the image receiving unit 104 captures the virtual image 108 of the expanded reference image (denoted as "initial virtual image") and sends it to the processor 102, the processor 102 determines the calibration parameters based on the initial edge of the reference image, the initial virtual image, and the reference virtual image, and corrects the subsequent target image to be projected using the calibration parameters and sends the corrected image to the head-up display system to project onto the front windshield 110.

[0028] In some examples, for example in the environment 100, an industrial control computer can also be included, and the industrial control computer can acquire the image (denoted as "reference image") sent by the processor 102 to the head-up display system 106. The industrial control computer can also process the image, for example, edge extend the image, and send the edge-extended image to the processor 102. In addition, the industrial control computer can also receive the virtual image 108 collected by the image receiving unit 104. Furthermore, the industrial control computer can also simulate imaging of the original image, for example, simulate reflection imaging, to obtain a reference virtual image. The industrial control computer can correct the original image based on the edge-extended image, the reference virtual image, and the virtual image 108, obtain initial correction parameters, and send the initial correction parameters to the processor 102. The processor 102 corrects the original image based on the initial correction parameters, and sends the corrected original image to the head-up display system 106 for display and projection. It can be understood that the industrial control computer can correct the original image multiple times, that is, the industrial control computer repeatedly receives the image, edge-extends the image, receives the virtual image, simulates planar reflection imaging of the extended image, and determines the initial correction parameters, intermediate correction parameters, etc. based on the received image, the virtual image, and the virtual image obtained by simulating imaging, to correct the image multiple times until a satisfactory or certain standard-compliant projection is obtained, and the final correction parameters are determined. It can also be understood that after the processor 102 receives and stores the final correction parameters, when receiving other images that need to be projected by the head-up display system 106, the received images can be corrected based on the stored correction parameters and then sent to the head-up display system 106. Therefore, in the environment 100, for the same vehicle 112, after the image correction is completed and the correction parameters are determined, the subsequent images can be directly projected by the head-up display system 106 after being corrected by the processor 102.

[0029] In this way, the processor 102 extends the initial edge of the acquired reference image, and then determines the initial virtual image through the vehicle-mounted head-up display system. After the correction parameters are determined based on the initial edge of the reference image and the initial virtual image, the target image displayed by the vehicle-mounted head-up display system can be corrected using the correction parameters, which can solve the imaging distortion problem caused by the reflection of image light by the vehicle-mounted head-up display system and improve the user's viewing experience.

[0030] Figure 2 A schematic diagram of a method 200 for image correction according to some embodiments of the present disclosure is shown. The method 200 can be performed by a processor, for example, by the processor 102 in the Figure 1 environment 100. As shown in Figure 2 At block 202, the processor acquires a reference image, where the reference image is used to determine correction parameters for a vehicle-mounted head-up display system. For example, in the environment 100, the processor 102 can acquire the reference image from the image receiving unit 104 or the industrial control computer. Figure 1In the illustrated environment 100, the processor 102 can acquire a reference image, where the reference image can be a dot matrix image used in a line calibration process, for example, can be an image including a dot matrix distribution of multiple rows and multiple columns of dots.

[0031] At block 204, the processor extends the initial edge of the reference image. For example, in the environment 100, the processor 102 extends the initial edge of the acquired reference image, so that the image edge of the reference image is extended, and accordingly, when the reference image is a dot matrix image, the number of dots in the dot matrix is correspondingly increased after the reference image is extended.

[0032] At block 206, based on the extended reference image, an initial virtual image is determined by the vehicle-mounted head-up display system. For example, in the environment 100, the image receiving unit 104 can capture a virtual image (hereinafter referred to as "initial virtual image") 108 of the extended reference image projected on the front windshield 110 by the head-up display system 106 through optical imaging, and the image receiving unit 104 can send the captured initial virtual image 108 to the processor 102.

[0033] At block 208, the processor determines a correction parameter based on the initial edge of the reference image and the initial virtual image to correct a target image displayed by the vehicle-mounted head-up display system. For example, in the environment 100, the processor 102 determines a correction parameter based on the received initial virtual image 108 and the initial edge of the reference image, so that the correction parameter is used to correct a target image that needs to be projected on the vehicle 112 by the head-up display system 106 subsequently.

[0034] In this way, the processor extends the initial edge of the acquired reference image, and then determines the initial virtual image by the vehicle-mounted head-up display system. After determining the correction parameter based on the initial edge of the reference image and the initial virtual image, the target image displayed by the vehicle-mounted head-up display system can be corrected by the correction parameter, which can solve the problem of imaging distortion caused by the reflection of image light by the vehicle-mounted head-up display system and improve the viewing experience of the user.

[0035] To better understand the concept of the present application, the following will be illustrated by taking the scenario of a vehicle manufacturer calibrating the head-up display system assembled in the vehicle as an example. Figure 3 A schematic diagram of a flow 300 for image correction according to some embodiments of the present disclosure is shown. As shown in the flow 300, at block 302, a collector is positioned. For example, in the illustrated environment 100, a camera is used as the collector (e.g., the camera 104 in the environment 100). Figure 3 As shown in the flow 300, at block 304, a reference image is acquired. For example, in the illustrated environment 100, the processor 102 acquires the reference image (e.g., the dot matrix image 202 in the environment 100). Figure 1 As shown in the illustrated environment 100, a camera is used as the collector (e.g., the camera 104 in the environment 100). Figure 1The image receiving unit 104 shown simulates the driver's perspective, and places a camera at the driver's eye level in the driver's seat to capture the virtual image 108 projected by the head-up display system 106. At block 304, the edge-extended reference image is projected using the image production unit. For example, in the environment 100, the initial edges of the reference image are extended by the processor 102, and the edge-extended reference image is projected using the image production unit in the head-up display system 106. It can be understood that when the initial edges of the reference image are edge-extended, the outer edges of the reference image can be uniformly extended, for example, the edges of the reference image are uniformly extended outward by 10 pixel points. In some examples, the edges of the reference image can also be randomly extended, for example, the upper half of the edges of the reference image are extended outward by 20 pixel points, and the lower half of the edges are extended outward by 10 pixel points. The extent of edge extension is not specifically limited in the present disclosure, and can be set according to specific needs by those skilled in the art.

[0036] With continued reference to Figure 3 At block 306, the initial virtual image of the edge-extended reference image is acquired using the image acquisition device. For example, in the environment 100, the initial virtual image 108 of the edge-extended reference image on the front windshield 110 is captured by the image receiving unit 104. Figure 1 It can be understood that according to the imaging principle of the mirror, although the head-up display system 106 projects the image of the extended reference image onto the front windshield 110, the initial virtual image 108 of the edge-extended reference image captured by the image receiving unit 104 is away from the image receiving unit 104 relative to the front windshield 110. At block 308, the correction parameter is determined. For example, in the environment 100, the image receiving unit sends the captured initial virtual image 108 of the edge-extended reference image to the processor 102, and the processor 102 determines the correction parameter based on the initial edges of the reference image and the initial virtual image 108. The detailed manner of determining the correction parameter will be described below. At block 310, the target image is corrected based on the correction parameter. For example, in the environment 100, after the processor 102 determines the correction parameter, the target image to be displayed is corrected based on the correction parameter, such as performing deformation processing on each part of the target image. At block 312, the corrected target image is projected using the image production unit. For example, in the environment 100, the processor 102 sends the corrected target image to the head-up display system 106, the head-up display system 106 displays the corrected target image on the image production unit, and projects the corrected target image onto the front windshield 110 of the vehicle 112 through the optical assembly.

[0037] In the determination of the correction matrix, in some examples, a reference virtual image for the extended reference image can also be determined by simulating the imaging based on the extended reference image, and the correction parameters can be determined based on the initial edge of the reference image, the initial virtual image and the reference virtual image. For example, it can be implemented in the following way. As shown in Figure 3 At block 314, a reference virtual image of the simulated reflective imaging of the reference image after edge extension is obtained, for example, in the environment 100, the reference virtual image is determined by simulating the reflective imaging of the extended reference image by the processor 102, it should be understood that since the reference virtual image is the simulated imaging of the extended reference image, the reference virtual image will not be distorted like the initial virtual image. At block 316, the pre-distorted image is determined based on the function inverse operation and bilinear interpolation. For example, in the environment 100 as shown in Figure 1 In the environment 100 as shown, the processor 102 can obtain the image data of the extended reference image, the reference image, the initial virtual image 108 and the reference virtual image, determine the mapping relationship between the images by the function inverse operation based on the determined image data, and when calculating the value of the unknown point in the calculated image, the bilinear interpolation algorithm can be used to perform interpolation operation on the four adjacent points of the target point. The basic principle of bilinear interpolation is to perform linear interpolation in the horizontal and vertical directions, and when determining the value of the target point, the four adjacent points around the target point can be found first. Then calculate the offset of the target point relative to the point in the upper left corner, denoted as the horizontal and vertical interpolation factors A and B. Then interpolate the left and right two points in the horizontal direction by A to get two intermediate values. Then interpolate the two intermediate values in the vertical direction to get the final result. In the interpolation process, the weighted average method is used for calculation, so the adjacent points close to the target point have greater influence on the interpolation result. When processing the interpolation calculation, existing interpolation functions can be used, for example, the LinearNDInterpolator function or the Interp2 function can be called, and the present disclosure does not limit this.

[0038] Continuing to refer to Figure 3 At block 318, the correction parameters are determined based on the pre-distorted image. For example, in the environment 100 as shown, Figure 1 In the environment 100 as shown, the processor 102 stores the corresponding correction parameters based on the determined pre-distorted image and reference image. It should be understood that since the pre-distorted image is determined based on the extended reference image, the initial virtual image and the reference virtual image, the initial virtual image is determined after the optical imaging of the extended reference image, and the reference virtual image is determined after the simulation imaging of the extended reference image. Therefore, the pre-distorted image is at least partially extended in the edge compared with the reference image. When storing the correction parameters, only the correction parameters of the part in the range corresponding to the initial edge of the reference image in the pre-distorted image need to be stored, for example, the size of the frame of the reference image is (M N), the frame size of the extended reference image is ((M+2) (N+2)), after determining the pre-distorted image, only the correction parameters of the frame range of (M N) need to be stored for the pre-distorted image, and when storing the correction parameters, they are stored in the form of correction matrices. The correction parameters indicate the elements of the correction matrices. It should be understood that when selecting the frame range, the center points of the reference image and the pre-distorted image should coincide. In this way, when displaying the corrected target image using the head-up display system, it can be ensured that the frame of the projected image will not be reduced, and the user's viewing experience can be ensured. As described above, when determining the correction parameters, the initial correction parameters can be determined based on blocks 314, 316, and 318, and the reference image is initially corrected. After the head-up display system 106 projects the initially corrected reference image again, the image receiving unit 104 can continue to collect the virtual image of the corrected reference image, and the processor 102 or the detection personnel can determine whether the correction of the reference image meets the relevant requirements. If not, the steps of blocks 306, 308, 314, 316, and 318 are repeated after the reference image is corrected using the previous correction parameters, until the collected virtual image meets the requirements.

[0039] For better understanding Figure 3 of the steps of determining the correction parameters in the flow for image correction shown, the following will be described in detail in combination with Figure 4A to Figure 4C example 400 shown. Among them, Figure 4A a schematic diagram of an example of edge extension of a reference image according to some embodiments of the present disclosure is shown, Figure 4B a schematic diagram of an example of determining a mapping relationship according to some embodiments of the present disclosure is shown, and Figure 4C a schematic diagram of an example of storing a correction matrix for a reference image according to some embodiments of the present disclosure is shown.

[0040] As Figure 4A shown in example 400, the processor can obtain the initial edge of the reference image (hereinafter referred to as "second image") 402 and perform edge extension on the initial edge of the second image 402, thereby determining the extended reference image (hereinafter referred to as "first image") 404. For a more intuitive understanding of edge extension of the image, in example 400, the second image 402 shown includes a plurality of image points (such as the circular shapes shown in the figure) arranged in a dot matrix, and the initial edge of the second image 402 includes a plurality of edge image points located at the image edge of the second image 402. For example, the second image 402 includes M N image points, and a plurality of edge image points located at the outer periphery constitute the initial edge of the second image 402. After uniform edge extension of the second image 402 and increasing the image points, a total of (M+2) (N+2) image points of the first image 404. And, after the initial edge of the second image 402 is determined, a plurality of image points can be added to the outer periphery of the initial edge, so as to expand the image edge of the second image 402. It can be understood that, although the image points of the second image 402 in the example 400 are substantially uniformly distributed, and the edge expansion of the second image 402 is also uniformly expanded. However, in other examples, the image points of the second image 402 can also be unevenly distributed. And, when the edge expansion of the second image 402 is performed, the expansion can also be uneven, for example, the left and right edges are respectively expanded by 2 image points, and the upper and lower edges are respectively expanded by 1 image point. And, in other examples, the image points of the second image 402 can be circularly radiated relative to the center of the second image 402, and the distribution of the image points of the second image 402 is not specifically limited in the present disclosure. In addition, it should be understood that the effective edge of the second image 402 should be limited by the image points, although the outer periphery of the image points of the second image 402 in the example 400 is shown by the dashed image points, the dashed image points in the outer periphery are only auxiliary graphics for better understanding the edge expansion of the second image 402. When the edge expansion of the second image 402 is performed, the reserved pixels of the image production unit of the head-up display system can be used as a correction margin, for example, the image production unit of the head-up display system includes 1920 1080 pixels, and in actual use, only 1600 800 pixels can be used to display images, at this time, the second image can be expanded by using the reserved pixels.

[0041] Next, referring to Figure 4B The first image 404 can be displayed by the head-up display system and projected on the front windshield, and the initial virtual image 406 after the optical imaging of the first image 404 is collected by the collector. The initial virtual image 406 can be a virtual image determined after the first image 404 is imaged by the free-form mirror in the head-up display system. From Figure 4BIt can be seen that before the second image 402 is corrected, the initial virtual image 406 exhibits significant distortion compared to the first image 404, with particularly pronounced edge distortion. The acquisition unit can send the acquired initial virtual image 406 to the processor. Furthermore, the processor can perform simulated reflection imaging on the first image 404 based on the first image 404 to determine the reference virtual image 408, which is the ideal virtual image that does not exhibit distortion when the head-up display system projects the first image 404. Since the reference virtual image 408 is simulated by a computer, it does not exhibit distortion compared to the first image 404. Because the first image 404, the second image 402, the initial virtual image 406, and the reference virtual image 408 are fixed, the processor can determine the pixel map and coordinates of each image point in the first image 404, the second image 402, the initial virtual image 406, and the reference virtual image 408. For example, for... Figure 4B The processor can obtain the first image point (the solid black circle shown in the figure, not labeled) with the first coordinates 420 (P, Q) in the first image 404, the second coordinates 422 (X', Y') in the initial virtual image 406, and the third coordinates 424 (X, Y) in the reference virtual image 408. To determine the mapping relationship between each image point or pixel in the first image 404 and the initial virtual image 406 (hereinafter referred to as the "first mapping relationship"), and the mapping relationship between each image point or pixel in the first image 404 and the reference virtual image 408 (hereinafter referred to as the "second mapping relationship"), the first mapping relationship and the second mapping relationship can be abstracted into functions respectively. sum function Therefore, for the first image point mentioned above, the following functional relationship exists:

[0042] (1)

[0043] (2)

[0044] In the above relationships (1) and (2), the first coordinate 420 (P,Q), the second coordinate 422 (X',Y'), and the third coordinate 424 (X,Y) are all known. Therefore, the first mapping relationship and the second mapping relationship can be determined by the inverse operation of the function. The first mapping relationship indicates the actual distortion of the optical imaging of the first image 404, such as the imaging distortion of the first image 404 by the freeform mirror in the head-up display system, or the imaging distortion of the first image 404 by the shape, position, and curvature of the windshield; the second mapping relationship indicates the simulated imaging (or ideal distortion) of the first image 404.

[0045] Based on the first mapping relationship and the second mapping relationship, the fourth coordinate 426 (P', Q') of the first image point in the pre-distorted image (hereinafter referred to as "third image") 410 can be determined, wherein the fourth coordinate 426 (P', Q') is determined by the first coordinate 420 (P, Q) and the first mapping relationship and the second mapping relationship, and the mapping relationship between the fourth coordinate 426 (P', Q') and the first coordinate 420 (P, Q) can be abstracted as a function , the function may indicate the on-screen distortion of the image, that is, the correction processing of the image. And the function has the following relationship with the function and the function

[0046] (3)

[0047] The above relationship (3) can be explained as the ideal distortion of the first image 404 is the superposition of the on-screen distortion and the optical distortion of the first image 404.

[0048] Therefore, the fourth coordinate 426 (P', Q') can be determined by , wherein represents the inverse function of the function . In this way, the fourth coordinate of all image points in the first image 404 can be determined, and the coordinates of all image points in the third image 410 can be determined. It should be noted that the functions , the function and the function all indicate the mapping relationship of each image point, that is, for different image points, the first mapping relationship and the second mapping relationship can be different, and therefore, the mapping relationship of each image point needs to be determined.

[0049] As described above, since the initial virtual image 406 is distorted relative to the first image 404, the positions of the image points in the initial virtual image 406 are offset relative to the first image 404, and when determining the first mapping relationship and the second mapping relationship, a bilinear interpolation algorithm can be used for calculation, which will not be described here.

[0050] It should be understood that in Figure 4B ​In the middle, it is shown that the first image 404 and the third image 410 have differences in frame size and image size compared with the initial virtual image 406 and the reference virtual image 408, which is only to show that the initial virtual image 406 and the reference virtual image 408 are magnified by the optical system or simulated to be magnified by the optical system. And when the reference virtual image 408 is determined based on the first image 404 by the computer, the size of the projection screen (for example, the size of the target display area of the front windshield of the vehicle) can be determined, the number of pixel points of the imaging screen of the image production unit corresponding to one unit size (for example, 1 centimeter) of the projection screen can be determined, and the corresponding relationship of the computer displaying the first image 404 on the image production unit can be calculated, and then the first image 404 is proportionally enlarged to obtain the simulated reference virtual image 408. It can also be understood that Figure 4B In the middle, the display of the first image point on the first image 404 and the initial virtual image 406 is inverted in position, which is only an example of the inverted projection of the virtual image after the first image 404 is imaged by the optical system. In other examples, the initial virtual image can also not be inverted relative to the first image.

[0051] Next, refer to Figure 4C After the fourth coordinates of each image point of the first image 404 are calculated, the third image 410 can be determined. Based on the initial edge of the second image 402 and the third image 410, the correction matrix 412 for the second image 402 is stored, wherein each element of the correction matrix 412 corresponds to the correction parameter of each image point. As described above, the first mapping relationship and the second mapping relationship between the image points are determined by the bilinear interpolation calculation, and the correction parameter can indicate the mapping relationship of each corresponding image point. When storing the correction matrix and the correction parameter, the pre-distortion image points of the third image 410 located within the initial edge range of the second image 402 can be obtained, and the correction parameter for the second image 402 is determined and stored based on the obtained pre-distortion image points. As Figure 4C shown, the second image 402 includes M N image points, and the third image 410 includes (M+2) (N+2) pre-distortion image points, so when the correction matrix is stored, the correction parameters of the M N pre-distortion image points in the third image 410 are stored to obtain the correction matrix 412 of M N. It should be understood that when the mapping relationship is calculated above, the first image 404, the initial virtual image 406 and the reference virtual image 408 are all center point coincidences, so the M When storing the correction parameters for N image points, they should also coincide with the center point of the second image 402. After determining the correction matrix 412, the original coordinates of each image point in the second image 402 are obtained. Then, based on the correction matrix 412, the second image 402 is corrected using the correction parameters corresponding to each image point. It can be understood that after determining the final correction matrix, if the processor needs to project other images (referred to as "target images") through the head-up display system, after receiving the target image, the processor can obtain the original pixel map of the target image and determine the original coordinates of each pixel in the original pixel map. Then, according to the final correction parameters, the corrected coordinates of each pixel in the target image are determined. Based on the corrected coordinates, the corrected target image is determined and projected, allowing a satisfactory virtual image to be observed on the windshield. Thus, by utilizing the reserved pixels of the image production unit to determine and store the correction matrix of the corresponding image size, the projection distortion of the head-up display system can be improved without reducing the image size, ensuring that the user's viewing experience is not affected.

[0052] It can also be understood that in some scenarios, such as during vehicle off-line calibration or inspection, the manufacturer of the vehicle's head-up display (HUD) system uses the settings of the processor in the factory-installed HUD system to perform initial pre-distortion of the displayed image, causing distortion in the image displayed on the image production unit in an attempt to offset the distortion effect of the windshield on the virtual image. However, due to differences in production tolerances for each vehicle, the pre-distortion by the processor in the HUD system still cannot completely correct the image display distortion. In this case, the method disclosed herein can still be used to correct the image displayed by the HUD system, because in the method disclosed herein, the correction parameters are determined using the initial edges and initial virtual image of the reference image, which is not directly related to whether the processor of the HUD system performs initial pre-distortion of the image. Furthermore, as mentioned above, in the embodiments of this disclosure, the operation can be repeated multiple times when determining the correction parameters until the expected projection is obtained. Therefore, even if the processor of the factory-installed HUD system performs initial pre-distortion of the image, it does not affect the subsequent correction of the reference image.

[0053] In some examples, the reference image can also be segmented when performing edge extension on the reference image. For example, in Example 400, the second image 402 is segmented to obtain multiple patches for the second image 402 and the center point coordinates of the patches located at the edges. The edges of the second image 402 are extended based on the center point coordinates of the edge patches to determine the first image 404. Then, the initial virtual image, the reference virtual image, and the mapping relationship are determined according to the subsequent steps described above.

[0054] It can be understood that, although the method for image correction of the present disclosure is described above by taking the head-up display system in a vehicle as an example, the concept of the present disclosure for image correction is also applicable to other non-vehicle scenarios, such as some indoor or outdoor image projection scenarios.

[0055] Figure 5 A block diagram of an apparatus 500 for image correction according to some embodiments of the present disclosure is shown. The apparatus 500 includes an image acquisition module 502 configured to acquire a reference image, wherein the reference image is used to determine a correction parameter for a head-up display system. The apparatus 500 further includes an edge expansion module 504 configured to expand an initial edge of the reference image. The apparatus 500 further includes a virtual image determination module 506 configured to determine an initial virtual image by the head-up display system based on the expanded reference image. In addition, the apparatus 500 further includes a parameter determination module 508 configured to determine the correction parameter based on the initial edge of the reference image and the initial virtual image, to correct a target image displayed by the head-up display system.

[0056] In some examples, the reference image includes a plurality of image points distributed in a dot pattern, and the edge expansion module 504 includes an initial edge determination module configured to determine an initial edge of the reference image, wherein the initial edge includes a plurality of edge image points located at an image edge of the reference image, and an image point increasing module configured to increase a plurality of image points at the initial edge of the reference image based on the plurality of edge image points to expand the image edge of the reference image.

[0057] In some examples, the parameter determination module 508 includes a reference virtual image determination module configured to determine a reference virtual image for the expanded reference image by simulation imaging based on the expanded reference image, and the parameter determination module 508 is further configured to determine the correction parameter based on the initial edge of the reference image, the initial virtual image and the reference virtual image.

[0058] In some examples, the parameter determination module 508 further includes a coordinate acquisition module configured to acquire a first coordinate of a first image point in the reference image, a second coordinate of the first image point in the initial virtual image and a third coordinate of the first image point in the reference virtual image, a mapping relationship determination module configured to determine a mapping relationship between the first coordinate, the second coordinate and the third coordinate by using a bilinear interpolation algorithm based on the acquired first coordinate, the second coordinate and the third coordinate, and the parameter determination module 508 is further configured to determine the correction parameter based on the mapping relationship and the initial edge.

[0059] In some examples, the parameter determining module 508 further comprises: a first mapping relationship determining module configured to determine a first mapping relationship between the first coordinate and the second coordinate by using a bilinear interpolation algorithm; a second mapping relationship determining module configured to determine a second mapping relationship between the first coordinate and the third coordinate by using the bilinear interpolation algorithm; and the parameter determining module 508 is further configured to determine the correction parameter based on the first mapping relationship, the second mapping relationship and the initial edge.

[0060] In some examples, the parameter determining module 508 further comprises: a fourth coordinate determining module configured to determine a fourth coordinate of the first image point based on the first mapping relationship and the second mapping relationship; a pre-distorted image determining module configured to determine a pre-distorted image for the extended reference image based on the fourth coordinate; and the parameter determining module 508 is further configured to determine the correction parameter based on the pre-distorted image and the initial edge.

[0061] In some examples, the pre-distorted image determining module is further configured to determine the fourth coordinate of all image points in the extended reference image; and determine the pre-distorted image based on the fourth coordinate of all image points.

[0062] In some examples, the parameter determining module 508 is further configured to store the correction parameter for the reference image based on a pre-distorted image point in the pre-distorted image within the initial edge range based on the pre-distorted image point.

[0063] In some examples, the apparatus 500 further comprises: an original pixel map obtaining module configured to obtain an original pixel map of a target image in response to receiving the target image to be displayed by the head-up display system; an original coordinate determining module configured to determine original coordinates of each pixel point in the original pixel map; determine a corrected coordinate for each pixel point based on the original coordinate and the correction parameter; and determine a corrected target image based on the corrected coordinate.

[0064] In some examples, the edge extending module 504 comprises: a tile obtaining module configured to obtain a plurality of tiles for the reference image by performing a tiling process on the reference image; a center point coordinate obtaining module configured to obtain a center point coordinate of a tile located at the edge; and the edge extending module 504 is further configured to extend the edge of the reference image based on the center point coordinate.

[0065] It can be understood that by using the apparatus 500 of the present disclosure, at least one of the many advantages as can be achieved by the method or process described above can be achieved.

[0066] Figure 6A schematic block diagram of an example device 600 that can be used to implement embodiments of the present disclosure is shown. As shown, the device 600 includes a computing unit 601 that can perform various appropriate actions and processes in accordance with computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other by a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0067] A plurality of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0068] The computing unit 601 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the method 200. For example, in some embodiments, the method 200 can be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method 200 described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the method 200 by other any appropriate means, such as by means of firmware.

[0069] The functionality described herein above can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), etc.

[0070] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, causes the machine to perform the functions / acts specified in the flow diagrams and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0071] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores program code for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include one or more lines of a system, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. Further, while operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order shown or in sequential order, or that all illustrated operations are even performed, and that all the described implementations are the only implementations. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, the specific sequences of operations, and the like, are not required in all implementations. In addition, although specific implementations have been illustrated and described herein, it will be appreciated that various modifications can be made. It is therefore contemplated to cover by the present disclosure any and all modifications, variations, or equivalents that fall within the spirit or scope of the underlying principles disclosed above and claimed below. Further, it is intended that any combination of the conditions set forth can be set forth in the claims, separately or in combination, without limitation only by the scope of the claims and their equivalents.

[0072] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for image correction, applied to an in-vehicle head-up display system, the method comprising: A reference image is acquired, which is displayed through the vehicle head-up display system, and is used to determine the correction parameters for the vehicle head-up display system; The initial edges of the reference image are expanded; Based on the extended reference image, an initial virtual image projected by the vehicle head-up display system and a reference virtual image determined by simulated reflection imaging are obtained. Determine the mapping relationship between multiple image points of the expanded reference image and the expanded reference image, the initial virtual image, and the reference virtual image, so as to determine the pre-distortion image based on the mapping relationship; as well as The correction parameters are determined based on the pre-distorted image and the range of the initial edge to correct the target image displayed through the vehicle head-up display system.

2. The method of claim 1, wherein expanding the initial edges of the reference image comprises: Determine the initial edges of the reference image, the initial edges comprising a plurality of edge image points located at the image edges of the reference image; as well as Based on the plurality of edge image points, a plurality of image points are added to the initial edge of the reference image to extend the image edge of the reference image.

3. The method of claim 2, wherein determining the correction parameters based on the pre-distortion image and the range of the initial edge comprises: Obtain the first coordinates of the first image point in the expanded reference image, the second coordinates of the first image point in the initial virtual image, and the third coordinates of the first image point in the reference virtual image; Based on the obtained first coordinate, second coordinate, and third coordinate, the mapping relationship between the first coordinate, second coordinate, and third coordinate is determined using a bilinear interpolation algorithm; as well as The correction parameters are determined based on the mapping relationship and the range of the initial edge.

4. The method of claim 3, wherein determining the correction parameter based on the mapping relationship and the range of the initial edge comprises: A first mapping relationship between the first coordinate and the second coordinate is determined using a bilinear interpolation algorithm; A second mapping relationship between the first coordinate and the third coordinate is determined using a bilinear interpolation algorithm; as well as The correction parameters are determined based on the first mapping relationship, the second mapping relationship, and the range of the initial edge.

5. The method of claim 4, wherein determining the correction parameter based on the first mapping relationship, the second mapping relationship, and the range of the initial edge comprises: Based on the first mapping relationship and the second mapping relationship, determine the fourth coordinate of the first image point; Based on the fourth coordinate, the pre-distortion image is determined for the expanded reference image; as well as The correction parameters are determined based on the pre-distortion image and the range of the initial edge.

6. The method of claim 5, wherein determining the pre-distortion image for the expanded reference image based on the fourth coordinate comprises: Determine the fourth coordinates of all image points in the expanded reference image; as well as The pre-distortion image is determined based on the fourth coordinates of all the image points.

7. The method of claim 6, wherein determining the correction parameters based on the pre-distortion image and the extent of the initial edge comprises: Obtain pre-distortion image points located within the initial edge range of the pre-distortion image; as well as Based on the pre-distorted image points, correction parameters for the reference image are stored.

8. The method according to claim 7, further comprising: In response to receiving a target image to be displayed by the head-up display system, the original pixel map of the target image is obtained; Determine the original coordinates of each pixel in the original pixel image; Based on the original coordinates and the correction parameters, the correction coordinates for each pixel are determined; as well as Based on the corrected coordinates, the corrected target image is determined.

9. The method of claim 1, wherein expanding the initial edges of the reference image comprises: By dividing the reference image into blocks, multiple image blocks for the reference image are obtained; Obtain the coordinates of the center point of the tile located at the edge; as well as Based on the coordinates of the center point, the edges of the reference image are extended.

10. An apparatus for image correction, applied to an in-vehicle head-up display system, the apparatus comprising: An image acquisition module is configured to acquire a reference image, which is displayed through the vehicle head-up display system, for determining correction parameters for the vehicle head-up display system; An edge extension module is configured to extend the initial edges of the reference image; The virtual image determination module is configured to acquire an initial virtual image projected by the vehicle head-up display system and a reference virtual image determined by simulated reflection imaging, based on the extended reference image. as well as The parameter determination module is configured to determine the mapping relationship between multiple image points of the expanded reference image and the expanded reference image, the initial virtual image, and the reference virtual image, to determine a pre-distorted image based on the mapping relationship, and to determine the correction parameters based on the range of the pre-distorted image and the initial edge, so as to correct the target image displayed through the vehicle head-up display system.

11. An electronic device, comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer program product tangibly stored on a non-transitory computer-readable medium and comprising machine-executable instructions that, when executed, cause a machine to perform the method according to any one of claims 1 to 9.

Images