Distortion correction-based left and right eye fusion image head-up display method and device

By capturing the user's eye position in real time and using partitioned distortion correction parameters to correct and fuse the images displayed to the left and right eyes multiple times, the problem of image distortion in HUD displays is solved, achieving a more accurate and vivid visual effect and improving driving safety.

CN118972544APending Publication Date: 2024-11-15HANGZHOU FERVCLOUD TECHNOLOGY CO LTD
View PDF 0 Cites 4 Cited by

Patent Information

Application Number
CN202411023401.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Image distortion caused by optical principles and the shape of the vehicle's windshield in traditional HUD displays affects the driver's understanding and judgment of information. Existing fixed distortion correction parameters cannot adapt to the distortion effect at different positions, resulting in poor image accuracy.

Method used

By capturing real-time position data of the user's left and right eyes, the target partition is determined in the virtual image coordinate space. The distortion correction parameters corresponding to the partition are used to correct the distortion of the left and right eye display images, and multiple fusion processes are performed to generate the target fused image and project it onto the display screen.

Benefits of technology

It eliminates image distortion, improves visual accuracy and clarity, enhances the sense of depth and three-dimensionality, and improves driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972544A_ABST
    Figure CN118972544A_ABST
Patent Text Reader

Abstract

The invention discloses a distortion correction-based left and right eye fusion image head-up display method and device. Comprising the following steps: capturing real-time position data of left and right eyes of a user, and determining a target partition in a plurality of partitions of a virtual image coordinate space according to the real-time position data of the left and right eyes; acquiring left and right eye display images to be subjected to fusion display, and performing distortion correction on the left and right eye display images by using the distortion correction parameters corresponding to the target partition to obtain left and right eye correction images; respectively acquiring pixel points at different positions from the left eye correction image and the right eye correction image, and fusing for multiple times to obtain a plurality of fused images; and obtaining a target fusion image according to the real-time position data of the left eye and the right eye and the plurality of fusion images, and projecting the target fusion image to a display screen for head-up display. The display content can be dynamically adjusted according to the visual angle of the user, correction is carried out through the distortion correction parameters, imaging can better conform to the distortion correction effect corresponding to the current eye position of the user, and the visual experience of the user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a head-up display method and device for left-eye and right-eye fused images based on distortion correction. Background Art

[0002] In traditional HUD displays, images are usually flat and lack a sense of three-dimensionality and depth. Naked-eye 3D HUD display technology uses the parallax principle of the human eye to allow the left and right eyes to see different images, thus creating a sense of three-dimensionality. However, due to factors such as the optical principle of the head-up display system and the shape of the vehicle windshield, the displayed image may be distorted, resulting in distorted, blurred or unclear images. This distortion affects the driver's accurate understanding and judgment of the information, reducing the effectiveness of the HUD.

[0003] At present, the existing technology for image distortion correction is to correct the distortion of the entire image through fixed distortion correction parameters. However, since the distortion effects at different positions are different, the method of using fixed distortion correction parameters alone for correction will result in poor image imaging accuracy, thereby affecting the user experience. Summary of the invention

[0004] The present invention provides a head-up display method and device for left-eye fused images based on distortion correction. By performing distortion correction on the left-eye fused images, it is possible to prevent important information from being difficult to recognize due to distortion, thereby improving visual accuracy and user experience.

[0005] According to one aspect of the present invention, a head-up display method for left-eye and right-eye fused images based on distortion correction is provided, the method comprising:

[0006] Capturing real-time position data of the left and right eyes of the user, and determining a target partition among multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes;

[0007] Obtaining left and right eye display images to be fused and displayed, and using distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images;

[0008] Pixel points at different positions are obtained from the left and right eye correction images respectively, and multiple fusion images are obtained;

[0009] According to the real-time position data of the left and right eyes and multiple fused images, a target fused image is obtained, and the target fused image is projected onto a display screen for head-up display.

[0010] Optionally, before using the distortion correction parameters corresponding to the target partitions to perform distortion correction on the left and right eye display images, the method also includes: after using the display screen to display the standard dot matrix image, by simulating the display image observed by the human eye at different observation positions, obtaining a distorted display image corresponding to each observation position; in the virtual image coordinate space, determining the partition corresponding to each observation position; based on the image difference between the standard dot matrix image and each distorted display image, generating the distortion correction parameters corresponding to each partition.

[0011] The advantage of this setting is that by displaying the standard dot matrix image and simulating the observation of the human eye at different observation positions, the distorted display image corresponding to each observation position can be obtained, which helps to fully understand the distortion of the displayed image at different viewing angles. By generating distortion correction parameters, the distortion characteristics of each partition can be accurately described, thereby achieving more accurate distortion correction.

[0012] Optionally, the real-time position data of the left and right eyes of the user are captured, and based on the real-time position data of the left and right eyes, a target partition is determined in multiple partitions of the virtual image coordinate space, including: capturing the real-time left eye position and the real-time right eye position of the user through a driver monitoring system; and determining the target partition based on the partition into which the real-time left eye position and the real-time right eye position fall in the virtual image coordinate space.

[0013] The advantage of this setting is that by monitoring the real-time position data of the user's left and right eyes, the display content can be adjusted in time to adapt to the user's perspective changes. After determining the target partition, the image can be corrected in a targeted manner to improve the accuracy and clarity of the display, avoid image distortion or unclear problems, and ensure that the user can always get a good visual effect.

[0014] Optionally, before obtaining pixel points at different positions from the left and right eye correction images and performing multiple fusions to obtain multiple fused images, the method also includes: determining the number of fusions of the fused images, and dividing the display screen into multiple areas according to the number of fusions; wherein the number of areas matches the number of fusions.

[0015] The advantage of this setting is that by determining the number of fusion times, the degree and effect of fusion can be controlled. Different fusion times can produce different visual effects to meet the needs of different users or adapt to different application scenarios.

[0016] Optionally, pixel points at different positions are respectively obtained from the left and right eye correction images for multiple fusions to obtain multiple fused images, including: each pixel point of the left and right eye correction images is expanded into RGB channel values ​​to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, the fused images in each fusion round corresponding to the number of fusions are determined.

[0017] The advantage of this setting is that by expanding each pixel of the left and right eye correction images into RGB channel values, the color information of the image can be analyzed and processed in more detail, and the color details can be better preserved and integrated during the fusion process. By fusing the left and right eye correction images, the stereoscopic sense and depth of the image can be enhanced, allowing users to see a stereoscopic three-dimensional image.

[0018] Optionally, according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram, a fused image under each fusion round corresponding to the number of fusion times is determined, including: generating a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determining a fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to fill in the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round.

[0019] The advantage of this setting is that different fusion rounds and fusion strategies can adapt to the different eye positions of the user, so that the user's left and right eyes can achieve the best observation effect at any line of sight position.

[0020] Optionally, the fusion strategy includes: the number of RGB channel value groups, the number of first line indents, and the number of vertical indents.

[0021] The advantage of this setting is that different fusion effects can be achieved by setting different fusion strategies, which provides users with flexibility and customizability to meet the requirements of image fusion in different situations.

[0022] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram respectively to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused images under each fusion round, including: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groupings in the current fusion strategy; adjusting the initial filling mode of the first row according to the number of indentation bits of the first row corresponding to the current fusion round to obtain the target filling mode of the first row; determining each non-first row filling mode in turn starting from the second row according to the target filling mode of the first row and the number of vertical indentation bits; selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0023] The advantage of this setting is that by adjusting the initial filling mode of the first row through the back indentation, the indentation effect between the first row and other rows can be achieved, thereby creating a sense of hierarchy and three-dimensionality in the fused image. Fusion according to the fusion strategy can make the filling mode of each row consistent, ensuring the quality of image fusion.

[0024] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; in adjacent fusion rounds, the number of first line indents in the latter fusion round is 1 more than the number of first line indents in the previous fusion round.

[0025] The advantage of this setting is that by keeping the number of RGB channel value groups and the number of vertical indents the same in different fusion rounds, the stability of the fusion effect can be improved. The gradually increasing number of first line indents can add a certain sense of depth to the fused image and improve the visual effect of the image.

[0026] Optionally, the number of RGB channel value groups is 4 and the number of vertical indents is 1.

[0027] The advantage of this setting is that by setting the specific number of RGB channel value groups and the number of vertical indents, the fusion process can be controlled more finely to ensure the accuracy of image fusion.

[0028] Optionally, a target fused image is obtained based on the real-time position data of the left and right eyes and multiple fused images, including: obtaining the fusion rounds corresponding to each area in the display screen in a mapping relationship between the pre-established areas and fusion rounds based on the real-time position data of the left and right eyes; and using the fusion images under the fusion rounds corresponding to each area to stitch together the target fused image.

[0029] The advantage of this setting is that it can dynamically determine the fusion round corresponding to each area based on the real-time position data of the user's left and right eyes, ensuring that the spliced ​​target fusion image matches the user's perspective, providing the user with a better visual experience.

[0030] Optionally, pixel points at different positions are obtained from the left and right eye correction images respectively, and are fused multiple times to obtain multiple fused images, including: calling a shader plug-in in a three-dimensional image processing engine, obtaining pixel points at different positions from the left and right eye correction images respectively, and are fused multiple times to obtain multiple fused images.

[0031] The benefit of this setting is that by calling the shader plug-in in the 3D image processing engine for multiple fusions, the human eye's perception of different depths and positions can be simulated, thereby enhancing the realism and three-dimensionality of the image. The shader plug-in can be executed in parallel on the graphics processor, improving the efficiency and performance of image processing and reducing processing time and latency.

[0032] According to another aspect of the present invention, a head-up display device for left-eye and right-eye fused images based on distortion correction is provided, the device comprising:

[0033] A target partition determination module is used to capture the real-time position data of the left and right eyes of the user, and determine the target partition among multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes;

[0034] An image distortion correction module is used to obtain left and right eye display images to be fused and displayed, and use the distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images;

[0035] An image fusion module is used to obtain pixel points at different positions from the left and right eye correction images, perform multiple fusions, and obtain multiple fused images;

[0036] The target fusion image determination and display module is used to obtain the target fusion image based on the real-time position data of the left and right eyes and multiple fusion images, and project the target fusion image onto the display screen for head-up display.

[0037] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the left-eye and right-eye fused image head-up display method based on distortion correction described in any embodiment of the present invention.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a head-up display method for left-eye and right-eye fused images based on distortion correction as described in any embodiment of the present invention when executed.

[0042] The technical solution of the embodiment of the present invention can dynamically adjust the display content according to the user's viewing angle by capturing the real-time position data of the user's left and right eyes. By using the distortion correction parameters corresponding to the target partition to correct the left and right eye display images, the distortion of the image can be eliminated, making the image more realistic and accurate, and improving the visual quality. By obtaining pixel points at different positions from the left and right eye correction images for multiple fusions, the details and three-dimensional sense of the image can be enhanced, providing a more realistic and vivid visual effect. By projecting the target fused image onto the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flowchart of a head-up display method for left-eye and right-eye fused images based on distortion correction provided according to Embodiment 1 of the present invention;

[0046] Figure 2 is a schematic diagram of a distorted image generation process provided according to Embodiment 1 of the present invention;

[0047] Figure 3 This is a schematic diagram of a distorted image generation effect provided by Embodiment 1 of the present invention;

[0048] Figure 4is a schematic diagram of a hardware correction process provided according to Embodiment 1 of the present invention;

[0049] Figure 5 is a schematic diagram of a software correction process provided according to Embodiment 1 of the present invention;

[0050] Figure 6 is a schematic diagram of a distortion correction effect provided according to the first embodiment of the present invention;

[0051] Figure 7 is a schematic diagram of a distortion mapping relationship grid provided according to the first embodiment of the present invention;

[0052] Figure 8 is a schematic diagram of a head-up display imaging process provided according to Embodiment 1 of the present invention;

[0053] Fig. 9 is a schematic diagram of a naked-eye 3D imaging principle provided according to Embodiment 1 of the present invention;

[0054] Fig.10 is a schematic diagram of a head-up display process of fused image distortion correction provided according to the first embodiment of the present invention;

[0055] Fig.11 is a flowchart of another head-up display method for left-eye and right-eye fused images based on distortion correction provided according to Embodiment 2 of the present invention;

[0056] Fig.12 is a structural schematic diagram of a head-up display device for left-eye and right-eye fused images based on distortion correction provided according to Embodiment 3 of the present invention;

[0057] Fig.13 It is a structural schematic diagram of a vehicle for implementing a head-up display method for left-eye and right-eye fused images based on distortion correction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0059] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0060] Embodiment 1

[0061] Figure 1 A flowchart of a head-up display method for left-eye fused images based on distortion correction is provided for the first embodiment of the present invention. This embodiment is applicable to the case where distortion processing and real-time head-up display are performed according to the user's eye position. The method can be executed by a head-up display device for left-eye fused images based on distortion correction. The head-up display device for left-eye fused images based on distortion correction can be implemented in the form of hardware and / or software. The head-up display device for left-eye fused images based on distortion correction can be configured in a vehicle. Figure 1 As shown, the method includes:

[0062] S110 , capturing real-time position data of the left and right eyes of the user, and determining a target partition among multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes.

[0063] Among them, the real-time position data of the left and right eyes refers to the specific position information of the user's left and right eyes in the virtual image coordinate space that is monitored and obtained in real time, including the real-time position of the left eye and the real-time position of the right eye, expressed in coordinate form. The virtual image coordinate space is a mathematical tool for describing and processing the position of virtual images. The eye position can be represented by setting a two-dimensional or three-dimensional coordinate system. In practical applications, the coordinate system of the virtual image coordinate space usually has a mapping relationship with other coordinate systems to facilitate interaction and processing in different systems and environments. For example, the coordinates of the virtual image coordinate space are converted into pixel coordinates on the actual screen.

[0064] Specifically, the processor can capture the real-time position data of the user's left and right eyes through human eye detection related equipment, and then determine the partition corresponding to the real-time position of the left eye and the partition corresponding to the real-time position of the right eye in the pre-divided virtual image coordinate space according to the real-time position of the user's left and right eyes, so that targeted processing and optimization of display effects can be performed later. The multiple partitions of the virtual image coordinate space are set to correspond to different eye position data more precisely, which is helpful for subsequent targeted processing and optimization of display effects.

[0065] Optionally, the real-time position data of the left and right eyes of the user are captured, and based on the real-time position data of the left and right eyes, a target partition is determined in multiple partitions of the virtual image coordinate space, including: capturing the real-time left eye position and the real-time right eye position of the user through a driver monitoring system; and determining the target partition based on the partition into which the real-time left eye position and the real-time right eye position fall in the virtual image coordinate space.

[0066] The driver monitoring system refers to a system that monitors and tracks the real-time positions of the driver's left and right eyes. The real-time positions of the left and right eyes include the real-time pupil position of the left eye and the real-time pupil position of the right eye.

[0067] Specifically, the driver monitoring system may include sensors and cameras that can accurately capture the position of the user's eyes. The sensors and cameras can monitor the subtle movements and changes of the user's eyes in real time, thereby determining the user's real-time left eye position and real-time right eye position. The processor will determine the target partitions into which they fall based on the collected coordinate values ​​of the user's real-time left eye position and right eye position in the virtual image coordinate space.

[0068] Exemplarily, the position data of the user's left and right eyes can be associated with the position and range of the virtual image coordinate space partition by establishing a position model, and a threshold range of the left and right eye positions can be set for each partition. When the left and right pupil positions of the user fall within the threshold range of a certain partition, the partition can be determined as the target partition. It should be noted that the determination of the target partition is not limited to which partition the pupil position is located in, and can also be determined by which partition the two-dimensional area of ​​the eye image falls in has a larger area. In addition, the target partition is dynamically adjusted, that is, as the position of the user's eyes changes, the processor will update the position data in real time and dynamically adjust the judgment of the target partition.

[0069] Furthermore, the driver monitoring system can determine the real-time position of the user's left and right eyes through infrared tracking technology or image recognition technology. Infrared tracking technology refers to emitting infrared light to the user's eyes, and then using sensors to receive the reflected light, and calculating the real-time position of the left and right eyes based on the change in light and time difference. Image recognition technology refers to using a camera to capture an image of the user's eyes, and using image processing algorithms to identify the eye's feature points, such as the pupil center, eye corners, etc., to determine the real-time position of the left and right eyes.

[0070] In a specific implementation, the driver monitoring system may include a DMS eye tracking device, which is a device that acquires the driver's eye status in real time by installing an optical camera and an infrared camera on the car. It analyzes the acquired information through a deep learning algorithm, can determine the driver's status, realize driver identification, driver fatigue monitoring, driver attention monitoring, and monitoring of dangerous driving behaviors, and issue different levels of warnings.

[0071] In summary, by determining the target partition corresponding to the real-time position of the left and right eyes of the user, a display effect that better meets the user's needs can be provided, and the real-time adaptability is strong, and adjustments can be made quickly according to the real-time changes in the user's eye position.

[0072] S120 , obtaining left and right eye display images to be fused and displayed, and using distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images.

[0073] The distortion correction parameter is a matrix used to correct the deformation of the image caused by the optical system or other factors. Different partitions correspond to different distortion correction parameters, so it is necessary to use the distortion correction parameters corresponding to the target partition for correction. The distortion correction parameters can effectively correct the image according to the characteristics of the partition, thereby obtaining a clearer and more accurate image.

[0074] It can be known that the imaging process is essentially the transformation of several coordinate systems. First, a point in space is transformed from the world coordinate system to the camera coordinate system, then projected to the imaging plane (image physical coordinate system), and finally the data on the imaging plane is transformed to the image plane (image pixel coordinate system). Figure 2 A schematic diagram of a distorted image generation process is provided for the first embodiment of the present invention. Figure 3 A schematic diagram of a distorted image generation effect is provided for the first embodiment of the present invention. Figure 3 In the figure, the left side of the arrow is the original image, and the right side of the arrow is the distorted image observed by the human eye. Figure 2In the image, after the original image is imaged through the curved mirror of the head-up display, a distorted image will be generated due to the change in the coordinate correspondence. That is, the undistorted coordinates (U, V) in the image pixel coordinate system (uOv coordinate system) fall on (Ud, Vd) in the uOv coordinate system after radial distortion and tangential distortion. In other words, the relationship between the real image imgR and the distorted image imgD is: imgR(U, V) = imgD(Ud, Vd). The distortion correction process is the process of determining the distortion correction parameters through the mapping relationship between the real image and the distorted image. Through distortion correction, the distorted image can be restored to the original image.

[0075] In the prior art, distortion correction includes hardware correction and software correction. Hardware distortion correction is to reduce or eliminate image distortion by designing and adjusting the hardware level of the imaging system. Software distortion correction is to perform distortion correction through image processing algorithms after acquiring the image. Software distortion correction can rearrange and adjust image pixels through mathematical transformation. For example, Figure 4 A schematic diagram of a hardware correction process is provided for the first embodiment of the present invention. Figure 4 In the head-up display system, after the original image is input, it is corrected by the chip to obtain a pre-distorted image, and then the normal image can be displayed after being projected through a curved mirror. Figure 5 A schematic diagram of a software correction process is provided for the first embodiment of the present invention. Figure 5 In the process, the original image is subjected to distortion correction to generate a pre-distorted image, which is then input into the head-up display system and projected through a curved mirror to display a normal image. The software correction method corrects the displayed image through an image processing algorithm, but it may be limited by computing power and real-time performance. The hardware correction method requires additional optical elements or mechanical structures, which increases the complexity and cost of the system.

[0076] Furthermore, the technical solution of the embodiment of the present invention can obtain images frame by frame, read the pixel value of each pixel coordinate point in the image, calculate the corresponding pixel value through the mapping relationship, and output the final normal image frame by frame when performing real-time distortion correction. It can achieve high-quality head-up display image correction without increasing system complexity and cost, thereby improving the driver's visual experience and driving safety. Figure 6 A schematic diagram of a distortion correction effect is provided for the first embodiment of the present invention. Figure 6 The left side of the middle arrow is the original image, and the right side of the arrow is the displayed image after distortion correction.

[0077] Optionally, before using the distortion correction parameters corresponding to the target partitions to perform distortion correction on the left and right eye display images, the method also includes: after using the display screen to display the standard dot matrix image, by simulating the display image observed by the human eye at different observation positions, obtaining a distorted display image corresponding to each observation position; in the virtual image coordinate space, determining the partition corresponding to each observation position; based on the image difference between the standard dot matrix image and each distorted display image, generating the distortion correction parameters corresponding to each partition.

[0078] Specifically, after the standard dot matrix image is displayed on the display screen, the camera device can simulate the display image that can be observed by the human eye at different observation positions, thereby obtaining a distorted display image corresponding to each observation position, and determining the partition corresponding to each observation position in the virtual image coordinate space. The processor will compare and analyze the distorted display image with the standard dot matrix image, including comparing the differences between the two in terms of pixel arrangement, shape, size, position, etc., to evaluate the image distortion generated by each screen partition. Based on the comparative analysis, the distortion correction parameters corresponding to each partition can be determined.

[0079] In a specific implementation, the following steps can be used to determine the distortion correction parameters of each partition: first, a square grid image with a regular geometric structure can be used as a standard dot matrix image for head-up display. Then, a camera device is used to simulate the distorted display image corresponding to each observation position of the human eye, and then feature points are extracted from the standard dot matrix image and the distorted display image. By matching the feature points, the mapping relationship between the standard dot matrix image and the distorted display image can be determined, and based on the mapping relationship, the distortion parameter matrix of each partition can be further constructed as the distortion correction parameter.

[0080] Furthermore, the mapping relationship between the normal image (standard dot matrix image) and the anti-distorted image (distorted display image) can be saved by constructing a Mesh grid. Figure 7 A schematic diagram of a distortion mapping relationship grid is provided for the first embodiment of the present invention. Figure 7 The mapping relationship between the standard dot matrix image and the distorted display image is shown in Figure 1. The constructed grid mapping relationship can be bound to the canvas. By outputting the image to the canvas, the canvas calls the corresponding distortion correction parameters for processing, and the distortion-corrected image can be generated.

[0081] To sum up, after determining the distortion correction parameters, in the subsequent image display process, the distortion correction parameters corresponding to the target partition can be used to correct the image distortion. The corrected image can more realistically reflect the shape and proportion of the object, reduce visual distortion and deformation, and thus provide users with a more accurate display effect.

[0082] S130 , respectively obtaining pixel points at different positions from the left and right eye corrected images, and performing multiple fusions to obtain multiple fused images.

[0083] The pixel point refers to the smallest unit of the image. The fused image refers to a new image obtained by processing the left and right eye corrected images according to the fusion rules.

[0084] Specifically, the processor can obtain the left and right eye correction images that need to be fused and displayed. The left and right eye correction images can be prepared in advance, or generated according to real-time application scenarios and user needs. The images include but are not limited to image files such as pictures, videos, and video streams. The processor will select corresponding pixel points from different positions in the left and right eye correction images according to the pre-set fusion rules, and fuse them in turn to obtain multiple fused images.

[0085] Optionally, pixel points at different positions are obtained from the left and right eye correction images respectively, and are fused multiple times to obtain multiple fused images, including: calling a shader plug-in in a three-dimensional image processing engine, obtaining pixel points at different positions from the left and right eye correction images respectively, and are fused multiple times to obtain multiple fused images.

[0086] Specifically, the processor will call the shader plug-in in the 3D image processing engine. The shader plug-in is a component of the 3D image processing engine and has 3D image processing functions. The processor can call the shader plug-in and extract pixels at different positions from the left and right eye correction images according to the preset fusion rules, and perform multiple fusions. In each fusion process, the shader plug-in will combine the selected pixels according to the preset fusion rules, and finally generate multiple fused images.

[0087] In summary, after multiple fusions of shader plug-ins, multiple fused images can be generated, thereby better meeting different application needs and visual effect requirements.

[0088] S140: acquiring a target fused image according to the real-time position data of the left and right eyes and the multiple fused images, and projecting the target fused image onto a display screen for head-up display.

[0089] Figure 8 A schematic diagram of the head-up display imaging process is provided for the first embodiment of the present invention. The head-up display (HUD) is a driving assistance instrument used on a vehicle. It is a comprehensive electronic display device composed of electronic components, display components, controllers, etc. The HUD can project vehicle speed, navigation information, warning information, etc. in the form of images and characters in front of the driver through optical components. The HUD can project images or information directly into the user's field of vision, allowing the user to see the relevant content without lowering his head.

[0090] Fig. 9 A schematic diagram of the naked-eye 3D imaging principle is provided for the first embodiment of the present invention. The naked-eye 3D display technology refers to a 3D display technology that allows users to directly watch three-dimensional images with their naked eyes without wearing special 3D glasses, presenting a 3D effect. The front-end display part of the naked-eye 3D display technology needs to be calculated by the pixel imaging unit so that the left and right eyes of the person see different pictures, and merge them into a 3D effect picture in the brain. Fig. 9 In the HUD, the optical design is performed on the HUD so that the user's left eye sees image P1 and the right eye sees image P2 through the imaging structure, such as the windshield. Due to the binocular parallax formed by images P1 and P2, the objects seen by the user have a sense of depth and space. The right eye sees the image of the right eye, which is synthesized into a three-dimensional picture with a sense of depth in the user's brain. Naked-eye 3D imaging can adjust the binocular parallax by changing the position between the two images, so that the distance of the virtual image perceived by the user changes (the actual distance of the virtual image remains unchanged). The closer the two images are, the closer the virtual image distance perceived by the user is; conversely, the farther the two images are, the farther the virtual image distance perceived by the user is.

[0091] Optionally, a target fused image is obtained based on the real-time position data of the left and right eyes and multiple fused images, including: obtaining the fusion rounds corresponding to each area in the display screen in a mapping relationship between the pre-established areas and fusion rounds based on the real-time position data of the left and right eyes; and using the fusion images under the fusion rounds corresponding to each area to stitch together the target fused image.

[0092] Specifically, each position data has a corresponding preset fusion rule, and the fusion rule includes the number of fused images selected for each area for display. The processor can determine the fusion round corresponding to each area in the display screen in the pre-established mapping relationship between the area and the fusion round according to the real-time position data of the left and right eyes. Then, the fused images under the fusion round corresponding to each area are used for splicing operations to obtain the target fused image. It should be noted that the splicing process needs to consider the alignment and matching of pixels to ensure that the target fused image after splicing is visually continuous and consistent.

[0093] Specific application scenarios: Fig.10 The present invention provides a schematic diagram of a fused image distortion correction head-up display process, such as Fig.10As shown in the figure, the DMS device obtains the position information of the left and right eyes in real time and uses the distortion correction matrix parameters corresponding to different areas to correct the distortion of the left and right eye input material images, and switches the image fusion rules and display content in real time according to the corrected image and the monitored eye position, thereby achieving the naked eye 3D effect in the AR HUD display. By combining the eye box position partitioning scheme of the 3DHUD to calculate the left and right fused images in real time, the target fusion object is determined in real time in the target screen partition for dynamic head-up display to the user.

[0094] The head-up display technology of this embodiment can be applied to a variety of driving scenarios where the driver drives through a vehicle. For example, for car driving, important data such as vehicle speed, navigation information, vehicle fault prompts, traffic sign recognition, etc. can be displayed on the car's windshield, so that the driver does not need to look down at the dashboard, keep his eyes on the road, and improve driving safety. For aircraft driving, the pilot's helmet or cockpit windshield can display key information such as flight altitude, speed, heading, target distance, etc., which helps the pilot quickly obtain important data during the flight and reduce distraction. For cycling driving, such as motorcycles, the speed, speed, navigation instructions, etc. can be displayed on the helmet mask, so that the rider does not need to look away to check the dashboard while riding.

[0095] The technical solution of the embodiment of the present invention can dynamically adjust the display content according to the user's viewing angle by capturing the real-time position data of the user's left and right eyes. By using the distortion correction parameters corresponding to the target partition to correct the left and right eye display images, the distortion of the image can be eliminated, making the image more realistic and accurate, and improving the visual quality. By obtaining pixel points at different positions from the left and right eye correction images for multiple fusions, the details and three-dimensional sense of the image can be enhanced, providing a more realistic and vivid visual effect. By projecting the target fused image onto the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0096] Embodiment 2

[0097] Fig.11 The flowchart of a method for head-up display of left-eye fused images based on distortion correction provided in the second embodiment of the present invention is provided. This embodiment adds a specific process of obtaining pixel points at different positions from the left-eye corrected images for multiple fusions to obtain multiple fused images based on the first embodiment. The specific contents of steps S210-S220 are roughly the same as steps S110-S120 in the first embodiment, so they will not be described in detail in this embodiment. Figure 2 As shown, the method includes:

[0098] S210 , capturing real-time position data of the left and right eyes of the user, and determining a target partition among multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes.

[0099] Optionally, the real-time position data of the left and right eyes of the user are captured, and based on the real-time position data of the left and right eyes, a target partition is determined in multiple partitions of the virtual image coordinate space, including: capturing the real-time left eye position and the real-time right eye position of the user through a driver monitoring system; and determining the target partition based on the partition into which the real-time left eye position and the real-time right eye position fall in the virtual image coordinate space.

[0100] S220 , obtaining left and right eye display images to be fused and displayed, and using distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images.

[0101] Optionally, before using the distortion correction parameters corresponding to the target partitions to perform distortion correction on the left and right eye display images, the method also includes: after using the display screen to display the standard dot matrix image, by simulating the display image observed by the human eye at different observation positions, obtaining a distorted display image corresponding to each observation position; in the virtual image coordinate space, determining the partition corresponding to each observation position; based on the image difference between the standard dot matrix image and each distorted display image, generating the distortion correction parameters corresponding to each partition.

[0102] S230, determining the fusion times of the fused image, and dividing the display screen into a plurality of regions according to the fusion times, wherein the number of regions matches the fusion times.

[0103] S240 , expand each pixel point of the left and right eye correction images into RGB channel values ​​respectively, to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram.

[0104] Specifically, the processor will expand each pixel in the left-eye correction image and the right-eye correction image into RGB channel values ​​respectively to describe the color composition of each pixel.

[0105] S250 , determining fused images in each fusion round corresponding to the number of fusion times according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram.

[0106] Specifically, the processor will perform various fusion rounds in sequence according to the RGB channel arrangement diagram of the left eye and the RGB channel arrangement diagram of the right eye according to the pre-set fusion times. In each round of fusion, the RGB channel values ​​of the left eye and the right eye can be extracted and combined according to the set fusion strategy.

[0107] Optionally, according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram, a fused image under each fusion round corresponding to the number of fusion times is determined, including: generating a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determining a fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to fill in the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round.

[0108] The full-eye RGB channel arrangement map to be filled is a blank image whose size and pixel layout are the same as those of the fused image and is used to store the RGB channel values ​​selected during the fusion process.

[0109] Specifically, the processor can select corresponding RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram in each round of fusion according to the fusion strategy, and fill them into the corresponding positions in the full eye RGB channel arrangement diagram to be filled. It should be noted that the fused image obtained by each fusion is a complete full eye RGB channel arrangement diagram, and the processor can perform different degrees of fusion in each fusion round according to the fusion strategy.

[0110] Optionally, the fusion strategy includes: the number of RGB channel value groups, the number of first line indents, and the number of vertical indents.

[0111] Among them, the values ​​involved in the fusion strategy need to be adapted to the arrangement of the spectroscopic grating. The number of RGB channel value groups is related to the number of pixels covered by the grating unit. For example, when a grating unit can cover 8 pixels (i.e., the backlight lamp beads), the number of RGB channel value groups needs to be set to 4. At this time, among every 8 lamp beads, the 4 lamp beads on the left and the 4 lamp beads on the right will emit in different directions, and can be seen by the left eye and the right eye respectively.

[0112] Specifically, in each round, different indentation numbers of different rows are also used to adapt to various different arrangements of the spectroscopic gratings. The crosstalk phenomenon can be effectively reduced by arranging the gratings in an inclined manner.

[0113] It should be noted that the indentation of different rounds is mainly for traversing various states. Different states correspond to the display of pixels (lamp beads) covered by the grating. It should be noted that there is an upper limit to the number of states. For example, in the case of 4 groups, when each round is indented by 1 bit, the upper limit of the number of states is reached in 8 rounds. At this time, the situation of the 9th round will be the same as the 1st round. Therefore, the upper limit of the round is actually the upper limit of the number of states. When partitioning and selecting the number of fusion rounds, it is actually selecting the corresponding state for fusion operation. The fusion round needs to select the most appropriate state according to specific needs and conditions to achieve the best fusion effect and achieve the desired goal.

[0114] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram respectively to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused images under each fusion round, including: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groupings in the current fusion strategy; adjusting the initial filling mode of the first row according to the number of indentation bits of the first row corresponding to the current fusion round to obtain the target filling mode of the first row; determining each non-first row filling mode in turn starting from the second row according to the target filling mode of the first row and the number of vertical indentation bits; selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0115] Specifically, according to the number of RGB channel value groups, it can be determined which channel values ​​are selected from the left eye and right eye RGB channel arrangement diagrams to fill the first row of pixels in the full eye RGB channel arrangement diagram to be filled. Then, according to the number of indents in the first row of the current fusion round, the filling position of the first row is moved to the right by a certain number of places to adjust the indentation of the initial filling method of the first row. Starting from the second row, the processor can determine the filling method of each non-first row in turn according to the target filling method of the first row and the number of vertical indents, where the number of vertical indents determines the vertical offset of each row of the filling position relative to the previous row.

[0116] Further, according to the first row target filling method and each non-first row filling method, the corresponding RGB channel values ​​are selected from the left eye and right eye RGB channel arrangement diagrams in turn to fill in the corresponding pixel points in the full eye RGB channel arrangement diagram to be filled, so as to complete the image filling fusion under the current fusion round and obtain the fused image of this round. The processor repeats the above steps to perform multiple fusion rounds, and finally obtains the fused images corresponding to each fusion round.

[0117] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; in adjacent fusion rounds, the number of first line indents in the latter fusion round is 1 more than the number of first line indents in the previous fusion round.

[0118] Among them, in different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same, which means that the number of RGB channel values ​​fused in each fusion round and the adjustment method of the vertical spacing between rows are fixed. In adjacent fusion rounds, the number of indents after the first line of the next fusion round is 1 more than the number of indents of the first line of the previous fusion round, which means that as the fusion rounds increase, the indent to the right of the first line increases by 1.

[0119] Specifically, the same number of RGB channel value groups ensures consistency in the division and processing of color channel values ​​in each fusion round, thereby maintaining the stability and consistency of color information processing in the fusion process as a whole.

[0120] Optionally, the number of RGB channel value groups is 4 and the number of vertical indents is 1.

[0121] Specifically, the number of RGB channel value groups of 4 means that every four RGB values ​​in the horizontal direction are grouped, and the number of vertical indents of 1 means that the RGB value indents one position downward in the vertical direction for each row. By grouping the four RGB values ​​in the horizontal direction of the left and right eye display images into a group, and indenting one position downward in the vertical direction for each row, the left and right eye images are fused N times, and each fusion is indented one position backward from the first row, and the pixels are saved by taking N fusion images. N is the total number of fusions corresponding to the number of screen partitions.

[0122] S260, obtaining a fusion round corresponding to each area in the display screen according to the mapping relationship between the areas and the fusion rounds established in advance according to the real-time position data of the left and right eyes.

[0123] S270: Use the fused images of the fusion rounds corresponding to each region to obtain a target fused image.

[0124] The technical solution of the embodiment of the present invention can more intuitively understand the difference and distribution of the left and right eye images in the color channels by expanding each pixel of the left and right eye correction images respectively to obtain the RGB channel arrangement diagrams of the left and right eyes, so that the fused image can better integrate the information of the left and right eyes. By fusing the left and right eye correction images, the stereoscopic sense and depth sense of the image can be enhanced, so that the user can see a stereoscopic three-dimensional image.

[0125] Embodiment 3

[0126] Fig.12 The structure diagram of a head-up display device for left-eye fusion images based on distortion correction provided by the third embodiment of the present invention is shown in FIG. Fig.12 As shown, the device includes: a target partition determination module 310, which is used to capture the real-time position data of the left and right eyes of the user, and determine the target partition in multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes;

[0127] An image distortion correction module 320 is used to obtain left and right eye display images to be fused and displayed, and to perform distortion correction on the left and right eye display images using distortion correction parameters corresponding to the target partition to obtain left and right eye corrected images;

[0128] An image fusion module 330 is used to obtain pixel points at different positions from the left and right eye correction images, perform multiple fusions, and obtain multiple fused images;

[0129] The target fusion image determination and display module 340 is used to obtain the target fusion image according to the real-time position data of the left and right eyes and the multiple fusion images, and project the target fusion image onto the display screen for head-up display.

[0130] Optionally, the device also includes: a distortion correction parameter generation module, which is used to obtain a distorted display image corresponding to each observation position by simulating the display image observed by the human eye at different observation positions after displaying a standard dot matrix image on a display screen before using the distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images; determine the partitions corresponding to each observation position in the virtual image coordinate space; and generate distortion correction parameters corresponding to each partition based on the image difference between the standard dot matrix image and each distorted display image.

[0131] Optionally, the target partition determination module 310 is specifically used to: capture the user's real-time left eye position and real-time right eye position through the driver monitoring system; and determine the target partition according to the partition where the real-time left eye position and real-time right eye position fall in the virtual image coordinate space.

[0132] Optionally, the device also includes: a display screen division module, which is used to obtain pixel points at different positions from the left and right eye correction images for multiple fusions to obtain multiple fused images, determine the number of fusion times of the fused images, and divide the display screen into multiple areas according to the number of fusions; wherein the number of areas matches the number of fusions.

[0133] Optionally, the image fusion module 330 specifically includes: an RBG arrangement diagram generation unit, which is used to expand each pixel point of the left and right eye corrected images into RGB channel values ​​respectively to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; a fused image generation unit, which is used to determine the fused images in each fusion round corresponding to the number of fusion times according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram.

[0134] Optionally, the fused image generating unit specifically includes: a whole-eye RBG arrangement diagram generating subunit, used to generate a whole-eye RGB channel arrangement diagram to be filled corresponding to the fused image; an image fusion filling subunit, used to determine a fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to fill the matching image position in the whole-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round.

[0135] Optionally, an image fusion filling subunit is specifically used to: obtain a current fusion strategy corresponding to a current fusion round, and determine an initial filling mode for the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; adjust the initial filling mode for the first row according to the number of first row indentation bits corresponding to the current fusion round to obtain a target filling mode for the first row; determine each non-first row filling mode in turn starting from the second row according to the first row target filling mode and the number of vertical indentation bits; select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image position in the full-eye RGB channel arrangement diagram to be filled, to obtain a fused image under the current fusion round.

[0136] Optionally, the target fusion image determination and display module 340 is specifically used to: obtain the fusion round corresponding to each area in the display screen according to the mapping relationship between the pre-established area and the fusion round based on the real-time position data of the left and right eyes; use the fusion images under the fusion round corresponding to each area to stitch together to obtain the target fusion image.

[0137] Optionally, the image fusion module 330 is specifically used to call a shader plug-in in the three-dimensional image processing engine to obtain pixel points at different positions from the left and right eye corrected images, perform multiple fusions, and obtain multiple fused images.

[0138] The technical solution of the embodiment of the present invention can dynamically adjust the display content according to the user's viewing angle by capturing the real-time position data of the user's left and right eyes. By using the distortion correction parameters corresponding to the target partition to correct the left and right eye display images, the distortion of the image can be eliminated, making the image more realistic and accurate, and improving the visual quality. By obtaining pixel points at different positions from the left and right eye correction images for multiple fusions, the details and three-dimensional sense of the image can be enhanced, providing a more realistic and vivid visual effect. By projecting the target fused image onto the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0139] A head-up display device for left-eye fused images based on distortion correction provided in an embodiment of the present invention can execute a head-up display method for left-eye fused images based on distortion correction provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0140] Embodiment 4

[0141] Fig.13 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0142] like Fig.13 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0144] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a head-up display method for left-right eye fusion images based on distortion correction.

[0145] In some embodiments, a head-up display method for left-right eye fusion images based on distortion correction may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the head-up display method for left-right eye fusion images based on distortion correction described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform a head-up display method for left-right eye fusion images based on distortion correction in any other appropriate manner (e.g., by means of firmware).

[0146] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0148] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0150] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0152] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0153] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A head-up display method for left-eye and right-eye fused images based on distortion correction, characterized in that: include: Capturing real-time position data of the left and right eyes of the user, and determining a target partition among multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes; Obtaining left and right eye display images to be fused and displayed, and using distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images; Pixel points at different positions are obtained from the left and right eye correction images respectively, and multiple fusion images are obtained; A target fused image is acquired according to the real-time position data of the left and right eyes and the multiple fused images, and the target fused image is projected onto a display screen for head-up display.

2. The method according to claim 1, characterized in that: Before using the distortion correction parameters corresponding to the target partition to perform distortion correction on the left-eye and right-eye display images, the method further includes: After the standard dot matrix image is displayed on the display screen, a distorted display image corresponding to each observation position is obtained by simulating the display image observed by the human eye at different observation positions; In the virtual image coordinate space, determining a partition corresponding to each observation position; According to the image difference between the standard dot matrix image and each of the distorted display images, a distortion correction parameter corresponding to each partition is generated.

3. The method according to claim 1, characterized in that: Capturing the real-time position data of the left and right eyes of the user, and determining a target partition among a plurality of partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes, including: Capturing the user's real-time left eye position and real-time right eye position through a driver monitoring system; A target partition is determined according to the partition into which the real-time left eye position and the real-time right eye position fall in the virtual image coordinate space.

4. The method according to claim 1, characterized in that: Before acquiring pixel points at different positions from the left and right eye correction images and fusing them multiple times to obtain multiple fused images, the method further includes: Determining the number of fusion times of the fused image, and dividing the display screen into a plurality of areas according to the number of fusion times; The number of regions matches the number of fusions.

5. The method according to claim 4, characterized in that Pixel points at different positions are obtained from the left and right eye correction images for multiple fusions to obtain multiple fused images, including: Each pixel point of the left and right eye correction images is expanded into RGB channel values ​​respectively to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; According to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, a fused image in each fusion round corresponding to the fusion number is determined.

6. The method according to claim 5, characterized in that According to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, a fused image in each fusion round corresponding to the fusion number is determined, including: Generate a full-eye RGB channel arrangement map to be filled corresponding to the fused image; Determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to fill the matching image position in the full eye RGB channel arrangement diagram to be filled, and obtain the fused image under each fusion round.

7. The method according to claim 6, characterized in that The fusion strategy includes: the number of RGB channel value groups, the number of first line indents and the number of vertical indents.

8. The method according to claim 7, characterized in that According to the fusion strategy, RGB channel values ​​are selected from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to be filled into the matching image position in the full eye RGB channel arrangement diagram to be filled, and the fused image under each fusion round is obtained, including: Obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; According to the number of first-line back-indent bits corresponding to the current fusion round, the back-indent adjustment is performed on the first-line initial filling mode to obtain the first-line target filling mode; According to the first line target filling mode and the number of vertical indents, starting from the second line, each non-first line filling mode is determined in sequence; According to the first-row target filling method and each non-first-row filling method, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to be filled into the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain a fused image under the current fusion round.

9. The method according to claim 7, characterized in that: In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; In adjacent fusion rounds, the number of indents after the first line of the latter fusion round is 1 more than the number of indents after the first line of the previous fusion round.

10. The method according to claim 9, characterized in that The number of RGB channel value groups is 4, and the number of vertical indent digits is 1.

11. The method according to claim 4, characterized in that Acquiring a target fused image according to the left and right eye real-time position data and the multiple fused images, including: According to the mapping relationship between the areas and fusion rounds established in advance, the left and right eye real-time position data is used to obtain the fusion rounds corresponding to each area in the display screen; The target fused image is obtained by splicing the fused images in the fusion rounds corresponding to each region.

12. The method according to any one of claims 1 to 11, characterized in that: Pixel points at different positions are obtained from the left and right eye correction images for multiple fusions to obtain multiple fused images, including: The shader plug-in in the three-dimensional image processing engine is called to obtain pixel points at different positions from the left and right eye correction images, respectively, and perform multiple fusions to obtain multiple fused images.

13. A head-up display device for left-eye and right-eye fusion images based on distortion correction, characterized in that: include: A target partition determination module, used to capture the real-time position data of the left and right eyes of the user, and determine the target partition among the multiple partitions of the virtual image coordinate space according to the real-time position data of the left and right eyes; An image distortion correction module, used to obtain left and right eye display images to be fused and displayed, and use distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain left and right eye corrected images; An image fusion module is used to obtain pixel points at different positions from the left and right eye correction images, perform multiple fusions, and obtain multiple fused images; The target fused image determination and display module is used to obtain the target fused image according to the real-time position data of the left and right eyes and the multiple fused images, and project the target fused image onto the display screen for head-up display.

14. A carrier, characterized in that: The vehicle comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 12.

15. A computer storage medium, characterized in that The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 12 when executed.

Citation Information

Cited By

  • Image distortion correction method and device, vehicle, medium and program product

    CN121437339A

  • Image processing apparatus and method, head-up display apparatus, and computer storage medium

    CN121644779A

  • Image acquisition method and device, head-up display and storage medium

    CN121708084A

  • Left and right eye fusion image head-up display method and device based on distortion correction

    WO2026025889A1