Head-up display method and device for predictive fused left-eye and right-eye image, vehicle, and medium

Abstract

A head-up display method and device for a predictive fused left-eye and right-eye image, a vehicle, and a medium. The display method comprises: on the basis of historically captured left-eye and right-eye historical position data of a user, determining left-eye and right-eye real-time predicted position data of the user (S110); acquiring left-eye and right-eye display images to be fused for display, and respectively acquiring pixel points at different positions from the left-eye and right-eye display images to perform fusion multiple times, so as to obtain a plurality of fused images (S120); and on the basis of the left-eye and right-eye real-time predicted position data and the plurality of fused images, acquiring a target fused image, and projecting the target fused image onto a display screen for head-up display (S130).

Description

Head-up display method and device for predicting and fusing images for left and right eyes, vehicle and medium

[0001] The present application claims priority to the Chinese patent application No. 202411023397.6 filed on July 29, 2024 to the Chinese Patent Office, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, for example to a head-up display method and device for predicting and fusing images for left and right eyes, vehicle and medium. BACKGROUND

[0003] In traditional HUD display, the image is usually flat, lacking stereoscopic and depth perception. Naked-eye 3D HUD display technology uses the principle of parallax of human eyes to make the left and right eyes see different images, thereby producing stereoscopic effect.

[0004] With the continuous development of the automotive industry, the demand of drivers for information display is also increasing. Traditional 2D HUD display has been unable to meet the needs of drivers for more intuitive and vivid information display. Naked-eye 3D HUD display technology can present important driving information to drivers in a more realistic way, improving driving safety and comfort.

[0005] Currently, some methods have attempted to fuse the real-time position data of the left and right eyes of the DMS, but due to the persistence of vision effect, when using real-time data as parameters to generate image fusion pictures, the human perception is that the crosstalk is large, and image distortion and ghosting phenomena are easily produced during projection. SUMMARY

[0006] The present application provides a head-up display method and device for predicting and fusing images for left and right eyes, vehicle and medium, which realizes more personalized and accurate image display by predicting the eye position of the user, and also provides a better visual experience.

[0007] According to an aspect of the present application, a head-up display method for predicting and fusing images for left and right eyes is provided, the method comprising:

[0008] determining real-time predicted position data of the left and right eyes of the user according to historical captured historical position data of the left and right eyes of the user;

[0009] obtaining left and right eye display images to be fused, and obtaining pixel points at different positions from the left and right eye display images for multiple times of fusion to obtain multiple fusion images;

[0010] obtaining a target fusion image according to the real-time predicted position data of the left and right eyes and the multiple fusion images, and projecting the target fusion image to a display screen for head-up display.

[0011] Optionally, the left and right eye real-time prediction position data of the user is determined according to historical captured left and right eye historical position data of the user, comprising: obtaining N left and right eye positions captured by the driver monitoring system, wherein N is an integer greater than 2; calculating a left and right eye historical position convergence value according to the first N-1 left and right eye positions in the order from far to near according to the capture time; and determining the left and right eye real-time prediction position data of the user according to the data difference between the left and right eye historical position convergence value and the Nth left and right eye historical position.

[0012] The advantage of this arrangement is that by considering multiple historical data points, the influence of individual outliers can be reduced, improving the accuracy of the prediction. By calculating the historical data convergence value, the historical data trend can be determined, thereby more accurately predicting the left and right eye positions of the user.

[0013] Optionally, the left and right eye real-time prediction position data of the user is determined according to the data difference between the left and right eye historical position convergence value and the Nth left and right eye historical position value, comprising: in response to the data difference between the left and right eye historical position convergence value and the Nth left and right eye historical position value being greater than a difference threshold, obtaining a target prediction variable corresponding to the left and right eye historical position convergence value based on a variable configuration list; determining the sum of the target prediction variable and the Nth left and right eye historical position value as the left and right eye real-time prediction position data; and in response to the data difference between the left and right eye historical position convergence value and the Nth left and right eye historical position value being less than or equal to a preset difference threshold, determining the Nth left and right eye historical position value as the left and right eye real-time prediction position data.

[0014] The advantage of this arrangement is that when the data difference is large, a target prediction variable can be introduced to adjust the prediction result, improving the accuracy and adaptability of the prediction. The variable configuration list can meet different needs. The response speed of the system can be improved, and the real-time prediction position can be quickly determined when the data difference is small.

[0015] Optionally, before obtaining a plurality of fusion images by performing multiple fusions on the pixel points of different positions in the left and right eye display images, the method further comprises: determining the number of fusions of the fusion image, and dividing the display screen into a plurality of regions according to the number of fusions; wherein the number of regions matches the number of fusions.

[0016] The advantage of this arrangement is that by determining the number of fusions, the degree and effect of fusion can be controlled. Different fusion numbers can produce different visual effects, meeting the needs of different users or adapting to different application scenarios.

[0017] Optionally, a plurality of fusion images are obtained by performing fusion on pixels at different positions in the left-eye display image and the right-eye display image respectively, including: expanding each pixel in the left-eye display image and the right-eye display image into RGB channel values respectively to obtain a left-eye RGB channel arrangement and a right-eye RGB channel arrangement; and determining a fusion image in each fusion round corresponding to the fusion times according to the left-eye RGB channel arrangement and the right-eye RGB channel arrangement.

[0018] The advantage of this arrangement is that by expanding each pixel in the left-eye display image and the right-eye display image into RGB channel values, the color information of the image can be analyzed and processed in more detail, and the color details can be better preserved and integrated in the fusion process. By fusing the left-eye display image and the right-eye display image, the stereoscopic effect and depth of the image can be enhanced, and the user can see a stereoscopic three-dimensional image.

[0019] Optionally, the fusion image in each fusion round corresponding to the fusion times is determined according to the left-eye RGB channel arrangement and the right-eye RGB channel arrangement, including: generating a to-be-filled full-eye RGB channel arrangement corresponding to the fusion image; determining a fusion strategy corresponding to each fusion round, and filling the RGB channel values selected from the left-eye RGB channel arrangement and the right-eye RGB channel arrangement into the matching image position in the to-be-filled full-eye RGB channel arrangement according to the fusion strategy to obtain the fusion image in each fusion round.

[0020] The advantage of this arrangement is that different fusion rounds and fusion strategies can adapt to different eye positions of the user, so that the left and right eyes of the user can achieve the best observation effect at any eye position.

[0021] Optionally, the fusion strategy includes: the number of RGB channel value groups, the number of first-line back indents, and the number of vertical back indents.

[0022] The advantage of this arrangement is that by setting different fusion strategies, different fusion effects can be achieved, providing flexibility and customizability for the user to meet the requirements of image fusion in different situations.

[0023] Optionally, according to the fusion strategy, RGB channel values are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram and filled into the matching image positions in the to-be-filled full-eye RGB channel arrangement diagram to obtain a fusion image under each fusion round, including: obtaining a current fusion strategy corresponding to a current fusion round, and determining a first row initial filling mode corresponding to the to-be-filled full-eye RGB channel arrangement diagram according to the number of RGB channel value groups in the current fusion strategy; performing back indentation adjustment on the first row initial filling mode according to the first row back indentation number corresponding to the current fusion round to obtain a first row target filling mode; starting from the second row, sequentially determining each non-first row filling mode according to the first row target filling mode and the vertical back indentation number; and selecting RGB channel values from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode and filling the RGB channel values into the matching image positions in the to-be-filled full-eye RGB channel arrangement diagram to obtain the fusion image under the current fusion round.

[0024] The advantage of such a setting is that by adjusting the first row initial filling mode through back indentation, the indentation effect between the first row and other rows can be achieved, thereby creating a sense of hierarchy and stereoscopic effect in the fusion image. According to the fusion strategy, the filling mode of each row can be consistent, thereby ensuring the quality of image fusion.

[0025] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the vertical back indentation number are the same; and in adjacent fusion rounds, the first row back indentation number of the next fusion round is 1 more than the first row indentation number of the previous fusion round.

[0026] The advantage of such a setting is that by keeping the number of RGB channel value groups and the vertical back indentation number the same under different fusion rounds, the stability of the fusion effect can be improved. The gradually increasing first row back indentation number can increase the depth of the fusion image and improve the visual effect of the image.

[0027] Optionally, the number of RGB channel value groups is 4 and the vertical back indentation number is 1.

[0028] The advantage of such a setting is that by setting specific numbers of RGB channel value groups and vertical back indentation numbers, the fusion process can be controlled more finely, thereby ensuring the accuracy of image fusion.

[0029] Optionally, according to the left-eye and right-eye real-time prediction position data and the plurality of fusion images, a target fusion image is obtained, including: according to the mapping relationship between the regions and the fusion rounds in the pre-established region, obtaining the fusion round corresponding to each region in the display screen according to the left-eye and right-eye real-time prediction position data; and using the fusion image under the fusion round corresponding to each region to splice a target fusion image.

[0030] The advantage of such an arrangement is that the fusion round corresponding to each region can be dynamically determined according to the real-time position data of the left and right eyes of the user, ensuring that the target fusion image obtained by splicing matches the perspective of the user and providing better visual experience for the user.

[0031] Optionally, before the multiple times of fusion are performed on the pixel points at different positions in the left and right eye display images, the method further comprises: determining a target partition according to the partition in which the real-time predicted left eye position and the real-time predicted right eye position fall in the virtual image coordinate space; performing distortion correction on the left and right eye display images using the distortion correction parameter corresponding to the target partition to obtain corrected left and right eye display images; wherein the distortion correction parameter corresponding to each observation position is determined by displaying a standard dot matrix image on the display screen in advance, obtaining a distortion display image corresponding to each observation position by simulating the observation of the display image by the human eye at different observation positions, and comparing the image difference between the distortion display image and the standard dot matrix image.

[0032] The advantage of such an arrangement is that the distortion display image corresponding to each observation position can be obtained by displaying a standard dot matrix image and simulating the observation of the human eye at different observation positions, which helps to comprehensively understand the distortion of the display image at different perspectives. The distortion correction parameter can accurately describe the distortion characteristics of each partition, thereby achieving more accurate distortion correction.

[0033] According to another aspect of the present application, a head-up display device for predicting left and right eye fusion images is provided, which comprises:

[0034] A left and right eye position prediction module configured to determine real-time predicted left and right eye position data of a user according to historical captured left and right eye historical position data of the user;

[0035] An image fusion module configured to obtain left and right eye display images to be fused, perform multiple times of fusion on pixel points at different positions in the left and right eye display images respectively, and obtain a plurality of fusion images;

[0036] A target fusion image determination and display module configured to obtain a target fusion image according to the real-time predicted left and right eye position data and the plurality of fusion images, and project the target fusion image into a display screen for head-up display.

[0037] According to another aspect of the present application, a vehicle is provided, which comprises:

[0038] At least one processor;

[0039] and a memory connected in communication with the at least one processor;

[0040] The memory stores a computer program capable of being executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the head-up display method of the predicted fusion image of the left eye and the right eye according to any one of the embodiments of the present application.

[0041] According to another aspect of the present application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the head-up display method of the predicted fusion image of the left eye and the right eye according to any one of the embodiments of the present application when executed by the processor.

[0042] The technical solution of the embodiments of the present application can predict the current left eye and right eye positions of the user by analyzing the historical left eye and right eye positions of the user, and can perform image fusion display in advance through the prediction, so as to reduce image distortion and ghosting phenomenon, thereby avoiding imaging crosstalk problems. By performing multiple fusion on the pixel points at different positions obtained from the left eye and right eye display images, the details and stereoscopic effect of the image can be enhanced, and more realistic and vivid visual effects can be provided. By projecting the target fusion image into the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving the driving safety.

[0043] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent through the following description. BRIEF DESCRIPTION OF DRAWINGS

[0044] FIG. 1 is a flowchart of a head-up display method of a predicted fusion image of a left eye and a right eye according to an embodiment of the present application;

[0045] FIG. 2 is a flowchart of another head-up display method of a predicted fusion image of a left eye and a right eye according to an embodiment of the present application;

[0046] FIG. 3 is a schematic diagram of a left eye and right eye historical position change curve according to an embodiment of the present application;

[0047] FIG. 4 is a schematic diagram of a left eye and right eye historical position prediction curve according to an embodiment of the present application;

[0048] FIG. 5 is a schematic diagram of another left eye and right eye historical position change curve according to an embodiment of the present application;

[0049] FIG. 6 is a schematic diagram of another left eye and right eye historical position prediction curve according to an embodiment of the present application;

[0050] FIG. 7 is a schematic diagram of a head-up display imaging process according to an embodiment of the present application;

[0051] FIG. 8 is a schematic diagram of a principle of naked-eye 3D imaging according to an embodiment of the present application;

[0052] FIG. 9 is a schematic diagram of a process of head-up display of a fused image according to an embodiment of the present application;

[0053] FIG. 10 is a flowchart of a method of head-up display of left-eye and right-eye predicted fused images according to another embodiment of the present application;

[0054] FIG. 11 is a schematic diagram of a structure of a device of head-up display of left-eye and right-eye predicted fused images according to an embodiment of the present application;

[0055] FIG. 12 is a schematic diagram of a structure of a vehicle implementing a method of head-up display of left-eye and right-eye predicted fused images according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application as well as above-mentioned appended drawings are intended to distinguish similar objects and not to describe a particular order or sequence. It is to be understood that the data thus designated can be interchanged, where appropriate, so that the embodiments of the present application described herein can be carried out in other than the order or sequence illustrated or described herein. Furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, a method, a system, a product, or an apparatus that comprises a list of steps or units is not necessarily limited to those steps or units that are clearly listed, but can include other steps or units that are not clearly listed or inherent to such a process, method, product, or apparatus.

[0057] Embodiment One

[0058] FIG. 1 is a flowchart of a method of head-up display of left-eye and right-eye predicted fused images according to an embodiment of the present application. The embodiment can be applied to a case of head-up display in prediction of a user's eye position. The method can be executed by a device of head-up display of left-eye and right-eye predicted fused images. The device of head-up display of left-eye and right-eye predicted fused images can be implemented in the form of hardware and / or software. The device of head-up display of left-eye and right-eye predicted fused images can be configured in a vehicle. As shown in FIG. 1, the method comprises:

[0059] S110, determining real-time predicted position data of left and right eyes of a user according to historical position data of left and right eyes of the user captured historically.

[0060] The left and right eye historical position data refers to the specific position information of the left and right eyes of the user recorded in a specified historical period, including the real-time position of the left eye and the real-time position of the right eye, which are expressed in the form of coordinates. The left and right eye real-time prediction position data refers to the prediction value of the positions of the left and right eyes of the user at the next moment at the current moment obtained by analyzing the known historical position data.

[0061] In a specific embodiment, the left and right eye real-time prediction position data of the user can be determined by constructing an eye position prediction model. That is, a large amount of historical captured left and right eye historical position data of the user is collected. Then, a machine learning algorithm is used to process and model the left and right eye historical position data. The modeling algorithm can be regression analysis, time series prediction, neural network, etc. Through learning and training of the left and right eye historical position data, the eye position prediction model can identify the rules therein and predict the left and right eye real-time prediction position data of the user.

[0062] FIG. 2 is a flowchart of a head-up display method for a left and right eye prediction fusion image according to an embodiment of the present application. Step S110 mainly includes the following steps S111 to S113:

[0063] S111, obtaining N left and right eye positions of the user captured by the driver monitoring system, wherein N is an integer greater than 2.

[0064] Illustratively, the driver monitoring system can include sensors and cameras that can accurately capture the positions of the eyes of the user. The sensors and cameras can monitor the subtle movements and changes of the eyes of the user in real time, thereby determining the left and right eye positions. The processor can extract the N left and right eye positions of the user captured by the driver monitoring system, and N is set to an integer greater than 2 in order to accumulate sufficient sample data for analysis and prediction, thereby providing more historical information and increasing the reliability of the prediction.

[0065] In some embodiments, the driver monitoring system can determine the left and right eye real-time positions of the user by infrared tracking technology or image recognition technology. The infrared tracking technology refers to emitting infrared light to the eyes of the user, then receiving the reflected light by a sensor, and calculating the left and right eye real-time positions according to the changes of the light and the time difference. The image recognition technology refers to using a camera to capture the image of the eyes of the user, identifying the feature points of the eyes such as the center of the pupil and the corners of the eyes by an image processing algorithm, and thereby determining the left and right eye historical position values.

[0066] In a specific embodiment, the driver monitoring system can include a DMS eye tracking device, which is a device for real-time acquisition of the eye state of the driver by installing optical and infrared cameras on the car. It can analyze the acquired information through a deep learning algorithm to determine the state of the driver, realize the identification of the driver, the monitoring of the fatigue of the driver, the monitoring of the attention of the driver, and the monitoring of dangerous driving behaviors, and perform different levels of early warning.

[0067] In summary, by acquiring a plurality of historical left and right eye positions, the information contained in the historical data can be fully utilized to improve the accuracy and reliability of the prediction.

[0068] S112, in the order of far to near according to the capture time, acquire the left and right eye position of the previous N-1 to calculate the convergence value of the left and right eye historical position.

[0069] For example, the processor uses a time series data analysis method, that is, the data is processed in the order of far to near according to the capture time. Specifically, the left and right eye positions of the previous N-1 can be acquired, and the convergence value of the left and right eye historical position can be calculated through a specific algorithm or calculation method. The convergence value refers to the trend or concentration of the left and right eye positions.

[0070] In summary, by processing the data in the order of far to near according to the capture time and calculating the convergence value, the time trend of the change of the eye position can be effectively captured.

[0071] S113, according to the data difference between the convergence value of the left and right eye historical position and the Nth left and right eye historical position, determine the real-time prediction position data of the left and right eyes of the user.

[0072] For example, the processor can analyze and determine the real-time prediction position data of the left and right eyes of the user according to the data difference between the calculated convergence value of the left and right eye historical position and the Nth left and right eye historical position, that is, the real-time prediction position data of the left and right eyes is obtained by comprehensively analyzing the trend of the historical data. Among them, if the data difference is large, it means that the latest data deviates from the historical trend. If the data difference is small or equal to the preset threshold, it means that the latest data is consistent with the historical trend.

[0073] Optionally, the real-time predicted position data of the left and right eyes of the user is determined according to a data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes, including: if the data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is greater than a difference threshold, a target prediction variable corresponding to the convergent value of the historical positions of the left and right eyes is obtained based on a variable configuration list; and an addition value between the target prediction variable and the Nth historical position value of the left and right eyes is determined as the real-time predicted position data of the left and right eyes; and if the data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is less than or equal to a preset difference threshold, the Nth historical position value of the left and right eyes is determined as the real-time predicted position data of the left and right eyes.

[0074] For example, when the processor determines the real-time predicted position data of the left and right eyes of the user according to a data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes, there are two cases.

[0075] The first case is that the data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is greater than a preset difference threshold. At this time, the current data fluctuation is large, that is, the eye position of the user is dynamically changing, and the processor obtains a target prediction variable corresponding to the convergent value of the historical positions of the left and right eyes according to a preset variable configuration list. The variable configuration list is a prediction variable configuration table obtained by a large amount of data training and learning, which contains prediction variables corresponding to different convergent values. After obtaining the target prediction variable, the processor performs addition operation on the target prediction variable and the Nth historical position value of the left and right eyes, and determines an addition value as the real-time predicted position data of the left and right eyes.

[0076] In addition, the second case is that the data difference between the convergent value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is less than or equal to a preset difference threshold. At this time, it indicates that the current data fluctuation is relatively small, that is, the eye position of the user is basically unchanged, and at this time, the Nth historical position value of the left and right eyes can be directly determined as the real-time predicted position data of the left and right eyes, because it can accurately reflect the current eye position.

[0077] In one specific embodiment, the DMS real-time data can be followed by motion to perform uniform motion, increase the prediction variable, and add the prediction variable to the DMS real-time data to predict the human eye position. FIG. 3 is a schematic diagram of a left eye historical position value position change curve provided by an embodiment of the present application, and FIG. 4 is a schematic diagram of a left eye historical position value position prediction curve provided by an embodiment of the present application. In FIG. 3, the eye position at time t0 is Ep0, the eye position at time t1 is Ep1, the distance between Ep0 and Ep1 is Ld, and the time interval between t0 and t1 is t. As shown in FIG. 4, the motion trajectory of the DMS real-time data in the time period t0-t1 is fluctuating motion, which does not conform to the image fusion use scenario. FIG. 5 is another schematic diagram of a left eye historical position value position change curve provided by an embodiment of the present application, and FIG. 6 is another schematic diagram of a left eye historical position value position prediction curve provided by an embodiment of the present application. The eye position at time t0 is Ep0, the eye position at time t1 is Ep1, the distance between Ep0 and Ep1 is Ld, and the time interval between t0 and t1 is t. In the time period t0-t1, the use of the motion following DMS real-time data can effectively reduce the jitter of the data and better fit the actual motion trajectory of the human eye. When the prediction variable is added to the eye position coordinate during the motion of the human eye, it can be better used for image fusion display. Thus, the influence of the human eye ghosting is avoided.

[0078] In summary, the real-time prediction position data is determined according to the difference between the historical convergence value and the latest data, so that the prediction method can adapt to different situations. When the data difference is large, the prediction variable is processed by superposition, and when the difference is small, the simple rule is followed, which has strong flexibility and adaptability. When the difference between the latest data and the historical trend is small, the latest data is directly used as the prediction value, which ensures a certain accuracy, avoids excessive complex calculation, and improves the stability and response speed of the system.

[0079] In S120, left and right eye display images to be fused and displayed are obtained, and pixels at different positions in the left and right eye display images are obtained respectively to perform fusion multiple times to obtain multiple fusion images.

[0080] The pixel refers to the smallest constituent unit of the image. The fusion image refers to a new image obtained by processing the left and right eye display images according to the fusion rule.

[0081] For example, the processor can obtain left and right eye display images to be fused and displayed. The left and right eye display images can be prepared in advance or generated according to real-time application scenarios and user needs, and the images include but are not limited to image files such as pictures, videos, and video streams. The processor will select corresponding pixels from different positions in the left and right eye display images according to the pre-set fusion rule to perform fusion in sequence to obtain multiple fusion images.

[0082] Optionally, before the multiple fusions of the pixel points in different positions in the left and right eye display images are obtained, the method further comprises: determining a target partition according to the partition in which the real-time predicted left eye position and the real-time predicted right eye position fall in the virtual image coordinate space; using the distortion correction parameter corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain corrected left and right eye display images; wherein the distortion correction parameter corresponding to each observation position is determined by displaying a standard dot matrix image on the display screen in advance, simulating the display image observed by the human eye at different observation positions to obtain a distortion display image corresponding to each observation position, and comparing the image difference between the distortion display image and the standard dot matrix image.

[0083] The distortion correction parameter is a matrix used to correct the deformation of the image caused by the optical system or other factors. Different partitions correspond to different distortion correction parameters, so the distortion correction parameter corresponding to the target partition needs to be used for correction. The distortion correction parameter can effectively correct the image according to the characteristics of the partition, so as to obtain a clearer and more accurate image.

[0084] For example, after the display screen displays the standard dot matrix image, the camera can simulate the display image that can be observed by the human eye at different observation positions, so as to obtain a distortion display image corresponding to each observation position, and determine a partition corresponding to each observation position in the virtual image coordinate space. The processor compares and analyzes the distortion display image and the standard dot matrix image, including comparing the differences between them in terms of pixel arrangement, shape, size, position, etc., to evaluate the image distortion caused by each screen partition. According to the comparison and analysis, the distortion correction parameter corresponding to each partition can be determined.

[0085] In a specific embodiment, the following steps can be used to determine the distortion correction parameter of each partition: a grid image with a regular geometric structure can be used as a standard dot matrix image for head-up display. Then, the camera simulates the distortion display image corresponding to each observation position of the human eye, and then extracts feature points from the standard dot matrix image and the distortion display image. By matching the feature points, the mapping relationship between the standard dot matrix image and the distortion display image can be determined, and based on the mapping relationship, the distortion parameter matrix of each partition can be constructed as the distortion correction parameter.

[0086] In summary, after the distortion correction parameter is determined, the distortion correction parameter corresponding to the target partition can be used to correct the image in the subsequent image display process. The corrected image can more truly reflect the shape and proportion of the object, reduce the visual distortion and deformation, and thus provide more accurate display effect for the user.

[0087] Optionally, a plurality of fusion images are obtained by performing multiple fusions on the pixels at different positions in the left-eye display image and the right-eye display image, including: calling a shader plug-in in the three-dimensional image processing engine to perform multiple fusions on the pixels at different positions in the left-eye display image and the right-eye display image to obtain a plurality of fusion images.

[0088] For example, the processor can call a shader plug-in in the three-dimensional image processing engine. The shader plug-in is a component in the three-dimensional image processing engine and has a 3D image processing function. The processor can call the shader plug-in and extract pixels at different positions from the left-eye display image and the right-eye display image according to a preset fusion rule to perform multiple fusions. In each fusion process, the shader plug-in combines the selected pixels according to the preset fusion rule to finally generate a plurality of fusion images.

[0089] As described above, the multiple fusions by the shader plug-in can generate a plurality of fusion images, thereby better meeting different application requirements and visual effect requirements.

[0090] S130, obtaining a target fusion image according to the real-time predicted position data of the left eye and the right eye and the plurality of fusion images, and projecting the target fusion image to the display screen for head-up display.

[0091] FIG. 7 is a schematic diagram of a head-up display imaging process according to an embodiment of the present application. The head-up display (HUD) is a driving assistance instrument used on a vehicle and is a comprehensive electronic display device composed of electronic components, display components, controllers, and the like. The HUD can project vehicle speed, navigation information, warning information, and the like in the form of images and characters to the front of the driver through optical components. Through the HUD, images or information can be directly projected into the user's line of sight, so that the user can see the relevant content without lowering his head.

[0092] Fig. 8 is a schematic diagram of a naked-eye 3D imaging principle according to an embodiment of the present application. The naked-eye 3D display technology refers to a 3D display technology in which a user can directly watch a three-dimensional image with naked eyes without wearing special 3D glasses, and a 3D effect is presented. The front-end display part of the naked-eye 3D display technology needs to be calculated by a pixel imaging unit, so that the left eye and the right eye of a person see different images, and the images are fused into a 3D effect image in the brain. In Fig. 8, the HUD is optically designed, so that the left eye of a user sees an image P1 through an imaging structure, such as a windshield, and the right eye sees an image P2. Due to the binocular disparity formed by the image P1 and the image P2, the user feels a sense of depth and a sense of space for the object, and the right eye sees a right-eye image, which is synthesized into a stereoscopic image with a sense of depth in the brain of the user. The naked-eye 3D imaging can change the position between the two images, adjust the binocular disparity, and make the user feel that the virtual image distance has changed (the actual distance of the virtual image does not change). The closer the two images are, the closer the user feels the virtual image distance to be; on the contrary, the farther the two images are, the farther the user feels the virtual image distance to be.

[0093] Optionally, the target fusion image is obtained according to the real-time predicted position data of the left and right eyes and the plurality of fusion images, including: obtaining the fusion round corresponding to each region in the display screen according to the mapping relationship between the regions and the fusion rounds in the real-time predicted position data of the left and right eyes; and splicing the fusion images under the fusion round corresponding to each region to obtain the target fusion image.

[0094] Optionally, the target fusion image is obtained according to the real-time predicted position data of the left and right eyes and the plurality of fusion images, including: obtaining the fusion round corresponding to each region in the display screen according to the mapping relationship between the regions and the fusion rounds in the real-time predicted position data of the left and right eyes; and splicing the fusion images under the fusion round corresponding to each region to obtain the target fusion image.

[0095] Specific application scenario: Fig. 9 is a kind of fusion image head-up display process schematic diagram provided in the application, as shown in Figure 9, the position information of left and right eyes is obtained in real time by DMS equipment, and the predicted human eye position is calculated by prediction algorithm, and the eye position data is transmitted to three-dimensional image processing engine, three-dimensional image processing engine can be based on the predicted human eye position Real-time switching image fusion rule and display content, in combination with image distortion correction, so as to achieve in AR HUD display naked eye 3D effect. By combining the eye box position partition scheme of 3D HUD, the three-dimensional image processing engine calculates left and right fusion images in real time, and dynamically determines the target fusion object in the target screen partition to the user for dynamic head-up display.

[0096] The technical scheme of the embodiment of the application can predict the current left and right eye positions of the user by analyzing the historical left and right eye positions of the user, and can perform image fusion display in advance through prediction, so as to reduce image distortion and ghosting phenomenon, thereby avoiding imaging crosstalk problem. By obtaining pixel points at different positions from left and right eye display images for multiple fusion, the details and stereoscopic effect of the image can be enhanced, and more realistic and vivid visual effect can be provided. By projecting the target fusion image into the display screen for head-up display, the user can obtain important information without looking down at the instrument, and the driving safety is improved.

[0097] Embodiment two

[0098] Fig. 10 is a flow chart of a kind of left and right eye prediction fusion image head-up display method provided in the second embodiment of the application, and the specific process of the embodiment is obtained by increasing multiple fusion of different positions from left and right eye correction images in the above-mentioned embodiment one. The specific content of step S210 is substantially the same as that of step S110 in embodiment one, so it will not be described here. As shown in Figure 10, the method comprises:

[0099] S210, according to the historical captured left and right eye historical position data of the user, determine the real-time prediction position data of the left and right eyes of the user.

[0100] Optionally, according to the historical captured left and right eye historical position data of the user, determine the real-time prediction position data of the left and right eyes of the user, comprising: obtaining N left and right eye positions of the user captured by the driver monitoring system, wherein N is an integer greater than 2;According to the order from far to near, the left and right eye historical position convergence value is calculated by obtaining the first N-1 left and right eye positions;According to the data difference between the left and right eye historical position convergence value and the Nth left and right eye historical position, the real-time prediction position data of the left and right eyes of the user is determined.

[0101] Optionally, the real-time predicted position data of the left and right eyes of the user is determined according to the data difference between the convergence value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes, including: if the data difference between the convergence value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is greater than a difference threshold value, obtaining a target prediction variable corresponding to the convergence value of the historical positions of the left and right eyes based on a variable configuration list; determining the sum value between the target prediction variable and the Nth historical position value of the left and right eyes as the real-time predicted position data of the left and right eyes; and if the data difference between the convergence value of the historical positions of the left and right eyes and the Nth historical position value of the left and right eyes is less than or equal to a preset difference threshold value, determining the Nth historical position value of the left and right eyes as the real-time predicted position data of the left and right eyes.

[0102] S220, determine the fusion number of the fusion image, and divide the display screen into a plurality of regions according to the fusion number; wherein the number of regions matches the fusion number.

[0103] S230, respectively expand each pixel point of the left eye display image and the right eye display image into RGB channel values to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram.

[0104] For example, the processor respectively expands each pixel point in the left eye display image and the right eye display image into RGB channel values to describe the color composition of each pixel point.

[0105] S240, determine the fusion image in each fusion round corresponding to the fusion number according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram.

[0106] For example, the processor performs operations in each fusion round according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram according to a preset fusion number. In each round of fusion, the RGB channel values of the left eye and the right eye can be extracted and combined according to a set fusion strategy.

[0107] Optionally, the fusion image in each fusion round corresponding to the fusion number is determined according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, including: generating a to-be-filled full-eye RGB channel arrangement diagram corresponding to the fusion image; determining a fusion strategy corresponding to each fusion round, and filling the RGB channel values selected from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram into the matching image positions of the to-be-filled full-eye RGB channel arrangement diagram according to the fusion strategy to obtain the fusion image in each fusion round.

[0108] The to-be-filled full-eye RGB channel arrangement diagram is a blank image with the same size and pixel layout as the fusion image, and is used to store the selected RGB channel values in the fusion process.

[0109] Exemplarily, the processor can select corresponding RGB channel values from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the fusion strategy in each round of fusion, and fill the corresponding positions in the full-eye RGB channel arrangement diagram to be filled. It should be noted that the fusion image obtained each time is a complete full-eye RGB channel arrangement diagram, and the processor can perform different degrees of fusion in each fusion round according to the fusion strategy.

[0110] Optionally, the fusion strategy includes the number of RGB channel value groups, the first row back indentation bit number, and the vertical back indentation bit number.

[0111] Among them, the numerical values involved in the fusion strategy need to be adapted to the arrangement of the light grating. The number of RGB channel value groups is related to the number of pixels covered by the grating unit, for example, when a grating unit can cover 8 pixels (i.e. the lamp beads of the backlight), the number of RGB channel value groups needs to be set to 4. At this time, among every 8 lamp beads, the left 4 lamp beads and the right 4 lamp beads will be shot in different directions, and then can be seen by the left eye and the right eye respectively.

[0112] Exemplarily, in each round, different indentation bit numbers of different rows are also used to adapt to various different light grating arrangement situations, and through the inclined light grating arrangement mode, the crosstalk phenomenon can be effectively reduced.

[0113] It should be noted that the indentation of different rounds is mainly to traverse various states. Different states correspond to the display of the pixels (lamp beads) covered by the grating. It should be noted that the number of states has an upper limit, for example, in the case of 4 groups of 1, when the indentation is 1 bit each round, 8 rounds reach the upper limit of the number of states, and at this time the 9th round is the same as the 1st round. Therefore, the upper limit of the round is actually the upper limit of the number of states. When the number of fusion rounds of the partition selection is selected, it is actually to select the corresponding state to perform the fusion operation. The fusion round needs to select the most suitable state according to the specific requirements and conditions, so as to achieve the best fusion effect and achieve the expected goal.

[0114] Optionally, according to the fusion strategy, RGB channel values are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram and filled into the matching image positions in the to-be-filled full-eye RGB channel arrangement diagram to obtain a fusion image under each fusion round, including: obtaining a current fusion strategy corresponding to a current fusion round, and determining a first row initial filling mode corresponding to the to-be-filled full-eye RGB channel arrangement diagram according to the number of RGB channel value groups in the current fusion strategy; performing back indentation adjustment on the first row initial filling mode according to the first row back indentation bit number corresponding to the current fusion round to obtain a first row target filling mode; starting from the second row, sequentially determining each non-first row filling mode according to the first row target filling mode and the vertical back indentation bit number; and according to the first row target filling mode and each non-first row filling mode, selecting RGB channel values from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram and filling them into the matching image positions in the to-be-filled full-eye RGB channel arrangement diagram to obtain a fusion image under the current fusion round.

[0115] For example, according to the number of RGB channel value groups, it can be determined which channel values are selected from the left-eye and right-eye RGB channel arrangement diagrams to fill the first row of pixel points in the to-be-filled full-eye RGB channel arrangement diagram. Then, according to the first row back indentation bit number of the current fusion round, the filling position of the first row is moved to the right by a certain number of bits to perform back indentation adjustment on the first row initial filling mode. Starting from the second row, the processor can sequentially determine the filling mode of each non-first row according to the first row target filling mode and the vertical back indentation bit number, wherein the vertical back indentation bit number determines the vertical offset of each row of filling positions relative to the previous row.

[0116] In some embodiments, according to the first row target filling mode and each non-first row filling mode, corresponding RGB channel values are sequentially selected from the left-eye and right-eye RGB channel arrangement diagrams and filled into the corresponding pixel points in the to-be-filled full-eye RGB channel arrangement diagram, so that the image filling fusion under the current fusion round is completed, and a fusion image of this round is obtained. The processor can perform multiple fusion rounds by repeatedly executing the above steps, and finally obtain the fusion images corresponding to each fusion round.

[0117] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the vertical back indentation bit number are the same; in adjacent fusion rounds, the first row back indentation bit number of the next fusion round is one more than the first row indentation bit number of the previous fusion round.

[0118] The same RGB channel value grouping number and the same longitudinal back indentation bit number in different fusion rounds means that the number of RGB channel values fused in each fusion round and the adjustment mode of the spacing between rows in the vertical direction are fixed. The first row of the next fusion round is indented by one more bit than the first row of the previous fusion round.

[0119] For example, the same RGB channel value grouping number ensures that the division and processing mode of the color channel values in each fusion round is consistent, thereby maintaining the stability and continuity of the color information processing in the fusion process as a whole.

[0120] Optionally, the RGB channel value grouping number is 4, and the longitudinal back indentation bit number is 1.

[0121] For example, the RGB channel value grouping number is 4, which means that every four RGB values in the horizontal direction form a group, and the longitudinal back indentation bit number is 1, which means that the RGB values in each row are indented by one bit in the vertical direction. By grouping the RGB values in the left and right eye display images in the horizontal direction every four values, indenting the RGB values in each row by one bit in the vertical direction, and sequentially taking values, the left and right eye images are fused N times, and the pixel of the N times fused image is taken from the first row and indented by one bit each time. N is the total number of fusions corresponding to the number of screen partitions.

[0122] S250, according to the mapping relationship between the pre-established area and the fusion round, the left and right eye real-time position data are obtained, and the fusion round corresponding to each area in the display screen is obtained.

[0123] S260, using the fusion image under the fusion round corresponding to each area, a target fusion image is spliced.

[0124] The technical scheme of the embodiment of the application can more intuitively understand the difference and distribution of the left and right eye images in the color channel by expanding each pixel point of the left and right eye display images to obtain the RGB channel arrangement of the left and right eye, so that the fusion image can better integrate the information of the left and right eye. By fusing the left and right eye display images, the stereoscopic and depth perception of the image can be enhanced, so that the user can see a stereoscopic three-dimensional image.

[0125] Embodiment three

[0126] FIG. 11 is a structural schematic diagram of a head-up display device for predicting a left and right eye fusion image according to an embodiment of the application. As shown in FIG. 11, the device comprises: a left and right eye position prediction module 310, configured to determine left and right eye real-time prediction position data of a user according to historical captured left and right eye historical position data of the user;

[0127] The image fusion module 320 is configured to acquire left-eye and right-eye display images to be fused and display, and acquire pixel points at different positions from the left-eye and right-eye display images respectively to perform multiple fusions to obtain a plurality of fusion images.

[0128] The target fusion image determination and display module 330 is configured to acquire a target fusion image according to the left-eye and right-eye real-time predicted position data and the plurality of fusion images, and project the target fusion image to a display screen to perform head-up display.

[0129] Optionally, the left-eye and right-eye position prediction module 310 comprises: a left-eye and right-eye historical position acquisition unit configured to acquire N left-eye and right-eye positions of a user captured by a driver monitoring system, wherein N is an integer greater than 2; a left-eye and right-eye historical position convergence value calculation unit configured to calculate a left-eye and right-eye historical position convergence value according to the first N-1 left-eye and right-eye positions in a sequence from far to near; and a left-eye and right-eye position prediction unit configured to determine left-eye and right-eye real-time predicted position data of the user according to a data difference between the left-eye and right-eye historical position convergence value and the Nth left-eye and right-eye historical position.

[0130] Optionally, the left-eye and right-eye position prediction unit is configured to: if the data difference between the left-eye and right-eye historical position convergence value and the Nth left-eye and right-eye historical position value is greater than a difference threshold value, acquire a target prediction variable corresponding to the left-eye and right-eye historical position convergence value based on a variable configuration list; and determine an addition value between the target prediction variable and the Nth left-eye and right-eye historical position value as the left-eye and right-eye real-time predicted position data; and if the data difference between the left-eye and right-eye historical position convergence value and the Nth left-eye and right-eye historical position value is less than or equal to a preset difference threshold value, determine the Nth left-eye and right-eye historical position value as the left-eye and right-eye real-time predicted position data.

[0131] Optionally, the device further comprises a display screen division module configured to determine a fusion number of the fusion images and divide the display screen into a plurality of regions according to the fusion number before acquiring pixel points at different positions from the left-eye and right-eye corrected images to perform multiple fusions to obtain the plurality of fusion images, wherein the number of regions matches the fusion number.

[0132] Optionally, the image fusion module 320 comprises: an RBG arrangement diagram generation unit configured to expand each pixel point of the left-eye and right-eye display images into RGB channel values respectively to obtain a left-eye RGB channel arrangement diagram and a right-eye RGB channel arrangement diagram; and a fusion image generation unit configured to determine fusion images in each fusion round corresponding to the fusion number according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram.

[0133] Optionally, the fusion image generating unit comprises: a full-eye RGB arrangement map generating subunit, configured to generate a to-be-filled full-eye RGB channel arrangement map corresponding to the fusion image; and an image fusion filling subunit, configured to determine a fusion strategy corresponding to each fusion round, and fill RGB channel values from the left-eye RGB channel arrangement map and the right-eye RGB channel arrangement map into matched image positions in the to-be-filled full-eye RGB channel arrangement map according to the fusion strategy, to obtain the fusion image under each fusion round.

[0134] Optionally, the image fusion filling subunit is configured to: acquire a current fusion strategy corresponding to a current fusion round, and determine a first row initial filling mode corresponding to the to-be-filled full-eye RGB channel arrangement map according to a number of RGB channel value groups in the current fusion strategy; perform back indentation adjustment on the first row initial filling mode according to a number of back indentation positions corresponding to the current fusion round, to obtain a first row target filling mode; and determine each non-first row filling mode in sequence from the second row according to the first row target filling mode and the number of back indentation positions; and fill the RGB channel values from the left-eye RGB channel arrangement map and the right-eye RGB channel arrangement map into the matched image positions in the to-be-filled full-eye RGB channel arrangement map according to the first row target filling mode and each non-first row filling mode, to obtain the fusion image under the current fusion round.

[0135] Optionally, the target fusion image determining and displaying module 330 is configured to: acquire the fusion round corresponding to each region in the display screen according to a mapping relationship between the regions and the fusion rounds in the pre-established region and fusion round mapping relationship according to the real-time left-eye and right-eye position data; and splice the fusion images under the fusion rounds corresponding to each region to obtain the target fusion image.

[0136] Optionally, the device further comprises a distortion correction module, configured to: before performing multiple fusions on the pixel points at different positions acquired from the left-eye and right-eye display images, determine a target partition according to the partition into which the real-time predicted left-eye position and the real-time predicted right-eye position fall in the virtual image coordinate space; and perform distortion correction on the left-eye and right-eye display images using the distortion correction parameter corresponding to the target partition, to obtain the corrected left-eye and right-eye display images; wherein the distortion correction parameter corresponding to each observation position is determined by pre-displaying a standard dot matrix image on the display screen, obtaining a distortion display image observed by a human eye at each observation position, and comparing the image difference between the distortion display image and the standard dot matrix image.

[0137] The technical scheme of the embodiment of the present application can predict the current left and right eye positions of the user by analyzing the historical left and right eye positions of the user, and can reduce image distortion and ghosting phenomenon by performing image fusion display in advance in a prediction manner, thereby avoiding imaging crosstalk problems. By performing multiple fusion on the pixel points at different positions obtained from the left and right eye display images, the details and stereoscopic effect of the image can be enhanced, and more realistic and vivid visual effects can be provided. By projecting the target fusion image into the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0138] The head-up display device for left and right eye prediction fusion images provided in the embodiment of the present application can perform the head-up display method for left and right eye prediction fusion images provided in any embodiment of the present application, and has the corresponding function modules and effects of the execution method.

[0139] Embodiment four

[0140] FIG. 12 shows a structural schematic diagram of an electronic device 10 that can be used to implement the embodiments of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0141] As shown in FIG. 12, the electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which are communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0142] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0143] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as a head-up display method of a left-eye and right-eye predictive fusion image.

[0144] In some embodiments, a head-up display method of a left-eye and right-eye predictive fusion image can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the head-up display method of a left-eye and right-eye predictive fusion image described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform a head-up display method of a left-eye and right-eye predictive fusion image by any other appropriate means, such as by means of firmware.

[0145] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0146] Computer programs used to practice the methods of the application can be written in any combination of one or more programming languages. These computer programs can be implemented on general-purpose computers, special purpose computers, or other programmable data processing apparatus to produce the functions / acts specified in the flow diagrams and / or block diagrams. Computer programs can be applied to a data changed on the functioning of the computer or processing apparatus by transforming the programming language into a machine language.

[0147] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store computer programs for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0148] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0149] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0150] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0151] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the present disclosure can be achieved, and the present disclosure is not limited herein.

Claims

1. A head-up display method of left and right eye predicted fusion images, comprising: determining left and right eye real-time predicted position data of a user according to historical captured left and right eye historical position data of the user; obtaining left and right eye display images to be fused and displayed, and obtaining pixel points at different positions from the left and right eye display images respectively to perform multiple fusions to obtain a plurality of fusion images; obtaining a target fusion image according to the left and right eye real-time predicted position data and the plurality of fusion images, and projecting the target fusion image to a display screen for head-up display.

2. The method of claim 1, wherein, determining left and right eye real-time predicted position data of a user according to historical captured left and right eye historical position data of the user, comprising: obtaining N left and right eye positions of the user captured by a driver monitoring system, wherein N is an integer greater than 2; obtaining the left and right eye historical position convergence values by calculating the first N-1 left and right eye positions in order from far to near according to the capture time; determining the left and right eye real-time predicted position data of the user according to the data difference between the left and right eye historical position convergence values and the Nth left and right eye historical position.

3. The method of claim 2, wherein, determining the left and right eye real-time predicted position data of the user according to the data difference between the left and right eye historical position convergence values and the Nth left and right eye historical position, comprising: in response to the data difference between the left and right eye historical position convergence values and the Nth left and right eye historical position value being greater than a difference threshold value, obtaining a target prediction variable corresponding to the left and right eye historical position convergence values based on a variable configuration list; determining the sum value between the target prediction variable and the Nth left and right eye historical position value as the left and right eye real-time predicted position data; in response to the data difference between the left and right eye historical position convergence values and the Nth left and right eye historical position value being less than or equal to a preset difference threshold value, determining the Nth left and right eye historical position value as the left and right eye real-time predicted position data.

4. The method of claim 1, wherein, Before obtaining a plurality of fusion images by obtaining pixel points at different positions from the left and right eye display images respectively to perform multiple fusions, further comprising: determining the number of fusions of the fusion images, and dividing the display screen into a plurality of regions according to the number of fusions; wherein the number of regions matches the number of fusions.

5. The method of claim 4, wherein, obtaining a plurality of fusion images by obtaining pixel points at different positions from the left and right eye display images respectively to perform multiple fusions, comprising: expanding each pixel point of the left and right eye display images into RGB channel values respectively to obtain left eye RGB channel arrangement and right eye RGB channel arrangement; determining the fusion images under each fusion round corresponding to the number of fusions according to the left eye RGB channel arrangement and the right eye RGB channel arrangement.

6. The method of claim 5, wherein, determining the fusion images under each fusion round corresponding to the number of fusions according to the left eye RGB channel arrangement and the right eye RGB channel arrangement, comprising: generating a to-be-filled full-eye RGB channel arrangement corresponding to the fusion image; determining a fusion strategy corresponding to each fusion round, and selecting RGB channel values from the left eye RGB channel arrangement and the right eye RGB channel arrangement respectively according to the fusion strategy to fill into the matching image positions of the to-be-filled full-eye RGB channel arrangement to obtain the fusion images under each fusion round.

7. The method of claim 6, wherein, The fusion strategy includes: RGB channel value grouping quantity, first row post-indentation bit number and vertical post-indentation bit number.

8. The method of claim 7, wherein, According to the fusion strategy, RGB channel values are selected from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram and filled into matched image positions in the full-eye RGB channel arrangement diagram to be filled to obtain a fusion image under each fusion round, including: A current fusion strategy corresponding to a current fusion round is acquired, and a first row initial filling mode corresponding to the full-eye RGB channel arrangement diagram to be filled is determined according to the RGB channel value grouping quantity in the current fusion strategy; According to the first row post-indentation bit number corresponding to the current fusion round, the first row initial filling mode is adjusted by post-indentation to obtain a first row target filling mode; According to the first row target filling mode and the vertical post-indentation bit number, each non-first row filling mode is determined in turn from the second row; According to the first row target filling mode and each non-first row filling mode, RGB channel values are selected from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram and filled into matched image positions in the full-eye RGB channel arrangement diagram to be filled to obtain a fusion image under the current fusion round.

9. The method of claim 7, wherein, In each fusion strategy under different fusion rounds, the RGB channel value grouping quantity and the vertical post-indentation bit number are the same; In adjacent fusion rounds, the first row post-indentation bit number of the next fusion round is 1 more than the first row post-indentation bit number of the previous fusion round.

10. The method of claim 9, wherein, The RGB channel value grouping quantity is 4, and the vertical post-indentation bit number is 1.

11. The method of claim 4, wherein, According to the left and right eye real-time prediction position data and the plurality of fusion images, a target fusion image is acquired, including: According to the mapping relationship between the regions and the fusion rounds in the left and right eye real-time prediction position data, a fusion round corresponding to each region in the display screen is acquired; Using the fusion image under the fusion round corresponding to each region, the target fusion image is spliced to obtain.

12. The method of claim 1, before acquiring pixels at different positions from the left and right eye display images for multiple times of fusion, further comprising: determining a target partition according to the partition in which the real-time predicted left eye position and the real-time predicted right eye position fall in the virtual image coordinate space; using distortion correction parameters corresponding to the target partition to perform distortion correction on the left and right eye display images to obtain corrected left and right eye display images; wherein the distortion correction parameters corresponding to each observation position are determined by pre-displaying a standard dot array image on the display screen, simulating the display image observed by the human eye at different observation positions to obtain a distortion display image corresponding to each observation position, and comparing the image difference between the distortion display image and the standard dot array image.

13. A head-up display device for predicting left and right eye fusion images, comprising: a left and right eye position prediction module configured to determine left and right eye real-time prediction position data of a user according to historical left and right eye historical position data of the user. The image fusion module is configured to acquire left and right eye display images to be fused and display, acquire pixel points at different positions from the left and right eye display images respectively for multiple times of fusion, and obtain multiple fusion images. The target fusion image determination and display module is configured to acquire a target fusion image according to the left and right eye real-time prediction position data and the multiple fusion images, and project the target fusion image to a display screen for head-up display.

14. A vehicle comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

15. A computer storage medium storing computer instructions for causing a processor to perform the method of any one of claims 1-12 when executed.

Images