Head-up display method and system for fused left- and right-eye image

Abstract

A head-up display method and system for a fused left- and right-eye image. The display method comprises: capturing real-time position data of left and right eyes of a user (S110); acquiring left- and right-eye display images to be fused for display, respectively acquiring pixel points at different positions from the left- and right-eye display images and performing multiple fusion operations to obtain a plurality of fused images (S120); and acquiring a target fused image on the basis of the real-time position data of the left and right eyes and the plurality of fused images, and projecting the target fused image onto a display screen for head-up display (S130).

Description

Head-up display method and system for left and right eye fused images

[0001] This application claims priority to Chinese Patent Application No. 202411023396.1, filed with the Chinese Patent Office on July 29, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, for example to a head-up display method and system for fusing left and right eye images. Background Technology

[0003] In traditional HUD displays, images are typically flat, lacking a sense of depth and 3D. However, glasses-free 3D HUD technology utilizes the parallax principle of the human eye, allowing the left and right eyes to see different images, thus creating a sense of depth.

[0004] As the automotive industry continues to develop, drivers' demands for information display are also increasing. Traditional 2D HUD displays can no longer meet drivers' needs for more intuitive and vivid information display. Naked-eye 3D HUD display technology can present important driving information to drivers in a more realistic way, improving driving safety and comfort.

[0005] The methods in the related technologies do not take into account the driver's real-time eye position for dynamic image fusion calculation, and directly fuse all images for display. This results in a large amount of pixel calculation and can easily cause the main thread to lag. Summary of the Invention

[0006] This application provides a head-up display method and system for left-right eye fusion images to adapt to the user's eye position and achieve more personalized and accurate image display.

[0007] According to one aspect of this application, a head-up display method for left-right eye fusion images is provided, the method comprising: capturing real-time position data of the user's left and right eyes;

[0008] Obtain the left and right eye display images to be fused and displayed. Extract pixels from different positions in the left and right eye display images and fuse them multiple times to obtain multiple fused images.

[0009] Based on real-time position data of the left and right eyes and multiple fused images, a target fused image is obtained and projected onto the display screen for head-up display.

[0010] Optionally, before obtaining multiple fused images by fusing pixels from different positions in the left and right eye display images, the method further includes: determining the number of times the fused images are fused, and dividing the display screen into multiple regions based on the number of fusions; wherein the number of regions matches the number of fusions.

[0011] The advantage of this setup is that by determining the number of blending iterations, the degree and effect of the blending can be controlled. Different blending iterations can produce different visual effects, meeting the needs of different users or adapting to different application scenarios.

[0012] Optionally, pixels at different positions are obtained from the left and right eye display images and fused multiple times to obtain multiple fused images, including: expanding each pixel of the left and right eye display images into RGB channel values ​​to obtain the left eye RGB channel layout map and the right eye RGB channel layout map; and determining the fused image under each fusion round corresponding to the number of fusions based on the left eye RGB channel layout map and the right eye RGB channel layout map.

[0013] The advantage of this setup is that by expanding each pixel of the left and right eye images into RGB channel values, the color information of the image can be analyzed and processed in more detail, and color details can be better preserved and integrated during the fusion process. Blending the left and right eye images enhances the stereoscopic and depth perception of the image, allowing users to see a three-dimensional image.

[0014] Optionally, based on the left-eye RGB channel layout map and the right-eye RGB channel layout map, determine the fused image for each fusion round corresponding to the number of fusions, including: generating a full-eye RGB channel layout map to be filled corresponding to the fused image; determining the fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel layout map and the right-eye RGB channel layout map to fill the matching image positions in the full-eye RGB channel layout map to be filled, thereby obtaining the fused image for each fusion round.

[0015] The advantage of this setup is that different fusion rounds and fusion strategies can adapt to different eye positions of the user, allowing the user's left and right eyes to achieve the best observation effect in any line of sight.

[0016] Optional fusion strategies include: the number of RGB channel value groups, the number of indents after the first line, and the number of vertical indents.

[0017] The advantage of this setup is that different fusion strategies can achieve different fusion effects, providing users with flexibility and customizability to meet the requirements of image fusion in different situations.

[0018] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image positions in the full-eye RGB channel layout image to be filled, respectively, to obtain the fused image under each fusion round. This includes: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling method of the first row corresponding to the full-eye RGB channel layout image to be filled based on the number of RGB channel value groups in the current fusion strategy; adjusting the initial filling method of the first row according to the number of indentations after the first row corresponding to the current fusion round to obtain the target filling method of the first row; determining each non-first row filling method sequentially starting from the second row based on the target filling method of the first row and the number of vertical indentations; and selecting RGB channel values ​​from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image positions in the full-eye RGB channel layout image to be filled, based on the target filling method of the first row and each non-first row filling method, to obtain the fused image under the current fusion round.

[0019] The advantage of this setup is that by adjusting the initial fill method of the first row through post-indentation, an indentation effect can be achieved between the first row and other rows, thereby creating a sense of depth and three-dimensionality in the merged image. Following the fusion strategy ensures consistency in the fill method of each row, guaranteeing the quality of the image fusion.

[0020] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation bits are the same; in adjacent fusion rounds, the number of first-line indentation bits in the later fusion round is 1 more than the number of first-line indentation bits in the previous fusion round.

[0021] The advantage of this setup is that by maintaining the same number of RGB channel value groups and vertical indentation bits across different fusion rounds, the stability of the fusion effect can be improved. Gradually increasing the first-line indentation bits can add a sense of depth to the fused image, enhancing its visual appeal.

[0022] Optionally, the number of RGB channel value groups is 4, and the vertical indentation is 1 bit.

[0023] The advantage of this setting is that by setting the specific number of RGB channel value groups and the vertical indentation, the fusion process can be controlled more precisely, ensuring the accuracy of image fusion.

[0024] Optionally, based on real-time position data of the left and right eyes and multiple fused images, a target fused image is obtained, and the target fused image is projected onto a display screen for head-up display. This includes: based on the real-time position data of the left and right eyes in a pre-established mapping relationship between regions and fusion rounds, obtaining the fusion round corresponding to each region on the display screen; and using the fused images under the fusion rounds corresponding to each region to stitch together the target fused image.

[0025] The advantage of this setup is that it can dynamically determine the fusion round corresponding to each region based on the real-time position data of the user's left and right eyes, ensuring that the stitched target fused image matches the user's perspective and provides the user with a better visual experience.

[0026] According to another aspect of this application, a head-up display system for left and right eye fusion images is provided, the system comprising: an image capture device, a control unit, and an image projection device;

[0027] The image capture device is configured to capture real-time position data of the user's left and right eyes and transmit the real-time position data of the left and right eyes to the control unit.

[0028] The control unit is configured to acquire pixels at different positions from the acquired left and right eye display images, perform multiple fusions to obtain multiple fused images, and acquire the target fused image based on the real-time position data of the left and right eyes and the multiple fused images.

[0029] An image projection device is configured to project a fused image of a target onto a display screen for head-up display.

[0030] Optionally, the image projection device includes an optical grating device and a mirror device; the optical grating device is configured to split the target fused image to generate a split image, so as to realize three-dimensional imaging; the mirror device is configured to project the split image onto the display screen for head-up display according to a preset angle.

[0031] The advantages of this setup are as follows: By using optical grating devices to split the target fusion image, a three-dimensional imaging effect can be achieved, providing users with a more three-dimensional and realistic visual experience. Projecting the image at a preset angle using mirror devices ensures that the image is accurately projected onto a specific position on the display screen for upward display, improving display accuracy and stability.

[0032] Optionally, the control unit is further configured to: determine the number of times the fused images are fused before obtaining multiple fused images by fusing pixels from different positions in the left and right eye display images, and divide the display screen into multiple regions according to the number of fusions; wherein the number of regions matches the number of fusions.

[0033] Optionally, the control unit is configured to: expand each pixel of the left and right eye display images into RGB channel values ​​respectively, to obtain the left eye RGB channel layout map and the right eye RGB channel layout map; generate the full-eye RGB channel layout map to be filled corresponding to the fused image; determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel layout map and the right eye RGB channel layout map respectively to fill the matching image positions in the full-eye RGB channel layout map to be filled, to obtain the fused image under each fusion round; wherein, the fusion strategy includes the number of RGB channel value groups, the number of first-row indentation bits, and the number of vertical indentation bits. In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation bits are the same; under adjacent fusion rounds, the number of first-row indentation bits of the later fusion round is 1 more than the number of first-row indentation bits of the previous fusion round.

[0034] Optionally, the control unit is configured to: acquire the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full-eye RGB channel layout image to be filled based on the number of RGB channel value groups in the current fusion strategy; adjust the initial filling method of the first row according to the number of indentations of the first row corresponding to the current fusion round to obtain the target filling method of the first row; determine each non-first row filling method sequentially starting from the second row based on the target filling method of the first row and the number of vertical indentations; select RGB channel values ​​from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image position in the full-eye RGB channel layout image to be filled based on the target filling method of the first row and each non-first row filling method to obtain the fused image under the current fusion round.

[0035] Optionally, the control unit is configured to: obtain the fusion round corresponding to each region on the display screen based on the real-time position data of the left and right eyes in the pre-established mapping relationship between regions and fusion rounds; and stitch together the fusion images corresponding to the fusion rounds of each region to obtain the target fusion image.

[0036] The technical solution of this application, by capturing real-time position data of the user's left and right eyes, can achieve real-time positioning of the user's eye positions, thereby providing the user with a more accurate display experience. By obtaining pixels from different positions in the left and right eye display images and fusing them multiple times, the pixel computation is small, reducing the likelihood of main thread lag, enhancing image detail and three-dimensionality, and providing a more realistic and vivid visual effect. By projecting the fused image onto the display screen for head-up display, users can obtain important information without looking down at the instrument panel, improving driving safety.

[0037] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0038] Figure 1 is a flowchart of a head-up display method for left and right eye fusion images provided according to Embodiment 1 of this application;

[0039] Figure 2 is a schematic diagram of a head-up display imaging process according to Embodiment 1 of this application;

[0040] Figure 3 is a schematic diagram of a naked-eye 3D imaging principle according to Embodiment 1 of this application;

[0041] Figure 4 is a schematic diagram of a fused image head-up display process according to Embodiment 1 of this application;

[0042] Figure 5 is a flowchart of another head-up display method for left and right eye fusion images provided according to Embodiment 2 of this application;

[0043] Figure 6 is a schematic diagram of a head-up display system for left and right eye fusion images provided according to Embodiment 3 of this application;

[0044] Figure 7 is a schematic diagram of another head-up display system for left and right eye fusion images provided according to Embodiment 3 of this application. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0046] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] Example 1

[0048] Figure 1 is a flowchart of a head-up display method for left-right eye fusion images provided in Embodiment 1 of this application. This embodiment is applicable to situations where real-time head-up display is performed based on the user's eye position. This method can be executed by a head-up display system for left-right eye fusion images, which can be configured in a vehicle. As shown in Figure 1, the method includes:

[0049] S110: Captures real-time position data of the user's left and right eyes.

[0050] Among them, the real-time position data of the left and right eyes refers to the specific position information of the user's left and right eyes in real time, including the real-time position of the left eye and the real-time position of the right eye, which is represented in coordinate form.

[0051] For example, real-time position data of the left and right eyes can be captured by a driver monitoring system, which includes sensors and cameras capable of accurately capturing the position of the user's eyes. The sensors and cameras can monitor subtle movements and changes in the user's eyes in real time, thereby determining the position of the left and right eyes.

[0052] In one embodiment, the driver monitoring system can determine the real-time position of the user's left and right eyes using either infrared tracking technology or image recognition technology. Infrared tracking technology involves emitting infrared light towards the user's eyes and then using sensors to receive the reflected light, calculating the real-time position of the left and right eyes based on changes in light intensity and time differences. Image recognition technology involves using a camera to capture images of the user's eyes and using image processing algorithms to identify key features of the eyes, such as the center of the pupil and the corners of the eyes, thereby determining the historical position values ​​of the left and right eyes.

[0053] In one specific implementation, the driver monitoring system may include a DMS (Eye-tracking System), which is a device that uses optical and infrared cameras installed in the vehicle to acquire real-time information about the driver's eye state. It analyzes the acquired information using deep learning algorithms to determine the driver's state, enabling driver identification, driver fatigue monitoring, driver attention monitoring, and monitoring of dangerous driving behaviors, and providing different levels of warnings.

[0054] S120. Obtain the left and right eye display images to be fused and displayed. Obtain pixels at different positions from the left and right eye display images and fuse them multiple times to obtain multiple fused images.

[0055] In this context, a pixel refers to the smallest unit of an image. A fused image is a new image obtained by processing the left and right eye images according to fusion rules.

[0056] For example, the processor can acquire the left and right eye display images that need to be fused for display. These images can be pre-prepared or generated based on real-time application scenarios and user needs; the images include, but are not limited to, image files such as pictures, videos, and video streams. The processor will select corresponding pixels from different positions in the left and right eye display images according to pre-defined fusion rules and fuse them sequentially to obtain multiple fused images.

[0057] Optionally, pixels at different positions are obtained from the left and right eye display images and fused multiple times to obtain multiple fused images. This includes calling the shader plugin in the 3D image processing engine to obtain pixels at different positions from the left and right eye display images and fused multiple times to obtain multiple fused images.

[0058] For example, the processor invokes the shader plugin within the 3D image processing engine. The shader plugin is a component of the 3D image processing engine, possessing 3D image processing capabilities. The processor can invoke the shader plugin and, according to preset fusion rules, extract pixels at different locations from the left and right eye display images, performing multiple fusions. In each fusion process, the shader plugin combines the selected pixels according to the preset fusion rules, ultimately generating multiple fused images.

[0059] In summary, multiple fusions through shader plugins can generate multiple fused images, thereby better meeting different application needs and visual effect requirements.

[0060] S130. Based on the real-time position data of the left and right eyes and multiple fused images, obtain the target fused image and project the target fused image onto the display screen for head-up display.

[0061] Figure 2 is a schematic diagram of a head-up display imaging process provided in Embodiment 1 of this application. A head-up display (HUD) is a driver assistance instrument used in vehicles, and is a comprehensive electronic display device composed of electronic components, display components, controllers, etc. The HUD can project vehicle speed, navigation information, warning information, etc., in the form of images and characters onto the driver's front through optical components. The HUD allows images or information to be directly projected into the user's field of vision, enabling the user to see relevant content without looking down.

[0062] Figure 3 is a schematic diagram of the naked-eye 3D imaging principle provided in Embodiment 1 of this application. Naked-eye 3D display technology refers to a 3D display technology that allows users to directly view three-dimensional images with the naked eye without wearing special 3D glasses, presenting a 3D effect. The front-end display part of the naked-eye 3D display technology needs to calculate through pixel imaging units so that the left and right eyes see different images, which are then merged into a 3D image in the brain. In Figure 3, through optical design of the HUD, the user's left eye sees image P1 through the imaging structure, such as the windshield, while the right eye sees image P2. Due to the binocular parallax formed by images P1 and P2, the objects seen by the user have a sense of depth and space, and the image seen by the right eye is synthesized into a stereoscopic image with a sense of depth in the user's brain. Naked-eye 3D imaging can adjust the binocular parallax by changing the position between two images, thus changing the distance of the virtual image perceived by the user (the actual distance of the virtual image remains unchanged). The closer the two images are, the closer the virtual image perceived by the user; conversely, the farther the two images are, the farther the virtual image perceived by the user.

[0063] Optionally, based on real-time position data of the left and right eyes and multiple fused images, a target fused image is obtained, and the target fused image is projected onto a display screen for head-up display. This includes: based on the real-time position data of the left and right eyes in a pre-established mapping relationship between regions and fusion rounds, obtaining the fusion round corresponding to each region on the display screen; and using the fused images under the fusion rounds corresponding to each region to stitch together the target fused image.

[0064] For example, each location data point has a corresponding preset fusion rule. The fusion rule includes the selection of which fused image to display for each region. The processor can determine the fusion round corresponding to each region on the display screen based on the real-time position data of the left and right eyes and the pre-established mapping relationship between regions and fusion rounds. Then, the fused images corresponding to the fusion rounds for each region are stitched together to obtain the target fused image. It should be noted that the stitching process needs to consider pixel alignment and matching to ensure that the stitched target fused image is visually continuous and consistent.

[0065] Specific application scenario: Figure 4 is a schematic diagram of a fused image head-up display process provided in this application. As shown in Figure 4, the position information of the left and right eyes is acquired in real time through the DMS device, and the eye position data is transmitted to the 3D image processing engine. The 3D image processing engine can switch the image fusion rules and display content in real time based on the monitored eye position, thereby achieving a naked-eye 3D effect in AR HUD display. By combining the eye box position partitioning scheme of 3DHUD and the 3D image processing engine technology to calculate the left and right fused images in real time, the target fusion object is determined in the target screen partition for dynamic head-up display to the user.

[0066] The technical solution of this application, by capturing real-time position data of the user's left and right eyes, can achieve real-time positioning of the user's eye positions, thereby providing the user with a more accurate display experience. By obtaining pixels from different positions in the left and right eye display images and fusing them multiple times, the pixel computation is small, reducing the likelihood of main thread lag, enhancing image detail and three-dimensionality, and providing a more realistic and vivid visual effect. By projecting the fused image onto the display screen for head-up display, users can obtain important information without looking down at the instrument panel, improving driving safety.

[0067] Example 2

[0068] Figure 5 is a flowchart of a head-up display method for left and right eye prediction fusion images provided in Embodiment 2 of this application. This embodiment adds a specific process to Embodiment 1 above, where pixels at different positions are obtained from the left and right eye corrected images and fused multiple times to obtain multiple fused images. The specific content of step S210 is largely the same as step S110 in Embodiment 1, therefore it will not be described again in this embodiment. As shown in Figure 5, the method includes:

[0069] S210: Captures real-time position data of the user's left and right eyes.

[0070] S220. Determine the number of times the image is fused, and divide the display screen into multiple regions based on the number of fusions; wherein the number of regions matches the number of fusions.

[0071] S230. Expand each pixel of the left and right eye display images into RGB channel values ​​to obtain the left eye RGB channel layout diagram and the right eye RGB channel layout diagram.

[0072] For example, the processor will expand each pixel in the left-eye display image and the right-eye display image into RGB channel values ​​respectively to describe the color composition of each pixel.

[0073] S240. Based on the RGB channel layout diagram of the left eye and the RGB channel layout diagram of the right eye, determine the fused image under each fusion round corresponding to the number of fusions.

[0074] For example, the processor performs each fusion round sequentially based on the RGB channel layout of the left and right eyes, according to a pre-set number of fusion cycles. In each fusion round, the RGB channel values ​​of the left and right eyes can be extracted and combined according to the set fusion strategy.

[0075] Optionally, based on the left-eye RGB channel layout map and the right-eye RGB channel layout map, determine the fused image for each fusion round corresponding to the number of fusions, including: generating a full-eye RGB channel layout map to be filled corresponding to the fused image; determining the fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel layout map and the right-eye RGB channel layout map to fill the matching image positions in the full-eye RGB channel layout map to be filled, thereby obtaining the fused image for each fusion round.

[0076] The image to be filled with the full eye RGB channel layout is a blank image with the same size and pixel layout as the fused image, used to store the RGB channel values ​​selected during the fusion process.

[0077] For example, the processor can, according to the fusion strategy, select the corresponding RGB channel values ​​from the left-eye RGB channel layout map and the right-eye RGB channel layout map in each round of fusion, and fill them into the corresponding positions in the full-eye RGB channel layout map to be filled. It should be noted that the fused image obtained in each fusion is a complete full-eye RGB channel layout map, and the processor can perform different degrees of fusion in each fusion round according to the fusion strategy.

[0078] Optional fusion strategies include: the number of RGB channel value groups, the number of indents after the first line, and the number of vertical indents.

[0079] The values ​​involved in the fusion strategy need to be adapted to the arrangement of the beam splitter. The number of RGB channel value groups is related to the number of pixels covered by the grating unit. For example, when one grating unit can cover 8 pixels (i.e., the backlight LEDs), the number of RGB channel value groups needs to be set to 4. In this case, out of every 8 LEDs, the 4 LEDs on the left and the 4 LEDs on the right will shine in different directions, thus being seen by the left and right eyes respectively.

[0080] For example, in each round, the different indentation bits of different rows are also to adapt to various different beam splitter grating arrangements. By using the tilted grating arrangement, crosstalk can be effectively reduced.

[0081] It's important to note that the indentation in different rounds is primarily for traversing various states. Different states correspond to the display of pixels (LED beads) covered by the raster. It's crucial to understand that there's an upper limit to the number of states. For example, in a group of four, if each round indents by one position, the maximum number of states is reached after eight rounds. In this case, the ninth round would be identical to the first. Therefore, the upper limit of the number of rounds is essentially the upper limit of the number of states. When selecting the number of fusion rounds for partitioning, it's essentially choosing the corresponding states for the fusion operation. The number of fusion rounds needs to be selected based on specific needs and conditions to achieve the best fusion effect and the desired goal.

[0082] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image positions in the full-eye RGB channel layout image to be filled, respectively, to obtain the fused image under each fusion round. This includes: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling method of the first row corresponding to the full-eye RGB channel layout image to be filled based on the number of RGB channel value groups in the current fusion strategy; adjusting the initial filling method of the first row according to the number of indentations after the first row corresponding to the current fusion round to obtain the target filling method of the first row; determining each non-first row filling method sequentially starting from the second row based on the target filling method of the first row and the number of vertical indentations; and selecting RGB channel values ​​from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image positions in the full-eye RGB channel layout image to be filled, based on the target filling method of the first row and each non-first row filling method, to obtain the fused image under the current fusion round.

[0083] For example, based on the number of RGB channel value groups, it can be determined which channel values ​​to select from the left and right eye RGB channel layout maps to fill the first row of pixels in the full-eye RGB channel layout map to be filled. Then, based on the indentation number of the first row in the current fusion round, the fill position of the first row is shifted to the right by a certain number of positions to adjust the initial fill method of the first row. Starting from the second row, the processor can determine the fill method for each non-first row sequentially based on the target fill method of the first row and the vertical indentation number, where the vertical indentation number determines the vertical offset of the fill position of each row relative to the previous row.

[0084] In one embodiment, based on the first-row target filling method and each non-first-row filling method, the corresponding RGB channel values ​​are sequentially selected from the left and right eye RGB channel layout maps and filled into the corresponding pixels in the full-eye RGB channel layout map to be filled. This completes the image filling fusion for the current fusion round, resulting in the fused image for that round. By repeating the above steps for multiple fusion rounds, the processor can ultimately obtain the fused images corresponding to each fusion round.

[0085] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation bits are the same; in adjacent fusion rounds, the number of first-line indentation bits in the later fusion round is 1 more than the number of first-line indentation bits in the previous fusion round.

[0086] In different fusion rounds, the same number of RGB channel value groups and vertical indentation means that the number of RGB channel values ​​fused in each fusion round and the vertical spacing between rows remain constant. In adjacent fusion rounds, the indentation of the first line of the later fusion round being one more than that of the previous fusion round means that as the fusion round increases, the rightward indentation of the first line increases by one.

[0087] For example, having the same number of RGB channel value groups ensures consistency in the division and processing of color channel values ​​in each fusion round, thereby maintaining the stability and continuity of color information processing throughout the fusion process.

[0088] Optionally, the number of RGB channel value groups is 4, and the vertical indentation is 1 bit.

[0089] For example, a grouping of 4 RGB channel values ​​means that every four RGB values ​​are grouped horizontally, and a vertical indentation of 1 means that each row of RGB values ​​is indented one position downwards. By sequentially grouping and taking values ​​from the left and right eye images into groups of four horizontally, and indenting each row of RGB values ​​one position downwards, the left and right eye images are fused N times. Each fusion starts with an indentation of one position from the first row, and the pixels of the fused image are saved after N iterations. N is the total number of fusion iterations corresponding to the number of screen partitions.

[0090] S250: Based on the real-time position data of the left and right eyes, obtain the fusion round corresponding to each region on the display screen in the pre-established mapping relationship between regions and fusion rounds.

[0091] S260. Using the fusion images corresponding to the fusion rounds for each region, stitch together to obtain the target fusion image.

[0092] The technical solution of this application embodiment expands each pixel of the left and right eye display images separately to obtain the RGB channel arrangement diagrams of the left and right eyes. This allows for a more intuitive understanding of the differences and distribution of color channels in the left and right eye images, enabling the fused image to better integrate information from both eyes. By fusing the left and right eye display images, the stereoscopic and depth perception of the image can be enhanced, allowing users to see a three-dimensional image.

[0093] Example 3

[0094] Figure 6 is a schematic diagram of a head-up display system for left and right eye fusion images provided in Embodiment 3 of this application. As shown in Figure 6, the system includes: an image capture device 310, a control unit 320, and an image projection device 330.

[0095] Optionally, an image capture device is used to capture real-time position data of the user's left and right eyes and transmit the real-time position data of the left and right eyes to a control unit; the control unit is used to obtain pixels at different positions from the acquired left and right eye display images and perform multiple fusions to obtain multiple fused images, and obtain a target fused image based on the real-time position data of the left and right eyes and the multiple fused images; and an image projection device is used to project the target fused image onto a display screen for head-up display.

[0096] For example, image capture devices typically employ eye-tracking technology to capture the real-time position data of a user's left and right eyes. For instance, using infrared light reflection, the device emits infrared light of a specific wavelength towards the user's eyes. When the light hits the surface of the eyeball, it is reflected. Sensors and cameras then capture changes in the reflected light to calculate the real-time position and movement trajectory of the user's left and right eyes. When processing the image, the control unit uses an image fusion algorithm to obtain pixels from different locations in the left and right eye display images and fuse them multiple times to obtain multiple fused images. Based on the real-time position data of the left and right eyes and the multiple fused images, a target fused image is obtained. The image projection device operates based on the principle of optical projection. When the projection device is working, it controls the intensity and color of the light source according to the pixel information of the target fused image, projecting the image onto the display screen at an appropriate ratio and focal length to achieve a head-up display effect.

[0097] Figure 7 is a schematic diagram of another head-up display system for left and right eye fusion images provided in Embodiment 3 of this application. In Figure 7, the image projection device includes an optical grating device 331 and a mirror device 332.

[0098] Optionally, the image projection device includes an optical grating device and a mirror device; the optical grating device is used to split the target fused image to generate a split image to achieve three-dimensional imaging; the mirror device is used to project the split image onto the display screen for head-up display according to a preset angle.

[0099] For example, an optical grating device consists of grating stripes. When light from the target fusion image passes through the grating, diffraction occurs. Light of different wavelengths and angles is separated and refracted in a specific way, thus splitting the target fusion image into a split image. By precisely controlling the parameters and structure of the grating, the images seen from different viewing angles can differ, creating a stereoscopic visual effect to achieve 3D imaging. When the split image reaches the mirror device, the mirror accurately reflects and projects the light onto the display screen according to a preset angle and direction. The preset angle needs to take into account factors such as the user's viewing position, the position and angle of the display screen, etc., to ensure that the user can obtain a clear and realistic 3D imaging effect from a comfortable viewing angle.

[0100] Optionally, the control unit is also configured to: determine the number of times the fused images are fused before obtaining multiple fused images by fusing pixels at different positions from the left and right eye display images multiple times; and divide the display screen into multiple regions according to the number of fusions; wherein the number of regions matches the number of fusions.

[0101] Optionally, the control unit is configured to: expand each pixel of the left and right eye display images into RGB channel values ​​respectively, to obtain the left eye RGB channel layout map and the right eye RGB channel layout map; generate a full-eye RGB channel layout map to be filled corresponding to the fused image; determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel layout map and the right eye RGB channel layout map respectively to fill the matching image positions in the full-eye RGB channel layout map to be filled, to obtain the fused image under each fusion round; wherein, the fusion strategy includes the number of RGB channel value groups, the number of first-row indentation bits, and the number of vertical indentation bits, and the number of RGB channel value groups and the number of vertical indentation bits are the same in each fusion strategy under different fusion rounds; under adjacent fusion rounds, the number of first-row indentation bits of the later fusion round is 1 more than the number of first-row indentation bits of the previous fusion round.

[0102] Optionally, the control unit is configured to: acquire the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full-eye RGB channel layout image to be filled based on the number of RGB channel value groups in the current fusion strategy; adjust the initial filling method of the first row according to the number of indentations of the first row corresponding to the current fusion round to obtain the target filling method of the first row; determine each non-first row filling method sequentially starting from the second row according to the target filling method of the first row and the number of vertical indentations; and select RGB channel values ​​from the left-eye RGB channel layout image and the right-eye RGB channel layout image to fill the matching image position in the full-eye RGB channel layout image to be filled, based on the target filling method of the first row and each non-first row filling method, to obtain the fused image under the current fusion round.

[0103] Optionally, the control unit is used to: obtain the fusion round corresponding to each region on the display screen based on the real-time position data of the left and right eyes in the pre-established mapping relationship between regions and fusion rounds; and stitch together the fusion images corresponding to the fusion rounds of each region to obtain the target fusion image.

[0104] The technical solution of this application embodiment captures real-time position data of the user's left and right eyes using an image capture device, enabling real-time positioning of the user's eye positions and thus providing a more accurate display experience. By using a control unit to obtain pixels from different positions in the left and right eye display images and perform multiple fusions, the pixel computation is small, reducing the likelihood of main thread lag, enhancing image detail and three-dimensionality, and providing a more realistic and vivid visual effect. The fused image is then projected onto a display screen for head-up display using an image projection device, allowing users to obtain important information without looking down at the instrument panel, thus improving driving safety.

[0105] The head-up display system for left-right eye fusion images provided in this application embodiment can execute the head-up display method for left-right eye fusion images provided in any embodiment of this application, and has the corresponding functional modules and effects of executing the method.

[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

Claims

1. A head-up display method for left-right eye fused images, comprising: Capture the real-time position data of the user's left and right eyes; Obtain the left and right eye display images to be fused and displayed. Extract pixels from different positions in the left and right eye display images and fuse them multiple times to obtain multiple fused images. Based on the real-time position data of the left and right eyes and the multiple fused images, a target fused image is obtained and projected onto the display screen for head-up display.

2. The method according to claim 1, before obtaining multiple fused images by repeatedly fusing pixels at different positions from the left and right eye display images, further comprising: The number of times the images are merged is determined, and the display screen is divided into multiple regions based on the number of merges. The number of regions corresponds to the number of fusions.

3. The method according to claim 2, wherein, Pixels at different locations are obtained from the left and right eye display images and fused multiple times to obtain multiple fused images, including: Each pixel of the left and right eye display images is expanded into RGB channel values ​​to obtain the left eye RGB channel layout map and the right eye RGB channel layout map; Based on the RGB channel layout of the left eye and the RGB channel layout of the right eye, the fused image under each fusion round corresponding to the number of fusions is determined.

4. The method according to claim 3, wherein, Based on the RGB channel layout of the left and right eyes, the fused images for each fusion round corresponding to the fusion number are determined, including: Generate and merge the full-eye RGB channel layout map to be filled; Determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel layout map and the right eye RGB channel layout map respectively to fill the matching image positions in the full eye RGB channel layout map to be filled, so as to obtain the fused image under each fusion round.

5. The method according to claim 4, wherein, The fusion strategy includes: the number of RGB channel value groups, the number of indentation bits after the first row, and the number of vertical indentation bits.

6. The method according to claim 5, wherein, According to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel layout image and the right-eye RGB channel layout image respectively to fill the matching image positions in the full-eye RGB channel layout image to be filled, resulting in fused images for each fusion round, including: Obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full eye RGB channel layout map to be filled based on the number of RGB channel value groups in the current fusion strategy; Based on the number of indentations after the first line corresponding to the current fusion round, the initial filling method of the first line is adjusted by indentation adjustment to obtain the target filling method of the first line; Based on the first row target filling method and the vertical indentation number, starting from the second row, each non-first row filling method is determined sequentially; Based on the first row target filling method and each non-first row filling method, RGB channel values ​​are selected from the left eye RGB channel layout map and the right eye RGB channel layout map to fill the matching image position in the full eye RGB channel layout map to be filled, thus obtaining the fused image under the current fusion round.

7. The method according to claim 4, wherein, In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation bits are the same; In adjacent merge rounds, the indentation of the first line of the later merge round is 1 more than the indentation of the first line of the previous merge round.

8. The method according to claim 7, wherein, The number of RGB channel value groups is 4, and the number of vertical indentation bits is 1.

9. The method according to claim 2, wherein, Based on the real-time position data of the left and right eyes and the multiple fused images, a target fused image is obtained, and the target fused image is projected onto a display screen for head-up display, including: Based on the real-time position data of the left and right eyes in the pre-established mapping relationship between regions and fusion rounds, obtain the fusion round corresponding to each region in the display screen; The target fused image is obtained by stitching together the fused images from the fusion rounds corresponding to each region.

10. A head-up display system for fusing left and right eye images, comprising: Image capture equipment, control unit, and image projection device; An image capture device is configured to capture real-time position data of the user's left and right eyes and transmit the real-time position data of the left and right eyes to a control unit. The control unit is configured to acquire pixels at different positions from the acquired left and right eye display images, perform multiple fusions to obtain multiple fused images, and acquire a target fused image based on the real-time position data of the left and right eyes and the multiple fused images. An image projection device is configured to project the target fused image onto a display screen for head-up display.

11. The system according to claim 10, wherein, The image projection device includes an optical grating device and a mirror device; The optical grating device is configured to split the target fused image to generate a split image, thereby achieving three-dimensional imaging; The mirror device is configured to project the split image onto a display screen for head-up display according to a preset angle.

12. The system according to claim 10, wherein, The control unit is further configured to: determine the number of times the fusion image is fused before obtaining multiple fused images by fusing pixels from different positions in the left and right eye display images, and divide the display screen into multiple regions according to the number of fusions; The number of regions corresponds to the number of fusions.

13. The system according to claim 12, wherein, The control unit is configured to: expand each pixel of the left and right eye display images into RGB channel values ​​respectively, to obtain the left eye RGB channel layout map and the right eye RGB channel layout map; Generate and merge the full-eye RGB channel layout map to be filled; Determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel layout map and the right eye RGB channel layout map respectively to fill the matching image position in the full eye RGB channel layout map to be filled, so as to obtain the fused image under each fusion round; The fusion strategy includes the number of RGB channel value groups, the number of first-line indents, and the number of vertical indents. In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same. In adjacent fusion rounds, the number of first-line indents in the later fusion round is 1 more than the number of first-line indents in the previous fusion round.

14. The system according to claim 13, wherein, The control unit is configured to: obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full eye RGB channel layout map to be filled based on the number of RGB channel value groups in the current fusion strategy; Based on the number of indentations after the first line corresponding to the current fusion round, the initial filling method of the first line is adjusted by indentation adjustment to obtain the target filling method of the first line; Based on the first row target filling method and the vertical indentation number, starting from the second row, each non-first row filling method is determined sequentially; Based on the first row target filling method and each non-first row filling method, RGB channel values ​​are selected from the left eye RGB channel layout map and the right eye RGB channel layout map to fill the matching image position in the full eye RGB channel layout map to be filled, thus obtaining the fused image under the current fusion round.

15. The system according to claim 14, wherein, The control unit is configured to: obtain the fusion round corresponding to each region in the display screen based on the real-time position data of the left and right eyes in a pre-established mapping relationship between regions and fusion rounds; The target fused image is obtained by stitching together the fused images from the fusion rounds corresponding to each region.

Images