Head-up display method and system for left-eye and right-eye fused image

By capturing the user's left and right eye position data in real time, performing multiple image fusion and area division, the problem of large amount of image fusion calculation in HUD display technology is solved, and more accurate and vivid image display is achieved, improving driving safety.

CN118972540APending Publication Date: 2024-11-15HANGZHOU FERVCLOUD TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411023396.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing HUD display technology fails to effectively process the driver's real-time eye position dynamic image fusion calculation, resulting in large amount of image fusion calculation, which easily leads to lag in the main thread, and cannot meet the driver's needs for more intuitive and vivid information display. .

Method used

By capturing the real-time position data of the user's left and right eyes, obtaining pixel points at different positions for multiple fusions, determining the number of fusions and strategies, dividing the display screen area, generating the target fusion image, and projecting it to the display screen for head-up display, enhancing the image Three-dimensional and depth.

Benefits of technology

Achieve more accurate and personalized image display, reduce pixel calculations, avoid lag in the main thread, improve image details and visual effects, enable users to obtain important information without bowing their heads, and improve driving safety .

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118972540A_ABST
    Figure CN118972540A_ABST
Patent Text Reader

Abstract

The invention discloses a head-up display method and system for a left-eye and right-eye fused image. Comprises: capturing real-time position data of left and right eyes of a user; obtaining left and right eye display images to be fused and displayed, respectively obtaining pixel points at different positions from the left and right eye display images, and carrying out multiple fusion to obtain a plurality of fused images; and obtaining a target fusion image according to the real-time position data of the left eye and the right eye and the plurality of fusion images, and projecting the target fusion image to a display screen for head-up display. Real-time positioning of the positions of the two eyes of the user can be achieved, and therefore more accurate display experience can be better provided for the user. The pixel points at different positions are obtained from the left eye display image and the right eye display image to be fused for multiple times, the pixel calculation amount is small, main thread jamming is not prone to being caused, the details and the stereoscopic impression of the image can be enhanced, and a more vivid visual effect is provided. The head-up display enables the user to obtain important information without lowering the head to look at the instrument, so that the driving safety is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a head-up display method and system for left-eye and right-eye fused images. Background Art

[0002] In traditional HUD displays, images are usually flat and lack a sense of three-dimensionality and depth. However, naked-eye 3D HUD display technology uses the parallax principle of the human eye to allow the left and right eyes to see different images, thus creating a sense of three-dimensionality.

[0003] With the continuous development of the automotive industry, drivers have higher and higher demands for information display. Traditional 2D HUD display can no longer meet drivers' demands for more intuitive and vivid information display. The naked-eye 3D HUD display technology can present important driving information to drivers in a more realistic way, improving driving safety and comfort.

[0004] The existing technology does not consider the real-time eye position of the driver in dynamic image fusion calculation, and directly fuses all images for display. The amount of pixel calculation is large, which can easily cause the main thread to freeze. Summary of the invention

[0005] The present invention provides a head-up display method and system for left-eye and right-eye fused images, so as to adapt to the eye position of the user and realize more personalized and accurate image display.

[0006] According to one aspect of the present invention, a head-up display method for left-eye and right-eye fused images is provided, the method comprising: capturing real-time position data of left and right eyes of a user;

[0007] Obtaining left and right eye display images to be fused and displayed, respectively obtaining pixel points at different positions from the left and right eye display images for multiple fusions to obtain multiple fused images;

[0008] According to the real-time position data of the left and right eyes and multiple fused images, a target fused image is obtained, and the target fused image is projected onto a display screen for head-up display.

[0009] Optionally, before obtaining pixel points at different positions from the left and right eye display images and performing multiple fusions to obtain multiple fused images, the method also includes: determining the number of fusions of the fused images, and dividing the display screen into multiple areas according to the number of fusions; wherein the number of areas matches the number of fusions.

[0010] The advantage of this setting is that by determining the number of fusion times, the degree and effect of fusion can be controlled. Different fusion times can produce different visual effects to meet the needs of different users or adapt to different application scenarios.

[0011] Optionally, pixel points at different positions are respectively obtained from the left and right eye display images and fused multiple times to obtain multiple fused images, including: each pixel point of the left and right eye display images is expanded into RGB channel values ​​respectively to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; according to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, the fused images in each fusion round corresponding to the number of fusion times are determined.

[0012] The advantage of this setting is that by expanding each pixel of the left and right eye display images into RGB channel values, the color information of the image can be analyzed and processed in more detail, and the color details can be better preserved and integrated during the fusion process. By fusing the left and right eye display images, the stereoscopic sense and depth of the image can be enhanced, allowing users to see a stereoscopic three-dimensional image.

[0013] Optionally, according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram, a fused image under each fusion round corresponding to the number of fusion times is determined, including: generating a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determining a fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to fill in the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round.

[0014] The advantage of this setting is that different fusion rounds and fusion strategies can adapt to the different eye positions of the user, so that the user's left and right eyes can achieve the best observation effect at any line of sight position.

[0015] Optionally, the fusion strategy includes: the number of RGB channel value groups, the number of first line indents, and the number of vertical indents.

[0016] The advantage of this setting is that different fusion effects can be achieved by setting different fusion strategies, which provides users with flexibility and customizability to meet the requirements of image fusion in different situations.

[0017] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram respectively to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused images under each fusion round, including: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groupings in the current fusion strategy; adjusting the initial filling mode of the first row according to the number of indentation bits of the first row corresponding to the current fusion round to obtain the target filling mode of the first row; determining each non-first row filling mode in turn starting from the second row according to the target filling mode of the first row and the number of vertical indentation bits; selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0018] The advantage of this setting is that by adjusting the initial filling mode of the first row through the back indentation, the indentation effect between the first row and other rows can be achieved, thereby creating a sense of hierarchy and three-dimensionality in the fused image. Fusion according to the fusion strategy can make the filling mode of each row consistent, ensuring the quality of image fusion.

[0019] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; in adjacent fusion rounds, the number of first line indents in the latter fusion round is 1 more than the number of first line indents in the previous fusion round.

[0020] The advantage of this setting is that by keeping the number of RGB channel value groups and the number of vertical indents the same in different fusion rounds, the stability of the fusion effect can be improved. The gradually increasing number of first line indents can add a certain sense of depth to the fused image and improve the visual effect of the image.

[0021] Optionally, the number of RGB channel value groups is 4 and the number of vertical indents is 1.

[0022] The advantage of this setting is that by setting the specific number of RGB channel value groups and the number of vertical indents, the fusion process can be controlled more finely to ensure the accuracy of image fusion.

[0023] Optionally, a target fused image is obtained based on the real-time position data of the left and right eyes and multiple fused images, and the target fused image is projected onto the display screen for head-up display, including: obtaining the fusion rounds corresponding to each area in the display screen in a mapping relationship between the pre-established areas and fusion rounds based on the real-time position data of the left and right eyes; and using the fusion images under the fusion rounds corresponding to each area to stitch together the target fused image.

[0024] The advantage of this setting is that it can dynamically determine the fusion round corresponding to each area based on the real-time position data of the user's left and right eyes, ensuring that the spliced ​​target fusion image matches the user's perspective, providing the user with a better visual experience.

[0025] According to another aspect of the present invention, there is provided a head-up display system for left-eye and right-eye fused images, the system comprising: an image capturing device, a control unit and an image projection device;

[0026] An image capture device, used to capture the real-time position data of the left and right eyes of the user, and transmit the real-time position data of the left and right eyes to the control unit;

[0027] A control unit, used to obtain pixel points at different positions from the acquired left and right eye display images, respectively, and perform multiple fusions to obtain multiple fused images, and obtain a target fused image based on the real-time position data of the left and right eyes and the multiple fused images;

[0028] The image projection device is used to project the target fused image onto the display screen for head-up display.

[0029] Optionally, the image projection device includes an optical grating device and a mirror device; the optical grating device is used to split the target fused image to generate a split image to achieve three-dimensional imaging; the mirror device is used to project the split image onto a display screen for head-up display according to a preset angle.

[0030] The advantage of this setting is that by splitting the target fusion image through the optical grating device, a three-dimensional imaging effect can be achieved, giving users a more three-dimensional and realistic visual experience. By projecting at a preset angle through the mirror device, it is ensured that the image can be accurately projected to a specific position on the display screen for head-up display, improving the accuracy and stability of the display.

[0031] Optionally, the control unit is also used to: determine the number of fusion times of the fused images before obtaining multiple fused images by acquiring pixel points at different positions from the left and right eye display images respectively and performing multiple fusions, and divide the display screen into multiple areas according to the number of fusions; wherein the number of areas matches the number of fusions.

[0032] Optionally, the control unit is used to: expand each pixel point of the left and right eye display images into RGB channel values ​​respectively, to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; generate a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, to obtain a fused image under each fusion round; wherein the fusion strategy includes the number of RGB channel value groups, the number of first-line indentation digits, and the number of vertical indentation digits, and in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation digits are the same; under adjacent fusion rounds, the number of first-line indentation digits of the latter fusion round is 1 more than the number of first-line indentation digits of the previous fusion round.

[0033] Optionally, the control unit is used to: obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; adjust the initial filling mode of the first row according to the number of first row indentation bits corresponding to the current fusion round to obtain the target filling mode of the first row; determine each non-first row filling mode in sequence starting from the second row according to the first row target filling mode and the number of vertical indentation bits; select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0034] Optionally, the control unit is used to: obtain the fusion round corresponding to each area in the display screen according to the mapping relationship between the real-time position data of the left and right eyes and the fusion round in a pre-established manner; use the fusion images under the fusion round corresponding to each area to stitch together to obtain the target fusion image.

[0035] The technical solution of the embodiment of the present invention can realize the real-time positioning of the user's eyes by capturing the real-time position data of the user's left and right eyes, thereby better providing the user with a more accurate display experience. By obtaining pixel points at different positions from the left and right eye display images and performing multiple fusions, the pixel calculation amount is small, and it is not easy to cause the main thread to freeze. The details and three-dimensional sense of the image can be enhanced, providing a more realistic and vivid visual effect. By projecting the target fused image onto the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0036] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 is a flowchart of a head-up display method for left-eye and right-eye fused images provided according to Embodiment 1 of the present invention;

[0039] Figure 2 is a schematic diagram of a head-up display imaging process provided according to Embodiment 1 of the present invention;

[0040] Figure 3 is a schematic diagram of a naked-eye 3D imaging principle provided according to Embodiment 1 of the present invention;

[0041] Figure 4 is a schematic diagram of a fused image head-up display process provided according to Embodiment 1 of the present invention;

[0042] Figure 5 is a flowchart of another head-up display method for left-eye and right-eye fused images provided according to Embodiment 2 of the present invention;

[0043] Figure 6 2 is a schematic diagram of the structure of a head-up display system for left-eye and right-eye fusion images provided according to Embodiment 3 of the present invention;

[0044] Figure 7 1 is a schematic diagram of the structure of another head-up display system for left-eye and right-eye fusion images provided according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0047] Embodiment 1

[0048] Figure 1 A flowchart of a head-up display method for left-right eye fusion images is provided for the first embodiment of the present invention. This embodiment is applicable to the case of real-time head-up display according to the user's eye position. The method can be executed by a head-up display system for left-right eye fusion images, and the head-up display system for left-right eye fusion images can be configured in a vehicle. Figure 1 As shown, the method includes:

[0049] S110 , capturing real-time position data of the left and right eyes of the user.

[0050] Among them, the real-time position data of the left and right eyes refers to the specific position information of the left and right eyes of the user monitored in real time, including the real-time position of the left eye and the real-time position of the right eye, expressed in the form of coordinates.

[0051] Specifically, the real-time position data of the left and right eyes can be captured by the driver monitoring system, which includes sensors and cameras that can accurately capture the position of the user's eyes. The sensors and cameras can monitor the subtle movements and changes of the user's eyes in real time to determine the position of the left and right eyes.

[0052] Furthermore, the driver monitoring system can determine the real-time position of the user's left and right eyes through infrared tracking technology or image recognition technology. Infrared tracking technology refers to emitting infrared light to the user's eyes, and then using sensors to receive the reflected light, and calculating the real-time position of the left and right eyes based on the change in light and time difference. Image recognition technology refers to using a camera to capture an image of the user's eyes, and using image processing algorithms to identify the eye's feature points, such as the pupil center, eye corners, etc., to determine the historical position values ​​of the left and right eyes.

[0053] In a specific implementation, the driver monitoring system may include a DMS eye tracking device, which is a device that acquires the driver's eye status in real time by installing an optical camera and an infrared camera on the car. It analyzes the acquired information through a deep learning algorithm, can determine the driver's status, realize driver identification, driver fatigue monitoring, driver attention monitoring, and monitoring of dangerous driving behaviors, and issue different levels of warnings.

[0054] S120 , obtaining left and right eye display images to be fused and displayed, and obtaining pixel points at different positions from the left and right eye display images respectively, performing fusion multiple times to obtain multiple fused images.

[0055] The pixel point refers to the smallest unit of the image. The fused image refers to a new image obtained by processing the left and right eye display images according to the fusion rules.

[0056] Specifically, the processor can obtain the left and right eye display images that need to be fused and displayed. The left and right eye display images can be prepared in advance, or generated according to real-time application scenarios and user needs, and the images include but are not limited to image files such as pictures, videos, and video streams. The processor will select corresponding pixel points from different positions in the left and right eye display images according to the pre-set fusion rules, and fuse them in turn to obtain multiple fused images.

[0057] Optionally, pixel points at different positions are obtained from the left and right eye display images respectively, and are fused multiple times to obtain multiple fused images, including: calling a shader plug-in in a three-dimensional image processing engine, obtaining pixel points at different positions from the left and right eye display images respectively, and are fused multiple times to obtain multiple fused images.

[0058] Specifically, the processor will call the shader plug-in in the 3D image processing engine. The shader plug-in is a component of the 3D image processing engine and has 3D image processing functions. The processor can call the shader plug-in and extract pixels at different positions from the left and right eye display images according to the preset fusion rules, and perform multiple fusions. In each fusion process, the shader plug-in will combine the selected pixels according to the preset fusion rules, and finally generate multiple fused images.

[0059] In summary, after multiple fusions of shader plug-ins, multiple fused images can be generated, thereby better meeting different application needs and visual effect requirements.

[0060] S130, acquiring a target fused image according to the real-time position data of the left and right eyes and the multiple fused images, and projecting the target fused image onto a display screen for head-up display.

[0061] Figure 2 A schematic diagram of the head-up display imaging process is provided for the first embodiment of the present invention. The head-up display (HUD) is a driving assistance instrument used on a vehicle. It is a comprehensive electronic display device composed of electronic components, display components, controllers, etc. The HUD can project vehicle speed, navigation information, warning information, etc. in the form of images and characters in front of the driver through optical components. The HUD can project images or information directly into the user's field of vision, allowing the user to see the relevant content without lowering his head.

[0062] Figure 3 A schematic diagram of the naked-eye 3D imaging principle is provided for the first embodiment of the present invention. The naked-eye 3D display technology refers to a 3D display technology that allows users to directly watch three-dimensional images with their naked eyes without wearing special 3D glasses, presenting a 3D effect. The front-end display part of the naked-eye 3D display technology needs to be calculated by the pixel imaging unit so that the left and right eyes of the person see different pictures, and merge them into a 3D effect picture in the brain. Figure 3 In the HUD, the optical design is performed on the HUD so that the user's left eye sees image P1 and the right eye sees image P2 through the imaging structure, such as the windshield. Due to the binocular parallax formed by images P1 and P2, the objects seen by the user have a sense of depth and space. The right eye sees the image of the right eye, which is synthesized into a three-dimensional picture with a sense of depth in the user's brain. Naked-eye 3D imaging can adjust the binocular parallax by changing the position between the two images, so that the distance of the virtual image perceived by the user changes (the actual distance of the virtual image remains unchanged). The closer the two images are, the closer the virtual image distance perceived by the user is; conversely, the farther the two images are, the farther the virtual image distance perceived by the user is.

[0063] Optionally, a target fused image is obtained based on the real-time position data of the left and right eyes and multiple fused images, and the target fused image is projected onto the display screen for head-up display, including: obtaining the fusion rounds corresponding to each area in the display screen in a mapping relationship between the pre-established areas and fusion rounds based on the real-time position data of the left and right eyes; and using the fusion images under the fusion rounds corresponding to each area to stitch together the target fused image.

[0064] Specifically, each position data has a corresponding preset fusion rule, and the fusion rule includes the number of fused images selected for each area for display. The processor can determine the fusion round corresponding to each area in the display screen in the pre-established mapping relationship between the area and the fusion round according to the real-time position data of the left and right eyes. Then, the fused images under the fusion round corresponding to each area are used for splicing operations to obtain the target fused image. It should be noted that the splicing process needs to consider the alignment and matching of pixels to ensure that the target fused image after splicing is visually continuous and consistent.

[0065] Specific application scenarios: Figure 4 The present invention provides a schematic diagram of a fusion image head-up display process, such as Figure 4 As shown, the DMS device obtains the position information of the left and right eyes in real time, and transmits the eye position data to the 3D image processing engine. The 3D image processing engine can switch the image fusion rules and display content in real time based on the monitoring of the human eye position, so as to achieve the naked eye 3D effect in the ARHUD display. By combining the eye box position partitioning scheme of 3DHUD and the 3D image processing engine technology to calculate the left and right fused images in real time, the target fusion object is determined in real time in the target screen partition for dynamic head-up display to the user.

[0066] The technical solution of the embodiment of the present invention can realize the real-time positioning of the user's eyes by capturing the real-time position data of the user's left and right eyes, thereby better providing the user with a more accurate display experience. By obtaining pixel points at different positions from the left and right eye display images and performing multiple fusions, the pixel calculation amount is small, and it is not easy to cause the main thread to freeze. The details and three-dimensional sense of the image can be enhanced, providing a more realistic and vivid visual effect. By projecting the target fused image onto the display screen for head-up display, the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0067] Embodiment 2

[0068] Figure 5 This is a flowchart of a head-up display method for left-eye predicted fusion images provided in Embodiment 2 of the present invention. Based on Embodiment 1 above, this embodiment adds a specific process of obtaining pixels at different positions from the left-eye corrected images for multiple fusions to obtain multiple fusion images. The specific content of step S210 is roughly the same as step S110 in Embodiment 1, so it will not be repeated in this embodiment. Figure 5 As shown, the method includes:

[0069] S210: Capture real-time position data of the left and right eyes of the user.

[0070] S220, determining the fusion times of the fused image, and dividing the display screen into a plurality of regions according to the fusion times; wherein the number of regions matches the fusion times.

[0071] S230 , expand each pixel of the left-eye and right-eye display images into RGB channel values ​​respectively, to obtain a left-eye RGB channel arrangement diagram and a right-eye RGB channel arrangement diagram.

[0072] Specifically, the processor will expand each pixel in the left-eye display image and the right-eye display image into RGB channel values ​​respectively to describe the color composition of each pixel.

[0073] S240 , determining fused images in each fusion round corresponding to the number of fusion times according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram.

[0074] Specifically, the processor will perform various fusion rounds in sequence according to the RGB channel arrangement diagram of the left eye and the RGB channel arrangement diagram of the right eye according to the pre-set fusion times. In each round of fusion, the RGB channel values ​​of the left eye and the right eye can be extracted and combined according to the set fusion strategy.

[0075] Optionally, according to the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram, a fused image under each fusion round corresponding to the number of fusion times is determined, including: generating a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determining a fusion strategy corresponding to each fusion round, and according to the fusion strategy, selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to fill in the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round.

[0076] The full-eye RGB channel arrangement map to be filled is a blank image whose size and pixel layout are the same as those of the fused image and is used to store the RGB channel values ​​selected during the fusion process.

[0077] Specifically, the processor can select corresponding RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram in each round of fusion according to the fusion strategy, and fill them into the corresponding positions in the full eye RGB channel arrangement diagram to be filled. It should be noted that the fused image obtained by each fusion is a complete full eye RGB channel arrangement diagram, and the processor can perform different degrees of fusion in each fusion round according to the fusion strategy.

[0078] Optionally, the fusion strategy includes: the number of RGB channel value groups, the number of first line indents, and the number of vertical indents.

[0079] Among them, the values ​​involved in the fusion strategy need to be adapted to the arrangement of the spectroscopic grating. The number of RGB channel value groups is related to the number of pixels covered by the grating unit. For example, when a grating unit can cover 8 pixels (i.e., the backlight lamp beads), the number of RGB channel value groups needs to be set to 4. At this time, among every 8 lamp beads, the 4 lamp beads on the left and the 4 lamp beads on the right will emit in different directions, and can be seen by the left eye and the right eye respectively.

[0080] Specifically, in each round, different indentation numbers of different rows are also used to adapt to various different arrangements of the spectroscopic gratings. The crosstalk phenomenon can be effectively reduced by arranging the gratings in an inclined manner.

[0081] It should be noted that the indentation of different rounds is mainly for traversing various states. Different states correspond to the display of pixels (lamp beads) covered by the grating. It should be noted that there is an upper limit to the number of states. For example, in the case of 4 groups, when each round is indented by 1 bit, the upper limit of the number of states is reached in 8 rounds. At this time, the situation of the 9th round will be the same as the 1st round. Therefore, the upper limit of the round is actually the upper limit of the number of states. When partitioning and selecting the number of fusion rounds, it is actually selecting the corresponding state for fusion operation. The fusion round needs to select the most appropriate state according to specific needs and conditions to achieve the best fusion effect and achieve the desired goal.

[0082] Optionally, according to the fusion strategy, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram respectively to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused images under each fusion round, including: obtaining the current fusion strategy corresponding to the current fusion round, and determining the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groupings in the current fusion strategy; adjusting the initial filling mode of the first row according to the number of indentation bits of the first row corresponding to the current fusion round to obtain the target filling mode of the first row; determining each non-first row filling mode in turn starting from the second row according to the target filling mode of the first row and the number of vertical indentation bits; selecting RGB channel values ​​from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0083] Specifically, according to the number of RGB channel value groups, it can be determined which channel values ​​are selected from the left eye and right eye RGB channel arrangement diagrams to fill the first row of pixels in the full eye RGB channel arrangement diagram to be filled. Then, according to the number of indents in the first row of the current fusion round, the filling position of the first row is moved to the right by a certain number of places to adjust the indentation of the initial filling method of the first row. Starting from the second row, the processor can determine the filling method of each non-first row in turn according to the target filling method of the first row and the number of vertical indents, where the number of vertical indents determines the vertical offset of each row of the filling position relative to the previous row.

[0084] Further, according to the first row target filling method and each non-first row filling method, the corresponding RGB channel values ​​are selected from the left eye and right eye RGB channel arrangement diagrams in turn to fill in the corresponding pixel points in the full eye RGB channel arrangement diagram to be filled, so as to complete the image filling fusion under the current fusion round and obtain the fused image of this round. The processor repeats the above steps to perform multiple fusion rounds, and finally obtains the fused images corresponding to each fusion round.

[0085] Optionally, in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; in adjacent fusion rounds, the number of first line indents in the latter fusion round is 1 more than the number of first line indents in the previous fusion round.

[0086] Among them, in different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same, which means that the number of RGB channel values ​​fused in each fusion round and the adjustment method of the vertical spacing between rows are fixed. In adjacent fusion rounds, the number of indents after the first line of the next fusion round is 1 more than the number of indents of the first line of the previous fusion round, which means that as the fusion rounds increase, the indent to the right of the first line increases by 1.

[0087] Specifically, the same number of RGB channel value groups ensures consistency in the division and processing of color channel values ​​in each fusion round, thereby maintaining the stability and consistency of color information processing in the fusion process as a whole.

[0088] Optionally, the number of RGB channel value groups is 4 and the number of vertical indents is 1.

[0089] Specifically, the number of RGB channel value groups of 4 means that every four RGB values ​​in the horizontal direction are grouped, and the number of vertical indents of 1 means that the RGB value indents one position downward in the vertical direction for each row. By grouping the four RGB values ​​in the horizontal direction of the left and right eye display images into a group, and indenting one position downward in the vertical direction for each row, the left and right eye images are fused N times, and each fusion is indented one position backward from the first row, and the pixels are saved by taking N fusion images. N is the total number of fusions corresponding to the number of screen partitions.

[0090] S250, obtaining a fusion round corresponding to each area in the display screen according to the mapping relationship between the areas and the fusion rounds established in advance according to the real-time position data of the left and right eyes.

[0091] S260: Use the fused images of the fusion rounds corresponding to each region to stitch together to obtain a target fused image.

[0092] The technical solution of the embodiment of the present invention can more intuitively understand the difference and distribution of the left and right eye images in the color channels by expanding each pixel of the left and right eye display images respectively to obtain the RGB channel arrangement diagrams of the left and right eyes, so that the fused image can better integrate the information of the left and right eyes. By fusing the left and right eye display images, the stereoscopic sense and depth sense of the image can be enhanced, so that the user can see a stereoscopic three-dimensional image.

[0093] Embodiment 3

[0094] Figure 6 This is a schematic diagram of the structure of a head-up display system for left-eye and right-eye fusion images provided in Embodiment 3 of the present invention. Figure 6 As shown, the system includes: an image capturing device 310 , a control unit 320 and an image projection device 330 .

[0095] Optionally, an image capture device is used to capture the real-time position data of the left and right eyes of the user, and transmit the real-time position data of the left and right eyes to the control unit; the control unit is used to obtain pixel points at different positions from the acquired left and right eye display images for multiple fusions to obtain multiple fused images, and obtain a target fused image based on the real-time position data of the left and right eyes and the multiple fused images; the image projection device is used to project the target fused image onto the display screen for head-up display.

[0096] Specifically, image capture devices usually use eye tracking technology to capture the real-time position data of the user's left and right eyes. For example, in the case of infrared light reflection, the device will emit infrared light of a specific wavelength to the user's eyes. When the light hits the surface of the eyeball, it will be reflected. Then, the sensor and camera will capture the changes in the reflected light to calculate the real-time position and movement trajectory of the user's left and right eyes. When processing images, the control unit uses an image fusion algorithm to obtain pixel points at different positions from the left and right eye display images for multiple fusions to obtain multiple fused images, and obtain the target fused image based on the real-time position data of the left and right eyes and the multiple fused images. The image projection device works based on the principle of optical projection. When the projection device is working, it will control the intensity and color of the light source according to the pixel information of the target fused image, and project the image onto the display screen at an appropriate ratio and focal length to achieve the effect of head-up display.

[0097] Figure 7 This is a schematic diagram of the structure of another head-up display system for left-eye and right-eye fusion images provided in Embodiment 3 of the present invention. Figure 7 In the figure, the image projection device includes an optical grating device 331 and a mirror device 332.

[0098] Optionally, the image projection device includes an optical grating device and a mirror device; the optical grating device is used to split the target fused image to generate a split image to achieve three-dimensional imaging; the mirror device is used to project the split image onto a display screen for head-up display according to a preset angle.

[0099] Specifically, the optical grating device is composed of an arrangement of grating stripes, and when the light of the target fused image passes through the grating, diffraction occurs. Light of different wavelengths and angles will be separated and refracted in a specific way, thereby splitting the target fused image to generate a split image. By precisely controlling the parameters and structure of the grating, the images seen from different perspectives can be different, creating a three-dimensional visual effect to achieve three-dimensional imaging. When the split image reaches the mirror device, the mirror will accurately reflect and project the light onto the display screen according to the preset angle and direction. When setting the preset angle, factors such as the user's viewing position, the position and angle of the display screen need to be taken into account to ensure that the user can obtain clear and realistic three-dimensional imaging effects at a comfortable viewing angle.

[0100] Optionally, the control unit is also used to: determine the number of fusion times of the fused images before obtaining multiple fused images by acquiring pixel points at different positions from the left and right eye display images respectively and performing multiple fusions, and divide the display screen into multiple areas according to the number of fusions; wherein the number of areas matches the number of fusions.

[0101] Optionally, the control unit is used to: expand each pixel point of the left and right eye display images into RGB channel values ​​respectively, to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; generate a full-eye RGB channel arrangement diagram to be filled corresponding to the fused image; determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to fill the matching image positions in the full-eye RGB channel arrangement diagram to be filled, to obtain a fused image under each fusion round; wherein the fusion strategy includes the number of RGB channel value groups, the number of first-line indentation digits, and the number of vertical indentation digits, and in each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indentation digits are the same; under adjacent fusion rounds, the number of first-line indentation digits of the latter fusion round is 1 more than the number of first-line indentation digits of the previous fusion round.

[0102] Optionally, the control unit is used to: obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling mode of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; adjust the initial filling mode of the first row according to the number of first row indentation bits corresponding to the current fusion round to obtain the target filling mode of the first row; determine each non-first row filling mode in sequence starting from the second row according to the first row target filling mode and the number of vertical indentation bits; select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram according to the first row target filling mode and each non-first row filling mode to fill the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under the current fusion round.

[0103] Optionally, the control unit is used to: obtain the fusion round corresponding to each area in the display screen according to the mapping relationship between the real-time position data of the left and right eyes and the fusion round in a pre-established manner; use the fusion images under the fusion round corresponding to each area to stitch together to obtain the target fusion image.

[0104] The technical solution of the embodiment of the present invention can realize the real-time positioning of the user's eyes by capturing the real-time position data of the left and right eyes of the user through the image capture device, so as to better provide the user with a more accurate display experience. The control unit obtains pixel points at different positions from the left and right eye display images for multiple fusions. The pixel calculation amount is small, and it is not easy to cause the main thread to freeze. The details and three-dimensional sense of the image can be enhanced, and a more realistic and vivid visual effect can be provided. The target fused image is projected onto the display screen for head-up display through the image projection device, so that the user can obtain important information without looking down at the instrument, thereby improving driving safety.

[0105] A head-up display system for left-eye fused images provided in an embodiment of the present invention can execute a head-up display method for left-eye fused images provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0107] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A head-up display method for left-eye and right-eye fused images, characterized in that: include: Capture the real-time position data of the user's left and right eyes; Obtaining left and right eye display images to be fused and displayed, respectively obtaining pixel points at different positions from the left and right eye display images for multiple fusions to obtain multiple fused images; A target fused image is acquired according to the real-time position data of the left and right eyes and the multiple fused images, and the target fused image is projected onto a display screen for head-up display.

2. The method according to claim 1, characterized in that Before acquiring pixel points at different positions from the left and right eye display images and fusing them multiple times to obtain multiple fused images, the method further includes: Determining the number of fusion times of the fused image, and dividing the display screen into a plurality of areas according to the number of fusion times; The number of regions matches the number of fusions.

3. The method according to claim 2, characterized in that Pixels at different positions are obtained from the left and right eye display images for multiple fusions to obtain multiple fused images, including: Expand each pixel of the left and right eye display images into RGB channel values ​​respectively to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; According to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, a fused image in each fusion round corresponding to the fusion number is determined.

4. The method according to claim 3, characterized in that According to the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram, a fused image in each fusion round corresponding to the fusion number is determined, including: Generate a full-eye RGB channel arrangement map to be filled corresponding to the fused image; Determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to fill the matching image position in the full eye RGB channel arrangement diagram to be filled, and obtain the fused image under each fusion round.

5. The method according to claim 4, characterized in that The fusion strategy includes: the number of RGB channel value groups, the number of first line indents and the number of vertical indents.

6. The method according to claim 5, characterized in that According to the fusion strategy, RGB channel values ​​are selected from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to be filled into the matching image position in the full eye RGB channel arrangement diagram to be filled, and the fused image under each fusion round is obtained, including: Obtain the current fusion strategy corresponding to the current fusion round, and determine the initial filling method of the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; According to the number of first-line back-indent bits corresponding to the current fusion round, the back-indent adjustment is performed on the first-line initial filling mode to obtain the first-line target filling mode; According to the first line target filling mode and the number of vertical indents, starting from the second line, each non-first line filling mode is determined in sequence; According to the first-row target filling method and each non-first-row filling method, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to be filled into the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain a fused image under the current fusion round.

7. The method according to claim 4, characterized in that In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; In adjacent fusion rounds, the number of indents after the first line of the latter fusion round is 1 more than the number of indents after the first line of the previous fusion round.

8. The method according to claim 7, characterized in that The number of RGB channel value groups is 4, and the number of vertical indent digits is 1.

9. The method according to claim 2, characterized in that: Acquiring a target fused image according to the real-time position data of the left and right eyes and the multiple fused images, and projecting the target fused image onto a display screen for head-up display, including: According to the mapping relationship between the areas and fusion rounds established in advance, the left and right eye real-time position data is used to obtain the fusion rounds corresponding to each area in the display screen; The target fused image is obtained by splicing the fused images in the fusion rounds corresponding to each region.

10. A head-up display system for left-eye and right-eye fusion images, characterized in that: The system comprises: an image capturing device, a control unit and an image projection device; An image capture device, used to capture the real-time position data of the left and right eyes of the user, and transmit the real-time position data of the left and right eyes to the control unit; A control unit, configured to obtain pixel points at different positions from the acquired left and right eye display images, respectively, and fuse them multiple times to obtain multiple fused images, and obtain a target fused image based on the left and right eye real-time position data and the multiple fused images; The image projection device is used to project the target fused image onto a display screen for head-up display.

11. The system according to claim 10, characterized in that The image projection device includes an optical grating device and a mirror device; The optical grating device is used to perform spectral splitting on the target fused image to generate a spectral split image, so as to achieve three-dimensional imaging; The mirror device is used to project the split image onto a display screen for head-up display according to a preset angle.

12. The system according to claim 10, characterized in that The control unit is further used to: determine the number of times the fused images are fused before obtaining the pixel points at different positions from the left and right eye display images for multiple fusions, and divide the display screen into multiple areas according to the number of fusions; The number of regions matches the number of fusions.

13. The system according to claim 12, characterized in that The control unit is used to: expand each pixel point of the left and right eye display images into RGB channel values ​​respectively, to obtain a left eye RGB channel arrangement diagram and a right eye RGB channel arrangement diagram; Generate a full-eye RGB channel arrangement map to be filled corresponding to the fused image; Determine the fusion strategy corresponding to each fusion round, and according to the fusion strategy, select RGB channel values ​​from the left eye RGB channel arrangement diagram and the right eye RGB channel arrangement diagram to fill the matching image position in the full eye RGB channel arrangement diagram to be filled, so as to obtain the fused image under each fusion round; Among them, the fusion strategy includes the number of RGB channel value groups, the number of first line indents and the number of vertical indents. In each fusion strategy under different fusion rounds, the number of RGB channel value groups and the number of vertical indents are the same; in adjacent fusion rounds, the number of first line indents in the latter fusion round is 1 more than the number of first line indents in the previous fusion round.

14. The system according to claim 13, characterized in that The control unit is used to: obtain a current fusion strategy corresponding to a current fusion round, and determine an initial filling mode for the first row corresponding to the full-eye RGB channel arrangement diagram to be filled according to the number of RGB channel value groups in the current fusion strategy; According to the number of first-line back-indent bits corresponding to the current fusion round, the back-indent adjustment is performed on the first-line initial filling mode to obtain the first-line target filling mode; According to the first line target filling mode and the number of vertical indents, starting from the second line, each non-first line filling mode is determined in sequence; According to the first-row target filling method and each non-first-row filling method, RGB channel values ​​are selected from the left-eye RGB channel arrangement diagram and the right-eye RGB channel arrangement diagram to be filled into the matching image position in the full-eye RGB channel arrangement diagram to be filled, so as to obtain a fused image under the current fusion round.

15. The system according to claim 14, characterized in that The control unit is used to: obtain a fusion round corresponding to each area in the display screen according to a pre-established mapping relationship between areas and fusion rounds of the left and right eye real-time position data; The target fused image is obtained by splicing the fused images in the fusion rounds corresponding to each region.

Citation Information

Patent Citations

  • Method for synthesizing multiview images

    CN102238410A

  • 3D display control method, system, storage medium thereof, and 3D display system

    CN109151434A

  • Display effect fusion method and system, storage medium and main control unit

    CN112242009A

  • Multi-level imaging system

    CN113126294A

  • Display method, device and system and computer readable storage medium

    CN114630099A

Cited By

  • Head-up display method and system for fused left- and right-eye image

    WO2026025894A1