Virtual image editing method, display device, storage medium, and vehicle

Abstract

The application relates to the technical field of projection display, in particular to a virtual image editing method, a display device, a storage medium and a vehicle. The application utilizes the convenient operation advantage of a mobile terminal, calls a virtual space in which a virtual image of a HUD display device is projected, edits virtual image elements in the virtual space of the mobile terminal according to a judgment of a user of the mobile terminal on a real scene, and synchronously makes the editing result take effect in a projection object of the HUD display device to meet the virtual-real fitting requirement in projection display, thereby providing effective navigation indication content. The application can improve the navigation capability in the virtual image projected by the HUD display device, in particular, can strengthen the accuracy of indication information in a complex road section, and optimize the navigation experience of augmented reality.

Description

Technical Field

[0001] This application relates to the field of projection display technology, and in particular to a virtual image editing method, display device, storage medium, and vehicle. Background Technology

[0002] HUD (Head-Up Display) is a novel in-vehicle display method that utilizes reflection from the windshield. Specifically, the HUD display device's optical engine emits light, which is projected onto the windshield through corresponding optical lenses to create a virtual image, enhancing the display effect against the real world outside the windshield. The virtual image elements used for navigation instructions are aligned with the real-world scene, greatly improving navigation intuitiveness. However, the presentation of the navigation instructions relies on the vehicle's cameras, radar, and other sensors' ability to perceive and process the real-world scene ahead, as well as the accuracy of the navigation path in the map data and the precision of the navigation target. Any deviations often result in incorrect navigation instructions, with the virtual image elements not matching the aligned real-world scene or missing key navigation instructions. This not only fails to provide clear and intuitive information to the driver but can also mislead them. Summary of the Invention

[0003] The purpose of this application is to provide a virtual image editing method, display device, storage medium, and vehicle, which solves the technical problems in the prior art where the projected virtual image of HUD display device is highly dependent on sensor resources and map data, and cannot fully meet the navigation needs of users during driving, and the indication information of the virtual image at key locations is inaccurate or missing.

[0004] To solve the above-mentioned technical problems, this application adopts the following technical solution.

[0005] Firstly, this application provides a virtual image editing method, including:

[0006] HUD display devices support projecting a first virtual image element in front of the vehicle windshield, the first virtual image element being configured to cooperate with a first real scene in front of the vehicle to provide directional information.

[0007] In response to a user inputting first auxiliary information about the first virtual image element relative to the first real scene via a mobile terminal, the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information to match the first real scene.

[0008] In one alternative embodiment of the first aspect, the HUD display device establishes a connection with the mobile terminal to support information interaction between the two.

[0009] In one alternative implementation of the first aspect, the connection is established via Bluetooth or WiFi.

[0010] In one alternative embodiment of the first aspect, the HUD display device establishes a connection with the mobile terminal to support information interaction between the two, including:

[0011] The HUD display device receives the first auxiliary information sent by the mobile terminal through the connection.

[0012] Based on the above description, the optional implementation can make the first virtual image element projected by the HUD display device no longer rigidly controlled by a fixed program, but can be finely adjusted as needed to adapt to the real scene in front of the vehicle and optimize the augmented reality effect.

[0013] In one alternative embodiment of the first aspect, the first virtual image element is a navigation arrow pointing to a specific intersection.

[0014] In one alternative embodiment of the first aspect, the first virtual image element is a highlighted marker indicating a specific entrance or exit.

[0015] In one alternative embodiment of the first aspect, the first virtual image element is a label indicating a specific building.

[0016] Based on the above description, the alternative implementation can support the projection display of various forms of first virtual image elements by the HUD display device through the flexible editing of the mobile terminal, without relying on the limited capabilities provided by sensor recognition and map data.

[0017] In one alternative embodiment of the first aspect, the first auxiliary information is standard format information indicating the fitting position and / or direction of the first virtual image element relative to the first real scene, wherein the standard format information is at least one of descriptive text, marked image, and information link.

[0018] In one optional embodiment of the first aspect, when the standard format information is the description text, the first virtual image element is adjusted according to the fitting position offset correction parameter and / or the indication direction rotation correction parameter recorded in the description text.

[0019] In one alternative embodiment of the first aspect, when the standard format information is the marker image, the first virtual image element is adjusted according to the specified marker position and / or the marker direction indicated in the marker image.

[0020] In an alternative embodiment of the first aspect, adjusting the first virtual image element according to the specified fitting mark position in the marked image includes:

[0021] The alignment position of the first virtual image element relative to the first real scene is determined based on the similarity between the alignment mark position and the first real scene.

[0022] In one optional embodiment of the first aspect, when the standard format information is the marked image, the first virtual image element is adjusted according to the model rendering corresponding to the marked image and the projection matrix parameters used for rendering.

[0023] In one optional implementation of the first aspect, when the standard format information is the marker image, the first virtual image element is adjusted according to the first real-scene marker content corresponding to the marker image and the camera projection matrix parameters of the first real-scene captured by the mobile terminal.

[0024] In one optional implementation of the first aspect, when the standard format information is the information link, the server corresponding to the information link is accessed, and the first virtual image element is adjusted according to the content provided by the server.

[0025] In one alternative implementation of the first aspect, the information link supports shared access by multiple users.

[0026] Based on the above description, the optional implementation supports multiple adjustments to the first virtual image element. Users can select and input according to the actual situation. The HUD display device supports adaptively using different forms to determine the adjustment content of the first virtual image element.

[0027] In an alternative implementation of the first aspect, the response to a user inputting first auxiliary information about the first virtual image element relative to the first real scene via a mobile terminal includes:

[0028] The HUD display device sends the current projection state of the first virtual image element to the mobile terminal to present an instruction relative to the first real scene to the user on the mobile terminal.

[0029] In an optional implementation of the first aspect, the HUD display device sending the current projection state of the first virtual image element to the mobile terminal includes:

[0030] The rendering data of the first virtual image element relative to the first real scene is used to provide spatial parameters when the first virtual image element is projected.

[0031] In an optional implementation of the first aspect, the HUD display device sending the current projection state of the first virtual image element to the mobile terminal includes:

[0032] The HUD display device projects the pre-rendered data of the first virtual image element to support the mobile terminal in editing the first virtual image element before projection.

[0033] Based on the above description, the optional implementation can send the complete rendering data to the mobile terminal. The mobile terminal can reproduce the three-dimensional spatial relationship between the first virtual image element and the first real scene through coordinate system transformation, thereby supporting the user to preview the editing effect while editing the first virtual image element, improving the user's intuitiveness.

[0034] In an optional embodiment of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes:

[0035] When it is determined from the first auxiliary information that the fitting position of the first virtual image element relative to the first real scene needs to be corrected, the first virtual image element is inversely mapped to the projection area according to the projection parameters of the HUD display device, so that the fitting position of the first virtual image element matches that of the first real scene.

[0036] Based on the above description, the optional implementation can correct the problem of misalignment between virtual and real images by directly changing the projection position of the first virtual image element based on the positional mapping relationship between the projection area and the image source display surface. Users do not need to consider specific projection correction parameters when editing.

[0037] In an optional embodiment of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes:

[0038] The user includes at least a first user and a second user. The first user inputs second auxiliary information of the first virtual image element relative to the first real scene through a first mobile terminal, and the second user inputs third auxiliary information of the first virtual image element relative to the first real scene through a second mobile terminal.

[0039] The first auxiliary information is determined based on the second auxiliary information and the third auxiliary information.

[0040] In an optional embodiment of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes:

[0041] The user also includes a third user, who inputs fourth auxiliary information about the first virtual image element relative to the first real scene through a third mobile terminal. The first auxiliary information is selected from the second, third, and fourth auxiliary information with the highest similarity.

[0042] Based on the above description, the optional implementation utilizes user input data from multiple sources to determine the specific parameters for matching the first virtual image element with the first real scene, thereby improving accuracy. It also compensates for the high cost and low refinement of map data production through a user-generated content model.

[0043] Secondly, this application provides a virtual image editing method, including:

[0044] The mobile terminal establishes a connection with the HUD display device and triggers editing mode;

[0045] In response to a user's input operation on the mobile terminal, the system sends first auxiliary information about the first virtual image element relative to the first real scene to the HUD display device, so that the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information. The first virtual image element is an indication information projected by the HUD display device in front of the vehicle windshield that matches the first real scene.

[0046] In an alternative embodiment of the second aspect, the mobile terminal establishing a connection with the HUD display device includes:

[0047] The mobile terminal establishes a connection via Bluetooth or WiFi.

[0048] In one alternative implementation of the second aspect, the first virtual image element is a navigation arrow pointing to a specific intersection.

[0049] In one alternative implementation of the second aspect, the first virtual image element is a highlighted marker indicating a specific entrance or exit.

[0050] In an alternative embodiment of the second aspect, the first virtual image element is a label indicating a specific building.

[0051] In one alternative embodiment of the second aspect, the first auxiliary information is standard format information indicating the fitting position and / or direction of the first virtual image element relative to the first real scene, wherein the standard format information is at least one of descriptive text, marked image, and information link.

[0052] According to the above description, in the optional implementation, the user can use a mobile terminal to control the first virtual image element of the HUD display device by judging the real scene in front of the vehicle. The real-time performance and flexibility are relatively high, eliminating the ambiguity of non-drivers giving verbal directions and enhancing the overall driving experience.

[0053] In an alternative implementation of the second aspect, the response to a user's input on the mobile terminal includes:

[0054] The first real scene and the first display element are reproduced on the screen of the mobile terminal. The first display element reproduces the relative relationship between the state of the first virtual image element projected by the HUD display device and the first real scene.

[0055] Based on the user's adjustments to the first display element, first auxiliary information is generated to adjust the first virtual image element.

[0056] In one alternative embodiment of the second aspect, the first real-world scene is reproduced on the screen of the mobile terminal by calling a 3D model from map data.

[0057] In one alternative embodiment of the second aspect, the first real-world scene is reproduced on the screen of the mobile terminal as a real-time image of the first real-world scene captured by the mobile terminal.

[0058] In an optional embodiment of the second aspect, the relationship between the state reproduction of the first virtual image element projected by the HUD display device and the first real scene includes:

[0059] The device receives the first projection matrix parameters for rendering the first virtual image element from the HUD display device, and adjusts the display of the first display element on the mobile terminal screen according to the second projection matrix parameters used for rendering the three-dimensional model on the mobile terminal screen.

[0060] In an optional embodiment of the second aspect, the relationship between the state reproduction of the first virtual image element projected by the HUD display device and the first real scene includes:

[0061] The system receives the first projection matrix parameters of the HUD display device for rendering the first virtual image element, and adjusts the display of the first display element on the mobile terminal screen according to the third projection matrix parameters of the camera that captured the first real scene.

[0062] In an optional implementation of the second aspect, generating first auxiliary information for adjusting the first virtual image element based on the user's adjustment of the first display element includes:

[0063] Based on the user's drag-and-drop operation on the mobile terminal screen, it is recorded as descriptive text or frozen as a marker image.

[0064] In an optional implementation of the second aspect, generating first auxiliary information for adjusting the first virtual image element based on the user's adjustment of the first display element includes:

[0065] The user's rotation operation on the mobile terminal body is recorded as descriptive text or frozen as a marker image.

[0066] Based on the above description, the optional implementation can change the alignment position and direction of the first display element relative to the reproduced first real scene through convenient user operation, thereby generating first auxiliary information with reference basis for the HUD display device projection.

[0067] In an alternative implementation of the second aspect, the response to a user's input on the mobile terminal includes:

[0068] At least a portion of the first real-world scene is captured by the camera on the mobile terminal to indicate the position and / or direction where the first virtual image element needs to be aligned.

[0069] In an alternative implementation of the second aspect, capturing at least a portion of the first real-world scene using a camera on the mobile terminal to mark the desired alignment position and / or indicated direction of the first virtual image element includes:

[0070] The fitting position and / or indication direction of the first virtual image element are determined by the user's selection of the corresponding real-world content on the mobile terminal screen.

[0071] In an optional implementation of the second aspect, determining the fitting position of the first virtual image element by the user's selection of the corresponding real-world content on the mobile terminal screen includes:

[0072] By performing semantic analysis on the selected real-world image, the outline of the corresponding real-world scene is marked.

[0073] Based on the above description, the optional implementation supports direct image capture to determine the first auxiliary information. The HUD display device can determine the fitting position and indication direction of the first virtual image element by judging the image similarity, so that users can provide their own projection reference information.

[0074] Thirdly, this application provides a display device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the virtual image editing method described in the first or second aspect.

[0075] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the virtual image editing method described in the first or second aspect.

[0076] Fifthly, this application provides a means of transportation, including the display device described in the third aspect or the computer-readable storage medium described in the fourth aspect.

[0077] Compared to existing technologies, this application leverages the ease of operation of mobile terminals to access the virtual space where the projected virtual image of the HUD display device resides. Based on the mobile terminal user's assessment of the real-world scene, the virtual image elements within this virtual space are edited. The edited results are simultaneously applied to the projected object on the HUD display device, ensuring a seamless integration of virtual and real-world elements during projection and thus providing effective navigation guidance. This application enhances the navigation capabilities of the projected virtual image on the HUD display device, particularly strengthening the accuracy of guidance information in complex road sections and optimizing the augmented reality navigation experience. Attached Figure Description

[0078] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the technical solution will be briefly introduced below. Obviously, the drawings described below are merely some examples recorded in this application, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0079] Figure 1 The following are schematic diagrams of HUD projection displays in some examples of this application.

[0080] Figure 2 This is a schematic diagram illustrating the deviation in the projection and bonding position of virtual image elements in existing technologies.

[0081] Figure 3 This is a schematic diagram illustrating the deviation of the projection direction of virtual image elements in the prior art.

[0082] Figure 4 This is a schematic diagram illustrating the connection between a HUD display device and a mobile terminal in some examples of this application.

[0083] Figure 5 This application provides schematic diagrams of mobile terminal editing virtual image elements in some examples.

[0084] Figure 6 The diagram shows the rotation control input of a mobile terminal in some examples of this application.

[0085] Figure 7 The following are schematic diagrams illustrating the display elements and realistic scenes presented on the mobile terminal in some examples of this application.

[0086] Figure 8 The following are some examples of how a mobile terminal captures and reproduces a real-world scene in real time, as shown in this application.

[0087] Figure 9 The following are schematic diagrams of parking lot entrance navigation scenarios in some examples of this application.

[0088] Figure 10 The following are schematic diagrams illustrating the input of marked images via a mobile terminal in some examples of this application.

[0089] Figure 11 This is a schematic diagram of a HUD display device projecting a highlighted label, as shown in some examples of this application.

[0090] Figure 12 This is a schematic diagram of the alignment position of the marked image in some examples of this application.

[0091] Figure 13 This is a schematic diagram of the projection symbols for HUD display devices in some examples of this application.

[0092] Figure 14 The following are schematic diagrams of HUD display device modules in some examples of this application.

[0093] Figure 15 The following are schematic diagrams illustrating the composition of HUD display devices in some examples of this application.

[0094] Figure 16 These are schematic diagrams of projection displays in vehicles, as shown in some examples of this application. Detailed Implementation

[0095] The present application will be described in detail below with reference to the accompanying drawings. However, the description is only a few examples recorded in the present application and does not limit the present application. Any changes in structure, method or function made by those skilled in the art based on these examples are included within the protection scope of the present application.

[0096] It should be noted that while the same labels or markers may be used in different examples, these do not represent an absolute structural or functional relationship. Furthermore, the use of terms such as "first," "second," etc., in the examples is merely for descriptive convenience and does not represent an absolute structural or functional distinction, nor should it be interpreted as indicating or implying relative importance or the number of corresponding objects. Unless otherwise specified, "at least one" in the description refers to one or more, and "more than one" refers to two or more.

[0097] Furthermore, when representing features, the character " / " can indicate an OR relationship between related objects. For example, "head-up display" or "head-up display" can be represented as "head-up display" or "head-up display". When representing operations, the character " / " can indicate a division relationship between related objects. For example, magnification M = L / P can be represented as L (virtual image size) divided by P (image source size). Moreover, the "AND / OR" in different examples is merely to describe the relationship between related objects. This relationship can include three cases. For example, a concave mirror and / or a convex mirror can be represented as a concave mirror alone, a convex mirror alone, or both concave and convex mirrors.

[0098] HUD projection displays primarily utilize the principle of optical reflection, reflecting the image light to be displayed through a transparent surface into the viewer's eyes. The human eye can then see the virtual image information by following the reverse direction of the light. Correspondingly, the transparent surface can be a vehicle's windshield, which acts as a display screen to show navigation instructions, vehicle speed, etc. Figure 1 As shown, a HUD display device may include at least an optical engine 1, a first reflector 2, and a second reflector 3. The optical engine 1 includes a backlight and an image source (not shown). The backlight is used to provide illumination light and adjust the brightness of the illumination light according to control. For example, the backlight can be an LED (Light Emitting Diode), a laser, etc. Under the illumination light provided by the backlight, the image source adjusts the corresponding display content according to control and projects display light from the surface of the image source. For example, the image source can be an LCD (Liquid Crystal Display), a DMD (Digital Micromirror Devices), a MEMS (Micro-Electro-Mechanical System) micromirror, an LCOS (Liquid Crystal on Silicon), etc. The first reflecting mirror 2 and the second reflecting mirror 3 can project the display light emitted by the optical engine 1 onto the windshield 4, enabling customized optical paths within a small space while meeting different projection display requirements. The first reflecting mirror 2 and the second reflecting mirror 3 can be configured as concave mirrors, convex mirrors, concave lenses, convex lenses, etc., according to optical planning needs, and the surface shape of the lenses can be freeform. Optionally, at least one of the first reflecting mirror 2 and the second reflecting mirror 3 can also be angled to a certain extent, thereby changing the projection position of the display light on the windshield 4 to accommodate viewers of different heights. The display light from the optical engine 1 is ultimately reflected on the windshield 4 of the vehicle to form a virtual image 5. When the human eye 6 observes the virtual image 5 through the windshield 4, it can perceive a certain sense of depth, just like viewing a real object at a specific distance outside the windshield. The virtual image 5 can be navigation instructions, vehicle speed, etc., as described above. It should be added that, depending on the characteristics of different optical engines, HUD display devices can also be equipped with astigmatism filters. In some examples, HUD display devices can also include Fresnel lenses, waveguide optics, diffractive optics, holographic optics, tapered optical fibers, etc.

[0099] like Figure 2As shown, the HUD display device projects display light onto the windshield 4 and reflects it to form a virtual image element 501 and a corresponding projection area 50 supporting the virtual image element 501. The virtual image element 501 can be positioned on the projection area 50 by controlling its projection position on the optical-mechanical display surface, thereby changing its spatial position. The virtual image element 501 can be set at a specific position in the projection area 50 according to its matching position with the corresponding real scene, thus achieving a corresponding virtual-real matching effect. However, determining the matching position between the virtual image element 501 and the corresponding real scene first requires the recognition of the real scene in front of the vehicle by sensing devices (such as cameras, LiDAR, etc.), and may also require semantic analysis of the real scene in front, such as determining the corresponding road intersections and lane lines, and may even require further analysis and judgment based on road conditions and navigation routes in map data. These identification and processing processes cannot currently guarantee absolute reliability. For example, sensing devices may not be able to accurately collect detailed information about the real scene, semantic analysis models may not be able to accurately identify target objects and their locations, and map data may lack corresponding data in remote areas or the detailed intersections within cities may not be fully maintained. This can lead to deviations in the projection position of the virtual image element 501 in the projection area 50 of the HUD display device due to data inaccuracies, preventing a correct virtual-real alignment with the actual scene in front. Figure 2 In this scenario, the virtual image element 501 should be projected onto the first contact point 502 to visually indicate a right-hand road direction through the alignment of the virtual and real elements. However, due to a misalignment, the virtual image element 501 is actually attached to the surface of a building directly in front of it, preventing the driver from correctly understanding the navigation instructions. For scenarios where the virtual image element is a navigation arrow, the final deviation in the projection, besides the aforementioned overall misalignment, could also be due to other factors such as... Figure 3As shown, although the virtual image element 501 is aligned with the correct intersection position, the direction of the navigation indication is deviated. This may be because the identified target is inaccurate, or the map data is not updated in real time, thus failing to provide reliable reference data for virtual-to-real alignment. In more cases, the virtual image element projected by the HUD display device not only deviates from the alignment position with the specific real scene ahead, but also deviates in the indicated direction, rendering it completely ineffective for navigation. Optionally, when the vehicle reaches a specific location, no virtual image element for navigation indication may be generated in the corresponding projection area 50. For example, multiple parking lot entrances and exits in a shopping mall cannot be effectively projected because the map data does not indicate such detailed data. This "last hundred meters blind spot" in navigation often causes great confusion for drivers and can even contribute to traffic congestion. For example, location information with personal identification characteristics that is bound to a user, such as someone's residence, is difficult to maintain comprehensively through map data because it does not have public attributes. Consequently, it will not be presented with relevant directional information in the projection area 50. The only alternative is to use landmark buildings maintained in the map data. However, in actual applications, drivers are often unaware of these alternative, publicly verifiable location names.

[0100] In some cases, drivers who rely solely on navigation guidance without knowing the specific route may encounter the aforementioned problems. At complex intersections, the lack of clear navigation instructions can cause hesitation, and sometimes distraction leading to overlooking potential hazards and causing traffic accidents. Conversely, having other passengers in the vehicle, such as a passenger in the front seat who is familiar with the road conditions, can provide assistance to the driver to some extent. However, in such scenarios, guidance largely relies on passengers' verbal expressions. Based on the relevant information transmission mechanisms, without visual aids, relying solely on verbal expression can easily lead to misunderstandings due to unclear concepts. This is especially true when passengers and drivers use completely different expressions for the same concept, or when a concept is not accurately described, resulting in ambiguity. For example, in "turn at this intersection later," the duration of "later" lacks a precise definition and may vary depending on the individual. "This intersection" is highly ambiguous, and the more intersections ahead, the greater the ambiguity. This is particularly true in complex urban road sections with multiple intersections, such as overpasses and junctions, where accurate information cannot be obtained through this method of guidance. Accordingly, while passengers are giving verbal directions, a "demonstration whiteboard" that allows for written expression can be provided, thus eliminating the ambiguity conveyed by verbal communication. Figure 4As shown, the aforementioned "demonstration whiteboard" shared by the driver and passengers is achieved through interaction between the vehicle's HUD display device and the passenger's mobile terminal. Ultimately, written information from the mobile terminal can be displayed in the projection area corresponding to the HUD display device, providing the driver with a more intuitive understanding of the content. Specifically, the HUD display device can establish a connection with the mobile terminal, enabling information communication between the two. This connection can be based on communication protocols such as Bluetooth or WiFi. Optionally, if the HUD display device relies on the vehicle's infotainment system, the connection can also be established between the mobile terminal and the system via Bluetooth or WiFi. It should be noted that the information interaction between the HUD display device and the mobile terminal in this context does not explicitly distinguish between their respective functions. In Bluetooth or WiFi communication connections, the connection can be automatically established based on the existing pairing relationship between the HUD display device and the mobile terminal, or it can be supported through appropriate authorization management.

[0101] As described above, the HUD display device supports projecting desired virtual image elements in front of the vehicle's windshield. These virtual image elements can be integrated with the real-world scene in front of the vehicle as needed. Optionally, the HUD display device supports a certain level of real-world scene recognition and 3D reconstruction capabilities, so that the projected virtual image elements appear as if they already exist in the corresponding real-world scene. Correspondingly, based on the interconnections, the HUD display device can send the currently displayed or pre-rendered projection content or related parameters to the mobile terminal. The hardware structure of the HUD display device can be referenced... Figure 14The mobile terminal, taking a mobile phone as an example, can include output devices such as a screen for viewing content projected onto the HUD display device and serving as control input for virtual image editing. It can also include input devices such as a gyroscope and a touchscreen for inputting content to be projected onto the HUD display device. In a specific example, if the virtual image element to be projected is a navigation arrow, it has high requirements for its alignment and direction. Therefore, the parameters of the projected navigation arrow and its alignment with the real scene can be synchronized to the mobile terminal. When there is a deviation, the mobile terminal can make directional adjustments, thus compensating for the shortcomings of relying solely on sensing devices and map data. Correspondingly, the mobile terminal can send first auxiliary information to the HUD display device based on the received adjustment results. This first auxiliary information can take various forms and can be standardized between the HUD display device and the mobile terminal. For example, it can be descriptive text in XML, JSON, or other formats, which can record the relative alignment offset correction parameters and / or direction rotation correction parameters after input by the mobile terminal, or even absolute coordinate values ​​and spatial attitude parameters in a specific coordinate system. For example, the first auxiliary information could be a marked image, which more intuitively records the alignment position and direction of the virtual image element in the real-world image. It could even be related rendering data, which can be transformed and mapped in three-dimensional coordinate systems using projection matrix parameters. Another example is an information link. The content linked to can be managed by a specific entity or supported by multiple users for shared editing, making it particularly suitable for building a user-created content platform for internet applications. The HUD display device can then update the projection display of the corresponding virtual image element based on this first auxiliary information, for example, by correcting... Figure 2 The alignment or correction of the navigation arrow in the middle Figure 3 The navigation arrows indicate the direction, thus correctly matching the real scene to achieve intuitive navigation, which will be explained in detail below.

[0102] In some examples, such as Figure 5As shown, when the driver needs passenger assistance in guiding the driving route, the passenger can open specific software on the mobile terminal 60, which triggers a connection with the in-vehicle HUD display device. Accordingly, the mobile terminal 60 can receive the projected video stream synchronized with the HUD display device and display it on its screen, achieving the same projection effect as the projection area 50. The navigation arrow 601 corresponds to the navigation arrow 501 in the projection area 50. The relative position of the navigation arrow 601 in the projection area 50 can be determined by its relative position on the mobile terminal 60 screen, thus inferring whether the navigation arrow 501 correctly aligns with the actual scene ahead. While synchronizing the projection display with the HUD display device, the mobile terminal 60 also automatically enters an editing mode, where the navigation arrow 601 supports external input adjustments. Users holding mobile terminal 60 can drag the navigation arrow 601 directly on the screen or control movement using directional operation keys. The content of their operation can be recorded as first auxiliary information and sent to the HUD display device. For example, if the navigation arrow 601 is shifted to the right by a certain distance, it will be received by the HUD display device. Based on the ratio between the projection area 50 and the screen of mobile terminal 60, the HUD display device will proportionally control the navigation arrow 601 to move a certain distance to the right in the projection area 50, thus enabling convenient navigation guidance.

[0103] In some examples, when the intersection pointed to by navigation arrow 501 needs correction, the direction of navigation arrow 601 can be directly dragged on the mobile terminal 60 to achieve correction. The corresponding rotation correction parameters can be sent to the HUD display device through the first auxiliary information. The HUD display device can adaptively adjust the direction of navigation arrow 501 to align with the correct intersection. Optionally, when adjusting the direction of navigation arrow 601, the mobile terminal 60 can also utilize sensor devices such as a gyroscope in the mobile terminal 60. Figure 6As shown, the user can rotate the mobile terminal 60 to synchronously adjust the navigation arrow 601 based on the direction sensed by the gyroscope, etc. The corresponding rotation correction parameters are also recorded to notify the HUD display device to adjust the navigation arrow 501 accordingly. In this example, rotating the mobile terminal 60 counterclockwise triggers the navigation arrow 601 to adjust to the left, and rotating it clockwise triggers it to adjust to the right. Optionally, the rotation input at the bottom of the diagram can also be used, which rotates around an axis 612 perpendicular to the ground. This directly changes the orientation of the mobile terminal 60; turning it to the left adjusts the navigation arrow 601 to the left, and turning it to the right adjusts it to the right. Preferably, as the orientation of the mobile terminal changes, the camera of the mobile terminal can be activated to identify the facing real-world area, thereby determining the adjustment direction and degree of the navigation arrow 601 based on the captured real-world area. Accordingly, the rotation angle of the mobile terminal 60 and the adjustment angle of the navigation arrow 601 have a certain mapping relationship; the larger the rotation angle of the mobile terminal 60, the larger the adjustment angle of the navigation arrow 601. Optionally, the mobile terminal 60 will only distinguish multiple rotation states based on the rotation angle, each corresponding to a different selectable adjustment direction of the navigation arrow 601. Furthermore, the multiple selectable adjustment directions of the navigation arrow 601 correspond to multiple intersections in the real-world view detected by the HUD display device or the mobile terminal. The rotation of the mobile terminal 60 triggers a directional selection among these intersections. In a specific example, in edit mode, to prevent accidental triggering caused by the rotation of the mobile terminal 60, the mobile terminal 60 does not directly respond to the rotation of the mobile terminal 60 to initiate the adjustment of the navigation arrow 601. Instead, it sets a first time threshold. Only when the entire process of the mobile terminal 60 rotating from its initial state to a first specific angle and then returning to its initial state is perceived within the first time threshold will the extraction of the first specific angle be triggered as the basis for adjusting the navigation arrow 601. If, within the first time threshold, the user turns to both a first specific angle and a second specific angle between the initial state and the point of return to the initial state (e.g., turning left once and then right once), the second specific angle of the last turn will be used as the basis for adjusting navigation arrow 601. For example, in response to the last right turn, the direction of navigation arrow 601 will also be adjusted to the right. However, if an adjustment is made within the first time threshold but the user never returns to the initial state, it will not be considered a user-initiated adjustment input, and navigation arrow 601 will not respond to the adjustment.

[0104] In some examples, such as Figure 7As shown, the mobile terminal 60 interacts with the HUD display device. When the HUD display device projects the navigation arrow 501, the mobile terminal 60 displays the corresponding navigation arrow 601 on its screen, simultaneously reproducing the real-world view corresponding to the projection area 50. This provides a clear view of the driver's perspective, allowing them to see the alignment between the navigation arrow 601 and the reproduced real-world view 611 on the screen. This facilitates adjustments on the mobile terminal 60. Specific adjustment methods can be found in [reference needed]. Figure 5 , Figure 6Example. Specifically, in editing mode, in addition to receiving the projected content provided by the HUD display device, the mobile terminal 60 also receives a video stream of the real-time scene captured by the vehicle's camera. This video stream, based on the spatial correspondence of the projection area 50, only extracts the portion of the real-world scene directly opposite the projection area 50. Therefore, when displayed on the mobile terminal 60 screen, the alignment and direction of the navigation arrow 601 relative to the replicated real-world scene 611 can be roughly and completely seen. Optionally, the replicated real-world scene 611 on the mobile terminal 60 screen does not necessarily rely on the real-time capture from the vehicle's camera; it can also be displayed based on a 3D model from map data. To accurately reflect the alignment between the navigation arrow 501 and the real-world scene in real space, and to demonstrate the precision of actual alignment, when displaying both the real-world scene 611 and the navigation arrow 601 simultaneously on the mobile terminal 60 screen, the model undergoes further processing based on the coordinate system determined between the human eye and the 3D model before rendering. Specifically, the 3D model is mapped to the corresponding world space coordinate system using an object-to-world matrix. Then, based on the projection matrix parameters of the human eye, the positions of the 3D model in world space and the virtual image element 501 in the projection area 50 are uniformly mapped to the corresponding screen space. The image determined in the screen space can then be displayed on the mobile terminal 60 screen, thus reproducing the realistic alignment effect. It should be noted that the projection matrix parameters include the camera extrinsic and intrinsic parameters used in 3D systems such as Unity. The camera extrinsic parameters include rotation matrices, translation vectors, etc., while the camera intrinsic parameters include x / y axis focal length, principal point coordinates, inter-axis tilt factors, etc. In this example, the camera represents the approximate intrinsic and extrinsic parameters of the human eye. Furthermore, in order to enhance the user's realistic experience of editing virtual image elements, the real scene reproduced on the mobile terminal 60 is obtained by real-time shooting by the camera on the mobile terminal 60. Accordingly, when the mobile terminal 60 displays it on the screen, it also needs to perform corresponding coordinate system transformation and processing of navigation arrow 601 to restore the actual position of the projected virtual image element in the real scene. When the user uses the mobile terminal 60 to shoot and view the real scene in front of them, it is as if there is a corresponding navigation arrow 601 on the road.Therefore, similar to rendering based on a 3D model, it needs to receive rendering data of the projected virtual image element 501 sent by the HUD display device. This data can include the coordinate system of the virtual image element 501 in space when it is projected, as well as the projection matrix parameters used for rendering. After receiving these parameters, the mobile terminal 60 needs to use display elements representing the virtual image element 501 (such as the navigation arrow 601) to intuitively simulate the alignment of the projection with the real scene. Therefore, it will perform coordinate system transformation on the projection rendering data to map it to the coordinate system captured by the mobile terminal 60's camera. That is, it uses the projection matrix parameters of the mobile terminal 60's camera to uniformly transform it into the corresponding screen space, so that it is in the same screen space as the real scene image captured by the mobile terminal 60's camera. At the same time, the user can adjust the navigation arrow 601 on the image captured in real time by the mobile terminal 60 to match the correct navigation instructions. Based on the above, the adjustment results input by the user as a passenger are recorded to generate first auxiliary information. The first auxiliary information can be the fitting position correction parameters and the direction rotation correction parameters in the example above. However, in order to improve accuracy, the absolute position in the coordinate system determined by the mobile terminal 60 and the corresponding projection matrix parameters can also be sent as the first auxiliary information to the HUD display device. The HUD display device can redetermine its position in the projection space based on these data, thereby achieving the same fitting relationship as displayed on the mobile terminal 60.

[0105] Optionally, the content displayed on the mobile terminal 60 may not necessarily be the current projection content of the projection area 50, but could also be a forward view position that the projection area 50 is not currently facing. Since the content needs to be projected as the vehicle moves forward, adjustments can be made in advance on the mobile terminal 60. See [link / reference] for details. Figure 7 For example, the mobile terminal 60 displays the pre-rendered navigation arrow positions and their corresponding 3D models, or references... Figure 8For example, the camera of mobile terminal 60 is pointed at the real scene outside the projection area 50 to check whether the subsequent projection display meets the requirements. It also combines the pre-rendered navigation arrow to determine whether it is in the correct alignment position or direction. The corrected data is sent to the HUD display device for storage and is retrieved when the projection area 50 is aligned with the position. Optionally, the adjusted navigation arrow 501 is not necessarily the projection content currently or subsequently configured in the projection area 50. For example, due to the lack of map data support, no guiding navigation arrow 501 will be generated at intersections in remote areas. In this case, the navigation arrow to be edited can be automatically displayed on the editing mode interface of mobile terminal 60 through intersection recognition or triggering of ambiguous paths during navigation. The user can adjust the navigation arrow on the mobile terminal. When the adjustment is completed, the first auxiliary information is sent to the HUD display device. The HUD display device can respond to the first auxiliary information to generate the corresponding navigation arrow 501 in the projection area 50 to form an alignment relationship with the corresponding intersection ahead, realizing navigation guidance independent of map data.

[0106] Regarding the first auxiliary information, the alignment position offset correction parameters and indication direction rotation correction parameters formed by the user's operation on the mobile terminal may have deviations in the projection transformation of the HUD display device. Furthermore, using coordinate system transformation based on projection matrix parameters for the first auxiliary information increases the processing power of the 3D transformation. Moreover, the determination of the camera projection matrix parameters on the mobile terminal 60 itself also introduces errors. Therefore, a more easily unified first auxiliary information is needed between the HUD display device and the mobile terminal. In some examples, the user can use a more direct adjustment method through the mobile terminal 60, such as... Figure 9 The system needs to guide vehicles to a specific parking lot entrance 71, but this entrance 71 cannot be represented in a normal projection based on the missing map data. Therefore, no virtual image elements for guidance will be projected in the projection area 50. Users can mark the entrance using the camera of a mobile terminal 60, such as... Figure 10As shown, the camera can be aimed at the parking entrance 71 to be marked, and a corresponding marked image can be taken and sent to the HUD display device as the first auxiliary information. The HUD display device can then perform similarity matching between the real scene in front obtained by its own camera and the marked image to find the fitting position of the corresponding virtual image element to be projected. Correspondingly, it will also determine the three-dimensional coordinates of the fitting position with a high degree of matching based on the position information of the real scene in front obtained by itself. Thus, when correcting the projected virtual image element, the position of the projection area 50 in space can be determined according to the projection parameters of the HUD display device, and the corresponding virtual image element can be inversely mapped to the projection area 50. The position of the projection area 50 is also mapped to the display position on the image source display surface of the HUD display device, so the projection can be easily controlled to ensure the effect of virtual and real matching. Optionally, during the process of taking a picture of the parking entrance 71, the user can also frame the corresponding parking entrance 71 on the screen of the mobile terminal 60 to improve the accuracy of selection and recognition, and at the same time, the matching weight within the framed area will be increased during similarity matching. Furthermore, semantic analysis-based intelligent models can be used to assist users with their initial input of auxiliary information. These models can analyze the content of the photograph to identify the most meaningful target object, such as recognizing a parking lot entrance, which is the most relevant destination for the current vehicle. Therefore, the bounding box is automatically optimized to match the outline of the parking lot entrance. For example... Figure 11 As shown, after determining the corresponding fitting position and direction based on the similarity matching of the marked image, the projected virtual image elements are not limited to the navigation arrow. They can also project a corresponding highlighted sign 710 based on the outline of the parking lot entrance 71. The highlighted sign 710 can increase the alertness by flashing, making it easy for drivers to understand the location of the parking lot. In more examples, when the mobile terminal 60 sends the first auxiliary information, including the marked image, to the HUD display device, it can also integrate the projection matrix parameters of the camera image into the first auxiliary information and send it along with the image. The HUD display device can determine the specific spatial location of the parking lot entrance 71 not only based on similarity matching but also by combining three-dimensional coordinate system transformation; this will not be elaborated further here.

[0107] In some examples, the same approach can be used. Figure 10 Examples are used to correct existing navigation arrows, etc. Figure 12As shown, the navigation arrow 501 is not projected in the correct position. In this case, the user can take a picture of the location where the navigation arrow 501 needs to be correctly aligned, such as a right-turn intersection, using the mobile terminal 60. As described above, the captured image is sent to the HUD display device to determine the correct alignment position based on similarity matching. Referring to the example above, while taking the picture, the user can also select a bounding area 602, such as drawing a circle on the screen with their finger. This allows for precise determination of the alignment position. Preferably, semantic analysis can be used to determine the position of lane lines to fine-tune the bounding area 602, thereby aligning it with a specific lane. Furthermore, in edit mode, the mobile terminal 60 supports simultaneous annotation of the alignment position and the direction of indication. For example, after completing the annotation of the alignment position, the display of the bounding area 602 can be automatically canceled, allowing the user to operate again on the mobile terminal 60 screen. The direction of indication can be determined by the direction from the bounding area 602 pointing to the new bounding area. Optionally, the two selection operations can also be switched using buttons. For example, the screen can simultaneously display a first button for adjusting the fit position and a second button for adjusting the direction. Responding to the triggering of the first button will select the fit position, and responding to the triggering of the second button will select the direction. Meanwhile, refer to... Figure 8 For example, when taking a picture to obtain a marked image, display elements can also be presented at the corresponding real-world location to reflect the incorrect virtual image elements in the projection.

[0108] The first auxiliary information upon which the HUD display device bases its projection correction can be stored in its own memory and retrieved the next time it reaches the same road. Furthermore, the first auxiliary information can be sent by multiple users' mobile terminals, and can be edited by different vehicles passing the same road at different times. The HUD display device can comprehensively determine the projection display of the corresponding virtual image element based on the first auxiliary information provided by multiple users. Optionally, when many users provide the first auxiliary information, popularity can be determined based on the similarity of different first auxiliary information, with the most popular information often being the most accurate. Alternatively, different first auxiliary information can be evaluated on a network platform and ranked according to the evaluation, with the most popular information often being the most accurate. In many other examples, the first auxiliary information exists in the form of an information link pointing to a specific server. Access to the server allows users to obtain the corresponding intersection adjustment content, and this information link also supports shared access by multiple users, enabling one person to edit while multiple people directly use it, greatly improving convenience. Optionally, the content editing also supports multiple users to access and modify it, but to ensure content controllability, certain permissions can be set, allowing only a specified range of people or those with high credibility to edit. As mentioned above, using information links as primary auxiliary information is particularly suitable for merchants to promote their businesses. Merchants can maintain their business information on relevant internet applications. Alternatively, bloggers who review restaurants can create such information links, which also support public rating and ranking on internet applications. Users can find these available information links within the internet application. Specifically, this could be a restaurant at a destination determined by the user before or during the journey. The user sends a specific information link to the HUD display device, which directly accesses these links to generate virtual image elements, projecting clear navigation instructions when the vehicle arrives at the destination. The server content pointed to by the information link can include introductions to specific merchants, as well as embedded descriptive text or marker images. For example, when maintaining this information link, the merchant can include marker images of the entrance and exterior of the shopping mall where the merchant is located. The HUD display device can automatically extract and process this content upon access. See [link to relevant documentation] for details. Figure 10 Examples, such as... Figure 13As shown, for a HUD display device that receives the corresponding information link, a first annotation symbol 503 will be projected in the projection area 50 according to preset rules. For example, it can mark the specific location in a building. This can be determined by analyzing the marked image pointed to by the information link, or by semantic analysis based on the specific floor location description maintained. Specifically, it can also include a second annotation symbol 504 pointing to the location of the corresponding merchant, telling how to reach the corresponding merchant through text or other means, such as prompts like "Go to Exit A and go up to the 2nd floor to eat," or other notes. The content can also be extracted from the server pointed to by the information link.

[0109] When a HUD display device implementing the virtual image editing method described above is applied to an in-vehicle system, the virtual image elements projected by the HUD display device, especially navigation instructions that are projected with a virtual-real fit to the real scene, can have their specific instructions corrected through convenient input on a mobile terminal, compensating for the deficiencies of existing real-scene recognition and map data. For example, Figure 14 As shown, the HUD display device integrated in the vehicle can be powered and fed by the vehicle's infotainment system 92, or it can be powered and generate data by the HUD display device itself. Specifically, the HUD display device may include a processor 91, an Ethernet interface 901, a CAN interface 902 (Controller Area Network), a power management module 903, running memory 904, storage memory 905, a temperature detection module 906, a motor 907, a backlight 908, an image source 909, a positioning module 910, radar 911, and a camera 912, etc. It should be noted that... Figure 14 The modules listed herein are merely illustrative and do not constitute any limitation. In some examples, the HUD display device may also include other modules. Furthermore, the aforementioned modules may be implemented in one or more hardware components in different examples, or a single module may be implemented by a combination of multiple hardware components. Optionally, the positioning module 910, radar 911, and camera 912 may be directly connected to the vehicle infotainment system 92, without being directly connected to the processor 91 of the HUD display device. For example, the vehicle infotainment system 92 itself integrates a positioning module for location tracking and radar and cameras for autonomous driving. The HUD display device can then acquire the data collected by the positioning module, radar, and camera in real time through communication with the vehicle infotainment system 92.

[0110] The processor 91, serving as the control center of the HUD display device, includes one or more processing units of any type, including but not limited to microcontrollers, microcontrollers, DSPs (Digital Signal Processors), or any combination thereof. The processor 91 generates operation control signals according to a computer program to control other modules and cooperate with corresponding modules to process acquired or inherent data and instructions.

[0111] Ethernet interface 901 is a network data connection port for local area network communication. It defines a series of software and hardware standards. Multiple electronic devices can be connected together through Ethernet interface 901. In this example, processor 91 can interact with vehicle infotainment system 92 through Ethernet interface 901, such as sending data to vehicle infotainment system 92 or receiving data sent by vehicle infotainment system 92.

[0112] The CAN interface 902 is a network data connection port for the Controller Area Network (CAN), providing a standard bus for automotive control systems and embedded industrial control systems, enabling communication and interaction between control nodes. In this example, the processor 91 can also interact with the vehicle's infotainment system 92 via the CAN interface 902. Optionally, the processor 91 can also connect to other external devices via the CAN interface 902. In some examples, the processor 91 may also be equipped with a GPIO (General-purpose input / output) interface to improve the compatibility of peripheral connections.

[0113] The power management module 903 is connected to the vehicle head unit 92 and can receive power from the vehicle head unit 92 to provide regulated power to the various modules of the HUD display device, ensuring that the processor 91 and various modules work under normal voltage supply and avoiding damage under overvoltage.

[0114] The running memory 904 is used to store the computer program executed by the processor 91, as well as temporarily stored calculation data and data exchanged with the storage memory. The running memory 904 can be a memory such as SDRAM (Synchronous Dynamic Random-access Memory).

[0115] Storage memory 905 is used to store resources such as display content of the HUD display device, as well as long-term stored running programs and data. Storage memory 905 can be a flash memory or other similar storage device. In some examples, processor 91 may also provide an interface to access external storage.

[0116] The temperature detection module 906 is used to detect the temperature inside the HUD display device. Specifically, it may include several temperature sensors. Since the resistance of the temperature sensors changes with temperature, the processor 91 can determine the resistance value of each temperature sensor at a corresponding temperature based on the voltage change between each temperature sensor and a voltage divider resistor under a fixed power supply voltage, thereby deducing the temperature at the location of the temperature sensor. In some examples, the processor 91 can control several temperature sensors via a GPIO interface. These temperature sensors can be placed at different locations inside the HUD display device, and the processor 91 can use time-division multiplexing to acquire the temperature values ​​fed back by each temperature sensor.

[0117] Motor 907, under the control of processor 91, drives the optical lenses in the HUD display device to rotate, thereby changing the corresponding optical path. For example, when sunlight backflow causes the image source surface to heat up, the motor can drive the optical lenses to prevent external sunlight from reaching the image source surface. In some examples, processor 91 can also drive a fan on the HUD display device via motor 907 to increase the speed of air exchange between the inside and outside of the HUD display device for heat dissipation. Specifically, motor 907 is connected to processor 91 through a motor driver chip, which provides high-performance power output to motor 907 and can also communicate and control processor 91 through interfaces such as SPI (Serial Peripheral Interface).

[0118] The backlight 908 provides illumination light and adjusts its brightness according to the control of the processor 91, thereby adjusting the projection display brightness of the entire HUD display device. The backlight 908, in conjunction with the image source 909, realizes the main functions of optical-engine projection display. Specifically, the backlight 908 is connected to the processor 91 via a backlight driver chip. The backlight driver chip provides a driving voltage to the backlight 908 and controls its brightness under the pulse width signal output by the processor 91.

[0119] Image source 909 is used to display images of corresponding content and project display light corresponding to the image according to the control of processor 91. Taking image source 909 as an LCD as an example, image source 909 includes liquid crystals corresponding to several pixels. The liquid crystals can rotate under the control of an electric field, thereby changing the direction of light travel and the color presented. When the illumination light emitted by backlight 908 reaches image source 909, the rotation direction of the liquid crystal determines the transmission mode of the illumination light, thereby producing different images, that is, emitting display light containing different display information.

[0120] The positioning module 910 is used to monitor the position of the HUD display device and the corresponding vehicle. The positioning module 910 can be a global navigation satellite system such as GPS (Global Positioning System) or BeiDou Navigation Satellite System. By measuring the distance between the satellite and the receiver on the positioning module 910 at different locations, it determines the corresponding position and orientation data. In some examples, the positioning module 910 may also include an inertial navigation system. Based on Newton's laws of motion, it measures the acceleration of the positioning module 910 in the inertial reference frame, integrates it over time, and transforms it to the navigation coordinate system to obtain data such as velocity, yaw angle, and position in the navigation coordinate system. Optionally, the inertial navigation system can assist the global navigation satellite system in achieving more accurate positioning, providing the processor 91 with the corresponding position information.

[0121] Radar 911 is used to determine the position of a target object by using electromagnetic waves, and can usually determine the distance between the target object and the vehicle where the radar 911 is located.

[0122] Camera 912 includes a vehicle body camera and an in-vehicle camera. The vehicle body camera is used to determine the position of a target object through visual recognition. The vehicle body camera can be a monocular camera or a binocular camera. The biggest difference between a monocular camera and a binocular camera is that a binocular camera can capture images from two different perspectives, thereby obtaining distance information in three-dimensional space. The in-vehicle camera is used to identify the behavioral state of the driver and passengers inside the vehicle, including fatigue detection, distraction detection, facial expression recognition, gesture recognition, and eye tracking. In this example, the in-vehicle camera can also specifically implement eye tracking.

[0123] In some examples, such as Figure 15 As shown, the display device implementing the above navigation display method may specifically include a processor 931, a memory 932, an input device 933, and an output device 934. The input device 933 may include operation buttons integrated on the display device, and the display device can receive input control commands and data through the input device 933. The output device 934 may include an image source integrated on the display device, and the display device can output corresponding commands or data to the output device 934. Further, the memory 932 stores a computer program running on the processor 931. When the processor 931 executes the computer program, it implements the virtual image editing method of the above example. In some examples, a computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the virtual image editing method of the above example.

[0124] like Figure 16As shown, vehicles can be equipped with the aforementioned HUD display device. Specifically, the HUD display device is integrated inside the center console 10, for example, in front of the steering wheel. The HUD display device projects corresponding display light onto the windshield 4 directly opposite it through its projection window 102. Viewers observing from inside the cockpit can directly see the virtual image in the projection area 50. The virtual image elements are not absolutely fixed; they can be temporarily modified or permanently modified using a mobile terminal inside the vehicle, thus achieving comprehensive and accurate navigation instructions. In some examples, the vehicle can also distribute the program for the virtual image editing method described above through the computer-readable storage medium, enabling convenient updates and upgrades to the vehicle's HUD display device. It should be noted that the aforementioned vehicles are not limited to cars used for transportation; they can also include buses, trucks, excavators, motorcycles, trains, high-speed trains, ships, yachts, airplanes, spacecraft, etc. The windshield for projection is not limited to the windshield of a car; it can also be a transparent surface in other locations.

[0125] Based on the above examples, the technical solutions involved in this application can be directly embodied in hardware, software modules executed by a control unit, or a combination of both, i.e., one or more steps and / or combinations of one or more steps. These can correspond to various software modules in a computer program flow, or to various hardware modules, such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof. For ease of description, the above description divides the functions into various modules and describes them separately. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware components.

[0126] Through the above description of examples, those skilled in the art can clearly understand that this application can be implemented using software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution involved in this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This software is executed by a microcontroller unit and, depending on the required configuration, can include one or more microcontroller units of any type, including but not limited to microcontroller units, microcontrollers, DSPs (Digital Signal Processors), or any combination thereof. The software is stored in memory, such as volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory, flash memory), or any combination thereof.

[0127] In summary, this application leverages the ease of operation of mobile terminals to access the virtual space where the projected virtual image of the HUD display device resides. Based on the mobile terminal user's assessment of the real-world scene, the virtual image elements within this virtual space are edited. The edited results are simultaneously applied to the projected object on the HUD display device, ensuring a seamless integration of virtual and real-world elements during projection and thus providing effective navigation guidance. This application enhances the navigation capabilities of the projected virtual image on the HUD display device, particularly strengthening the accuracy of guidance information in complex road conditions and optimizing the augmented reality navigation experience.

[0128] It should be understood that although this specification includes some examples, none of these examples constitutes a single, independent technical solution. This descriptive style is merely for clarity. Those skilled in the art should consider this specification as a whole, and the technical solutions in the examples can be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0129] The detailed descriptions listed above are merely specific descriptions of feasible implementation methods of this application and are not intended to limit the scope of protection of this application. All equivalent implementation methods or modifications that do not depart from the teachings of this application should be included within the scope of protection of this application.

Claims

1. A virtual image editing method, characterized in that, include: HUD display devices support projecting a first virtual image element in front of the vehicle windshield, the first virtual image element being configured to cooperate with a first real scene in front of the vehicle to provide directional information. The HUD display device establishes a connection with the mobile terminal and sends the current projection state of the first virtual image element to the mobile terminal so as to reproduce the first real scene and the first display element corresponding to the first virtual image element on the screen of the mobile terminal. The first display element reproduces the relative relationship between the first virtual image element and the first real scene. In response to a passenger user's visual adjustment operation on the first display element based on the first real scene and the first display element reproduced on the mobile terminal, the mobile terminal generates first auxiliary information indicating the need to correct the fitting position and / or direction of the first virtual image element relative to the first real scene. The HUD display device receives the first auxiliary information and controls the projection display of the first virtual image element according to the first auxiliary information to correct the matching relationship between the first virtual image element and the first real scene.

2. The virtual image editing method according to claim 1, characterized in that, The first auxiliary information is standard format information representing the fitting position and / or indicating direction of the first virtual image element relative to the first real scene, and the standard format information is at least one of descriptive text, marked image, and information link.

3. The virtual image editing method according to claim 2, characterized in that, When the first auxiliary information is a marker image, the HUD display device determines the fitting position of the first virtual image element by performing similarity matching between the marker image and the real scene captured by the vehicle camera.

4. The virtual image editing method according to claim 1, characterized in that, The HUD display device controls the projection display of the first virtual image element according to the first auxiliary information, including: When it is determined from the first auxiliary information that the fitting position of the first virtual image element relative to the first real scene needs to be corrected, the first virtual image element is inversely mapped to the projection area according to the projection parameters of the HUD display device, so that the fitting position of the first virtual image element matches that of the first real scene.

5. A virtual image editing method, characterized in that, include: The mobile terminal establishes a connection with the HUD display device and triggers editing mode; The system receives the current projection state of the first virtual image element sent by the HUD display device, and reproduces the first real scene in front of the vehicle and the first display element corresponding to the first virtual image element on the screen of the mobile terminal according to the projection state. The first display element reproduces the relative relationship between the first virtual image element projected by the HUD display device and the first real scene. The first virtual image element is the indication information projected by the HUD display device in front of the vehicle windshield that matches the first real scene. In response to a passenger user’s visual adjustment operation on the reproduced first display element on the mobile terminal, first auxiliary information is generated indicating the need to correct the fitting position and / or direction of the first virtual image element relative to the first real scene. The first auxiliary information is sent to the HUD display device so that the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information, so as to correct the matching relationship between the first virtual image element and the first real scene.

6. The virtual image editing method according to claim 5, characterized in that, The visual adjustment operation includes the passenger user dragging the first display element on the mobile terminal screen or rotating the mobile terminal body.

7. The virtual image editing method according to claim 5, characterized in that, The visual adjustment operations include: The first auxiliary information is generated by capturing at least a portion of the first real scene using a camera on the mobile terminal to mark the position and / or direction of the first virtual image element to be attached on the captured image.

8. A display device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor. The communication interface includes an Ethernet interface and a CAN interface. It can establish a communication connection with a mobile terminal or a vehicle-mounted system via Bluetooth or WiFi. When the processor executes the computer program, it implements the steps of the virtual image editing method according to any one of claims 1-4, or implements the steps of the virtual image editing method according to any one of claims 5-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the virtual image editing method according to any one of claims 1-4, or implements the steps of the virtual image editing method according to any one of claims 5-7.

10. A means of transportation, characterized in that, Includes the display device of claim 8 or the computer-readable storage medium of claim 9.

Images