Virtual image editing method, display device, storage medium and vehicle

Abstract

The invention relates to the technical field of projection display, in particular to a virtual image editing method, display equipment, a storage medium and a vehicle. The virtual space where the projection virtual image of the HUD display device is located is called by using the advantage of convenient operation of the mobile terminal, the virtual image elements in the virtual space are edited according to the real scene judgment condition of a mobile terminal user, and the editing result is synchronized in the projection object of the HUD display device to take effect to meet the virtual-real fitting requirement during projection display, so that the user experience is improved. Therefore, effective navigation indication content is provided. According to the method, the navigation capability of the HUD display equipment in the projection virtual image can be improved, particularly the accuracy of indication information in a complex road section is enhanced, and the navigation experience of augmented reality is optimized.

Description

Technical Field

[0001] The present application relates to the field of projection display technology, and in particular to a virtual image editing method, a display device, a storage medium, and a vehicle. Background Art

[0002] HUD (Head Up Display) is a new way of realizing in-vehicle display by using reflection on the vehicle windshield. Specifically, the display light is emitted by the optical machine of the HUD display device, and is projected onto the windshield through the corresponding optical lens to produce a corresponding virtual image, which forms an enhanced display effect with the real world outside the windshield. Among them, the virtual image elements used for navigation instructions are matched with the real scene, which can greatly improve the intuitiveness of navigation. However, the virtual image presentation of navigation instructions depends on the perception and processing capabilities of the vehicle's cameras, radars, etc. of the real scene ahead, as well as the accuracy of the navigation path in the map data and the precision of the navigation target. Once there is a deviation, the content of the navigation instructions will often be incorrect. Its virtual image elements do not match the matched real scene or the navigation instructions in key positions are missing. Not only does it fail to provide the driver with intuitive and clear instruction information, but it misleads the driver. Summary of the Invention

[0003] The purpose of this application is to provide a virtual image editing method, display device, storage medium and vehicle, which solves the technical problems in the prior art that the projected virtual image of the HUD display device is highly dependent on sensor resources and map data, cannot fully meet the user's navigation needs during driving, and the indication information of the virtual image at key positions is inaccurate or missing.

[0004] In order to solve the above technical problems, this application adopts the following technical solutions.

[0005] In a first aspect, the present application provides a virtual image editing method, comprising: The HUD display device supports projecting a first virtual image element in front of a vehicle windshield, wherein the first virtual image element is configured to cooperate with a first real scene in front of the vehicle to provide indication information; In response to a user inputting first auxiliary information of the first virtual image element relative to the first real scene through the mobile terminal, the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information to match the first real scene.

[0006] In an optional implementation of the first aspect, the HUD display device establishes a connection with the mobile terminal to support information interaction between the two.

[0007] In an optional implementation of the first aspect, the connection is established via Bluetooth or WiFi.

[0008] In an optional implementation of the first aspect, establishing a connection between the HUD display device and the mobile terminal to support information interaction therebetween includes: The HUD display device receives the first auxiliary information sent by the mobile terminal through the connection.

[0009] According to the above description, the optional implementation method can make the first virtual image element projected by the HUD display device no longer rigidly controlled by a fixed program, but can be finely adjusted as needed to adapt to the real scene in front of the vehicle and optimize the augmented reality effect.

[0010] In an optional implementation of the first aspect, the first virtual image element is a navigation arrow pointing to a specific intersection.

[0011] In an optional implementation of the first aspect, the first virtual image element is a highlighted mark indicating a specific entrance or exit.

[0012] In an optional implementation manner of the first aspect, the first virtual image element is a marking symbol indicating a specific building.

[0013] According to the above description, the optional implementation method can support the HUD display device to project and display first virtual image elements of various forms through flexible editing of the mobile terminal, without relying on the limited capabilities provided by sensor recognition and map data.

[0014] In an optional implementation of the first aspect, the first auxiliary information is standard format information indicating the fitting position and / or indication direction of the first virtual image element relative to the first real scene, and the standard format information is at least one of a descriptive text, a marked image, and an information link.

[0015] In an optional implementation of the first aspect, when the standard format information is the description text, the first virtual image element is adjusted according to the fitting position offset correction parameter and / or the indication direction rotation correction parameter recorded in the description text.

[0016] In an optional implementation of the first aspect, when the standard format information is the mark image, the first virtual image element is adjusted according to a fitting mark position and / or an indication mark direction specified in the mark image.

[0017] In an optional implementation manner of the first aspect, adjusting the first virtual image element according to a specified fit mark position in the mark image includes: The fitting position of the first virtual image element relative to the first real scene is determined according to the similarity between the fitting mark position and the first real scene.

[0018] In an optional implementation of the first aspect, when the standard format information is the marker image, the first virtual image element is adjusted according to the model rendering corresponding to the marker image and the projection matrix parameters used for the rendering.

[0019] In an optional implementation of the first aspect, when the standard format information is the mark image, the first virtual image element is adjusted according to the first real scene mark content corresponding to the mark image and the camera projection matrix parameters of the mobile terminal shooting the first real scene.

[0020] In an optional implementation manner of the first aspect, when the standard format information is the information link, a server corresponding to the information link is accessed, and the first virtual image element is adjusted according to content provided by the server.

[0021] In an optional implementation of the first aspect, the information link supports shared access by multiple people.

[0022] According to the above description, the optional implementation method supports multiple adjustments of the first virtual image element. The user can select and input according to actual conditions. The HUD display device supports adaptively using different forms to determine the adjustment content of the first virtual image element.

[0023] In an optional implementation manner of the first aspect, the responding to the user inputting, through the mobile terminal, first auxiliary information of the first virtual image element relative to the first real scene includes: The HUD display device sends the current projection state of the first virtual image element to the mobile terminal, so as to present an instruction question relative to the first real scene to the user on the mobile terminal.

[0024] In an optional implementation manner of the first aspect, the HUD display device sending the current projection state of the first virtual image element to the mobile terminal includes: The rendering data of the first virtual image element relative to the first real scene is used to provide spatial parameters when the first virtual image element is projected.

[0025] In an optional implementation manner of the first aspect, the HUD display device sending the current projection state of the first virtual image element to the mobile terminal includes: The HUD display device is to project advance rendering data of the first virtual image element to support the mobile terminal to edit the first virtual image element before projection.

[0026] According to the above description, an optional implementation method can send the complete rendering data to the mobile terminal, and the mobile terminal can reproduce the three-dimensional spatial relationship between the projected first virtual image element and the first real scene through coordinate system conversion, thereby supporting the user to edit the first virtual image element while previewing the editing effect, thereby improving user intuitiveness.

[0027] In an optional implementation of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes: When it is determined based on the first auxiliary information that the fitting position of the first virtual image element relative to the first real scene needs to be corrected, the first virtual image element is inversely mapped to the projection area where it is located according to the projection parameters of the HUD display device so that the fitting position of the first virtual image element matches the first real scene.

[0028] According to the above description, an optional implementation method can correct the problem of virtual and real fitting misalignment by directly changing the projection position of the first virtual image element based on the position mapping relationship between the projection area and the image source display surface. The user does not need to consider specific projection correction parameters when editing.

[0029] In an optional implementation of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes: The users include at least a first user and a second user, the first user inputs second auxiliary information of the first virtual image element relative to the first real scene via a first mobile terminal, and the second user inputs third auxiliary information of the first virtual image element relative to the first real scene via a second mobile terminal; The first auxiliary information is comprehensively determined based on the second auxiliary information and the third auxiliary information.

[0030] In an optional implementation of the first aspect, the HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes: The user also includes a third user, and the third user inputs fourth auxiliary information of the first virtual image element relative to the first real scene through a third mobile terminal, and the first auxiliary information selects the one with the highest similarity among the second auxiliary information, the third auxiliary information, and the fourth auxiliary information.

[0031] According to the above description, an optional implementation method utilizes user input data from multiple sources to determine specific parameters for matching the first virtual image element with the first real scene, thereby improving accuracy and compensating for the high cost and low refinement of map data production through a user-generated content model.

[0032] In a second aspect, the present application provides a virtual image editing method, comprising: The mobile terminal establishes a connection with the HUD display device and triggers the editing mode; In response to the user's input operation on the mobile terminal, first auxiliary information of a first virtual image element relative to a first real scene is sent to the HUD display device, so that the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information. The first virtual image element is the indication information projected by the HUD display device in front of the vehicle windshield to match the first real scene.

[0033] In an optional implementation of the second aspect, establishing a connection between the mobile terminal and the HUD display device includes: The mobile terminal establishes a connection via Bluetooth or WiFi.

[0034] In an optional implementation of the second aspect, the first virtual image element is a navigation arrow pointing to a specific intersection.

[0035] In an optional implementation of the second aspect, the first virtual image element is a highlighted mark indicating a specific entrance or exit.

[0036] In an optional implementation of the second aspect, the first virtual image element is a marking symbol indicating a specific building.

[0037] In an optional implementation of the second aspect, the first auxiliary information is standard format information indicating the fitting position and / or indication direction of the first virtual image element relative to the first real scene, and the standard format information is at least one of a descriptive text, a marked image, and an information link.

[0038] According to the above description, in an optional implementation mode, the user can use a mobile terminal to control the first virtual image element of the HUD display device by judging the real scene in front of the vehicle. The real-time performance and flexibility are relatively high, eliminating the ambiguity of verbal instructions given by non-drivers and enhancing the driving experience of the entire vehicle.

[0039] In an optional implementation of the second aspect, the responding to an input operation of a user on the mobile terminal includes: The first real scene and the first display element are reproduced on the screen of the mobile terminal, and the first display element is reproduced in relative relationship with the first real scene according to the state of the first virtual image element projected by the HUD display device; First auxiliary information for adjusting the first virtual image element is generated according to the user's adjustment of the first display element.

[0040] In an optional implementation of the second aspect, the reproduction of the first real scene on the screen of the mobile terminal is a three-dimensional model in the call map data.

[0041] In an optional implementation manner of the second aspect, the first real scene reproduced on the screen of the mobile terminal is a real-time image of the first real scene captured by the mobile terminal.

[0042] In an optional implementation of the second aspect, the first display element reproduces a relative relationship with the first real scene according to a state of the first virtual image element projected by the HUD display device, including: A first projection matrix parameter for rendering the first virtual image element by the HUD display device is received, and the display of the first display element on the mobile terminal screen is adjusted according to a second projection matrix parameter used for displaying and rendering the called three-dimensional model on the mobile terminal screen.

[0043] In an optional implementation of the second aspect, the first display element reproduces a relative relationship with the first real scene according to a state of the first virtual image element projected by the HUD display device, including: A first projection matrix parameter for rendering the first virtual image element by the HUD display device is received, and display of the first display element on the mobile terminal screen is adjusted according to a third projection matrix parameter of a camera used by the mobile terminal to capture the first real scene.

[0044] In an optional implementation manner of the second aspect, generating first auxiliary information for adjusting the first virtual image element according to the user's adjustment of the first display element includes: According to the dragging operation of the user on the screen of the mobile terminal, the content is recorded as a description text or frozen as a mark image.

[0045] In an optional implementation manner of the second aspect, generating first auxiliary information for adjusting the first virtual image element according to the user's adjustment of the first display element includes: According to the user's rotation operation on the mobile terminal body, it is recorded as a description text or frozen as a mark image.

[0046] According to the above description, an optional implementation method can change the fitting position and indication direction of the first display element relative to the reproduced first real scene through convenient user operations, thereby generating first auxiliary information with a reference basis for projection of the HUD display device.

[0047] In an optional implementation of the second aspect, the responding to an input operation of a user on the mobile terminal includes: At least a portion of the first real scene is photographed by a camera on the mobile terminal to mark a position and / or a direction to be indicated by the first virtual image element.

[0048] In an optional implementation manner of the second aspect, photographing at least a portion of the first real scene using a camera on the mobile terminal to indicate a position where the first virtual image element needs to be attached and / or a direction to be indicated includes: The fitting position and / or indication direction of the first virtual image element is determined by the user's selection of the corresponding real scene content on the mobile terminal screen.

[0049] In an optional implementation manner of the second aspect, determining the fitting position of the first virtual image element by the user selecting corresponding real scene content on the mobile terminal screen includes: By semantically analyzing the selected real scene content image, the outline of the corresponding real scene is marked.

[0050] According to the above description, an optional implementation method supports direct image capture to determine the first auxiliary information. The HUD display device can determine the fitting position and indication direction of the first virtual image element by judging the image similarity, so that the user can provide his own projection reference information.

[0051] In a third aspect, the present application provides a display device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the processor implements the steps of the virtual image editing method described in the first aspect or the second aspect when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the virtual image editing method described in the first aspect or the second aspect.

[0053] In a fifth aspect, the present application provides a vehicle comprising the display device described in the third aspect or the computer-readable storage medium described in the fourth aspect.

[0054] Compared to existing technologies, this application leverages the convenient operation of mobile terminals to call upon the virtual space where the HUD display device projects the virtual image. Based on the mobile terminal user's judgment of the real scene, the virtual image elements in the virtual space are edited. The edited results are then applied to the projected object on the HUD display device to meet the requirements for virtual-reality fit during projection, thereby providing effective navigation instructions. This application can improve the navigation capabilities of the virtual image projected by the HUD display device, particularly enhancing the accuracy of the instructions on complex roads and optimizing the navigation experience of augmented reality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the technical solution of this application, the following briefly introduces the drawings required for describing the technical solution. Obviously, the drawings described below are merely examples of the invention, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0056] Figure 1 Schematic diagram of HUD projection display in some examples of this application.

[0057] Figure 2 Schematic diagram of the position deviation of virtual image element projection fitting in the prior art.

[0058] Figure 3 This is a schematic diagram of the direction deviation of the virtual image element projection indication in the prior art.

[0059] Figure 4 This is a schematic diagram of the connection between the HUD display device and the mobile terminal in some examples of this application.

[0060] Figure 5 This is a schematic diagram of a mobile terminal editing virtual image elements in some examples of this application.

[0061] Figure 6 This is a schematic diagram of the rotation control input of a mobile terminal in some examples of this application.

[0062] Figure 7 Schematic diagram of a mobile terminal presenting display elements and replicating real scenes in some examples of this application.

[0063] Figure 8 In some examples of this application, a mobile terminal reproduces a schematic diagram of a real scene through real-time shooting.

[0064] Figure 9 Schematic diagram of parking lot entrance navigation scenario in some examples of this application.

[0065] Figure 10 This is a schematic diagram of inputting a mark image through a mobile terminal in some examples of this application.

[0066] Figure 11 Schematic diagram of the HUD display device projecting highlight logos in some examples of this application.

[0067] Figure 12 Schematic diagram of the fitting position of the marking image record in some examples of this application.

[0068] Figure 13 This is a schematic diagram of the projection annotation symbols of the HUD display device in some examples of this application.

[0069] Figure 14 Schematic diagram of the HUD display device module in some examples of this application.

[0070] Figure 15 This is a schematic diagram of the composition of the HUD display device in some examples of this application.

[0071] Figure 16 This is a schematic diagram of projection display in a vehicle in some examples of this application. DETAILED DESCRIPTION

[0072] The following will describe the present application in detail with reference to the accompanying drawings, but the description is merely some examples recorded in the present application and does not limit the present application. Any changes in structure, method or function made by ordinary technicians in this field based on these examples are included in the scope of protection of the present application.

[0073] It should be noted that the same reference numbers or indicia may be used in different examples, but these do not represent absolute structural or functional connections. Furthermore, the terms "first," "second," and so on, which may be mentioned in various examples, are merely for descriptive convenience and do not represent absolute structural or functional distinctions. They should not be construed as indicating or implying relative importance or the number of corresponding objects. Unless otherwise specified, the term "at least one" in the description refers to one or more, and "a plurality" refers to two or more.

[0074] Additionally, when representing features, the character " / " can indicate an OR relationship between the preceding and following objects. For example, heads-up display / head-up display can be represented as heads-up display or heads-up display. When representing operations, the character " / " can indicate a division relationship between the preceding and following objects. For example, magnification M = L / P can be represented as L (virtual image size) divided by P (image source size). Furthermore, the "and / or" in different examples is simply to describe the relationship between the preceding and following objects. This relationship can include three situations. For example, a concave mirror and / or a convex mirror can be represented as the concave mirror alone, the convex mirror alone, or the concave and convex mirrors simultaneously.

[0075] HUD projection display mainly uses the principle of optical reflection to reflect the imaging light to be displayed through a transparent surface into the viewer's eyes. The human eye can see the virtual image information along the opposite direction of the light. Accordingly, the transparent surface can be the windshield of the vehicle, and the windshield can be used as a display screen to display the navigation instructions of the vehicle, the vehicle's driving speed, etc. Figure 1As shown, the HUD display device may include at least an optical engine 1, a first reflector 2, and a second reflector 3. The optical engine 1 includes a backlight source and an image source (not shown). The backlight source is used to provide illumination and adjust the brightness of the illumination light according to control. For example, the backlight source may be an LED (Light Emitting Diode) or a laser. Under the illumination provided by the backlight source, the image source adjusts the corresponding display content according to control and projects the display light from the surface of the image source. For example, the image source may be an LCD (Liquid Crystal Display), a DMD (Digital Micromirror Device), a MEMS (Micro-Electro-Mechanical System) micromirror, or an LCOS (Liquid Crystal on Silicon). The first reflector 2 and the second reflector 3 can project the display light projected by the optical machine 1 onto the windshield 4, realizing customization of the optical path in a smaller space while meeting different projection display requirements. The first reflector 2 and the second reflector 3 can be set to a concave mirror, a convex mirror, a concave lens, a convex lens, etc. according to the requirements of optical planning, and the surface shape of the lens can be a free-form surface. Optionally, at least one of the first reflector 2 and the second reflector 3 can also be adjusted to a certain degree of angle, thereby changing the projection position of the display light on the windshield 4 to meet the needs of viewers of different heights. The display light of the optical machine 1 is finally reflected on the windshield 4 of the vehicle to form a virtual image 5. When the human eye 6 observes the virtual image 5 facing the windshield 4, it can feel a certain sense of depth, just like looking at a real object at a specific distance outside the windshield. The virtual image 5 can be the navigation instructions content, vehicle speed, etc. as described above. It should be added that, according to the characteristics of different optical machines, the HUD display device can also be provided with a diffuser. In some examples, the HUD display device can also include Fresnel lenses, waveguide optical devices, diffraction optical devices, holographic optical devices, tapered optical fibers, etc.

[0076] like Figure 2As shown, the HUD display device projects display light onto the windshield 4 and reflects it to form a virtual image element 501 and a corresponding projection area 50 that carries the virtual image element 501. The virtual image element 501 can be controlled by its display position on the optical display surface, thereby changing its spatial position. The virtual image element 501 can be set at a specific position in the projection area 50 based on its matching position with the corresponding real scene, thereby achieving a corresponding virtual-real fit effect. However, determining the matching position between the virtual image element 501 and the corresponding real scene first requires the recognition of the real scene in front of the vehicle by a sensor device (such as a camera, lidar, etc.). It may also require semantic analysis of the real scene in front, such as determining the corresponding road intersection and lane markings, and may even require further analysis and judgment based on the road conditions and navigation path in the map data. Currently, these recognition and processing processes cannot guarantee absolute reliability. For example, the sensor equipment cannot accurately collect detailed information of the real scene, the semantic analysis model cannot accurately identify the target object and location, and the map data may lack corresponding data in remote areas or the detailed intersections in the city are not fully maintained. In this way, the projection position may deviate due to the deviation of the data on which the HUD display device relies for the projection of the virtual image element 501 in the projection area 50, and it may not form a correct virtual-real fit relationship with the real scene in front, such as Figure 2 In the example, the target projection position of the virtual image element 501 should be at the first fitting point 502, so that the virtual and real fitting can intuitively point to the right road to provide navigation instructions. However, due to the deviation, the virtual image element 501 is actually fitted to the surface of the building in front, making it impossible for the driver to correctly understand the content of the navigation instructions. For scenes where the virtual image element is a navigation arrow, the deviation in the projection can be caused by the deviation of the overall fitting position mentioned above. Figure 3As shown, although the virtual image element 501 is fitted at the correct intersection position, there is a deviation in the direction of the navigation indication. This may be because the identified target object is inaccurate, or the map data has not been updated in real time, and it is unable to provide reliable virtual-real fitting reference data. In more cases, the virtual image element projected by the HUD display device not only has a deviation in the fitting position with the specific real scene in front, but also has a deviation in the indicated direction, resulting in no navigation indication function at all. Optionally, when the vehicle travels to a specific location, no virtual image element for navigation indication may be generated in the corresponding projection area 50, such as multiple parking lot entrances and exits in a shopping mall. Since such detailed data is not marked in the map data, effective projection display cannot be performed. Often, the last 100 meters of blind spots in such navigation will cause great confusion to the driver, and may even be the cause of traffic congestion. For another example, location information with personal identity characteristics that is bound to a user, such as a person's residence, is difficult to fully maintain through map data, etc., because it does not have public attributes. Accordingly, no indication content in this regard will be presented in the projection area 50. Its alternative method can only be achieved by using landmark buildings maintained in the map data. However, in actual applications, drivers are often not aware of these alternative publicly searchable location names.

[0077] In some cases, if drivers rely solely on navigation guidance without knowing the specific route, they may encounter the aforementioned issues. This lack of clear navigation instructions can lead to hesitation at complex intersections, and sometimes even distraction and ignoring potential dangers, leading to traffic accidents. Accordingly, if there are other passengers in the vehicle, such as a passenger who is more familiar with the road, they can provide guidance to the driver to a certain extent. However, in this scenario, the guidance mostly relies on the passengers' verbal expressions, which are based on the relevant transmission mechanism of information exchange. If there is no visual information as an aid, relying solely on verbal expressions may lead to listeners not being able to understand intuitively due to unclear concepts involved in the verbal expressions. In particular, when passengers and drivers use completely different expressions to describe the same concept or the expression of a certain concept is not accurate enough, there will be ambiguity in understanding. For example, in "turn to this intersection later", the duration corresponding to "later" lacks a precise definition, and there may be different standards in the understanding of different people. "This intersection" is very ambiguous, and the more intersections there are ahead, the greater the ambiguity. Especially when there are multiple intersections in complex sections such as overpasses and forks in the city, it is basically impossible to obtain accurate information through this mode of guidance. Accordingly, while the passengers are giving verbal instructions, a "demonstration whiteboard" that can be expressed in writing can be provided to the passengers, thereby eliminating the vague information conveyed by verbal expressions. For example Figure 4As shown, the above-mentioned "presentation whiteboard" that can be shared by the driver and passengers is realized through the interaction between the HUD display device on the vehicle and the mobile terminal held by the passenger. Ultimately, the written information that can be issued by the mobile terminal can be presented in the projection area corresponding to the HUD display device, so as to provide the driver with a more intuitive content expression. Specifically, the HUD display device can establish a connection with the mobile terminal, and information communication can be carried out between the two. The connection between the HUD display device and the mobile terminal can be based on a communication protocol such as Bluetooth or WiFi. Optionally, when the HUD display device depends on the existence of the vehicle computer, the connection can also be established between the mobile terminal and the vehicle computer via Bluetooth or WiFi. It should be noted that the information interaction between the HUD display device and the mobile terminal in this context does not make a clear division of labor between the HUD display device and the vehicle computer. In the Bluetooth or WiFi communication connection, the connection can be automatically established based on the existing pairing relationship between the HUD display device and the mobile terminal, or the connection between the HUD display device and the mobile terminal can be supported through the corresponding authorization relationship management.

[0078] As mentioned above, the HUD display device supports projecting the required virtual image elements in front of the vehicle windshield. The virtual image elements can be coordinated with the real scene in front of the vehicle as needed. Optionally, the HUD display device supports certain real scene recognition and three-dimensional reconstruction capabilities, so that the projected virtual image elements appear to exist on the corresponding real scene. Accordingly, based on the connection between each other, the HUD display device can send the current or pre-rendered projection display content or related parameters to the mobile terminal. The hardware structure of the HUD display device can refer to Figure 14, while the mobile terminal, taking a mobile phone as an example, can include output devices such as a screen for viewing content related to the projection of the HUD display device, which serves as a control input for virtual image editing. It can also include input devices such as a gyroscope and a touch screen for inputting content that needs to be projected through the HUD display device. In a specific example, the virtual image element that needs to be projected is a navigation arrow, which has high requirements for the fitting position and indication direction. Therefore, the parameters of the navigation arrow being projected and coordinated with the real scene can be synchronized to the mobile terminal. When there is a deviation, the mobile terminal can make directional adjustments, thereby compensating for the shortcomings of relying solely on sensor equipment and map data. Accordingly, the mobile terminal can send first auxiliary information to the HUD display device based on the received adjustment results. The first auxiliary information can be in various forms and can be standardized between the HUD display device and the mobile terminal. For example, it can be a descriptive text in a format such as XML, JSON, etc., which can record the relative fitting position offset correction parameters and / or indication direction rotation correction parameters after input through the mobile terminal, or even the absolute coordinate values ​​and spatial posture parameters in a specific coordinate system. For example, the first auxiliary information can be a marker image, which can more intuitively record the fitting position and indicated direction of the virtual image element in the real image through the image content, or even related rendering data, and can be used to convert and map the three-dimensional coordinate system through the projection matrix parameters. For example, the first auxiliary information can be an information link, and the content pointed by the information link can be managed by a specific subject, or it can support shared editing by multiple people, which is particularly suitable for Internet applications to build a platform for user-generated content. The HUD display device can update the projection display of the corresponding virtual image element according to the first auxiliary information, such as correcting Figure 2 The position or correction of the navigation arrow in Figure 3 The direction of the navigation arrow in the figure is correctly matched with the real scene to achieve intuitive navigation, which will be described in detail below.

[0079] In some examples, such as Figure 5As shown, when the driver needs the passenger to assist in guiding the driving route, the user as a passenger can open the specific software on the mobile terminal 60, which can trigger the establishment of a connection with the HUD display device in the car. Accordingly, the mobile terminal 60 can receive the projection video stream synchronized with the HUD display device and display it on the screen of the mobile terminal 60, which is consistent with the projection effect of the projection area 50. The navigation arrow 601 corresponds to the navigation arrow 501 in the projection area 50. The relative position of the navigation arrow 501 in the projection area 50 can be determined by the relative position of the navigation arrow 601 in the screen of the mobile terminal 60, thereby inferring whether the navigation arrow 501 has a correct fit relationship with the real scene in front. When the mobile terminal 60 synchronizes the projection display of the HUD display device, it will also automatically enter the editing mode. The navigation arrow 601 in the editing mode supports adjustment by external input. A user holding a mobile terminal 60 can directly drag the navigation arrow 601 on the screen, or realize movement control through the direction operation buttons. The content of the operation can be recorded as the first auxiliary information and sent to the HUD display device. For example, the navigation arrow 601 is corrected to the right by a certain distance, which will be received by the HUD display device and proportionally controlled to move a certain distance to the right in the projection area 50 according to the proportional relationship between the projection area 50 and the screen of the mobile terminal 60. In this way, convenient navigation guidance can be achieved.

[0080] In some examples, when the intersection pointed by the navigation arrow 501 needs to be corrected, the correction can be achieved by directly dragging the direction of the navigation arrow 601 on the mobile terminal 60. The corresponding generated rotation correction parameters can be sent to the HUD display device through the first auxiliary information. The HUD display device can adaptively adjust the indicated direction of the navigation arrow 501 to face the correct intersection. Optionally, when the mobile terminal 60 adjusts the direction of the navigation arrow 601, it can also use sensor devices such as gyroscopes in the mobile terminal 60. Figure 6As shown, the user can rotate the direction of the mobile terminal 60 and adjust the navigation arrow 601 synchronously according to the direction sensed by the gyroscope, etc., and the corresponding rotation correction parameters will also be recorded to notify the HUD display device to make corresponding adjustments to the navigation arrow 501. In this example, rotating the mobile terminal 60 counterclockwise will trigger the navigation arrow 601 to adjust to the left, and rotating the mobile terminal 60 clockwise will trigger the navigation arrow 601 to adjust to the right. Optionally, the rotation input at the bottom of the figure can also be used, which is to rotate around an axis 612 perpendicular to the ground. This can directly change the orientation of the mobile terminal 60. Turning left will cause the navigation arrow 601 to adjust to the left, and turning right will cause the navigation arrow 601 to adjust to the right. Preferably, as the orientation of the mobile terminal changes, the camera of the mobile terminal can be turned on to identify the real scene area facing it, so as to determine the adjustment direction and degree of the navigation arrow 601 based on the real scene area captured. Accordingly, the rotation angle of the mobile terminal 60 and the adjustment angle of the navigation arrow 601 have a certain mapping relationship. The greater the rotation angle of the mobile terminal 60, the greater the adjustment angle of the navigation arrow 601. Optionally, the mobile terminal 60 will only distinguish multiple rotation states based on the rotation angle, corresponding to multiple optional adjustment directions of the navigation arrow 601. Furthermore, the multiple optional adjustment directions of the navigation arrow 601 correspond to multiple intersections in the real scene ahead identified by the HUD display device or the mobile terminal, and the rotation of the mobile terminal 60 triggers the direction selection between the multiple intersections. In a specific example, in the editing mode, in order to prevent false triggering caused by the rotation of the mobile terminal 60, the mobile terminal 60 does not directly respond to the rotation of the mobile terminal 60 to initiate the adjustment of the navigation arrow 601. Instead, it sets a first time threshold. Only when the entire process of the mobile terminal 60 rotating from the initial state to the first specific angle and then returning to the initial state is perceived within the first time threshold, will the extraction of the first specific angle as the basis for adjusting the navigation arrow 601 be triggered. When the navigation arrow 601 is rotated to the first specific angle and the second specific angle respectively from the initial state to the restored initial state within the first time threshold, for example, it is rotated to the left and then to the right, the second specific angle of the last rotation is used as the basis for the final adjustment of the navigation arrow 601. For example, in response to the last rightward rotation, the direction of the navigation arrow 601 will also be adjusted to the right. If an adjustment is made within the first time threshold but the navigation arrow 601 is not restored to the initial state, it is not considered to be a user-initiated adjustment input, and no response is given to the adjustment of the navigation arrow 601.

[0081] In some examples, such as Figure 7As shown, the mobile terminal 60 interacts with the HUD display device, and when the corresponding navigation arrow 601 is displayed on the screen of the mobile terminal 60 according to the situation that the HUD display device projects and displays the navigation arrow 501, the real scene corresponding to the projection area 50 is reproduced, which can intuitively show the driver's feeling of watching the front. The driver can view the fitting relationship between the navigation arrow 601 and the reproduced real scene 611 on the screen, which is convenient for the user to adjust on the mobile terminal 60. The specific adjustment method can be referred to Figure 5 、 Figure 6Example. Specifically, in editing mode, in addition to receiving the projection content provided by the HUD display device, the mobile terminal 60 also receives a video stream of the real scene in front captured in real time by the HUD display device and the vehicle-mounted camera. The real scene video stream will also only cut out the part of the real scene directly opposite the projection area 50 based on the corresponding relationship of the projection area 50 in space, so that when displayed on the screen of the mobile terminal 60, the fitting position and indicated direction of the navigation arrow 601 relative to the replicated real scene 611 can be roughly and completely seen. Optionally, the replicated real scene 611 on the screen of the mobile terminal 60 does not rely on the result of real-time shooting by the camera on the vehicle, but can also be displayed according to the three-dimensional model in the map data. In order to restore the fit between the navigation arrow 501 and the real scene in real space and reflect the accuracy of actual alignment, when the replicated real scene 611 and the navigation arrow 601 are simultaneously displayed on the screen of the mobile terminal 60, they will be processed according to the coordinate system determined between the human eye and the three-dimensional model before being rendered. Specifically, the three-dimensional model can be mapped to the corresponding world space coordinate system through the object-to-world matrix, and then the position of the three-dimensional model in the world space and the virtual image element 501 in the projection area 50 in the world space can be uniformly mapped to the corresponding screen space according to the projection matrix parameters of the human eye. The image determined in the screen space can be displayed on the screen of the mobile terminal 60, which can restore the virtual-real fit effect under real viewing. It should be noted that the projection matrix parameters include the camera extrinsic parameter matrix and the camera intrinsic parameter matrix used by three-dimensional systems such as Unity. The camera extrinsic parameter matrix includes the rotation matrix, translation vector, etc. The camera intrinsic parameter matrix includes the x / y axis focal length, principal point coordinates, inter-axis tilt factor, etc. The camera in this example is the approximate intrinsic and extrinsic parameters of the human eye. Furthermore, in order to improve the user's realistic experience of editing virtual image elements, the real scene reproduced on the mobile terminal 60 is obtained by real-time shooting with the camera on the mobile terminal 60. Accordingly, when the mobile terminal 60 is displayed on the screen, corresponding coordinate system conversion and processing of the navigation arrow 601 are required to restore the fitting position of the projected virtual image element to the real scene. When the user uses the mobile terminal 60 to shoot and view the real scene in front, it is as if the corresponding navigation arrow 601 really exists on the road.Therefore, similar to a rendering display based on a three-dimensional model, it is necessary to receive rendering data of the projected virtual image element 501 sent by the HUD display device. This data may include the coordinate system in space when the virtual image element 501 is projected and the projection matrix parameters used for rendering. After receiving these parameters, the mobile terminal 60 needs to use display elements representing the virtual image element 501 (such as the navigation arrow 601) to intuitively simulate the fit of the projection in the real scene. Therefore, the projected rendering data will be converted to the coordinate system to map it to the coordinate system captured by the mobile terminal 60 camera. That is, the projection matrix parameters of the mobile terminal 60 camera are used to uniformly convert it to the corresponding screen space, so that it is in the same screen space as the real scene image captured by the mobile terminal 60 camera. At the same time, the user can adjust the navigation arrow 601 on the image captured in real time by the mobile terminal 60 to match the correct navigation instructions. On the basis of the above, the adjustment results input by the user as a passenger are recorded to generate the first auxiliary information. The first auxiliary information can be the fitting position correction parameters and the indicated direction rotation correction parameters in the above example, etc. However, in order to improve the accuracy, the absolute position in the coordinate system determined by the mobile terminal 60 and the corresponding projection matrix parameters can also be directly sent to the HUD display device as the first auxiliary information. The HUD display device can redetermine the position in the projection space based on these data, thereby achieving the same fitting relationship as displayed on the mobile terminal 60.

[0082] Optionally, the content displayed in the mobile terminal 60 is not necessarily the current projection content of the projection area 50, but may be the front visual field position that the projection area 50 is not facing. Since the corresponding content needs to be projected as the vehicle moves forward, it can be adjusted in advance on the mobile terminal 60. Figure 7 For example, the mobile terminal 60 displays the navigation arrow position and the corresponding three-dimensional model rendered in advance, or refers to Figure 8For example, the camera of the mobile terminal 60 is facing the real scene outside the projection area 50 to check whether the subsequent projection display meets the requirements, and combined with the pre-rendered navigation arrow to determine whether it is in the correct fitting position or indicating direction. The corrected data will be sent to the HUD display device for storage and called when the projection area 50 is facing the correct position. Optionally, the adjusted navigation arrow 501 is not necessarily the projection content of the projection area 50 itself at the moment or in the future. For example, due to the lack of map data support, the guiding navigation arrow 501 will not be generated at the intersection in the remote area. In this case, the navigation arrow to be edited can be automatically displayed on the editing mode interface of the mobile terminal 60 through the intersection recognition or the ambiguous path trigger during the navigation process. The user can adjust the navigation arrow on the mobile terminal. When the adjustment is completed, it is sent to the HUD display device with the first auxiliary information. The HUD display device can generate a corresponding navigation arrow 501 in the projection area 50 in response to the first auxiliary information to form a fitting relationship with the corresponding intersection ahead, thereby realizing navigation guidance without map data.

[0083] For the first auxiliary information, the fitting position offset correction parameters and the indicated direction rotation correction parameters formed by the user's operation on the mobile terminal may have deviations in the conversion of the HUD display device projection. The first auxiliary information uses a coordinate system conversion based on the projection matrix parameters, which increases the processing power of the three-dimensional conversion. In addition, the determination of the camera projection matrix parameters of the mobile terminal 60 itself will also produce errors. Therefore, it is necessary to use a more unified first auxiliary information between the HUD display device and the mobile terminal. In some examples, the user can use a more direct adjustment method through the mobile terminal 60, such as Figure 9 The vehicle needs to be guided to a specific parking lot entrance 71, but this parking lot entrance 71 cannot be realized based on the missing map data in normal projection, so no virtual image elements for guiding are projected in the projection area 50. The user can mark it by shooting with the camera of the mobile terminal 60, such as Figure 10As shown, the camera can be aimed at the parking entrance 71 to be marked, and a corresponding marked image can be obtained as the first auxiliary information and sent to the HUD display device. The HUD display device can then perform similarity matching between the real scene in front of it and the marked image, thereby finding the corresponding fitting position for the virtual image element to be projected. Accordingly, the three-dimensional coordinates of the fitting position with a high degree of matching can be determined based on the position information of the real scene in front of it. When correcting the projected virtual image element, the position of the projection area 50 in space can be determined based on the projection parameters of the HUD display device, and the corresponding virtual image element can be inversely mapped to the projection area 50 where it is located. The position of the projection area 50 is mapped to the display position on the image source display surface of the HUD display device, so that the projection can be easily controlled to ensure the virtual-real fitting effect. Optionally, when taking a photo of the parking entrance 71, the user can also frame the corresponding parking entrance 71 on the screen of the mobile terminal 60 to improve the accuracy of the selection and recognition. At the same time, the matching weight within the framed range will be increased during the similarity matching. Furthermore, an intelligent model of semantic analysis can be used to assist the user in inputting the first auxiliary information. It can analyze the most meaningful target object based on the content of the photo, such as identifying the parking lot entrance, which is the most relevant destination for the current vehicle, and automatically optimize the frame range to be consistent with the outline of the parking lot entrance. Figure 11 As shown, after the corresponding fitting position and indicated direction are determined based on the similarity matching of the marker image, the projected virtual image element is not limited to the navigation arrow. The corresponding highlight mark 710 can also be projected according to the outline of the parking lot entrance 71. The highlight mark 710 can increase the reminder by flashing, so that the driver can easily understand the location of entering the parking lot. In more examples, when the mobile terminal 60 sends the first auxiliary information including the marker image to the HUD display device, it can also integrate the projection matrix parameters of the camera shooting into the first auxiliary information and send it together. The HUD display device can not only determine the specific spatial position of the parking lot entrance 71 based on similarity matching, but also combine it with the three-dimensional coordinate system transformation. It will not be repeated here.

[0084] In some examples, it is also possible to use Figure 10 Example way to correct existing navigation arrows, such as Figure 12As shown, the navigation arrow 501 is not projected at the correct position. At this time, the user can use the mobile terminal 60 to take a photo at the position where the navigation arrow 501 needs to be correctly aligned, such as the intersection for turning right. As described above, the marked image obtained by taking the photo is sent to the HUD display device, so that the correct alignment position is determined based on similarity matching. Referring to the above example, while taking the photo, the user is also supported to select the framed range 602, such as drawing a circle with a finger on the screen, so that the alignment position can be accurately determined. Preferably, the position of the lane line can be determined in combination with semantic analysis to fine-tune the framed range 602 so as to align it with a specific lane. Furthermore, the mobile terminal 60 in the editing mode supports simultaneous marking of the alignment position and the indication direction. For example, after the alignment position marking is completed, the display of the framed range 602 can be automatically canceled, and the user can operate on the screen of the mobile terminal 60 again. The indication direction can be determined by the direction in which the framed range 602 points to the new framed range. Optionally, the two framing operations can also be switched by buttons. For example, there is a first button for adjusting the fit position and a second button for adjusting the indication direction on the screen. In response to the triggering of the first button, the fit position selection will be realized, and in response to the triggering of the second button, the indication direction selection will be realized. Figure 8 For example, when taking a photo to obtain a marked image, a display element may be presented at the corresponding real scene position to reflect the incorrect virtual image element alignment in the projection.

[0085] The first auxiliary information used by the HUD display device for corrective projection can be stored in its own memory and retrieved the next time the device reaches the same road. Furthermore, the first auxiliary information can be sent by mobile terminals corresponding to multiple users. This information can be edited by different vehicles passing the same road at different times. The HUD display device can then determine the projection of the corresponding virtual image elements based on the first auxiliary information provided by multiple users. Optionally, when there are many users providing first auxiliary information, popularity can be determined based on the similarity of different first auxiliary information pieces, with the most popular being the most accurate. Alternatively, different pieces of first auxiliary information can be rated and ranked online, with the most popular being the most accurate. In further examples, the first auxiliary information exists as an information link that points to a specific server. The server can be accessed to obtain the adjusted content for the corresponding intersection. This information link can also be shared by multiple users, allowing one person to edit and multiple people to use it directly, greatly improving convenience. Optionally, multiple users can access and modify the content. However, to ensure controllability, permissions can be set to restrict editing to a specified range of users or those with high credit ratings. As mentioned above, the method of using information links as the first auxiliary information is particularly suitable for merchants to carry out corresponding publicity. They can maintain their own merchants in the corresponding Internet applications. Optionally, some store exploration bloggers can also establish corresponding information links. These information links also support public evaluation rankings in Internet applications. Users can find these available information links in Internet applications. Specifically, they can be the destination restaurant determined by the user before the vehicle departs or during the journey. The user selects a specific information link and sends it to the HUD display device. The HUD display device generates virtual image elements by directly accessing these information links, and will project clear navigation instructions when the vehicle arrives at the destination. The server content pointed to by the information link can be an introduction to a specific merchant, as well as embedded descriptive text or tagged images. For example, when maintaining this information link, the merchant can include tagged images of the entrance and exit of the mall and the appearance of the floor where the merchant is located. The HUD display device can automatically extract these contents for processing when accessing. For details, please refer to Figure 10 Examples include: Figure 13As shown, a HUD display device that receives a corresponding information link will project a first annotation symbol 503 in the projection area 50 according to preset rules. For example, a specific location within a building can be marked based on the analysis of the marked image pointed to by the information link, or it can be determined through semantic analysis based on the specific floor location description maintained. Specifically, a second annotation symbol 504 pointing to the location of the corresponding merchant can also be included, using text or other expressions to provide specific instructions for reaching the merchant, such as "Go to Exit A and go to the second floor for dining," or other notes, the content of which can also be retrieved from the server pointed to by the information link.

[0086] When the HUD display device that implements the virtual image editing method in the above example is applied to a vehicle, the virtual image elements projected by the HUD display device, especially the navigation instructions that are projected in a virtual and real way with the real scene, can correct the specific instruction content through the convenient input of the mobile terminal, thus making up for the shortcomings of the existing real scene recognition and map data. Figure 14 As shown, the HUD display device integrated in the vehicle can be powered and data-generated by the vehicle computer 92, or the HUD display device itself can provide power and generate data. The HUD display device may specifically include a processor 91, an Ethernet interface 901, a CAN interface 902 (Controller Area Network), a power management module 903, a running memory 904, a storage memory 905, a temperature detection module 906, a motor 907, a backlight source 908, an image source 909, a positioning module 910, a radar 911, a camera 912, etc. It should be noted that Figure 14 The various modules listed in the description are merely exemplary and do not constitute any limitation. In some examples, the HUD display device may also include other modules. In addition, the above modules may be implemented in one or more hardware in different examples, or a single module may be implemented by a combination of multiple hardware. Optionally, the positioning module 910, radar 911, and camera 912 may also be directly connected to the vehicle computer 92, and are not directly connected to the processor 91 of the HUD display device. For example, the vehicle computer 92 itself is integrated with a positioning module for position tracking and a radar and camera for autonomous driving. The HUD display device can obtain the collected data of the positioning module, radar, and camera in real time through communication with the vehicle computer 92.

[0087] Processor 91, serving as the control center of the HUD display device, includes one or more processing units of any type, including but not limited to a microcontroller, a microcontroller, a DSP (Digital Signal Processor), or any combination thereof. Processor 91 is used to generate operational control signals based on a computer program to control other modules and collaborate with corresponding modules to process acquired or inherent data and instructions.

[0088] The Ethernet interface 901 is a network data connection port for local area network communication, which defines a series of software and hardware standards. Multiple electronic devices can be connected together through the Ethernet interface 901. In this example, the processor 91 can exchange information with the vehicle computer 92 through the Ethernet interface 901, such as sending data to the vehicle computer 92 or receiving data sent by the vehicle computer 92.

[0089] CAN interface 902 is a network data connection port for the controller area network, providing a standard bus for the vehicle's internal control system and embedded industrial control, enabling communication and interaction between control nodes. In this example, processor 91 can also exchange information with vehicle computer 92 via CAN interface 902. Optionally, processor 91 can also connect to other external devices via CAN interface 902. In some examples, processor 91 may also be provided with a GPIO (General-purpose input / output) interface to improve compatibility with peripheral connections.

[0090] The power management module 903 is connected to the vehicle computer 92 and can receive the power provided by the vehicle computer 92 to provide a regulated power supply for each module of the HUD display device, ensuring that the processor 91 and each module operate under normal voltage supply and avoid damage due to overvoltage.

[0091] The running memory 904 is used to store computer programs executed by the processor 91, as well as temporarily stored operation data, data exchanged with the storage memory, etc. The running memory 904 can be a memory such as SDRAM (Synchronous Dynamic Random-access Memory).

[0092] The storage memory 905 is used to store resources such as the display content of the HUD display device, as well as long-term storage of running programs and data. The storage memory 905 can be a memory such as Flash. In some examples, the processor 91 can also provide an interface to access external memory.

[0093] Temperature detection module 906 is used to detect the temperature within the HUD display device. Specifically, it may include several temperature sensors. Because the resistance of a temperature sensor changes with temperature, processor 91 can determine the resistance of each temperature sensor at a given temperature based on the voltage change between each temperature sensor and the voltage divider resistor under a fixed power supply voltage, thereby reversely inferring the temperature at the location of the temperature sensor. In some examples, processor 91 can control several temperature sensors via a GPIO interface. Several temperature sensors can be located at different locations within the HUD display device. Processor 91 can use time-sharing detection to obtain temperature values ​​fed back by each of the temperature sensors.

[0094] The motor 907 is used to drive the optical lens in the HUD display device to rotate under the control of the processor 91, thereby realizing the change of the corresponding optical path. For example, when the backflow of sunlight causes the temperature of the image source surface to rise, the motor can be used to drive the optical lens to prevent external sunlight from reaching the image source surface. In some examples, the processor 91 can also drive the fan provided on the HUD display device through the motor 907 to increase the speed of air exchange between the inside and outside of the HUD display device to achieve heat dissipation. Specifically, the motor 907 is connected to the processor 91 through a motor driver chip. The motor driver chip provides high-performance power output for the motor 907, and can also communicate and control with the processor 91 through interfaces such as SPI (Serial Peripheral Interface).

[0095] Backlight source 908 is used to provide illumination and adjust the brightness of the illumination light according to the control of processor 91, thereby adjusting the projection display brightness of the entire HUD display device. Backlight source 908 cooperates with image source 909 to realize the main function of optical projection display. Specifically, backlight source 908 is connected to processor 91 via a backlight driver chip. The backlight driver chip provides driving voltage for backlight source 908 and controls the brightness of backlight source 908 based on the pulse width signal output by processor 91.

[0096] Image source 909 is used to display an image of corresponding content and project display light corresponding to the image under the control of processor 91. Taking LCD as an example, image source 909 includes liquid crystals corresponding to a number of pixels. The liquid crystals can rotate in direction under the control of an electric field, thereby changing the direction of light travel and the color presented. When the illumination light emitted by the backlight source 908 reaches the image source 909, the rotation direction of the liquid crystal determines the transmission method of the illumination light, thereby generating different images, that is, emitting display light containing different display information.

[0097] The positioning module 910 is used to monitor the position of the HUD display device and the corresponding vehicle. The positioning module 910 can be a global navigation satellite system such as the GPS (Global Positioning System) and the Beidou satellite navigation system. By measuring the distance between the satellite and the receiver on the positioning module 910 at different locations, the corresponding position and orientation data are determined. In some examples, the positioning module 910 may also include an inertial navigation system. Based on Newton's laws of mechanics, the acceleration of the positioning module 910 in the inertial reference system is measured, integrated over time, and transformed into a navigation coordinate system to obtain data such as speed, yaw angle, and position in the navigation coordinate system. Optionally, the inertial navigation system can assist the global navigation satellite system in achieving more accurate positioning and provide the processor 91 with corresponding position information.

[0098] The radar 911 is used to determine the position of a target object through electromagnetic waves, and can usually determine the distance of the target object from the vehicle where the radar 911 is located.

[0099] Camera 912 includes a vehicle body camera and an in-vehicle camera. The vehicle body camera is used to determine the location of a target object through visual recognition. The vehicle body camera can be a monocular camera or a binocular camera. The main difference between a monocular camera and a binocular camera is that a binocular camera can capture images from two different perspectives, thereby obtaining distance information in three-dimensional space. The in-vehicle camera is used to identify the behavior of the driver and passengers in the vehicle, including fatigue detection, distraction detection, expression recognition, gesture recognition, and gaze tracking. In this example, the in-vehicle camera can also specifically implement eye tracking.

[0100] In some examples, such as Figure 15 As shown, the display device that implements the above-mentioned navigation display method may specifically include a processor 931, a memory 932, an input device 933 and an output device 934, wherein the input device 933 may include an operation button integrated on the display device, etc., and the display device may receive input control instructions and data through the input device 933. The output device 934 may include an image source integrated on the display device, etc., and the display device may output corresponding instructions or data to the output device 934. Furthermore, the memory 932 stores a computer program running on the processor 931, and when the processor 931 executes the computer program, the virtual image editing method of the above-mentioned example is implemented. In some examples, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the virtual image editing method of the above-mentioned example is implemented.

[0101] like Figure 16As shown, a vehicle can be equipped with the aforementioned HUD display device. Specifically, the HUD display device is integrated within the center console 10, for example, in front of the steering wheel. The HUD display device projects corresponding display light onto the vehicle's windshield 4 through its projection window 102. Viewers observing the area in front of the windshield 4 from the cockpit directly see a virtual image within the projection area 50. The virtual image elements are not fixed and can be temporarily modified or permanently modified for reference via a mobile device within the vehicle, thereby achieving comprehensive and accurate navigation instructions. In some examples, the vehicle can also distribute a program for obtaining the virtual image editing method described in the aforementioned example via the computer-readable storage medium, enabling convenient updates and upgrades to the vehicle's HUD display device. It should be noted that the aforementioned vehicles are not limited to cars as a means of transportation, but may also include buses, trucks, excavators, motorcycles, trains, high-speed trains, ships, yachts, airplanes, spacecraft, and the like. The windshield to which the projection is directed is not limited to the front windshield of a car and may also be a transparent surface located elsewhere.

[0102] In conjunction with the above examples, the technical solutions involved in this application can be directly embodied as hardware, a software module executed by a control unit, or a combination of the two, that is, one or more steps and / or one or more step combinations, which can correspond to various software modules of a computer program flow, or to various hardware modules, such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any appropriate combination thereof. For the convenience of description, the above description is divided into various modules and described separately according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0103] Through the description of the above examples, those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution involved in this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. The software is executed by a microcontroller unit and, depending on the required configuration, can include one or more microcontroller units of any type, including but not limited to a microcontroller unit, a microcontroller, a DSP (Digital Signal Processor), or any combination thereof. The software is stored in a memory, such as a volatile memory (such as a random access memory), a non-volatile memory (such as a read-only memory, a flash memory, etc.), or any combination thereof.

[0104] In summary, this application leverages the convenient operation advantages of mobile terminals to call upon the virtual space where the HUD display device projects the virtual image. Based on the mobile terminal user's judgment of the real scene, the virtual image elements in the virtual space are edited. The edited results are then synchronized and applied to the projected object of the HUD display device to meet the virtual-reality fit requirements during projection, thereby providing effective navigation instructions. This application can improve the navigation capabilities of the virtual image projected by the HUD display device, especially enhance the accuracy of the instructions on complex roads, and optimize the navigation experience of augmented reality.

[0105] It should be understood that although this specification includes some examples, none of these examples constitutes a single independent technical solution. This description is provided for clarity purposes only. Those skilled in the art should consider this specification as a whole. The technical solutions in the various examples may be appropriately combined to form other implementations that are understandable to those skilled in the art.

[0106] The series of detailed descriptions listed above are merely specific descriptions of feasible implementation methods of the present application. They are not intended to limit the scope of protection of the present application. Any equivalent implementation methods or modifications that do not deviate from the teachings of the present application should be included in the scope of protection of the present application.

Claims

1. A virtual image editing method, characterized in that: include: The HUD display device supports projecting a first virtual image element in front of a vehicle windshield, wherein the first virtual image element is configured to cooperate with a first real scene in front of the vehicle to provide indication information; In response to a user inputting first auxiliary information of the first virtual image element relative to the first real scene through the mobile terminal, the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information to match the first real scene.

2. The virtual image editing method according to claim 1, wherein: The first auxiliary information is standard format information indicating the fitting position and / or indication direction of the first virtual image element relative to the first real scene, and the standard format information is at least one of a descriptive text, a marked image, and an information link.

3. The virtual image editing method according to claim 1, wherein: The first auxiliary information of the first virtual image element relative to the first real scene input by the user through the mobile terminal includes: The HUD display device sends the current projection state of the first virtual image element to the mobile terminal, so as to present an instruction question relative to the first real scene to the user on the mobile terminal.

4. The virtual image editing method according to claim 2, wherein: The HUD display device controlling the projection display of the first virtual image element according to the first auxiliary information includes: When it is determined based on the first auxiliary information that the fitting position of the first virtual image element relative to the first real scene needs to be corrected, the first virtual image element is inversely mapped to the projection area where it is located according to the projection parameters of the HUD display device so that the fitting position of the first virtual image element matches the first real scene.

5. A virtual image editing method, characterized in that: include: The mobile terminal establishes a connection with the HUD display device and triggers the editing mode; In response to the user's input operation on the mobile terminal, first auxiliary information of a first virtual image element relative to a first real scene is sent to the HUD display device, so that the HUD display device controls the projection display of the first virtual image element according to the first auxiliary information. The first virtual image element is the indication information projected by the HUD display device in front of the vehicle windshield to match the first real scene.

6. The virtual image editing method according to claim 5, characterized in that: The responding to the user's input operation on the mobile terminal includes: The first real scene and the first display element are reproduced on the screen of the mobile terminal, and the first display element is reproduced in relative relationship with the first real scene according to the state of the first virtual image element projected by the HUD display device; First auxiliary information for adjusting the first virtual image element is generated according to the user's adjustment of the first display element.

7. The virtual image editing method according to claim 5, characterized in that: The responding to the user's input operation on the mobile terminal includes: At least a portion of the first real scene is photographed by a camera on the mobile terminal to mark a position and / or a direction to be indicated by the first virtual image element.

8. A display device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the virtual image editing method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the virtual image editing method according to any one of claims 1 to 7 are implemented.

10. A means of transport, characterized in that: Includes the display device according to claim 8 or the computer-readable storage medium according to claim 9.

Images