Augmented reality display method, related device and computer program product

Abstract

The invention provides an augmented reality display method, related equipment and a computer program product, and relates to the technical field of augmented reality display. The method comprises the following steps: acquiring attitude information determined for augmented reality equipment through an inertial measurement unit; acquiring an environment image which is acquired by an image acquisition unit and comprises a preset anchoring object; the preset anchor is fixed at a preset position of the vehicle; determining a relative pose of a preset anchoring object relative to the augmented reality equipment based on the environment image; correcting an accumulated error of the attitude information according to the relative pose so as to obtain corrected attitude information relative to the vehicle coordinate system; wherein the vehicle coordinate system is rigidly and fixedly connected with a vehicle physical structure; and displaying the augmented reality content at a predetermined anchoring position of the vehicle based on the corrected attitude information. According to the embodiment of the invention, the fixed-screen display of the augmented reality content in the vehicle space can be realized.

Description

Technical Field

[0001] This disclosure relates to the field of augmented reality display technology, and in particular to an augmented reality display method, augmented reality device, augmented reality display apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] When using augmented reality devices for entertainment in a vehicle operating environment, current technical solutions generally face several limitations: First, due to the continuous movement of the vehicle, the attitude information constructed based on the Inertial Measurement Unit (IMU) and Simultaneous Localization and Mapping (SLAM) is prone to cumulative errors, causing the virtual display screen to shift position; second, the system stability is insufficient, and virtual content is difficult to stably fit the in-vehicle environment for a long time; in addition, the interactive experience is limited, and when the user turns their head, the virtual screen often cannot stay in the preset position.

[0004] The aforementioned problems combine to weaken the coherence and immersion of augmented reality experiences. Summary of the Invention

[0005] The purpose of this disclosure is to provide an augmented reality display method, augmented reality device, augmented reality display apparatus, electronic device, computer-readable storage medium, and computer program product that can achieve a stable and accurate "fixed-screen" display effect of augmented reality content in a vehicle space.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides an augmented reality display method, comprising: acquiring attitude information determined by the inertial measurement unit for the augmented reality device, wherein the attitude information is used to drive the real-time display of augmented reality content; acquiring an environmental image including a preset anchor, acquired by the image acquisition unit; the preset anchor being fixed at a preset position on a vehicle; determining the relative pose of the preset anchor relative to the augmented reality device based on the environmental image; correcting the cumulative error of the attitude information according to the relative pose to obtain corrected attitude information relative to a vehicle coordinate system; wherein the vehicle coordinate system is rigidly connected to the vehicle's physical structure; and displaying the augmented reality content at a predetermined anchor position on the vehicle based on the corrected attitude information.

[0008] In some embodiments, determining the relative pose of the preset anchor relative to the augmented reality device based on the environmental image includes: identifying the preset anchor in the environmental image and determining at least one of the position, size, and deformation information of the preset anchor in the environmental image; and determining a first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit based on at least one of the position, size, and deformation information of the preset anchor in the environmental image, wherein the first pose is the relative pose of the preset anchor relative to the augmented reality device.

[0009] In some embodiments, the preset anchors include at least two spatially separated preset markers; wherein, determining the first pose of the preset anchors in the camera coordinate system corresponding to the image acquisition unit based on at least one of the position, size, and deformation information of the preset anchors in the environmental image includes: determining the three-dimensional coordinate values ​​of a plurality of preset feature points on the at least two markers in the vehicle coordinate system; obtaining the two-dimensional pixel coordinates corresponding to the plurality of preset feature points in the environmental image; and determining the pose of the image acquisition unit in the vehicle coordinate system according to the correspondence between the three-dimensional coordinates and the two-dimensional pixel coordinates, wherein the pose of the image acquisition unit in the vehicle coordinate system is used to determine the first pose.

[0010] In some embodiments, the relative pose is the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit, and the pose information is the pose determined for the augmented reality display in the global environment coordinate system, which is constructed and maintained by the visual real-time localization and mapping unit in the augmented reality display. Correcting the cumulative error of the pose information based on the relative pose to obtain corrected pose information relative to the vehicle coordinate system includes: obtaining the second pose of the preset anchor in the vehicle coordinate system; aligning the global environment coordinate system to the vehicle coordinate system based on the first pose and the second pose to obtain the corrected pose information; wherein the aligned global environment coordinate system is used to render and display the augmented reality content, so that the augmented reality content is anchored and displayed at a predetermined position on the vehicle.

[0011] In some embodiments, aligning the global environment coordinate system to the vehicle coordinate system based on the first pose and the second pose to obtain the corrected pose information includes: obtaining extrinsic parameters of the image acquisition unit relative to the visual real-time localization and mapping unit; transforming the first pose to the global environment coordinate system based on the extrinsic parameters to obtain a third pose of the preset anchor in the global environment coordinate system; determining a transformation matrix between the second pose and the third pose; and aligning the global environment coordinate system to the vehicle coordinate system based on the transformation matrix.

[0012] In some embodiments, the preset anchors include at least two spatially separated preset markers.

[0013] In some embodiments, the location of the preset anchor includes at least one of the following: the vanity mirror area of ​​the vehicle, an interior panel less than a preset threshold distance from the vanity mirror, the left A-pillar of the vehicle, the right A-pillar of the vehicle, and the central control screen of the vehicle.

[0014] In some embodiments, at least one of the following is used as the preset anchor: the vehicle's vanity mirror, the vehicle's left A-pillar, the vehicle's right A-pillar, the vehicle's central control screen, and the content displayed on the central control screen.

[0015] This disclosure provides an augmented reality device, which includes a computing processing unit, an inertial measurement unit, and an image acquisition unit; the image acquisition unit is used to acquire environmental images; the inertial measurement unit is used to determine the attitude information of the augmented reality device, wherein the attitude information is used to drive the real-time display of augmented reality content; the computing processing unit is used to execute any of the above-described augmented reality display methods.

[0016] This disclosure provides an augmented reality display device, including: a posture information acquisition module, an environmental image acquisition module, a relative pose determination module, a correction module, and a display module.

[0017] The posture information acquisition module is used to acquire posture information determined by the inertial measurement unit for the augmented reality device, wherein the posture information is used to drive the real-time display of augmented reality content; the environment image acquisition module can be used to acquire an environment image containing a preset anchor, which is acquired by the image acquisition unit; the preset anchor is fixed at a preset position on the vehicle; the relative pose determination module can be used to determine the relative pose of the preset anchor relative to the augmented reality device based on the environment image; the correction module can be used to correct the cumulative error of the posture information according to the relative pose to obtain corrected posture information relative to the vehicle coordinate system; wherein the vehicle coordinate system is rigidly connected to the vehicle's physical structure; the display module can be used to display the augmented reality content at the vehicle's predetermined anchor position based on the corrected posture information.

[0018] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the augmented reality display method described above.

[0019] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the augmented reality display method as described in any of the preceding embodiments.

[0020] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the aforementioned augmented reality display method.

[0021] The augmented reality display method, related devices, apparatus, computer-readable storage medium, and computer program products provided in this disclosure achieve dynamic correction of the accumulated error of inertial navigation of augmented reality devices by utilizing a preset anchor fixed inside the vehicle as a spatial reference. This method can continuously and accurately anchor augmented reality content stably at a designated position inside the vehicle during vehicle movement, effectively overcoming the problem of virtual content drift caused by dynamic conditions such as vehicle acceleration and turning, and significantly improving the stability, immersion, and user experience of in-vehicle augmented reality displays.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0024] Figure 1 This is a schematic diagram illustrating an augmented reality display system according to an exemplary embodiment.

[0025] Figure 2 This is a flowchart illustrating an augmented reality display method according to an exemplary embodiment.

[0026] Figure 3 This is a flowchart illustrating a relative pose determination method according to an exemplary embodiment.

[0027] Figure 4 This is a flowchart illustrating a first pose determination method according to an exemplary embodiment.

[0028] Figure 5 This is a flowchart illustrating a method for determining corrected attitude information according to an exemplary embodiment.

[0029] Figure 6 This is a flowchart illustrating a method for determining corrected attitude information according to an exemplary embodiment.

[0030] Figure 7 This is a schematic diagram of a system corresponding to an augmented reality device according to an exemplary embodiment.

[0031] Figure 8 This is a block diagram illustrating an augmented reality display device according to an exemplary embodiment.

[0032] Figure 9 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0033] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0034] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0035] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0036] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0037] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0038] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0039] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0040] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0041] In autonomous driving or leisure scenarios, passengers want to use augmented reality devices for entertainment (such as playing games or watching videos) in the car.

[0042] In related technologies, a combination of inertial measurement units and real-time visual positioning and mapping is typically used to locate virtual objects in the real world.

[0043] However, these solutions have significant shortcomings in the dynamically changing vehicle interior environment and typically suffer from the following technical problems.

[0044] Positioning drift problem: When the vehicle is in continuous motion, especially when it encounters rapid acceleration, deceleration or turning, relying solely on the device's own SLAM system is prone to cumulative errors, causing the virtual screen to "drift away" in the user's field of vision or to deviate uncontrollably.

[0045] Poor stability: Existing solutions have difficulty maintaining a stable relative position between the virtual screen and the vehicle's interior environment for extended periods during continuous vehicle movement. The virtual screen is prone to cumulative shifts over time, affecting the continuous display effect.

[0046] User interaction is limited: When a user turns their head or adjusts their posture, the virtual screen may not remain in their expected field of vision position, limiting the freedom of interaction.

[0047] To address the aforementioned problems, this disclosure provides a method and system for achieving stable, accurate, and long-term "anchored" display of a virtual screen in a dynamic vehicle environment, thereby overcoming the technical defects of positioning drift and poor stability.

[0048] The technical features involved in this disclosure will be explained in detail below.

[0049] Figure 1This is a schematic diagram illustrating an augmented reality display system according to an exemplary embodiment. The system may include an augmented reality device 101 and a computing device 102.

[0050] The augmented reality device 101 described above can be, for example, AR glasses. The augmented reality device 101 may include the following modules or units.

[0051] An image acquisition unit is used to acquire video information. For example, the augmented reality device 101 may include an RGB image acquisition unit. Specifically, the augmented reality device 101 can be used to acquire information about the environment inside and outside the vehicle to obtain environmental images.

[0052] Display unit, used to display virtual elements.

[0053] Audio input units, such as microphones, are used to collect audio information.

[0054] Audio output units, such as speakers, are used to play audio.

[0055] A communication unit is used to communicate with other devices via wired or wireless means to transmit data.

[0056] In some implementations, the augmented reality device 101 may further include one or more computing processing units for running augmented reality programs to perform tasks such as rendering and controlling virtual elements and recognizing user gestures. Furthermore, the processing units may also run machine learning models to provide services such as AI (Artificial Intelligence) inference analysis.

[0057] In one embodiment, the augmented reality device 101 may further include one or more memories, such as RAM (Random Access Memory), ROM (Read-Only Memory), etc., for temporary or long-term storage of augmented reality-related data, such as augmented reality program data, video, audio, gesture, sensor data, etc. generated during program operation.

[0058] In one embodiment, the augmented reality device 110 may further include one or more sensors, such as an inertial measurement unit (IMU) and an ambient light sensor, for collecting specific types of sensor data. For example, the IMU collects inertial information, and the ambient light sensor collects ambient light information. The IMU can be used to determine the attitude information of the augmented reality device, wherein the attitude information is used to drive the real-time display of augmented reality content.

[0059] In some embodiments, the augmented reality device 110 may also integrate a SLAM system, which is a key technology enabling mobile devices to simultaneously achieve self-localization and environmental mapping in unknown environments. Its core lies in using onboard sensors (such as cameras, LiDAR, or inertial measurement units) to collect environmental information in real time, and employing probabilistic estimation, filtering, or optimization algorithms to alternately perform two coupled processes: "estimating its own position based on an existing map" and "incrementally updating the map based on its current position." This system does not require prior environmental information; it corrects accumulated errors through continuous feature matching and loop closure detection, ultimately generating a consistent environmental representation that can be used for navigation, obstacle avoidance, or augmented reality applications.

[0060] The computing device 102 can be deployed inside or outside the augmented reality device, and can be, for example, a vehicle-mounted host, a personal computer, or a server. The augmented reality device 101 can establish a connection with the computing device 102 via wired or wireless communication to achieve data transmission and collaborative processing between the two.

[0061] The computing device 102 can be configured with a powerful processor capable of performing complex computing tasks related to augmented reality.

[0062] In some implementations, the computing device 102 may be configured with an artificial intelligence unit, which includes at least one artificial intelligence model and a corresponding data processing module for performing multimodal feature fusion and collaborative task processing. For example, the artificial intelligence model may be a Large Language Model (LLM), a Vision Language Model (VLM), or a lightweight version of these large models. The computing device 102 may provide AI inference and analysis services to the augmented reality device 101.

[0063] It should be noted that, Figure 1 The system shown is merely an example, and this disclosure is not limited thereto. For example, any number of augmented reality devices can be set up as needed. Alternatively, to improve the communication quality between different devices, gateways, routers, and other devices can be added to the system. Or, the computing device 102 can be configured as a distributed cluster, with different units or machine learning models deployed on multiple nodes in the cluster, such as a data preprocessing unit on node 1, an artificial intelligence unit on node 2, a task management unit on node 3, and a visualization monitoring unit on node 4.

[0064] In one embodiment, the computing device 102 described in the present disclosure is a neck ring device that is paired with augmented reality glasses. The neck ring device serves as the computing and control main body of the augmented reality system and is connected to the augmented reality glasses through wired or wireless communication. It is used to process the sensing data collected by the augmented reality glasses and generate pose information and control instructions for driving the display of augmented reality content. The neck ring device at least includes a processor and a memory, and the processor is configured to execute program instructions stored in the memory to implement the augmented reality display method described in the present disclosure.

[0065] It should be understood that the present disclosure does not limit that the augmented reality display method must run entirely in the neck ring device. In other embodiments, some or all steps of the method may also run in the augmented reality glasses, an external computing device, or a cloud computing unit.

[0066] In some embodiments, the augmented reality system proposed by the present disclosure may at least include the following components: an augmented reality device, such as AR glasses worn by a user (with a built-in camera unit, an inertial unit, and a computing processing unit); at least two physical markers arranged in a vehicle; and positioning and rendering software running on the augmented reality device or an in-vehicle host.

[0067] Next, the present disclosure will explain and illustrate the augmented reality display method proposed by the present disclosure in combination with the above augmented reality system.

[0068] Figure 2 is a flowchart of an augmented reality display method shown according to an exemplary embodiment. The method provided by the embodiments of the present disclosure can be executed by any electronic device with computing and processing capabilities. For example, the method can be executed by the server or terminal device in the above Figure 1 embodiments, or can be jointly executed by the server and the terminal device. In the following embodiments, the server is taken as an example of the execution main body for illustration, but the present disclosure is not limited thereto.

[0069] Refer to Figure 2 , the augmented reality display method provided by the embodiments of the present disclosure may include the following steps.

[0070] In some embodiments, the above augmented reality method may be applied to the computing processing unit corresponding to the augmented reality device. The augmented reality device further includes an inertial measurement unit (IMU) and an image acquisition unit.

[0071] In some embodiments, the above computing processing unit may be deployed in the augmented reality device, or may also be deployed in the Figure 1 computing device in, such as the neck ring device.

[0072] Step S202: Obtain the attitude information determined for the augmented reality device by the inertial measurement unit, wherein the attitude information is used to drive the real-time display of augmented reality content.

[0073] In some embodiments, the inertial measurement unit (IMU) is a core sensor component in an augmented reality device, used to directly measure the device's own motion. It typically includes a three-axis gyroscope (measuring angular velocity) and a three-axis accelerometer (measuring linear acceleration), and may also integrate a magnetometer (measuring magnetic field direction). The IMU provides the device with high-frequency, continuous motion sensing data by outputting these raw physical quantities in real time.

[0074] In some embodiments, more accurate and robust attitude information for augmented reality devices can be determined by fusing data from the SLAM system in the augmented reality device with data from the inertial measurement unit (IMU).

[0075] In some embodiments, the inertial measurement system can provide raw data of the device’s angular velocity and linear acceleration at a high frequency of hundreds of hertz. The SLAM system first preprocesses this data (such as noise reduction and zero bias correction), and obtains short-time relative rotation changes by integrating the angular velocity and short-time displacement changes by integrating the acceleration (with gravity components removed), which serve as priors for motion estimation between adjacent visual frames.

[0076] Specifically, the angular velocity data collected in real time by the IMU can be integrated based on the initial attitude to directly calculate the rotational changes of the device in three-dimensional space; at the same time, the gravity direction can be extracted using accelerometer data to help correct the pitch and roll angles of the device; and then the dynamic response of the gyroscope and the long-term stability of the accelerometer can be combined through sensor fusion algorithms (such as Kalman filtering) to finally output the continuous six-degree-of-freedom attitude estimate of the device.

[0077] Step S204: Obtain an environmental image containing a preset anchor, captured by the image acquisition unit; the preset anchor is fixed at a preset position on the vehicle.

[0078] In some embodiments, an environmental image may refer to visual data captured in real time by the image acquisition unit (such as a camera) of an augmented reality device, containing the real-world scene in which the device is located. This image may encompass physical spatial information about the interior and / or exterior of the vehicle.

[0079] In some embodiments, the aforementioned environmental images may include one or more identifiable anchors (e.g., specific pattern markings inside or outside the vehicle, content displayed on an in-vehicle screen, or fixed objects with stable geometric features in the environment) that are pre-defined and fixed at specific locations in the scene. These images constitute the raw input data for subsequent visual localization, anchor recognition, and spatial registration.

[0080] In some embodiments, the preset anchors may include at least two spatially separated preset markers.

[0081] In some embodiments, the location of the preset anchor may include at least one of the following: the vanity mirror area of ​​the vehicle, the interior panel at a distance of less than a preset threshold from the vanity mirror, the left A-pillar of the vehicle, the right A-pillar of the vehicle, and the central control screen of the vehicle.

[0082] In some embodiments, at least one of the following can be used as a preset anchor: the vehicle's vanity mirror, the vehicle's left A-pillar, the vehicle's right A-pillar, the vehicle's central control screen, and the content displayed on the central control screen.

[0083] For example, a special logo image can be designed as a preset marker, and then this preset marker can be symmetrically placed on the vanity mirror area (or nearby interior panel) of the passenger side of the vehicle. It should be noted that the two logos can be spaced a certain horizontal distance apart to form a relatively long "baseline".

[0084] In some embodiments, the vehicle's central control display screen itself can be used as a large anchor, eliminating the need for additional logo stickers.

[0085] In some embodiments, two spatially separated feature points can be selected on the central control display screen structure as preset markers, such as the left and right sides or the lower left and lower right points.

[0086] In some embodiments, two spatially separated markers may be preset on a specific interface of the central control screen (such as the boot screen, brand node, or other designated interface), and this disclosure does not limit this.

[0087] For example, the outline, corner features, or image features of the central control screen when displaying a specific reference pattern (such as the brand screen at startup) can be pre-stored as preset markers.

[0088] In some embodiments, due to the large physical size of the central control screen, the "feature baseline" it forms in the image is very wide, so high-precision attitude estimation can be achieved using only a single large-sized marker.

[0089] In some embodiments, a marker may be placed on each of the left and right A-pillars of the vehicle (or vertically above and below the A-pillar on the same side). This approach utilizes the structural stability and spatial span of the A-pillars within the vehicle to provide references in both depth and width for attitude calculation, making it suitable for scenarios with more complex requirements for the spatial position of the virtual screen.

[0090] In this embodiment, the use of two spatially separated markers ensures more robust and accurate attitude calculation results even during vehicle bumps or slight head movements by the user, effectively suppressing virtual screen jitter and drift. Even when the user turns their head within a certain angle, they can maintain their position via at least one marker, thus fixing the screen.

[0091] The study found that single markers lack stability in dynamic vehicle environments and are susceptible to occlusion, lighting changes, and recognition errors. Therefore, some embodiments employ at least two markers, increasing the baseline distance between them to improve system robustness. Assume the pose estimation error of a single marker is delta. When two markers are approximately 1 meter apart, they jointly define a spatial baseline. In the horizontal direction, the attitude angle error constrained by the two markers can theoretically be equivalent to (delta1 + delta2) / baseline distance. For example, if the baseline distance is 1 meter, the total error is (delta1 + delta2), significantly diluting its impact on the attitude angle. This is equivalent to effectively merging two small markers into a larger "virtual" marker, thereby greatly improving the stability and accuracy of attitude estimation.

[0092] The aforementioned method abandons the traditional approach of solely relying on a global environment map built within the vehicle to provide positioning for augmented reality devices. Instead, it utilizes inherent, fixed objects within the vehicle as "beacons" or "anchor points." By identifying two or more pre-placed known markers, the pose of the anchor points in the original environment map is quickly and accurately calculated. This, combined with inertial navigation (IMU), provides the glasses with a corrected pose matrix relative to the original positioning, thereby stably correcting and binding the virtual screen to the vehicle coordinate system. In particular, using a combination of two markers spaced at a certain distance effectively improves the accuracy and stability of pose estimation and expands the effective positioning range.

[0093] In some embodiments, a single preset anchor may be used to correct the attitude information, and this disclosure does not limit this.

[0094] Step S206: Determine the relative pose of the preset anchor object with respect to the augmented reality device based on the environmental image.

[0095] In some embodiments, the preset anchors can first be identified from the environmental image and their key feature points can be located by feature extraction and matching algorithms (such as deep learning-based detection models or traditional image feature methods). Then, the spatial transformation relationship between the anchors and the device camera coordinate system can be calculated by combining the known actual physical size and geometric structure of the anchors and the intrinsic parameter calibration data of the image acquisition unit, using pose estimation algorithms such as Perspective-n-Point (PnP), thereby obtaining the six-degree-of-freedom relative pose including position and orientation.

[0096] Step S208: Correct the cumulative error of the attitude information based on the relative pose to obtain the corrected attitude information relative to the vehicle coordinate system; wherein the vehicle coordinate system is rigidly connected to the vehicle's physical structure.

[0097] In some embodiments, the relative pose of a preset anchor relative to the augmented reality device calculated based on the environmental image can be combined with the fixed pose of the anchor pre-calibrated in the vehicle coordinate system to calculate the spatial transformation relationship between the global environmental coordinate system and the vehicle coordinate system constructed by the SLAM system. This transformation relationship is then used to align and correct the device pose information output by the SLAM system, which has accumulated errors, thereby unifying the device pose to the vehicle coordinate system that is rigidly fixed to the vehicle's physical structure, and finally obtaining corrected and stable pose information relative to the vehicle.

[0098] In some embodiments, the pose transformation relationship (transformation matrix T) between a known fixed pose of a preset anchor in the vehicle coordinate system and the pose of the same anchor calculated by visual observation in the global environment coordinate system is calculated. This transformation matrix T represents the spatial alignment relationship between the global environment coordinate system (SLAM coordinate system) and the vehicle coordinate system.

[0099] Step S210: Based on the corrected attitude information, display the augmented reality content at the vehicle's predetermined anchoring position.

[0100] In some embodiments, the display position and angle of a virtual screen fixed to the vehicle can be determined (e.g., positioned at a certain distance behind the windshield). Based on this position and the user's real-time posture, the graphics rendering engine continuously and stably renders the virtual screen content onto the display module of the augmented reality device (such as AR glasses).

[0101] The above embodiments solve the technical challenge of providing passengers (such as the front passenger) with a stable and immersive augmented reality (AR) experience in a moving vehicle using augmented reality devices (such as AR glasses). The core principle is as follows: Two or more known physical or image markers (such as a specific logo) are placed at selected locations inside the vehicle (such as the front passenger vanity mirror, A-pillar, or center console screen). The augmented reality device can be equipped with a binocular camera to capture images of these markers in real time and uses computer vision algorithms (such as feature point recognition, pose estimation, and deep learning networks) to accurately calculate the position and pose of the glasses relative to these markers. Meanwhile, augmented reality devices can establish a stable original positioning in normal environments (such as when the vehicle is stationary) through their built-in visual SLAM and inertial navigation (IMU) systems. The position and attitude calculated by these markers can correct the original positioning in real time, "anchoring" the virtual screen in the coordinate system determined by these markers. This makes the virtual screen appear relatively stationary with respect to the vehicle's interior environment and will not "drift" or "shift" due to the vehicle's acceleration, deceleration, turning, or other movements, providing passengers with a stable and usable AR interactive interface (such as games, videos, etc.).

[0102] The above method has at least the following technical effects.

[0103] 1. By using fixed objects inside the vehicle as known markers for direct relative positioning, extremely high positioning stability is achieved, fundamentally solving the problem of virtual screen drift caused by vehicle movement.

[0104] 2. By using two spatially separated markers to form a long baseline, the accuracy and anti-interference capability of attitude estimation are significantly improved, and the system can maintain reliable operation even if the ambient light changes or there is local occlusion.

[0105] 3. Through innovative marker selection (such as utilizing the existing central control screen), elegant and seamless deployment is achieved, avoiding the hassle of users having to paste additional labels and improving the product's usability and aesthetics.

[0106] 4. Through software algorithm implementation, the solution does not require modification of vehicle hardware, and has the advantages of low cost and easy adaptation to different vehicle models, making it easy to promote and apply.

[0107] Figure 3 This is a flowchart illustrating a relative pose determination method according to an exemplary embodiment.

[0108] refer to Figure 3 Determining the relative pose of a preset anchor object with respect to an augmented reality device based on an environmental image may include the following steps.

[0109] Step S302: Identify the preset anchors in the environmental image and determine at least one of the preset anchors' position, size, and deformation information in the environmental image.

[0110] In some embodiments, the camera of an augmented reality device (such as AR glasses) can continuously capture images of the field of view in front of it to obtain environmental images.

[0111] In some embodiments, pre-trained deep learning models or image feature matching algorithms can be used to identify pre-defined anchors (such as two logo markers) from environmental images. This method can use a convolutional neural network (CNN) as the core recognition model, trained end-to-end on a massive dataset of real-world in-vehicle environment images. The training data comprehensively covers various real-world conditions—including varying lighting conditions at different times of day (e.g., strong daylight and low-light conditions at night), various weather effects (e.g., glare from rainy windows and interference from water stains on glass), and challenging visual scenes (e.g., backlighting, strong local shadows, dynamic occlusion, etc.). Through this data-driven training, the model can achieve robust feature extraction and cross-scene generalization, ensuring stable and accurate detection and localization of anchors even in complex real-world environments.

[0112] Step S304: Based on at least one of the position, size, and deformation information of the preset anchor in the environmental image, determine the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit, wherein the first pose is the relative pose of the preset anchor relative to the augmented reality device.

[0113] In some embodiments, the relative pose of an anchor is determined based on at least one of the position, size, and deformation information of the anchor in the environmental image. This can be achieved by: first, estimating the approximate distance and orientation of the anchor to the camera based on its pixel position in the image and its known actual physical size through perspective projection; further, combining the geometric deformation exhibited by the anchor in the image (such as edge trapezoidal distortion and corner perspective offset), using camera intrinsic parameters and pose calculation algorithms such as PnP (Perspective-n-Point), establishing projection constraints between the two-dimensional feature points of the image and the three-dimensional model of the anchor, thereby solving for the precise six-degree-of-freedom pose of the anchor in the camera coordinate system, i.e., its spatial position and orientation relative to the augmented reality device.

[0114] By combining a pre-trained robust recognition model with a geometrically constrained pose calculation method, the relative pose determination process can stably and accurately identify the preset anchor in a complex and ever-changing real-world vehicle environment and calculate its six-degree-of-freedom spatial pose relative to the augmented reality device, providing accurate and reliable visual observation input for subsequent coordinate system alignment and error correction.

[0115] Furthermore, since rendering requires high refresh rates, and general cameras are limited by their hardware design and cannot achieve very high exposure frequencies, relying solely on vision for pose correction results in visual lag and stuttering. Therefore, this disclosure calculates the image pose at each exposure time point and fills in the pose using inertial unit (IMU) predictions between every two frames. This method tightly or loosely fuses low-frequency visual positioning results with high-frequency IMU data, achieving high-frequency, smooth estimation and prediction of augmented reality device pose. This algorithm can output coherent pose information at frequencies from at least 500Hz to 1kHz, significantly reducing overall system latency and effectively avoiding dizziness caused by untimely or abrupt information updates.

[0116] In some embodiments, the preset anchor may include only one preset marker. Determining the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit, based on at least one of the preset anchor's position, size, and deformation information in the environmental image, may include the following steps: determining the three-dimensional coordinates of multiple preset feature points on the marker in the vehicle coordinate system; acquiring the two-dimensional pixel coordinates of the multiple preset feature points in the environmental image; and determining the pose of the image acquisition unit in the vehicle coordinate system according to the correspondence between the three-dimensional coordinates and the two-dimensional pixel coordinates of the marker. The pose of the image acquisition unit in the vehicle coordinate system is used to determine the first pose.

[0117] Figure 4 This is a flowchart illustrating a first pose determination method according to an exemplary embodiment.

[0118] In some embodiments, the preset anchors may include at least two spatially separated preset markers.

[0119] refer to Figure 4 Determining the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit based on at least one of the position, size, and deformation information of the preset anchor in the environmental image may include the following steps.

[0120] Step S402: Determine the three-dimensional coordinates of multiple preset feature points on at least two markers in the vehicle coordinate system.

[0121] The vehicle coordinate system mentioned above can be a pre-set three-dimensional spatial coordinate system that can change as the vehicle changes.

[0122] Below, this disclosure will explain how the above-mentioned vehicle coordinate system can be set up with reference to an embodiment.

[0123] In some embodiments, the midpoint of the line connecting the center points of the two markers is taken as the origin, the direction of the line is taken as the X-axis (defining the yaw reference), the vertical Z-axis is determined by combining the direction of gravity (fixing the pitch reference), and the Y-axis (implying the roll reference) is automatically derived according to the right-hand rule, thereby constructing a three-dimensional vehicle coordinate system that is rigidly connected to the vehicle's physical structure, has a clear direction and is stable, providing precise spatial alignment anchor points for augmented reality devices.

[0124] In some embodiments, any three-dimensional Cartesian coordinate system that is rigidly connected to the vehicle and has a defined direction can be the vehicle coordinate system in this disclosure. This disclosure does not limit how the vehicle coordinate system is constructed.

[0125] In some embodiments, after the vehicle coordinate system is established, the three-dimensional coordinates of each preset feature point (such as corner points and edge center points) on the marker in the vehicle coordinate system can be directly calculated using the known physical dimensions, geometric structure, and center point position of the marker in the vehicle coordinate system. For example, for a rectangular logo marker, the precise three-dimensional coordinate values ​​of the four corner points and other feature points can be derived through translation and rotation transformations in three-dimensional space based on its center point coordinates, size parameters, and preset installation posture relative to the vehicle coordinate system.

[0126] Step S404: Obtain the two-dimensional pixel coordinates of multiple preset feature points in the environmental image.

[0127] In some embodiments, after identifying preset markers in an environmental image, a trained feature point detection model (such as a CNN-based keypoint regression network) or a traditional image algorithm (such as corner detection and edge extraction) is used to accurately locate the preset feature points (such as corner points and center points) of each marker, and finally outputs the two-dimensional coordinate values ​​of these feature points in the image pixel coordinate system.

[0128] Step S406: Based on the correspondence between three-dimensional coordinates and two-dimensional pixel coordinates, determine the pose of the image acquisition unit in the vehicle coordinate system. The pose of the image acquisition unit in the vehicle coordinate system is used to determine the first pose.

[0129] In some embodiments, based on the position, size, deformation, and other information of the two identified markers in the image, combined with the known actual physical position and geometric relationship of the two markers inside the vehicle, pose estimation algorithms such as PnP (Perspective-n-Point) can be used to accurately calculate the three-dimensional position and rotational attitude of the AR glasses camera relative to the coordinate system defined by the two markers. The algorithm specifically involves acquiring the pre-calibrated intrinsic and extrinsic parameters of the augmented reality device (intrinsic parameters of the two cameras, and extrinsic parameters relative to the IMU on the glasses). Binocular vision simultaneously detects and calculates the pixel coordinates (left_u, left_v) and (right_u, right_v) of marker 1 (position (x1, y1, z1), physical size marker_size1) and marker 2 (position (x2, y2, z2), physical size marker_size2) in the binocular image, respectively. Using calibration parameters and known geometric relationships between the two markers (such as relative position and distance), the pose of the glasses relative to the combination of markers (i.e. relative to the vehicle) can be calculated more accurately and stably through multi-point perspective (PnP) or similar algorithms.

[0130] In some embodiments, the known three-dimensional coordinates of multiple preset feature points on the marker in the vehicle coordinate system can be matched with the corresponding two-dimensional pixel coordinates detected in the environmental image. Combined with the pre-calibrated intrinsic parameter matrix of the image acquisition unit, a system of projection geometric equations is constructed, and the rotation matrix and translation vector of the image acquisition unit (camera) relative to the vehicle coordinate system are calculated by using iterative optimization methods (such as EPnP, UPnP), thereby determining its six-degree-of-freedom pose in the vehicle coordinate system. This pose is the relative pose of the preset anchor object to the augmented reality device.

[0131] The aforementioned method provides a stable spatial reference frame that is strictly aligned with the vehicle's physical structure through a vehicle coordinate system constructed using dual markers. It leverages the strong geometric constraints formed by the long baselines between markers, combined with precise feature point detection and matching based on deep learning or traditional algorithms, to robustly calculate the six-DOF pose of the image acquisition unit relative to this coordinate system using a perspective projection (PnP) model. This method effectively overcomes the limitations of single visual features being susceptible to occlusion, deformation, and environmental interference, significantly improving the accuracy, robustness, and anti-drift capability of pose estimation in complex dynamic scenes such as vehicle movement and lighting changes. This provides crucial technical support for the persistent, stable, and accurate registration of augmented reality content within the vehicle's interior space.

[0132] Figure 5 This is a flowchart illustrating a method for determining corrected attitude information according to an exemplary embodiment.

[0133] In some embodiments, the relative pose can be the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit, and the pose information can be the pose determined for augmented reality display in the global environment coordinate system, which is constructed and maintained by the visual real-time localization and mapping unit in augmented reality display.

[0134] refer to Figure 5 The above-mentioned method for determining the corrected attitude information may include the following steps.

[0135] Step S502: Obtain the second pose of the preset anchor in the vehicle coordinate system.

[0136] Step S504: Based on the first pose and the second pose, align the global environment coordinate system to the vehicle coordinate system to obtain the corrected pose information; wherein the aligned global environment coordinate system is used to render and display the augmented reality content, so that the augmented reality content is anchored and displayed at the predetermined position of the vehicle.

[0137] In some embodiments, the rigid transformation matrix from the global environment coordinate system to the vehicle coordinate system can be calculated first based on the fixed second pose of the preset anchor in the vehicle coordinate system and the first pose obtained by visual observation in the global environment coordinate system. Then, the transformation matrix is ​​applied to the global environment coordinate system maintained by the SLAM system to perform a one-time or progressive spatial transformation on all its map points, thereby completing the alignment of the two coordinate systems. This allows the subsequent rendering of augmented reality content based on the corrected coordinate system to be stably anchored at a predetermined position in the vehicle's physical space.

[0138] Figure 6 This is a flowchart illustrating a method for determining corrected attitude information according to an exemplary embodiment.

[0139] refer to Figure 6 Aligning the global environment coordinate system with the vehicle coordinate system based on the first pose and the second pose to obtain the corrected pose information may include the following steps.

[0140] Step S602: Obtain the extrinsic parameters of the image acquisition unit relative to the visual real-time positioning and map building unit.

[0141] Step S604: Based on the extrinsic parameters, the first pose is transformed to the global environment coordinate system to obtain the third pose of the preset anchor in the global environment coordinate system.

[0142] Step S606: Determine the transformation matrix between the second pose and the third pose.

[0143] Step S608: Align the global environment coordinate system with the vehicle coordinate system based on the transformation matrix.

[0144] The above method achieves alignment through coordinate system transformation: First, using the pre-calibrated extrinsic parameters between the image acquisition unit and the SLAM system, the first pose of the anchor object relative to the camera is transformed to the global environment coordinate system constructed by SLAM, obtaining its observation pose (third pose) in the global coordinate system; then, the spatial transformation matrix between this observation pose and the pre-calibrated fixed pose (second pose) of the anchor object in the vehicle coordinate system is calculated; finally, this transformation matrix is ​​applied to the global environment coordinate system to rotate and translate all map points and device historical trajectories as a whole, thereby completing the accurate alignment of the global environment coordinate system to the vehicle coordinate system and realizing the stable binding of virtual content with the vehicle's physical space.

[0145] The above scheme uses a pre-set anchor as a "spatial beacon," transforms the visual observation pose to the SLAM global coordinate system through camera extrinsic parameters, and then registers it with the calibration pose in the vehicle coordinate system to calculate the transformation matrix between the two. Based on this matrix, the global coordinate system is transformed as a whole, thereby achieving precise and rigid alignment between the SLAM system and the vehicle's physical space. This fundamentally eliminates the accumulated error of inertial navigation and ensures that the virtual content can be stably "locked" in the designated position inside the vehicle for a long time.

[0146] Below, this disclosure will explain and illustrate augmented reality display methods in conjunction with specific application scenarios.

[0147] Example 1: A scheme based on dual logo markers.

[0148] 1. Marker Placement: On the vanity mirror area (or nearby interior panel) of the passenger side of the vehicle, symmetrically place two identical, specially designed logo patterns as markers. The two logos should be spaced a certain horizontal distance apart, forming a relatively long "baseline".

[0149] 2. System Components: Augmented Reality devices, such as AR glasses worn by the user (with built-in camera unit, inertial unit, and computing processing unit); two physical markers placed inside the vehicle; and positioning and rendering software running on the AR glasses or the vehicle's main unit.

[0150] 3. Work process.

[0151] Step S101 (Recognition): The AR glasses' camera continuously captures the forward field of view. The software identifies two logo markers from the image using a pre-trained deep learning model or image feature matching algorithm. A Convolutional Neural Network (CNN) is used as the core deep learning model. This model is trained by collecting and learning from massive amounts of image data from real-world in-vehicle environments. The data covers diverse lighting conditions (such as strong daylight and low light at night), weather conditions (such as glass reflections and water stains in rainy weather), and challenging scenarios (such as backlighting and interlacing shadows). This gives the model powerful feature extraction and generalization capabilities.

[0152] Step S102 (Pose Estimation): Based on the position, size, deformation, and other information of the two identified markers in the image, combined with the known actual physical position and geometric relationship of the two markers inside the vehicle, pose estimation algorithms such as PnP (Perspective-n-Point) are used to accurately calculate the three-dimensional position and rotational attitude of the AR glasses camera relative to the coordinate system defined by the two markers.

[0153] Furthermore, since rendering requires a high refresh rate, and general cameras are limited by their hardware design and cannot achieve very high exposure frequencies, relying solely on vision for pose correction will result in visual lag and stuttering. Therefore, this method calculates the image pose at each exposure time point and fills in the pose using predictions from the inertial measurement unit between every two frames.

[0154] The method described above fuses low-frequency visual positioning results with high-frequency IMU data through tight or loose coupling, achieving high-frequency, smooth estimation and prediction of glasses pose. This algorithm significantly reduces overall system latency and effectively avoids dizziness caused by untimely or abrupt information updates.

[0155] Step S103 (Pose Correction): By setting fixed physical markers and performing precise pose correction based on the marker recognition results, a "fixed screen" effect (i.e., the virtual content remains stable relative to the vehicle) is achieved in the in-vehicle scenario. This process is universal and can be adapted to all vehicle models and similar extended scenarios. The specific steps are as follows.

[0156] (1) Locating the marker: The preset marker recognition algorithm of the aforementioned vision system is used to calculate the three-dimensional position of the marker in the glasses coordinate system (i.e., the position and posture relative to the glasses). The pose of the marker can be converted into the SLAM coordinate system through the original SLAM positioning of the glasses.

[0157] (2) Coordinate system correction: The identified marker position is used as the new spatial reference origin (or the coordinate system origin fixed to the vehicle). Based on this, the original global or local coordinate system of the glasses is realigned or transformed. In this way, no matter how the glasses move, the virtual content can be stably displayed at a specific position inside the vehicle with the marker as the anchor point.

[0158] Step S104 (Screen Positioning and Rendering): Based on the calculated posture, determine the display position and angle of a virtual screen fixed to the vehicle (e.g., positioned at a certain distance behind the windshield). The graphics rendering engine continuously and stably renders the virtual screen content onto the display module of the AR glasses based on this position and the user's real-time posture.

[0159] The above method, by using two spatially separated markers, achieves more robust and accurate attitude calculation results even when the vehicle is bumpy or the user's head is slightly swaying, effectively suppressing virtual screen jitter and drift. Even when the user turns their head within a certain angle, the system can maintain its position using at least one marker, thus fixing the screen. Research has shown that a single marker lacks stability in dynamic in-vehicle environments and is susceptible to occlusion, lighting changes, and recognition errors. Therefore, this embodiment employs at least two markers and improves system robustness by increasing the baseline distance between them. Assume the pose estimation error of a single marker is delta. When two markers are approximately 1 meter apart, they jointly define a spatial baseline. In the horizontal direction, the attitude angle error obtained by the joint constraint of the two markers can theoretically be equivalent to (delta1 + delta2) / baseline distance. For example, if the baseline distance is 1 meter and the total error is (delta1+delta2), its influence on the attitude angle is significantly diluted. This is equivalent to effectively merging two small markers into a "virtual" large marker with a larger spatial scale, thereby greatly improving the stability and accuracy of attitude estimation.

[0160] Example 2: A scheme based on the central control screen as a marker.

[0161] 1. Landmark Placement: Use the vehicle's central control display screen itself as a large landmark. No additional logo is needed.

[0162] 2. Work process.

[0163] In some embodiments, the outline, corner features, or image features of the central control screen when displaying a specific reference pattern (such as the brand image at startup) can be pre-stored. Alternatively, images of various vehicle central control screens can be pre-collected or acquired via the network using a deep learning network, labeled to establish a central control screen dataset, and trained to obtain a network model capable of detecting the position of the central control screen. The ultimate goal is to obtain the pixel coordinates of the central control screen in visual perception.

[0164] In some embodiments, AR glasses recognize the unique shape or displayed content of the central control screen, such as recognizing specific features in the central control screen.

[0165] Because of the large physical size of the central control screen, the "feature baseline" it forms in the image is very wide, so high-precision attitude estimation can be achieved even with just a single large marker.

[0166] The subsequent positioning and rendering processes in this embodiment are similar to those in Embodiment 1, and will not be described again in this embodiment.

[0167] Example 3: A scheme based on dual markers of A-column.

[0168] 1. Marker placement: Place one marker on each of the left and right A-pillars of the vehicle (or on the upper and lower parts of the same side A-pillar).

[0169] The subsequent positioning and rendering processes in this embodiment are similar to those in Embodiment 1, and will not be described again in this embodiment.

[0170] This solution utilizes the structural stability and spatial span of the A-pillar in the vehicle to provide references in the depth and width directions for attitude calculation, making it suitable for scenarios with more complex requirements for the spatial position of the virtual screen.

[0171] The solutions provided in this disclosure can be applied to the following technical fields.

[0172] 1. Intelligent vehicles and autonomous driving: Used to provide stable AR entertainment (games, movies) and information display (navigation, vehicle status) services for passengers in autonomous driving mode.

[0173] 2. Augmented Reality (AR) Devices and Content Ecosystem: Applicable to all head-mounted display devices that require a stable AR experience within mobile vehicles.

[0174] 3. In-vehicle Human-Machine Interface (HMI): As the interactive interface of the next generation of in-vehicle information systems, it provides a more immersive and safer interaction method.

[0175] The above method has at least the following technical effects.

[0176] 1. By using fixed objects inside the vehicle as known markers for direct relative positioning, extremely high positioning stability is achieved, fundamentally solving the problem of virtual screen drift caused by vehicle movement.

[0177] 2. By using two spatially separated markers to form a long baseline, the accuracy and anti-interference capability of attitude estimation are significantly improved, and the system can maintain reliable operation even if the ambient light changes or there is local occlusion.

[0178] 3. Through innovative marker selection (such as utilizing the existing central control screen), an elegant and seamless deployment is achieved, avoiding the hassle of users having to paste additional labels and improving the product's usability and aesthetics.

[0179] 4. Through software algorithm implementation, the solution does not require modification of vehicle hardware, and has the advantages of low cost and easy adaptation to different vehicle models, making it easy to promote and apply.

[0180] Figure 7 This is a schematic diagram of a system corresponding to an augmented reality device according to an exemplary embodiment.

[0181] like Figure 7 As shown, the augmented reality device described above may include a computing processing unit 701, an inertial measurement unit 702, and an image acquisition unit 703.

[0182] The image acquisition unit 701 can be used to acquire environmental images.

[0183] The inertial measurement unit 702 can be used to determine the attitude information of the augmented reality device, which is used to drive the real-time display of augmented reality content.

[0184] The computing processing unit 703 can be used to execute any of the above-mentioned augmented reality display methods.

[0185] It should be particularly noted that the steps in the various embodiments of the above-described augmented reality display method can be overlapped, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of augmented reality display methods should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the embodiments.

[0186] It should be noted that the scope of protection of this disclosure should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this disclosure.

[0187] Based on the same inventive concept, this disclosure also provides an augmented reality display device, as shown in the following embodiments. Since the principle by which this device embodiment solves the problem is similar to that of the above-described method embodiments, the implementation of this device embodiment can refer to the implementation of the above-described method embodiments, and repeated details will not be described again.

[0188] Figure 8 This is a block diagram illustrating an augmented reality display device according to an exemplary embodiment. (Refer to...) Figure 8 The augmented reality display device 800 provided in this embodiment may include: a posture information acquisition module 801, an environmental image acquisition module 802, a relative pose determination module 803, a correction module 804, and a display module 805.

[0189] The system includes several modules: an attitude information acquisition module 801, which acquires attitude information determined by an inertial measurement unit for the augmented reality device (ARD), and an environment image acquisition module 802, which acquires an environment image containing a preset anchor, captured by an image acquisition unit. The preset anchor is fixed at a preset position on the vehicle. A relative pose determination module 803, which determines the relative pose of the preset anchor relative to the ARD device based on the environment image. A correction module 804, which corrects the cumulative error of the attitude information based on the relative pose to obtain corrected attitude information relative to the vehicle coordinate system, where the vehicle coordinate system is rigidly connected to the vehicle's physical structure. A display module 805, which displays the ARD content at the vehicle's predetermined anchor position based on the corrected attitude information.

[0190] It should be noted that the posture information acquisition module 801, environmental image acquisition module 802, relative pose determination module 803, correction module 804, and display module 805 mentioned above correspond to S202 to S210 in the method embodiment. The examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above method embodiment. It should be noted that the above modules, as part of the device, can be executed in a computer system such as a set of computer-executable instructions.

[0191] In some embodiments, determining the relative pose of a preset anchor object relative to an augmented reality device based on an environmental image includes: identifying the preset anchor object in the environmental image and determining at least one of the position, size, and deformation information of the preset anchor object in the environmental image; and determining the first pose of the preset anchor object in the camera coordinate system corresponding to the image acquisition unit based on at least one of the position, size, and deformation information of the preset anchor object in the environmental image, wherein the first pose is the relative pose of the preset anchor object relative to the augmented reality device.

[0192] In some embodiments, the preset anchors include at least two spatially separated preset markers. Determining the first pose of the preset anchors in the camera coordinate system corresponding to the image acquisition unit, based on at least one of the position, size, and deformation information of the preset anchors in the environmental image, includes: determining the three-dimensional coordinate values ​​of multiple preset feature points on the at least two markers in the vehicle coordinate system; acquiring the two-dimensional pixel coordinates corresponding to the multiple preset feature points in the environmental image; and determining the pose of the image acquisition unit in the vehicle coordinate system according to the correspondence between the three-dimensional coordinates and the two-dimensional pixel coordinates, wherein the pose of the image acquisition unit in the vehicle coordinate system is used to determine the first pose.

[0193] In some embodiments, the relative pose is the first pose of the preset anchor in the camera coordinate system corresponding to the image acquisition unit, and the pose information is the pose determined for augmented reality display in the global environment coordinate system, which is constructed and maintained by the visual real-time localization and mapping unit in the augmented reality display. The process of correcting the cumulative error of the pose information based on the relative pose to obtain corrected pose information relative to the vehicle coordinate system includes: obtaining the second pose of the preset anchor in the vehicle coordinate system; and aligning the global environment coordinate system to the vehicle coordinate system based on the first and second poses to obtain the corrected pose information. The aligned global environment coordinate system is used to render and display the augmented reality content, so that the augmented reality content is anchored and displayed at a predetermined position on the vehicle.

[0194] In some embodiments, the global environment coordinate system is aligned to the vehicle coordinate system based on the first pose and the second pose to obtain corrected pose information, including: acquiring the extrinsic parameters of the image acquisition unit relative to the visual real-time localization and mapping unit; transforming the first pose to the global environment coordinate system based on the extrinsic parameters to obtain the third pose of the preset anchor in the global environment coordinate system; determining the transformation matrix between the second pose and the third pose; and aligning the global environment coordinate system to the vehicle coordinate system based on the transformation matrix.

[0195] In some embodiments, the preset anchors include at least two spatially separated preset markers.

[0196] In some embodiments, the location of the preset anchor includes at least one of the following: the vanity mirror area of ​​the vehicle, the interior panel at a distance of less than a preset threshold from the vanity mirror, the left A-pillar of the vehicle, the right A-pillar of the vehicle, and the central control screen of the vehicle.

[0197] In some embodiments, at least one of the following is used as a preset anchor: the vehicle's vanity mirror, the vehicle's left A-pillar, the vehicle's right A-pillar, the vehicle's central control screen, and the content displayed on the central control screen.

[0198] Since the functions of the device 800 have been described in detail in their respective method embodiments, they will not be repeated here.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0200] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0201] Figure 9 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 9 The electronic device 900 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0202] like Figure 9 As shown, the electronic device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0203] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0204] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this disclosure.

[0205] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0206] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments, or may exist independently and not assembled into the device. The computer-readable storage medium carries one or more programs that, when executed by the device, enable the device to perform the following functions: acquiring attitude information determined for the augmented reality device by an inertial measurement unit, wherein the attitude information is used to drive the real-time display of augmented reality content; acquiring an environmental image containing a preset anchor, acquired by an image acquisition unit. The preset anchor is fixed at a preset position on the vehicle; determining the relative pose of the preset anchor relative to the augmented reality device based on the environmental image; correcting the cumulative error of the attitude information according to the relative pose to obtain corrected attitude information relative to the vehicle coordinate system, wherein the vehicle coordinate system is rigidly connected to the vehicle's physical structure; and displaying the augmented reality content at the predetermined anchor position on the vehicle based on the corrected attitude information.

[0207] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0208] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0209] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0210] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An augmented reality display method characterized by, The method is applied to a computing processing unit in an augmented reality device, the augmented reality device further comprising an inertial measurement unit and an image acquisition unit, comprising: obtaining attitude information determined for the augmented reality device by the inertial measurement unit, wherein the attitude information is used to drive real-time display of augmented reality content; obtaining an environment image containing a preset anchor acquired by the image acquisition unit; the preset anchor is fixed at a preset position of a vehicle; determining a relative pose of the preset anchor relative to the augmented reality device based on the environment image; correcting cumulative errors of the attitude information according to the relative pose to obtain corrected attitude information relative to a vehicle coordinate system; wherein the vehicle coordinate system is rigidly connected with the physical structure of the vehicle; based on the corrected attitude information, displaying augmented reality content at a predetermined anchor position of the vehicle.

2. The method of claim 1, wherein, determining a relative pose of the preset anchor relative to the augmented reality device based on the environment image, comprising: identifying the preset anchor in the environment image, determining at least one of position, size and deformation information of the preset anchor in the environment image; based on at least one of the position, size and deformation information of the preset anchor in the environment image, determining a first pose of the preset anchor in a camera coordinate system corresponding to the image acquisition unit, wherein the first pose is the relative pose of the preset anchor relative to the augmented reality device.

3. The method of claim 2, wherein, The preset anchor comprises at least two spatially separated preset markers; wherein, based on at least one of the position, size and deformation information of the preset anchor in the environment image, determining a first pose of the preset anchor in a camera coordinate system corresponding to the image acquisition unit, comprises: determining three-dimensional coordinate values of a plurality of preset feature points on the at least two markers in the vehicle coordinate system; obtaining corresponding two-dimensional pixel coordinates of the plurality of preset feature points in the environment image; determining a pose of the image acquisition unit in the vehicle coordinate system according to the correspondence between the three-dimensional coordinates and the two-dimensional pixel coordinates, the pose of the image acquisition unit in the vehicle coordinate system being used to determine the first pose.

4. The method of claim 1, wherein, The attitude information is a pose determined for the augmented reality display in a global environment coordinate system, the global environment coordinate system being constructed and maintained by a visual simultaneous localization and mapping unit in the augmented reality display; The relative pose is a first pose of the preset anchor in a camera coordinate system corresponding to the image acquisition unit; wherein, according to the relative pose, the cumulative errors of the attitude information are corrected to obtain corrected attitude information relative to a vehicle coordinate system, comprising: obtaining a second pose of the preset anchor in the vehicle coordinate system; align the global environment coordinate system to the vehicle coordinate system according to the first pose and the second pose to obtain the corrected pose information; wherein the global environment coordinate system after alignment is used to render and display augmented reality content, so that the augmented reality content is anchored and displayed at a predetermined position of the vehicle.

5. The method of claim 4, wherein, align the global environment coordinate system to the vehicle coordinate system according to the first pose and the second pose to obtain the corrected pose information, comprising: obtaining an external parameter of the image acquisition unit relative to the visual simultaneous localization and mapping unit; converting the first pose to the global environment coordinate system according to the external parameter to obtain a third pose of the preset anchor in the global environment coordinate system; determining a conversion matrix between the second pose and the third pose; aligning the global environment coordinate system to the vehicle coordinate system based on the conversion matrix.

6. The method of claim 1, wherein, The preset anchor includes at least two spatially separated preset markers.

7. The method of claim 1, wherein, The preset anchor is arranged at at least one of the following positions: a vanity mirror area of the vehicle, an interior trim panel within a distance less than a preset threshold from the vanity mirror, a left A-pillar of the vehicle, a right A-pillar of the vehicle, and a center control screen of the vehicle.

8. The method of claim 1, wherein, At least one of the following is used as the preset anchor: the vanity mirror of the vehicle, the left A-pillar of the vehicle, the right A-pillar of the vehicle, the center control screen of the vehicle, and the display content of the center control screen.

9. An augmented reality device, comprising a computing processing unit, an inertial measurement unit, and an image acquisition unit; The image acquisition unit is configured to acquire an environment image. The inertial measurement unit is configured to determine pose information of the augmented reality device, wherein the pose information is used to drive real-time display of augmented reality content. The computing processing unit is configured to perform the augmented reality display method according to any one of claims 1-8.

10. An augmented reality display device, characterized by The device is deployed in a computing processing unit of an augmented reality device, and the augmented reality device further comprises an inertial measurement unit and an image acquisition unit, comprising: A pose information acquisition module is configured to acquire pose information of the augmented reality device determined by the inertial measurement unit, wherein the pose information is used to drive real-time display of augmented reality content. An environment image acquisition module is configured to acquire an environment image containing a preset anchor fixed at a predetermined position of a vehicle, wherein the environment image is acquired by the image acquisition unit. A relative pose determination module is configured to determine a relative pose of the preset anchor relative to the augmented reality device based on the environment image. A correction module is configured to correct cumulative errors of the pose information according to the relative pose to obtain corrected pose information relative to a vehicle coordinate system; wherein the vehicle coordinate system is rigidly connected with the physical structure of the vehicle. A display module is configured to display augmented reality content at a predetermined anchor position of the vehicle based on the corrected pose information.

11. An electronic device, comprising: comprising: a memory and a processor; The memory is configured to store computer program instructions; and the processor is configured to invoke the computer program instructions stored in the memory to implement the augmented reality display method according to any one of claims 1-8.

12. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the augmented reality display method according to any one of claims 1-8.

13. A computer program product comprising computer program instructions stored in a computer readable storage medium, characterized in that, The computer program instructions, when executed by a processor, implement the method according to any one of claims 1-8.

Images