Method and apparatus for determining pose of virtual-object display unit, and electronic device

Abstract

A method and apparatus for determining the pose of a virtual-object display unit, and an electronic device. The method for determining the pose of a virtual-object display unit comprises: on the basis of a current vehicle body pose, determining the current pose of a virtual-object display unit; then, acquiring a jitter parameter measured by a motion detection apparatus; and on the basis of the jitter parameter and the current pose of the virtual-object display unit, determining a pose of the virtual-object display unit at a display moment of a virtual object. Therefore, a virtual object displayed by a virtual-object display unit can fit into an actual scenario and does not jitter due to the impact of the jolting of a vehicle body.

Description

Method, device and electronic device for determining the posture of a virtual object display unit

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 31, 2023, with application number 202311440706.5 and application name “Method, device and electronic device for determining the posture of a virtual object display unit”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of smart terminal technology, and in particular to a method, device, and electronic device for determining the posture of a virtual object display unit. Background Art

[0003] The in-vehicle display is a crucial interface for users to interact with the vehicle's computer. Typically, these displays are mounted on the instrument panel or center console. Drivers inevitably look down to check the display while driving, potentially compromising driving safety. A head-up display (HUD) projects the information the driver needs to focus on onto the windshield or other display media in front of them, preventing them from frequently looking away from the road and ensuring driving safety. Augmented reality (AR) involves superimposing virtual digital information (e.g., text, images, and / or 3D models) on the real physical world, creating a visual fusion of the real and virtual worlds. Augmented reality in-vehicle head-up display (AR-HUD) is a technology that combines augmented reality and head-up display. Figure 1 shows a schematic diagram of the projection effect of an in-vehicle AR-HUD. After rendering driving information with AR effects (such as navigation instructions, vehicle speed and / or autonomous driving information, etc.), it is projected onto the windshield through a virtual object display unit (also known as a "light machine"), thereby realizing the fusion display of virtual driving information and actual driving scenes, providing the driver with an immersive driving experience.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a method, device, and electronic device for determining the posture of a virtual object display unit. The embodiments of the present application also provide a computer-readable storage medium to ensure that the virtual object displayed by the virtual object display unit fits the actual scene and is not shaken by the bumps of the vehicle body.

[0006] In a first aspect, an embodiment of the present application provides a method for determining the posture of a virtual object display unit, comprising: determining the current posture of the virtual object display unit based on the current posture of a vehicle body; obtaining jitter parameters measured by a motion detection device; and determining the posture of the virtual object display unit at the moment of display of the virtual object based on the jitter parameters and the current posture of the virtual object display unit.

[0007] In the above-mentioned method for determining the posture of the virtual object display unit, the current posture of the virtual object display unit is determined according to the current posture of the vehicle body, and then the jitter parameters measured by the motion detection device are obtained. According to the above-mentioned jitter parameters and the current posture of the virtual object display unit, the posture of the virtual object display unit at the moment of display of the above-mentioned virtual object is determined, so that the virtual object displayed by the virtual object display unit can fit the actual scene and will not shake due to the bumps of the vehicle body.

[0008] In one possible implementation, the current vehicle posture is transmitted to the virtual object display unit by a vehicle posture calculation unit; the motion detection device includes a first motion detection device, which is rigidly connected to the virtual object display unit; and obtaining the jitter parameter measured by the motion detection device includes: obtaining the first jitter parameter measured by the first motion detection device during the vehicle posture transmission delay.

[0009] In this implementation, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit is determined based on the data of the vehicle-mounted sensors after the vehicle posture calculation unit obtains the data of the vehicle-mounted sensors, see Figure 8. In this implementation, the current vehicle posture refers to the posture at the end of the current frame posture calculation time (i.e., moment 1), and the current posture of the virtual object display unit can be the posture of the virtual object display unit at the end of the transmission delay (i.e., moment 2).

[0010] In one possible implementation, determining the posture of the virtual object display unit at the moment of display of the virtual object based on the jitter parameter and the current posture of the virtual object display unit includes: compensating the current posture of the virtual object display unit based on the first jitter parameter; predicting a third jitter parameter during virtual object rendering based on historical jitter parameters measured by the first motion detection device; and determining the posture of the virtual object display unit at the moment of display of the virtual object based on the third jitter parameter and the posture of the virtual object display unit after compensation.

[0011] In one possible implementation, compensating for the current posture of the virtual object display unit based on the first jitter parameter measured during the vehicle body posture transmission delay includes: obtaining the posture change of the virtual object display unit during the vehicle body posture transmission delay based on the first jitter parameter; and compensating for the current posture of the virtual object display unit based on the posture change.

[0012] In one possible implementation, determining the current posture of the virtual object display unit based on the current vehicle body posture includes: determining the current posture of the virtual object display unit based on the current vehicle body posture and a pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0013] In one possible implementation manner, determining the posture of the virtual object display unit at the display moment of the virtual object based on the third jitter parameter and the posture of the virtual object display unit after compensation includes: determining the posture of the virtual object display unit at the start moment of rendering of the virtual object based on the posture of the virtual object display unit after compensation; obtaining the posture of the virtual object display unit at the display moment of the virtual object based on the third jitter parameter and the posture of the virtual object display unit at the start moment of rendering of the virtual object.

[0014] In some examples, obtaining the position of the virtual object display unit at the time of display of the virtual object based on the third jitter parameter and the position of the virtual object display unit at the time of rendering start of the virtual object may be as follows: obtaining an initial position value of the virtual object display unit at the time of display of the virtual object based on the third jitter parameter and the position of the virtual object display unit at the time of rendering start of the virtual object, and smoothing the initial position value of the virtual object display unit at the time of display of the virtual object using the historical position value of the virtual object display unit, thereby finally obtaining the position of the virtual object display unit at the time of display of the virtual object. Smoothing the initial position value of the virtual object display unit at the time of display of the virtual object is performed to filter out points where there is a large difference between the initial position value of the virtual object at the time of display and the historical position value of the virtual object display unit, thereby reducing the difference between the position value of the virtual object display unit at the time of display of the virtual object and the historical position value of the virtual object display unit. When smoothing the initial value of the posture of the virtual object display unit at the time of displaying the above-mentioned virtual object, methods such as averaging or filtering can be used, but the embodiments of the present application are not limited to this. Any method that can reduce the difference between the posture of the virtual object display unit at the time of displaying the above-mentioned virtual object and the historical posture of the virtual object display unit should fall within the protection scope of the embodiments of the present application.

[0015] In the above implementation method, the virtual object display unit obtains the current vehicle body posture, determines the current posture of the virtual object display unit based on the current vehicle body posture and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, and then obtains the first jitter parameter measured by the first motion detection device during the above vehicle body posture transmission delay period, and obtains the posture change of the virtual object display unit during the above vehicle body posture transmission delay period based on the first jitter parameter measured during the vehicle body posture transmission delay period; and compensates for the current posture of the virtual object display unit based on the above posture change, and then the virtual object display unit predicts the third jitter parameter during the virtual object rendering period based on the historical jitter parameter measured by the first motion detection device. Finally, the virtual object display unit determines the posture of the virtual object display unit at the display moment of the above virtual object based on the above third jitter parameter and the posture of the compensated virtual object display unit, so that the virtual object displayed by the virtual object display unit can fit the actual scene and is not affected by the bumps of the vehicle body and shakes.

[0016] In one possible implementation, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit includes: a vehicle posture after anti-shake compensation; the vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining a second jitter parameter measured by a second motion detection device and performing anti-shake compensation on the current vehicle posture based on the second jitter parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is non-rigidly connected to the virtual object display unit. Referring to Figure 10, in this implementation, the current vehicle posture obtained by the virtual object display unit can be the vehicle posture at time 3.

[0017] In one possible implementation, determining the current position of the virtual object display unit based on the current vehicle position includes determining the current position of the virtual object display unit based on the vehicle position after anti-shake compensation and a pre-calibrated physical transformation relationship between the vehicle and the virtual object display unit. Referring to FIG10 , in this implementation, the current position of the virtual object display unit can be the position of the virtual object display unit at the end of the transmission delay (i.e., time 4).

[0018] In one possible implementation, after determining the current posture of the virtual object display unit based on the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, it also includes: determining the relative jitter between the vehicle chassis and the virtual object display unit based on the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device; and performing anti-shake compensation on the current posture of the virtual object display unit based on the relative jitter.

[0019] In one possible implementation, compensating the current posture of the virtual object display unit according to the first jitter parameter includes: obtaining the posture change of the virtual object display unit during the delay period of the vehicle posture transmission according to the first jitter parameter; and compensating the posture of the virtual object display unit after anti-shake compensation according to the posture change.

[0020] In one possible implementation manner, the first motion detection device is rigidly connected to the virtual object display unit, including: the first motion detection device is installed on a component rigidly connected to the virtual object display unit.

[0021] The above implementation method can ensure that the virtual object displayed by the virtual object display unit fits the actual scene and is not affected by the bumps of the vehicle body and shakes. When the vehicle posture calculation unit on the chassis and the virtual object display unit are non-rigidly connected, the above implementation method combines the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device to perform anti-shake compensation and delay compensation on the posture of the virtual object display unit, thereby solving the posture calculation error problem caused by the relative jitter and data transmission delay between the vehicle posture calculation unit and the virtual object display unit. In addition, this embodiment also predicts the posture of the virtual object display unit at the time of display of the virtual object based on the historical jitter parameters of the virtual object display unit, solving the problem that the posture of the virtual object display unit at the time of display of the virtual object cannot be obtained due to the lack of sensor data during rendering.

[0022] In one possible implementation, the motion detection device includes a second motion detection device, which is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is rigidly connected to the virtual object display unit; determining the posture of the virtual object display unit at the display moment of the virtual object based on the jitter parameters and the current posture of the virtual object display unit includes: performing anti-shake compensation on the current posture of the virtual object display unit based on the jitter parameters currently measured by the second motion detection device; predicting the jitter parameters of the virtual object display unit during posture transmission and virtual object rendering based on historical jitter parameters measured by the motion detection device; determining the posture of the virtual object display unit at the display moment of the virtual object based on the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation.

[0023] In this implementation, the current vehicle body pose can be obtained by the vehicle body pose calculation unit; specifically, the vehicle body pose calculation unit can obtain data from the onboard sensors and determine the current vehicle body pose based on the data from the onboard sensors. Referring to FIG11(b), in this implementation, the current vehicle body pose is determined by the vehicle body pose calculation unit based on the data from the onboard sensors. Therefore, the current vehicle body pose can be the vehicle body pose at the end of the current frame pose calculation time (i.e., time 5). Furthermore, the current pose of the virtual object display unit 4021 is determined by the vehicle body pose calculation unit based on the current vehicle body pose. Therefore, it can be considered that after the vehicle body pose calculation unit determines the current vehicle body pose, the current pose of the virtual object display unit can be determined. Therefore, referring to FIG11(b), in this implementation, the current pose of the virtual object display unit can be understood as the pose of the virtual object display unit at time 5.

[0024] In one possible implementation manner, determining the posture of the virtual object display unit at the display moment of the virtual object based on the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation includes: determining the posture of the virtual object display unit at the start moment of rendering of the virtual object based on the posture of the virtual object display unit after anti-shake compensation; obtaining the posture of the virtual object display unit at the display moment of the virtual object based on the predicted jitter parameters and the posture of the virtual object display unit at the start moment of rendering of the virtual object.

[0025] In this implementation, the vehicle body posture calculation unit obtains the posture of the virtual object display unit at the display time of the virtual object based on the predicted jitter parameters and the posture of the virtual object display unit at the start time of rendering of the virtual object. This can be: the vehicle body posture calculation unit obtains the initial posture value of the virtual object display unit at the display time of the above-mentioned virtual object based on the posture of the virtual object display unit at the start time of rendering of the above-mentioned virtual object and the above-mentioned predicted jitter parameters, and uses the historical posture of the virtual object display unit to smooth the initial posture value of the virtual object display unit at the display time of the above-mentioned virtual object, and finally obtains the posture of the virtual object display unit at the display time of the above-mentioned virtual object, and sends it to the virtual object display unit.

[0026] In the above implementation, after the vehicle body posture calculation unit obtains the current vehicle body posture, it determines the current posture of the virtual object display unit based on the current vehicle body posture. Then, the vehicle body posture calculation unit obtains the jitter parameters currently measured by the second motion detection device rigidly connected to the vehicle body posture calculation unit, and performs anti-shake compensation on the current posture of the virtual object display unit based on the currently measured jitter parameters. Then, the vehicle body posture calculation unit predicts the jitter parameters of the virtual object display unit during the posture transmission period and the virtual object rendering period based on the historical jitter parameters measured by the second motion detection device. Based on the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation, the posture of the virtual object display unit at the time of display of the above virtual object is determined. Finally, the vehicle body posture calculation unit transmits the posture of the virtual object display unit at the time of display of the above virtual object to the virtual object display unit, so that the virtual object displayed by the virtual object display unit can fit the actual scene and is not affected by the bumps of the vehicle body and shakes.

[0027] In a second aspect, an embodiment of the present application provides a device for determining the position and posture of a virtual object display unit. The device is included in an electronic device and has the function of implementing the electronic device behavior in the first aspect and possible implementations of the first aspect. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, an acquisition module and a determination module.

[0028] In a third aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method provided in the first aspect.

[0029] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0032] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a schematic diagram of the projection effect of an in-vehicle AR-HUD;

[0034] FIG2 is a schematic diagram of the installation of a vehicle-mounted virtual object display unit;

[0035] FIG3 is a schematic diagram showing the display principle of the vehicle-mounted virtual object display unit;

[0036] FIG4 is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0037] FIG5 is a schematic diagram of hardware provided by an embodiment of the present application;

[0038] FIG6 is a schematic structural diagram of a vehicle posture calculation unit 4011 provided in one embodiment of the present application;

[0039] FIG7 is a flowchart of a method for determining a virtual object display unit posture according to an embodiment of the present application;

[0040] FIG8 is a timing diagram of an operation provided by an embodiment of the present application;

[0041] FIG9 is a flowchart of a method for determining a virtual object display unit posture according to another embodiment of the present application;

[0042] FIG10 is a timing diagram of another embodiment of the present invention;

[0043] FIG11( a ) is a flowchart of a method for determining a virtual object display unit posture according to another embodiment of the present application;

[0044] FIG11( b ) is a working timing diagram provided by yet another embodiment of the present application;

[0045] FIG12 is a schematic structural diagram of an electronic device provided in one embodiment of the present application;

[0046] FIG13 is a schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0047] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0048] Figure 2 is a schematic diagram of the installation of the on-board virtual object display unit, and Figure 3 is a schematic diagram of the display principle of the on-board virtual object display unit. As shown in Figures 2 and 3, the AR-HUD optical machine (i.e., the virtual object display unit in Figure 2) is generally installed in front of the driving seat of the car cockpit, while the sensors and processors used to calculate the body posture (i.e., the body posture calculation unit in Figure 2) are generally installed outside the cockpit (such as the frame, chassis or body surface). In order to achieve a display effect that combines virtual and real, the AR-HUD optical machine needs to accurately calculate the body posture through the on-board sensors, and then calculate the accurate posture of the AR-HUD optical machine in the real world based on the positional relationship between the body and the AR-HUD optical machine, thereby calculating the 3D projection position of the virtual driving information on the windshield. Therefore, the calculation accuracy of the posture and the anti-shake processing effect will affect the quality of the AR-HUD virtual and real experience.

[0049] As shown in Figure 2, sensor data or pose data is typically obtained by the vehicle pose calculation unit in the chassis domain, while the virtual object display unit that uses this data is located in the cockpit domain. Data transmission from the chassis domain to the cockpit domain often experiences a long latency (20ms to 100ms), resulting in a lag between the pose transmitted to the virtual object display unit and the actual scene. Using this pose to render virtual driving information can cause a misalignment between the virtual object and the actual scene. Furthermore, due to the unique nature of vehicle suspension structures, the cockpit and chassis are generally not rigidly connected. During driving, due to bumpy roads, the two experience relative motion. Even without data transmission latency, the relative motion between the vehicle pose calculated by sensors mounted on the chassis domain and the pose of the virtual object display unit mounted in the cockpit domain can introduce errors, resulting in misalignment or noticeable jitter in the virtual driving information displayed by the AR-HUD.

[0050] Existing in-vehicle HUD anti-shake technologies mostly focus on compensating for overall vehicle body motion or relative jitter between components within the virtual object display unit. They don't address the pose lag and jitter caused by data transmission delays, nor do they address the relative jitter caused by a non-rigid connection between the virtual object display unit and the vehicle's sensors. A non-rigid connection refers to relative motion between the virtual object display unit and the vehicle's sensors, rather than a fixed connection.

[0051] The basic principle of an existing in-vehicle HUD anti-shake solution is to suppress vibration by installing a mechanical device. Common anti-shake device installation locations include between the cockpit and the virtual object display unit, or between the internal components of the virtual object display unit.

[0052] Among them, one implementation solution is to connect the HUD display medium to the cockpit through a shock-absorbing mechanism to prevent display jitter caused by relative movement between the two; another implementation solution is to connect the HUD display lens through a support rod fixing block and a support rod sleeve, and fix the HUD display lens through a special mechanical structure to prevent it from jittering; and another implementation solution is to design the HUD reflector bracket with a pivot connection and multi-point fixation to reduce the jitter of the reflector.

[0053] However, the above implementation only addresses relative jitter between the virtual object display unit and the cockpit, or between components within the virtual object display unit. When the virtual object display unit and the onboard sensors are non-rigidly connected, this solution cannot address virtual object jitter caused by relative jitter between the two, nor can it address virtual object misalignment caused by latency in the transmission of pose data.

[0054] Another existing in-vehicle HUD anti-shake solution works by detecting the vehicle's overall vibration relative to the ground and controlling the virtual object display unit to perform corresponding vibration compensation movements. One implementation involves using an inertial measurement device mounted on the vehicle to obtain the vehicle's vibration parameters relative to the ground (including offset and angle). This is then accomplished by adjusting the control circuit to adjust the HUD image generation device (i.e., the virtual object display unit) in the opposite direction of the vehicle's vibration, thereby compensating for any HUD image jitter caused by the vehicle's vibration.

[0055] However, there's a data transmission delay when the inertial measurement unit (IMU) mounted on the vehicle body transmits jitter parameters to the HUD image generator. The jitter parameters used for HUD image jitter compensation lag behind the actual jitter, making this solution unable to address jitter during the transmission delay. Furthermore, when the HUD image generator is controlled to perform compensatory motion based on the jitter parameters, the control circuit has a response delay, making this solution unable to address jitter during the response delay. Furthermore, when the HUD image generator and the vehicle body or sensors on the vehicle body are non-rigidly connected, this solution cannot address virtual object jitter caused by relative jitter between the two.

[0056] Another existing in-vehicle HUD anti-shake solution works by detecting jitter between components within the virtual object display unit and adjusting the display position of the virtual object to compensate for jitter. One implementation involves installing angular velocity sensors on both the head-up display and the rotatable mirror within the virtual object display unit. By detecting the relative jitter parameters between the two internal components, the compensation amount for the HUD display is calculated and displayed accordingly.

[0057] However, this solution only addresses image jitter caused by relative jitter between components within the virtual object display unit. When the virtual object display unit and the onboard sensor are non-rigidly connected, it cannot address virtual object jitter caused by relative jitter between the two, nor can it address virtual object misalignment caused by latency in the transmission of pose data.

[0058] The basic principle of another existing vehicle-mounted HUD anti-shake solution is to detect the jitter between the virtual object display unit and the cockpit, and adjust the display position of the virtual object display unit to perform anti-shake compensation. Among them, one implementation scheme is to detect the jitter of the positioning mark on the car windshield through an image sensor installed on the virtual object display unit, thereby calculating the jitter of the HUD projection image relative to the windshield, and then offset compensating the display image. However, the above scheme only targets the relative jitter between the virtual object display unit and the cockpit (for example, the windshield). When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, it cannot solve the virtual object jitter problem caused by the relative jitter between the two, nor can it solve the problem of virtual object misalignment caused by the delay in posture data transmission.

[0059] Generally speaking, the on-board sensors used to calculate the vehicle's posture (e.g., inertial measurement unit (IMU), wheel speedometer, camera, light detection and ranging (Lidar), global navigation satellite system (GNSS), and / or real-time kinematic (RTK)) are installed on the vehicle (e.g., the vehicle shell, chassis, axles, or wheels). The AR-HUD virtual object display unit is generally installed in front of the driver's seat in the cockpit. The on-board sensors and the virtual object display unit are generally non-rigidly connected, and there will be relative jitter between the two when the vehicle is driving. When displaying a virtual object, the virtual object display unit determines the display position of the virtual object in the projected image based on the posture of the virtual object display unit and the posture of the virtual object in the real scene. Generally, the vehicle's posture is calculated by the on-board sensors, and then the posture of the virtual object display unit is inferred from the vehicle's posture. The onboard sensors and vehicle posture calculation unit are generally located in the chassis domain, while the virtual object display unit is generally located in the cockpit domain. There is a delay in the transmission of sensor data or posture data from the chassis domain to the cockpit domain. On the one hand, relative jitter between the onboard sensors and the virtual object display unit during this delay period can lead to inaccurate posture. On the other hand, this will cause the posture acquired by the virtual object display unit to lag behind the actual situation. Therefore, when the virtual object display unit and the onboard sensors are non-rigidly connected, if there is relative jitter and data transmission delay between the two, the calculated posture of the virtual object display unit will be inaccurate, resulting in the virtual object not fitting the real scene or jittering.

[0060] In addition, there is no sensor data from the time the position of the virtual object display unit is calculated to the time the virtual object is actually displayed. Therefore, the position of the virtual object display unit at the display moment needs to be predicted in advance based on historical sensor data.

[0061] As mentioned above, the existing technical solutions do not solve the relative jitter problem between the virtual object display unit and the vehicle-mounted sensor, the posture lag problem caused by data transmission delay, and the posture lag problem caused by virtual object rendering delay.

[0062] Based on the above problems, an embodiment of the present application provides a method for determining the posture of a virtual object display unit. When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, the method for determining the posture of the virtual object display unit provided by the embodiment of the present application can detect the relative jitter between the two and compensate for it. When there is a delay in the data being transmitted from the vehicle-mounted sensor to the virtual object display unit, the embodiment of the present application can solve the posture lag problem caused by the delay, and can also predict the posture of the virtual object display unit at the moment of virtual object display based on historical sensor data in advance, so that the virtual object displayed by the virtual object display unit is more in line with the real scene. It should be noted that in the embodiment of the present application, the posture may include position and attitude, that is, the three-dimensional translation and three-dimensional rotation relationship of a point relative to a coordinate system.

[0063] The embodiment of the present application obtains jitter parameters through a motion detection device installed on the virtual object display unit and the vehicle body, which is used to perform anti-shake compensation and advance prediction on the posture of the virtual object display unit, so that the placement of the virtual object is more consistent with the real scene.

[0064] Specifically, the embodiments of the present application combine the jitter parameters of the onboard sensors on the vehicle body with the jitter parameters of the virtual object display unit to detect the relative motion between the two due to the non-rigid connection during data transmission, and perform anti-shake compensation and time delay compensation on the calculated position and posture of the virtual object display unit. In addition, the jitter parameters of the virtual object display unit and the historical jitter parameters of the onboard sensors on the vehicle body are used to infer the jitter parameters during the rendering and display of the virtual object. Based on the current position and predicted jitter parameters of the virtual object display unit, the position and posture of the virtual object display unit at the time of virtual object display are predicted in advance.

[0065] The method for determining the position and posture of a virtual object display unit provided in an embodiment of the present application can be applied to an AR-HUD system for a smart car. FIG4 is a schematic diagram of the system architecture provided in an embodiment of the present application. As shown in FIG4 , the system architecture may include a chassis domain 401 , a cockpit domain 402 , a data transmission unit 403 , and a clock synchronization unit 404 .

[0066] The chassis domain 401 may include a vehicle posture calculation unit 4011 and a second motion detection device 4012, and the cockpit domain 402 may include a virtual object display unit 4021 and a first motion detection device 4022. The chassis domain 401 and the cockpit domain 402 transmit posture or sensor data via a data transmission unit 403. Because the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and data transmission between them is subject to time delay, a clock synchronization unit 404 is required to synchronize the clocks between the chassis domain 401 and the cockpit domain 402.

[0067] The chassis domain may also be called a mobile data center (MDC) domain, and the cockpit domain may also be called a cockpit domain controller (CDC) domain.

[0068] Specifically, the vehicle posture calculation unit 4011 may generally include an MDC and various on-board sensors. The MDC may include a central processing unit (CPU), a graphics processing unit (GPU), a cache, a data communication bus, etc. It is primarily used to acquire on-board sensor data and calculate the vehicle posture or the posture of the virtual object display unit. In the embodiments of the present application, the vehicle posture calculation unit 4011 may also be referred to as an on-board sensor for calculating the vehicle posture.

[0069] The second motion detection device 4012 is typically mounted on a vehicle component (such as the vehicle's exterior, chassis, or axle) that is rigidly connected to the vehicle posture calculation unit 4011. Common second motion detection devices include accelerometers, gyroscopes, cameras, lidar sensors, photoelectric sensors, inertial measurement units (IMUs), or inertial elements. These devices are primarily used to detect jitter parameters of the onboard sensors in the chassis domain 401 used to calculate the vehicle posture, and to perform anti-shake compensation on the vehicle posture based on these jitter parameters.

[0070] The data transmission unit 403 generally includes a data communication line and a data transmission control unit, and is mainly used to transmit the posture or sensor data of the chassis domain 401 to the cockpit domain 402.

[0071] The clock synchronization unit 404 is used to ensure that the devices in the chassis domain 401 and the devices in the cockpit domain 402 maintain clock synchronization.

[0072] The first motion detection device 4022 is typically mounted on the virtual object display unit 4021, or on a component rigidly connected to the virtual object display unit 4021, such as a dashboard, steering wheel, driving recorder, or rearview mirror. Common first motion detection devices 4022 include accelerometers, gyroscopes, cameras, lidars, photoelectric sensors, IMUs, or inertial sensors, and are primarily used to obtain jitter parameters of the virtual object display unit 4021.

[0073] The virtual object display unit 4021, i.e., the AR-HUD virtual object display unit (e.g., AR-HUD optical engine / projector), is primarily used to render virtual objects and project them onto the vehicle's windshield. In a specific implementation, the virtual object display unit 4021 can be a HUD optical engine, projector, or HUD display.

[0074] FIG5 is a hardware diagram provided by one embodiment of the present application. During specific installation, referring to FIG5 , the vehicle posture calculation unit 4011, second motion detection device 4012, virtual object display unit 4021, first motion detection device 4022, data transmission unit 403, and clock synchronization unit 404 can be installed in the chassis domain 401 and the cockpit domain 402, respectively. The vehicle posture calculation unit 4011 and second motion detection device 4012 are installed in the chassis domain 401, while the virtual object display unit 4021 and first motion detection device 4022 are installed in the cockpit domain 402. There is a data transmission delay between the chassis domain 401 and the cockpit domain 402, and the devices in the two domains are synchronized via the clock synchronization unit 404.

[0075] In a specific implementation, the vehicle posture calculation unit 4011 can be implemented using the structure shown in FIG6 , which is a schematic diagram of the structure of the vehicle posture calculation unit 4011 provided in one embodiment of the present application. As shown in FIG6 , the vehicle posture calculation unit 4011 may include: a processor 410 and a communication interface 420. Optionally, the vehicle posture calculation unit 4011 may also include a memory 430. The processor 410, the communication interface 420, and the memory 430 may communicate with each other through an internal connection path to transmit control and / or data signals. The memory 430 is used to store computer programs, and the processor 410 is used to call and run the computer programs from the memory 430.

[0076] The processor 410 and the memory 430 may be combined into a processing device, or more commonly, they are independent components. The processor 410 is configured to execute program codes stored in the memory 430. In a specific implementation, the memory 430 may also be integrated into the processor 410 or independent of the processor 410.

[0077] Optionally, the vehicle body posture calculation unit 4011 may further include a power supply 450 for providing power to various devices or circuits in the vehicle body posture calculation unit 4011 .

[0078] It should be understood that the processor 410 in the vehicle posture calculation unit 4011 shown in FIG6 may be a system on a chip (SOC), and the processor 410 may include a CPU, and may further include other types of processors, such as a GPU.

[0079] Similarly, the virtual object display unit 4021 may also be implemented using the structure shown in FIG6 , which will not be described in detail here.

[0080] The method for determining the position and posture of a virtual object display unit provided in an embodiment of the present application may include: determining the current position and posture of the virtual object display unit based on the current vehicle body position, then obtaining a jitter parameter measured by a motion detection device, and finally determining the position and posture of the virtual object display unit at the time of display of the virtual object based on the jitter parameter and the current position and posture of the virtual object display unit. The method for determining the position and posture of a virtual object display unit provided in an embodiment of the present application is described in detail below with reference to Figures 4, 5, and 6.

[0081] FIG7 is a flow chart of a method for determining a virtual object display unit posture according to an embodiment of the present application. As shown in FIG7 , the method for determining a virtual object display unit posture may include:

[0082] In step 701 , the virtual object display unit 4021 obtains the current vehicle body posture; wherein the current vehicle body posture is transmitted to the virtual object display unit 4021 by the vehicle body posture calculation unit 4011 .

[0083] In this embodiment, the current vehicle pose transmitted by the vehicle pose calculation unit 4011 to the virtual object display unit 4021 is determined based on the data from the onboard sensors after the vehicle pose calculation unit 4011 obtains the data. Referring to FIG8 , in this embodiment, the current vehicle pose refers to the pose at the end of the current frame pose calculation time (i.e., time 1).

[0084] In step 702 , the virtual object display unit 4021 determines the current posture of the virtual object display unit 4021 according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0085] Among them, the current posture of the virtual object display unit 4021 is determined according to the current vehicle body posture after the virtual object display unit 4021 obtains the current vehicle body posture transmitted by the vehicle body posture calculation unit 4011. Therefore, in this embodiment, referring to Figure 8, the current posture of the virtual object display unit 4021 can be the posture of the virtual object display unit 4021 at the end of the transmission delay (i.e., moment 2).

[0086] In step 703 , the virtual object display unit 4021 obtains the first jitter parameter measured by the first motion detection device 4022 during the vehicle posture transmission delay period; wherein the first motion detection device is rigidly connected to the virtual object display unit 4021 , and the first jitter parameter may be the jitter parameter of the virtual object display unit 4021 .

[0087] Among them, since the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and the data transmission between them has a time delay, the vehicle posture transmission delay can be the time used by the vehicle posture calculation unit 4011 to transmit the current vehicle posture to the virtual object display unit 4021.

[0088] In this embodiment, the rigid connection between the first motion detection device and the virtual object display unit 4021 may be: the first motion detection device is installed on a component rigidly connected to the virtual object display unit 4021 .

[0089] In step 704, the virtual object display unit 4021 obtains the posture change of the virtual object display unit 4021 during the vehicle body posture transmission time delay according to the first jitter parameter; and compensates the current posture of the virtual object display unit 4021 according to the posture change.

[0090] In step 705 , the virtual object display unit 4021 predicts a third jitter parameter during virtual object rendering based on the historical jitter parameter measured by the first motion detection device 4022 .

[0091] In step 706 , the virtual object display unit 4021 determines the position and posture of the virtual object display unit 4021 at the time of displaying the virtual object according to the third jitter parameter and the position and posture of the virtual object display unit 4021 after compensation.

[0092] Specifically, the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the display moment of the above-mentioned virtual object based on the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation. The posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object can be obtained based on the third jitter parameter and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object.

[0093] In a specific implementation, the posture of the virtual object display unit 4021 at the time of display of the virtual object is obtained according to the third jitter parameter and the posture of the virtual object display unit 4021 at the time of rendering start of the above-mentioned virtual object: the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is obtained according to the posture of the virtual object display unit 4021 at the time of rendering start of the above-mentioned virtual object and the third jitter parameter, and the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is smoothed using the historical posture of the virtual object display unit 4021 to obtain the posture of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object.

[0094] The purpose of smoothing the initial value of the pose of the virtual object display unit 4021 at the time of displaying the virtual object is to filter out points where the initial value of the pose of the virtual object at the time of displaying the virtual object is significantly different from the historical pose of the virtual object display unit 4021, thereby reducing the difference between the pose of the virtual object display unit 4021 at the time of displaying the virtual object and the historical pose of the virtual object display unit 4021. When smoothing the initial value of the pose of the virtual object display unit 4021 at the time of displaying the virtual object, methods such as averaging or filtering may be used, but the embodiments of the present application are not limited thereto. Any method that can reduce the difference between the pose of the virtual object display unit 4021 at the time of displaying the virtual object and the historical pose of the virtual object display unit 4021 shall fall within the scope of protection of the embodiments of the present application.

[0095] The method provided in this embodiment can be applied in a scenario where the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are rigidly connected. In some examples, when the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are rigidly connected, the virtual object display unit 4021 only needs to obtain the first jitter parameter from the first motion detection device 4022 to perform delay compensation and advance prediction of the posture of the virtual object display unit 4021. Figure 8 is a working timing diagram provided by an embodiment of the present application. As shown in Figure 8, in this embodiment, there is no relative jitter between the vehicle body posture calculation unit 4011 and the virtual object display unit 4021. Therefore, it is not necessary to obtain the jitter parameter measured by the second motion detection device 4012. It is only necessary to use the first jitter parameter measured by the first motion detection device to perform transmission delay compensation and advance prediction.

[0096] In the above-mentioned method for determining the posture of the virtual object display unit, the virtual object display unit 4021 obtains the current posture of the vehicle body, determines the current posture of the virtual object display unit 4021 according to the current posture of the vehicle body, and then obtains the first jitter parameter measured by the first motion detection device 4022 during the above-mentioned vehicle body posture transmission delay period, and obtains the posture change of the virtual object display unit 4021 during the above-mentioned vehicle body posture transmission delay period according to the first jitter parameter measured during the vehicle body posture transmission delay period; and according to the above-mentioned posture change, the posture of the virtual object display unit 4021 is adjusted. The current posture is compensated, and then the virtual object display unit 4021 predicts the third jitter parameter during the virtual object rendering period according to the historical jitter parameter measured by the first motion detection device 4022. Finally, the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the display moment of the above-mentioned virtual object according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation, so that the virtual object displayed by the virtual object display unit 4021 can fit the actual scene and is not shaken by the bumps of the vehicle body.

[0097] In this embodiment, when the vehicle body posture calculation unit 4011 and the virtual object display unit 4021 are rigidly connected, the virtual object display unit 4021 uses the first jitter parameter measured by the first motion detection device 4022 to perform time delay compensation for the posture of the virtual object display unit. This can address posture calculation errors caused by posture or sensor data transmission delays. Furthermore, the posture of the virtual object display unit 4021 at the time of virtual object display can be predicted in advance based on the historical jitter parameters of the virtual object display unit 4021, thereby resolving the issue of being unable to determine the posture of the virtual object display unit 4021 at the time of virtual object display due to a lack of sensor data during virtual object rendering.

[0098] FIG9 is a flow chart of a method for determining a virtual object display unit posture according to another embodiment of the present application. As shown in FIG9 , the method for determining a virtual object display unit posture may include:

[0099] In step 901 , the virtual object display unit 4021 obtains the current vehicle body posture; wherein the current vehicle body posture is transmitted to the virtual object display unit 4021 by the vehicle body posture calculation unit 4011 .

[0100] In this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit 4011 to the virtual object display unit 4021 is the vehicle posture determined based on the vehicle-mounted sensor data obtained by the vehicle posture calculation unit 4011, and after anti-shake compensation is performed. Referring to FIG. 10 , in this embodiment, the current vehicle posture obtained by the virtual object display unit 4021 may be the vehicle posture at time 3.

[0101] Specifically, the vehicle body posture after the above-mentioned anti-shake compensation is obtained after the vehicle body posture calculation unit 4011 obtains the second shake parameter measured by the second motion detection device 4012, and performs anti-shake compensation on the current vehicle body posture according to the above-mentioned second shake parameter; wherein, the second motion detection device 4012 is rigidly connected to the vehicle body posture calculation unit 4011; the vehicle body posture calculation unit 4011 is non-rigidly connected to the virtual object display unit 4021.

[0102] In step 902 , the virtual object display unit 4021 determines the current posture of the virtual object display unit 4021 according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0103] Similarly, in this embodiment, the current position of the virtual object display unit 4021 is determined based on the vehicle body posture after the virtual object display unit 4021 obtains the vehicle body posture after anti-shake compensation and is transmitted by the vehicle body posture calculation unit 4011. Therefore, referring to FIG10 , in this embodiment, the current position of the virtual object display unit 4021 can be the position of the virtual object display unit 4021 at the end of the transmission delay (i.e., time 4).

[0104] In step 903 , the virtual object display unit 4021 determines the relative jitter between the vehicle chassis and the virtual object display unit 4021 based on the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 .

[0105] In step 904 , the virtual object display unit 4021 performs anti-shake compensation on the current posture of the virtual object display unit 4021 according to the relative shake.

[0106] In step 905 , the virtual object display unit 4021 obtains the first jitter parameter measured by the first motion detection device 4022 during the vehicle posture transmission delay period; wherein, the first motion detection device 4022 is rigidly connected to the virtual object display unit 4021 , and the first jitter parameter may be the jitter parameter of the virtual object display unit 4021 .

[0107] Among them, since the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and the data transmission between them has a time delay, the vehicle posture transmission delay can be the time used by the vehicle posture calculation unit 4011 to transmit the current vehicle posture to the virtual object display unit 4021.

[0108] In this embodiment, the rigid connection between the first motion detection device and the virtual object display unit 4021 may be: the first motion detection device is installed on a component rigidly connected to the virtual object display unit 4021 .

[0109] Step 906: The virtual object display unit 4021 obtains the posture change of the virtual object display unit 4021 during the above-mentioned vehicle posture transmission delay based on the first jitter parameter measured during the above-mentioned vehicle posture transmission delay; and compensates the posture of the virtual object display unit 4021 after anti-shake compensation based on the above-mentioned posture change.

[0110] In step 907 , the virtual object display unit 4021 predicts a third jitter parameter during virtual object rendering based on the historical jitter parameter measured by the first motion detection device 4022 .

[0111] In step 908 , the virtual object display unit 4021 determines the position and posture of the virtual object display unit 4021 at the time of displaying the virtual object according to the third jitter parameter and the position and posture of the virtual object display unit 4021 after compensation.

[0112] Specifically, according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation, the posture of the virtual object display unit 4021 at the display moment of the above-mentioned virtual object can be determined as follows: the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object according to the posture of the virtual object display unit 4021 after compensation; according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object, the posture of the virtual object display unit 4021 at the display moment of the virtual object is obtained.

[0113] In specific implementation, the posture of the virtual object display unit 4021 at the time of display of the virtual object is obtained according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 at the time of starting rendering of the above-mentioned virtual object: according to the posture of the virtual object display unit 4021 at the time of starting rendering of the above-mentioned virtual object and the third jitter parameter, the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is obtained, and the historical posture of the virtual object display unit 4021 is used to smooth the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object, and finally the posture of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is obtained.

[0114] The method provided in this embodiment can be applied in scenarios where the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are non-rigidly connected. In some examples, when the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are non-rigidly connected, the virtual object display unit 4021 needs to combine the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and time delay compensation on the posture of the virtual object display unit 4021. Figure 10 is a working timing diagram provided by another embodiment of the present application. As shown in Figure 10, in this embodiment, there is relative jitter between the vehicle body posture calculation unit 4011 and the virtual object display unit 4021. Therefore, it is necessary to combine the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and time delay compensation on the posture of the virtual object display unit 4021.

[0115] The above-described method for determining the pose of a virtual object display unit can ensure that the virtual object displayed by the virtual object display unit 4021 fits the actual scene and is not affected by vehicle vibrations and jitter. When the vehicle pose calculation unit 4011 on the chassis and the virtual object display unit 4021 are non-rigidly connected, this embodiment combines the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and delay compensation on the pose of the virtual object display unit 4021, thereby resolving pose calculation errors caused by relative jitter and data transmission delay between the vehicle pose calculation unit 4011 and the virtual object display unit 4021. In addition, this embodiment also predicts the pose of the virtual object display unit 4021 at the time of virtual object display based on the historical jitter parameters of the virtual object display unit 4021, resolving the problem of being unable to obtain the pose of the virtual object display unit 4021 at the time of virtual object display due to the lack of sensor data during rendering.

[0116] FIG11( a ) is a flow chart of a method for determining a virtual object display unit posture according to yet another embodiment of the present application, and FIG11( b ) is a timing diagram of an operation according to yet another embodiment of the present application. As shown in FIG11( a ) and FIG11( b ), the method for determining a virtual object display unit posture may include:

[0117] Step 1101: The vehicle posture calculation unit 4011 obtains the current vehicle posture.

[0118] Specifically, the vehicle body posture calculation unit 4011 may obtain the current vehicle body posture by: obtaining data from the vehicle-mounted sensors and determining the current vehicle body posture based on the data from the vehicle-mounted sensors. Referring to FIG. 11( b ), in this embodiment, the current vehicle body posture is determined by the vehicle body posture calculation unit 4011 based on the data from the vehicle-mounted sensors. Therefore, the current vehicle body posture may be the vehicle body posture at the end of the current frame posture calculation time (i.e., time 5).

[0119] In step 1102 , the vehicle posture calculation unit 4011 determines the current posture of the virtual object display unit 4021 according to the current vehicle posture and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0120] In this embodiment, the current posture of the virtual object display unit 4021 is determined by the vehicle body posture calculation unit 4011 based on the current vehicle body posture. Therefore, it can be considered that after the vehicle body posture calculation unit 4011 determines the current vehicle body posture, the current posture of the virtual object display unit 4021 can be determined. Therefore, referring to Figure 11(b), in this embodiment, the current posture of the virtual object display unit 4021 can be understood as the posture of the virtual object display unit 4021 at time 5.

[0121] In step 1103 , the vehicle posture calculation unit 4011 obtains the jitter parameter currently measured by the second motion detection device 4012 ; wherein the second motion detection device 4012 is rigidly connected to the vehicle posture calculation unit 4011 , and the vehicle posture calculation unit 4011 is rigidly connected to the virtual object display unit 4021 .

[0122] In step 1104 , the vehicle posture calculation unit 4011 performs anti-shake compensation on the current posture of the virtual object display unit 4021 according to the jitter parameters currently measured by the second motion detection device 4012 .

[0123] In step 1105 , the vehicle posture calculation unit 4011 predicts the jitter parameters of the virtual object display unit 4021 during posture transmission and virtual object rendering based on the historical jitter parameters measured by the second motion detection device 4012 .

[0124] In step 1106 , the vehicle body posture calculation unit 4011 determines the posture of the virtual object display unit 4021 at the start time of rendering the virtual object according to the posture of the virtual object display unit 4021 after anti-shake compensation.

[0125] In step 1107 , the vehicle body posture calculation unit 4011 obtains the posture of the virtual object display unit 4021 at the time of displaying the virtual object based on the predicted jitter parameter and the posture of the virtual object display unit 4021 at the time of starting rendering of the virtual object.

[0126] Specifically, the vehicle body posture calculation unit 4011 obtains the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object based on the predicted jitter parameters and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object, which can be: the vehicle body posture calculation unit 4011 obtains the initial posture value of the virtual object display unit 4021 at the display time of the above-mentioned virtual object based on the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object and the above-mentioned predicted jitter parameters, and uses the historical posture of the virtual object display unit 4021 to smooth the initial posture value of the virtual object display unit 4021 at the display time of the above-mentioned virtual object, and finally obtains the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object, and sends it to the virtual object display unit 4021.

[0127] In the above-mentioned method for determining the posture of the virtual object display unit, after the vehicle body posture calculation unit 4011 obtains the current vehicle body posture, it determines the current posture of the virtual object display unit 4021 according to the current vehicle body posture, and then the vehicle body posture calculation unit 4011 obtains the jitter parameters currently measured by the second motion detection device 4012 rigidly connected to the vehicle body posture calculation unit 4011, and performs anti-shake compensation on the current posture of the virtual object display unit 4021 according to the currently measured jitter parameters. Then, the vehicle body posture calculation unit 4011 predicts the jitter parameters of the virtual object display unit 4021 during the posture transmission period and the virtual object rendering period according to the historical jitter parameters measured by the second motion detection device 4012, and determines the posture of the virtual object display unit 4021 at the moment of display of the above-mentioned virtual object according to the predicted jitter parameters and the posture of the virtual object display unit 4021 after anti-shake compensation. Finally, the vehicle body posture calculation unit 4011 transmits the posture of the virtual object display unit 4021 at the time of displaying the above-mentioned virtual object to the virtual object display unit 4021, so that the virtual object displayed by the virtual object display unit 4021 can fit the actual scene and will not shake due to the bumps of the vehicle body.

[0128] When the vehicle body posture calculation unit 4011 and the virtual object display unit 4021 are rigidly connected, this embodiment can predict in advance the posture of the virtual object display unit 4021 after the data transmission delay and the rendering time of the virtual object based on the historical jitter parameters of the chassis, thereby solving the posture jitter problem caused by the transmission delay and the lack of sensor data during rendering.

[0129] It should be noted that the method for determining the position and posture of the virtual object display unit provided in the embodiment of the present application can be applied not only to vehicle AR-HUD, but also to aircraft, excavators, cranes and other equipment with AR-HUD functions.

[0130] It is understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in a different order than those presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0131] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0132] This embodiment can divide the electronic device into functional modules based on the above-mentioned method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is illustrative and is only a logical functional division. In actual implementation, other division methods may be used.

[0133] FIG12 is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. In the case where each functional module is divided according to each function, FIG12 shows a possible schematic diagram of the composition of an electronic device 1200 involved in the above embodiment. As shown in FIG12 , the electronic device 1200 may include: an acquisition module 1201, a determination module 1202, a compensation module 1203, and a prediction module 1204.

[0134] The acquisition module 1201 is used to acquire the current vehicle body posture; wherein the current vehicle body posture is transmitted from the vehicle body posture calculation unit to the virtual object display unit;

[0135] A determination module 1202 is configured to determine the current posture of the virtual object display unit based on the current vehicle posture. In this embodiment, the determination module 1202 is specifically configured to determine the current posture of the virtual object display unit based on the current vehicle posture and a pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0136] The acquisition module 1201 is further configured to acquire a first jitter parameter measured by the first motion detection device during the vehicle body posture transmission delay period; and obtain a posture change of the virtual object display unit during the vehicle body posture transmission delay period based on the first jitter parameter measured during the vehicle body posture transmission delay period; wherein the first motion detection device is rigidly connected to the virtual object display unit, and the first jitter parameter includes the jitter parameter of the virtual object display unit; the rigid connection between the first motion detection device and the virtual object display unit may be: the first motion detection device is mounted on a component rigidly connected to the virtual object display unit;

[0137] A compensation module 1203 is configured to compensate the current posture of the virtual object display unit according to the above-mentioned posture change;

[0138] The prediction module 1204 is configured to predict a third jitter parameter during virtual object rendering based on the historical jitter parameter measured by the first motion detection device;

[0139] The determination module 1202 is further configured to determine the posture of the virtual object display unit at the moment of displaying the virtual object according to the third jitter parameter and the posture of the virtual object display unit after compensation.

[0140] In one implementation of this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit is determined based on the data of the vehicle-mounted sensors after the vehicle posture calculation unit obtains the data.

[0141] The determination module 1202 is specifically configured to determine the posture of the virtual object display unit at the start time of rendering the virtual object according to the posture of the virtual object display unit after the compensation;

[0142] Acquisition module 1201 is specifically configured to obtain the position and posture of the virtual object display unit at the time the virtual object is displayed based on the third jitter parameter and the position and posture of the virtual object display unit at the time the virtual object is rendered. In this embodiment, acquisition module 1201 is specifically configured to obtain an initial value of the position and posture of the virtual object display unit at the time the virtual object is displayed based on the position and posture of the virtual object display unit at the time the virtual object is rendered and the third jitter parameter, and to smooth the initial value of the position and posture of the virtual object display unit at the time the virtual object is displayed using the historical position and posture of the virtual object display unit to obtain the position and posture of the virtual object display unit at the time the virtual object is displayed.

[0143] In another implementation of this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit may be: the vehicle posture after anti-shake compensation; the above-mentioned vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining the second jitter parameter measured by the second motion detection device, and performing anti-shake compensation on the current vehicle posture according to the second jitter parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit.

[0144] In this way, the determination module 1202 is specifically used to determine the current posture of the virtual object display unit according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the above-mentioned virtual object display unit.

[0145] Furthermore, the determination module 1202 is further configured to determine, after determining the current position of the virtual object display unit, a relative jitter between the vehicle chassis and the virtual object display unit based on the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device;

[0146] The compensation module 1203 is specifically configured to perform anti-shake compensation on the current posture of the virtual object display unit according to the relative jitter.

[0147] In this implementation, the compensation module 1203 is specifically configured to compensate the posture of the virtual object display unit after anti-shake compensation according to the posture change.

[0148] It should be noted that all relevant contents of each step involved in the method embodiments shown in Figures 7 to 10 of the present application can be referred to the functional description of the corresponding functional modules and will not be repeated here.

[0149] The electronic device 1200 provided in this embodiment is used to execute the method for determining the position and posture of the virtual object display unit provided in the embodiments shown in Figures 7 to 10 of this application, and thus can achieve the same effect as the above method.

[0150] It should be understood that the electronic device 1200 can serve as a virtual object display unit, and the electronic device 1200 can be implemented using the structure shown in Figure 6. The functions of the acquisition module 1201, the determination module 1202, the compensation module 1203, and the prediction module 1204 can be implemented by the processor 410 in the electronic device 400 shown in Figure 6.

[0151] In the case of adopting an integrated unit, the electronic device 1200 may include a processing module, a storage module, and a communication module.

[0152] The processing module can be used to control and manage the operation of the electronic device 1200. For example, it can be used to support the electronic device 1200 in executing the steps performed by the acquisition module 1201, determination module 1202, compensation module 1203, and prediction module 1204. The storage module can be used to support the electronic device 1200 in storing program code and data. The communication module can be used to support communication between the electronic device 1200 and other devices.

[0153] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0154] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1200 involved in this embodiment may be a device having the structure shown in FIG. 6 .

[0155] FIG13 is a schematic diagram of the structure of an electronic device provided in another embodiment of the present application. In the case where each functional module is divided according to each function, FIG13 shows a possible schematic diagram of the composition of an electronic device 1300 involved in the above embodiment. As shown in FIG13 , the electronic device 1300 may include: an acquisition module 1301, a determination module 1302, a compensation module 1303, a prediction module 1304, and a transmission module 1305.

[0156] The acquisition module 1301 is used to obtain the current vehicle posture;

[0157] A determination module 1302 is configured to determine a current position and posture of the virtual object display unit based on the current vehicle body position and posture. In this embodiment, the determination module 1302 is specifically configured to determine the current position and posture of the virtual object display unit based on the current vehicle body position and a pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0158] The acquisition module 1301 is further configured to acquire a jitter parameter currently measured by a second motion detection device; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is rigidly connected to the virtual object display unit;

[0159] The compensation module 1303 is configured to perform anti-shake compensation on the current posture of the virtual object display unit according to the jitter parameter currently measured by the second motion detection device;

[0160] A prediction module 1304 is configured to predict jitter parameters of the virtual object display unit during posture transmission and virtual object rendering based on historical jitter parameters measured by the second motion detection device;

[0161] Determination module 1302 is further configured to determine the pose of the virtual object display unit at the time the virtual object is displayed based on the predicted jitter parameters and the pose of the virtual object display unit after anti-shake compensation. In this embodiment, determination module 1302 is specifically configured to determine the pose of the virtual object display unit at the time the virtual object is rendered based on the pose of the virtual object display unit after anti-shake compensation; and to obtain the pose of the virtual object display unit at the time the virtual object is rendered based on the predicted jitter parameters and the pose of the virtual object display unit at the time the virtual object is rendered. In a specific implementation, determination module 1302 is specifically configured to obtain an initial pose value of the virtual object display unit at the time the virtual object is displayed based on the pose of the virtual object display unit at the time the virtual object is rendered and the predicted jitter parameters, and to smooth the initial pose value of the virtual object display unit at the time the virtual object is displayed using the historical pose of the virtual object display unit, ultimately obtaining the pose of the virtual object display unit at the time the virtual object is displayed.

[0162] The transmission module 1305 is configured to transmit the position and posture of the virtual object display unit at the time of displaying the virtual object to the virtual object display unit.

[0163] In this embodiment, the acquisition module 1301 is specifically used to acquire data from the vehicle-mounted sensor;

[0164] The determination module 1302 is specifically configured to determine the current vehicle posture based on the data from the vehicle-mounted sensors.

[0165] It should be noted that all relevant contents of each step involved in the method embodiment shown in Figures 11(a) to 11(b) of the present application can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0166] The electronic device 1300 provided in this embodiment is used to execute the method for determining the posture of the virtual object display unit provided in the embodiment shown in Figures 11(a) to 11(b) of this application, and thus can achieve the same effect as the above method.

[0167] It should be understood that the electronic device 1300 can serve as a vehicle posture calculation unit, and the electronic device 1300 can be implemented using the structure shown in Figure 6. The functions of the acquisition module 1301, the determination module 1302, the compensation module 1303, the prediction module 1304, and the transmission module 1305 can be implemented by the processor 410 in the electronic device 400 shown in Figure 6.

[0168] In the case of adopting an integrated unit, the electronic device 1300 may include a processing module, a storage module, and a communication module.

[0169] The processing module can be used to control and manage the operation of the electronic device 1300. For example, it can be used to support the electronic device 1300 in executing the steps performed by the acquisition module 1301, determination module 1302, compensation module 1303, prediction module 1304, and transmission module 1305. The storage module can be used to support the electronic device 1300 in storing program code and data. The communication module can be used to support communication between the electronic device 1300 and other devices.

[0170] Among them, the processing module can be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0171] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1300 involved in this embodiment may be a device having the structure shown in FIG. 6 .

[0172] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided in the embodiments shown in Figures 7 to 10 of the present application.

[0173] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, it enables the computer to execute the method provided in the embodiment shown in Figures 11(a) to 11(b) of the present application.

[0174] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is run on a computer, the computer executes the method provided in the embodiments shown in Figures 7 to 10 of the present application.

[0175] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is run on a computer, it enables the computer to execute the method provided in the embodiment shown in Figures 11(a) to 11(b) of the present application.

[0176] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0177] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0179] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for determining the position and posture of a virtual object display unit, characterized in that: include: Determine the current position and posture of the virtual object display unit according to the current vehicle body position and posture; Obtaining jitter parameters measured by a motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the jitter parameter and the current position and posture of the virtual object display unit.

2. The method according to claim 1, characterized in that: The current vehicle posture is transmitted to the virtual object display unit by the vehicle posture calculation unit; the motion detection device includes a first motion detection device, and the first motion detection device is rigidly connected to the virtual object display unit; The obtaining of the jitter parameters measured by the motion detection device comprises: Acquire a first jitter parameter measured by the first motion detection device during the vehicle body posture transmission delay.

3. The method according to claim 2, characterized in that Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit comprises: Compensating a current posture of the virtual object display unit according to the first jitter parameter; predicting a third jitter parameter during virtual object rendering based on the historical jitter parameter measured by the first motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the third jitter parameter and the position and posture of the virtual object display unit after compensation.

4. The method according to claim 3, characterized in that The compensating the current posture of the virtual object display unit according to the first jitter parameter measured during the vehicle body posture transmission delay includes: According to the first jitter parameter, obtaining a posture change of the virtual object display unit during a delay period of the vehicle body posture transmission; According to the posture change, the current posture of the virtual object display unit is compensated.

5. The method according to claim 1, characterized in that Determining the current posture of the virtual object display unit according to the current vehicle body posture includes: The current posture of the virtual object display unit is determined according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

6. The method according to claim 3, characterized in that Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the compensated virtual object display unit comprises: Determining the posture of the virtual object display unit at the start time of rendering the virtual object according to the posture of the virtual object display unit after compensation; The position and posture of the virtual object display unit at the time of displaying the virtual object is obtained according to the third jitter parameter and the position and posture of the virtual object display unit at the time of starting rendering of the virtual object.

7. The method according to claim 3, characterized in that The current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit includes: the vehicle posture after anti-shake compensation; the vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining the second jitter parameter measured by the second motion detection device, and performing anti-shake compensation on the current vehicle posture according to the second jitter parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is non-rigidly connected to the virtual object display unit.

8. The method according to claim 7, characterized in that Determining the current posture of the virtual object display unit according to the current vehicle body posture includes: The current posture of the virtual object display unit is determined according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

9. The method according to claim 8, characterized in that After determining the current posture of the virtual object display unit according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, the method further includes: determining a relative jitter between the vehicle chassis and the virtual object display unit according to a first jitter parameter measured by the first motion detection device and a second jitter parameter measured by the second motion detection device; According to the relative jitter, anti-shake compensation is performed on the current posture of the virtual object display unit.

10. The method according to claim 9, characterized in that The compensating the current posture of the virtual object display unit according to the first jitter parameter includes: According to the first jitter parameter, obtaining a posture change of the virtual object display unit during a delay period of the vehicle body posture transmission; According to the posture change, the posture of the virtual object display unit after anti-shake compensation is compensated.

11. The method according to claim 2, characterized in that The first motion detection device is rigidly connected to the virtual object display unit and comprises: The first motion detection device is mounted on a component rigidly connected to the virtual object display unit.

12. The method according to claim 1, characterized in that The motion detection device comprises a second motion detection device, the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is rigidly connected to the virtual object display unit; Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit comprises: Performing anti-shake compensation on the current posture of the virtual object display unit according to the jitter parameter currently measured by the second motion detection device; Predicting jitter parameters during the posture transmission period and the virtual object rendering period of the virtual object display unit according to the historical jitter parameters measured by the motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the predicted jitter parameters and the position and posture of the virtual object display unit after anti-shake compensation.

13. The method according to claim 12, characterized in that The step of determining the position and posture of the virtual object display unit at the time of displaying the virtual object according to the predicted jitter parameter and the position and posture of the virtual object display unit after anti-shake compensation comprises: Determining the position and posture of the virtual object display unit at the start time of rendering the virtual object according to the position and posture of the virtual object display unit after anti-shake compensation; The position and posture of the virtual object display unit at the time of displaying the virtual object are obtained according to the predicted jitter parameter and the position and posture of the virtual object display unit at the time of starting rendering of the virtual object.

14. A device for determining the position and posture of a virtual object display unit, characterized in that: include: A determination module, used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body; An acquisition module, used for acquiring jitter parameters measured by a motion detection device; The determination module is further used to determine the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit.

15. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 13.

Images