Method and device for determining pose of virtual object display unit and electronic equipment

Abstract

According to the method and device for determining the pose of the virtual object display unit and the electronic equipment, in the method for determining the pose of the virtual object display unit, the current pose of the virtual object display unit is determined according to the current pose of the vehicle body, and then the jitter parameter measured by the motion detection device is obtained; and according to the shaking parameter and the current pose of the virtual object display unit, determining the pose of the virtual object display unit at the display moment of the virtual object, thereby realizing that the virtual object displayed by the virtual object display unit can fit an actual scene and is not influenced by the bumping of a vehicle body to shake.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of intelligent terminal technology, and in particular to a method, device and electronic device for determining the posture of a virtual object display unit. Background Art

[0002] The in-vehicle display is an important interface for users to interact with the in-vehicle computer. Generally, the in-vehicle display is installed in the dashboard or center console. The driver will inevitably look down to check the contents of the in-vehicle display while driving, which may lead to driving safety issues. The in-vehicle head-up display (HUD) projects the information that the driver needs to pay attention to during driving onto the windshield or other display media in front of the field of vision, thereby avoiding the driver's frequent diversion of his eyes from the road and ensuring driving safety. Augmented reality (AR) refers to the superimposition and display of virtual digital information (such as text, images and / or three-dimensional models, etc.) on the real physical world, thereby achieving a visual effect of the fusion of the real world and the virtual world. Augmented reality in-vehicle head-up display (AR-HUD) is a technology that combines augmented reality and head-up display. Figure 1 This is a schematic diagram of the projection effect of the vehicle-mounted AR-HUD. After the driving information with AR effects (such as navigation instructions, vehicle speed and / or automatic driving information, etc.) is rendered, it is projected onto the windshield through a virtual object display unit (also called a "light machine"), thereby realizing the fusion display of virtual driving information and actual driving scenes, providing the driver with an immersive driving experience. Summary of the invention

[0003] The embodiments of the present application provide a method, device and electronic device for determining the posture of a virtual object display unit. The embodiments of the present application also provide a computer-readable storage medium to ensure that the virtual object displayed by the virtual object display unit can fit the actual scene and will not shake due to the bumps of the vehicle body.

[0004] In a first aspect, an embodiment of the present application provides a method for determining the posture of a virtual object display unit, comprising: determining the current posture of the virtual object display unit according to the current posture of a vehicle body; obtaining jitter parameters measured by a motion detection device; and determining the posture of the virtual object display unit at the moment of display of the virtual object according to the jitter parameters and the current posture of the virtual object display unit.

[0005] In the method for determining the posture of the virtual object display unit, the current posture of the virtual object display unit is determined according to the current posture of the vehicle body, and then the jitter parameters measured by the motion detection device are obtained. According to the jitter parameters and the current posture of the virtual object display unit, the posture of the virtual object display unit at the moment of displaying the virtual object is determined, so that the virtual object displayed by the virtual object display unit can fit the actual scene and will not jitter due to the bumps of the vehicle body.

[0006] In one possible implementation, the current vehicle posture is transmitted to the virtual object display unit by a vehicle posture calculation unit; the motion detection device includes a first motion detection device, and the first motion detection device is rigidly connected to the virtual object display unit; and obtaining the jitter parameter measured by the motion detection device includes: obtaining the first jitter parameter measured by the first motion detection device during the vehicle posture transmission delay.

[0007] In this implementation, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit is determined based on the data of the vehicle-mounted sensor after the vehicle posture calculation unit obtains the data of the vehicle-mounted sensor, see Figure 8 In this implementation, the current vehicle posture refers to the posture at the end of the current frame posture calculation time (i.e., time 1), and the current posture of the virtual object display unit can be the posture of the virtual object display unit at the end of the transmission delay (i.e., time 2).

[0008] In one possible implementation manner, determining the posture of the virtual object display unit at the time of display of the virtual object based on the jitter parameter and the current posture of the virtual object display unit includes: compensating the current posture of the virtual object display unit based on the first jitter parameter; predicting a third jitter parameter during virtual object rendering based on historical jitter parameters measured by the first motion detection device; and determining the posture of the virtual object display unit at the time of display of the virtual object based on the third jitter parameter and the posture of the virtual object display unit after compensation.

[0009] In one possible implementation, compensating for the current posture of the virtual object display unit according to a first jitter parameter measured during the vehicle body posture transmission delay includes: obtaining a posture change of the virtual object display unit during the vehicle body posture transmission delay according to the first jitter parameter; and compensating for the current posture of the virtual object display unit according to the posture change.

[0010] In one possible implementation, determining the current posture of the virtual object display unit according to the current vehicle body posture includes: determining the current posture of the virtual object display unit according to the current vehicle body posture and a pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0011] In one possible implementation manner, determining the posture of the virtual object display unit at the time of display of the virtual object according to the third jitter parameter and the posture of the virtual object display unit after compensation includes: determining the posture of the virtual object display unit at the start time of rendering of the virtual object according to the posture of the virtual object display unit after compensation; obtaining the posture of the virtual object display unit at the time of display of the virtual object according to the third jitter parameter and the posture of the virtual object display unit at the start time of rendering of the virtual object.

[0012] In some examples, obtaining the posture of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the virtual object display unit at the time of rendering the virtual object can be: obtaining the initial posture value of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the virtual object display unit at the time of rendering the virtual object, and smoothing the initial posture value of the virtual object display unit at the time of displaying the virtual object using the historical posture of the virtual object display unit, and finally obtaining the posture of the virtual object display unit at the time of displaying the virtual object. Smoothing the initial posture value of the virtual object display unit at the time of displaying the virtual object is to filter out points with large differences between the initial posture value at the time of displaying the virtual object and the historical posture of the virtual object display unit, and to reduce the difference between the posture of the virtual object display unit at the time of displaying the virtual object and the historical posture of the virtual object display unit. When smoothing the initial value of the posture of the virtual object display unit at the time of displaying the above-mentioned virtual object, methods such as averaging or filtering can be used, but the embodiments of the present application are not limited to this. Any method that can reduce the difference between the posture of the virtual object display unit at the time of displaying the above-mentioned virtual object and the historical posture of the virtual object display unit should fall within the protection scope of the embodiments of the present application.

[0013] In the above implementation, the virtual object display unit obtains the current vehicle body posture, determines the current posture of the virtual object display unit according to the current vehicle body posture and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, and then obtains the first jitter parameter measured by the first motion detection device during the above vehicle body posture transmission delay period, and obtains the posture change of the virtual object display unit during the above vehicle body posture transmission delay period according to the first jitter parameter measured during the vehicle body posture transmission delay period; and according to the above posture change, compensates for the current posture of the virtual object display unit, and then the virtual object display unit predicts the third jitter parameter during the virtual object rendering period according to the historical jitter parameter measured by the first motion detection device. Finally, the virtual object display unit determines the posture of the virtual object display unit at the display moment of the above virtual object according to the third jitter parameter and the posture of the virtual object display unit after compensation, so that the virtual object displayed by the virtual object display unit can fit the actual scene and is not affected by the bumps of the vehicle body and shakes.

[0014] In one possible implementation, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit includes: the vehicle posture after anti-shake compensation; the vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining the second shake parameter measured by the second motion detection device, and performing anti-shake compensation on the current vehicle posture according to the second shake parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is non-rigidly connected to the virtual object display unit. Fig.10 In this implementation, the current vehicle posture obtained by the virtual object display unit may be the vehicle posture at time 3.

[0015] In one possible implementation, determining the current posture of the virtual object display unit according to the current vehicle posture includes: determining the current posture of the virtual object display unit according to the vehicle posture after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit. Fig.10 In this implementation, the current posture of the virtual object display unit may be the posture of the virtual object display unit at the moment when the transmission delay ends (ie, moment 4).

[0016] In one possible implementation, after determining the current posture of the virtual object display unit according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, it also includes: determining the relative jitter between the vehicle chassis and the virtual object display unit according to the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device; and performing anti-shake compensation on the current posture of the virtual object display unit according to the relative jitter.

[0017] In one possible implementation manner, compensating the current posture of the virtual object display unit according to the first jitter parameter includes: obtaining a posture change of the virtual object display unit during the transmission time delay of the vehicle posture according to the first jitter parameter; and compensating the posture of the virtual object display unit after anti-shake compensation according to the posture change.

[0018] In one possible implementation manner, the first motion detection device is rigidly connected to the virtual object display unit, including: the first motion detection device is installed on a component rigidly connected to the virtual object display unit.

[0019] The above implementation method can achieve that the virtual object displayed by the virtual object display unit can fit the actual scene and will not shake due to the bumps of the vehicle body. When the vehicle posture calculation unit on the chassis is non-rigidly connected to the virtual object display unit, the above implementation method combines the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device to perform anti-shake compensation and delay compensation on the posture of the virtual object display unit, thereby solving the posture calculation error problem caused by the relative jitter and data transmission delay between the vehicle posture calculation unit and the virtual object display unit. In addition, this embodiment also predicts the posture of the virtual object display unit at the time of display of the virtual object based on the historical jitter parameters of the virtual object display unit, solving the problem that the posture of the virtual object display unit at the time of display of the virtual object cannot be obtained due to the lack of sensor data during rendering.

[0020] In one possible implementation, the motion detection device includes a second motion detection device, the second motion detection device is rigidly connected to a vehicle body posture calculation unit, and the vehicle body posture calculation unit is rigidly connected to the virtual object display unit; determining the posture of the virtual object display unit at the time of display of the virtual object according to the jitter parameters and the current posture of the virtual object display unit includes: performing anti-shake compensation on the current posture of the virtual object display unit according to the jitter parameters currently measured by the second motion detection device; predicting the jitter parameters of the virtual object display unit during posture transmission and virtual object rendering according to historical jitter parameters measured by the motion detection device; determining the posture of the virtual object display unit at the time of display of the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation.

[0021] In this implementation, the current body posture can be obtained by the body posture calculation unit; specifically, the body posture calculation unit can obtain the data of the vehicle-mounted sensor, and determine the current body posture according to the data of the vehicle-mounted sensor. Referring to FIG11(b), in this implementation, the current body posture is determined by the body posture calculation unit according to the data of the vehicle-mounted sensor, so the current body posture can be the body posture at the end of the current frame posture calculation time (i.e., time 5). In addition, the current posture of the virtual object display unit 4021 is determined by the body posture calculation unit according to the current body posture, so it can be considered that after the body posture calculation unit determines the current body posture, the current posture of the virtual object display unit can be determined, so referring to FIG11(b), in this implementation, the current posture of the virtual object display unit can be understood as the posture of the virtual object display unit at time 5.

[0022] In one possible implementation manner, determining the posture of the virtual object display unit at the time of display of the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation includes: determining the posture of the virtual object display unit at the start time of rendering of the virtual object according to the posture of the virtual object display unit after anti-shake compensation; obtaining the posture of the virtual object display unit at the time of display of the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit at the start time of rendering of the virtual object.

[0023] In the present implementation, the vehicle body posture calculation unit obtains the posture of the virtual object display unit at the display time of the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit at the start time of rendering the virtual object. The vehicle body posture calculation unit obtains the initial posture value of the virtual object display unit at the display time of the virtual object according to the posture of the virtual object display unit at the start time of rendering the virtual object and the above-mentioned predicted jitter parameters, and uses the historical posture of the virtual object display unit to smooth the initial posture value of the virtual object display unit at the display time of the virtual object, and finally obtains the posture of the virtual object display unit at the display time of the virtual object, and sends it to the virtual object display unit.

[0024] In the above implementation, after the vehicle body posture calculation unit obtains the current vehicle body posture, it determines the current posture of the virtual object display unit according to the current vehicle body posture, and then the vehicle body posture calculation unit obtains the jitter parameters currently measured by the second motion detection device rigidly connected to the vehicle body posture calculation unit, and performs anti-shake compensation on the current posture of the virtual object display unit according to the currently measured jitter parameters, and then the vehicle body posture calculation unit predicts the jitter parameters of the virtual object display unit during the posture transmission period and the virtual object rendering period according to the historical jitter parameters measured by the second motion detection device, and determines the posture of the virtual object display unit at the display time of the above virtual object according to the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation. Finally, the vehicle body posture calculation unit transmits the posture of the virtual object display unit at the display time of the above virtual object to the virtual object display unit, so that the virtual object displayed by the virtual object display unit can fit the actual scene and will not shake due to the bumps of the vehicle body.

[0025] In the second aspect, an embodiment of the present application provides a device for determining the posture of a virtual object display unit, the device is included in an electronic device, and the device has the function of realizing the behavior of the electronic device in the first aspect and the possible implementation of the first aspect. The function can be implemented by hardware, or by hardware executing the corresponding software implementation. The hardware or software includes one or more modules or units corresponding to the above functions. For example, an acquisition module and a determination module, etc.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method provided in the first aspect.

[0027] It should be understood that the second and third aspects of the embodiments of the present application are consistent with the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here.

[0028] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0030] In one possible design, the program in the fifth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a schematic diagram of the projection effect of the vehicle-mounted AR-HUD;

[0032] Figure 2 The figure is a schematic diagram of the installation of the vehicle-mounted virtual object display unit;

[0033] Figure 3 A schematic diagram of the display principle of the vehicle-mounted virtual object display unit;

[0034] Figure 4 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0035] Figure 5 A hardware schematic diagram provided for one embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a vehicle posture calculation unit 4011 provided in one embodiment of the present application;

[0037] Figure 7 A flowchart of a method for determining a virtual object display unit posture provided by an embodiment of the present application;

[0038] Figure 8 A working timing diagram provided for an embodiment of the present application;

[0039] Fig. 9 A flowchart of a method for determining a virtual object display unit posture provided by another embodiment of the present application;

[0040] Fig.10 A working timing diagram provided for another embodiment of the present application;

[0041] FIG11( a) is a flow chart of a method for determining a virtual object display unit posture according to another embodiment of the present application;

[0042] FIG11( b ) is a working timing diagram provided in yet another embodiment of the present application;

[0043] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0044] Fig.13 A schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0045] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0046] Figure 2 This is a schematic diagram of the installation of the vehicle-mounted virtual object display unit. Figure 3 Schematic diagram of the display principle of the vehicle-mounted virtual object display unit, such as Figure 2 and Figure 3 As shown, AR-HUD optical machine (i.e. Figure 2 The virtual object display unit in the vehicle is generally installed in front of the driving position in the vehicle cockpit, while the sensors and processors used to calculate the vehicle posture (i.e. Figure 2 The vehicle posture calculation unit in the AR-HUD is generally installed outside the cockpit (such as the frame, chassis or body surface). In order to achieve a display effect that combines virtual and real, the AR-HUD optical machine needs to accurately calculate the vehicle posture through the on-board sensor, and then deduce the accurate posture of the AR-HUD optical machine in the real world based on the positional relationship between the vehicle body and the AR-HUD optical machine, thereby calculating the 3D projection position of the virtual driving information on the windshield. Therefore, the calculation accuracy of the posture and the anti-shake processing effect will affect the quality of the AR-HUD virtual and real experience.

[0047] like Figure 2As shown in the figure, generally speaking, sensor data or posture data is obtained by the vehicle posture calculation unit in the chassis domain, and the virtual object display unit using these data is located in the cockpit domain. The data transmission from the chassis domain to the cockpit domain often has a long delay (20ms to 100ms), so the posture transmitted to the virtual object display unit lags behind the actual situation. If this posture is used to render and display virtual driving information, the virtual object will not fit the actual scene. In addition, due to the particularity of the automobile suspension structure, the cockpit and the chassis are generally not rigidly connected. During driving, due to the bumpy road, there will be relative movement between the cockpit and the chassis. Even if there is no data transmission delay, the body posture calculated by the sensor installed in the chassis domain is used to infer the posture of the virtual object display unit installed in the cockpit domain. Due to the relative movement of the two, errors will be introduced, resulting in the virtual driving information displayed by the AR-HUD not fitting the actual scene or having obvious jitter.

[0048] Most of the existing vehicle-mounted HUD anti-shake technologies focus on the overall vehicle body jitter compensation or the relative jitter compensation between the internal components of the virtual object display unit. They do not solve the posture lag and jitter problems caused by data transmission delays, nor do they solve the relative jitter problems caused by the non-rigid connection between the virtual object display unit and the vehicle-mounted sensor. Among them, the non-rigid connection means that there is relative movement between the virtual object display unit and the vehicle-mounted sensor, and it is not a fixed connection.

[0049] The basic principle of an existing vehicle-mounted HUD anti-shake solution is to suppress shaking by installing a mechanical device. Common anti-shake device installation locations include between the cockpit and the virtual object display unit, and between the internal components of the virtual object display unit.

[0050] Among them, one implementation scheme is to connect the HUD display medium to the cockpit through a shock-absorbing mechanism to prevent display jitter caused by relative movement between the two; another implementation scheme is to connect the HUD display lens through a support rod fixing block and a support rod sleeve, and fix the HUD display lens through a special mechanical structure to prevent it from shaking; another implementation scheme is to design the HUD reflector bracket as a pivot connection and multi-point fixing method to reduce the jitter of the reflector.

[0051] However, the above implementation only targets the relative jitter between the virtual object display unit and the cockpit or the relative jitter between the internal components of the virtual object display unit. When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, it is impossible to solve the virtual object jitter problem caused by the relative jitter between the two, nor the virtual object misalignment problem caused by the delay in posture data transmission.

[0052] The basic principle of another existing vehicle-mounted HUD anti-shake solution is to detect the shaking of the entire vehicle body relative to the ground and control the virtual object display unit to perform corresponding shaking compensation movements. One implementation scheme is to obtain the shaking parameters (including offset and angle) of the vehicle body relative to the ground through an inertial measurement device installed on the vehicle body, and then adjust the HUD image generation device (i.e., the virtual object display unit) in the opposite direction of the vehicle body shaking by adjusting the control circuit, thereby compensating for the HUD screen shaking problem caused by the vehicle body shaking.

[0053] However, there is a data transmission delay when the inertial measurement device installed on the vehicle body transmits the jitter parameters to the HUD image generation device. The jitter parameters used for HUD image jitter compensation lag behind the actual jitter situation. This solution cannot handle the jitter during the transmission delay. In addition, when the HUD image generation device is controlled to perform compensation movement according to the jitter parameters, the response of the control circuit has a delay. This solution cannot handle the jitter during the response delay. In addition, when the HUD image generation device is non-rigidly connected to the vehicle body or the sensor on the vehicle body, this solution cannot solve the virtual object jitter problem caused by the relative jitter between the two.

[0054] The basic principle of another existing vehicle-mounted HUD anti-shake solution is to detect the jitter between the internal components of the virtual object display unit and adjust the display position of the virtual object display unit for anti-shake compensation. One implementation scheme is to install angular velocity sensors on the head-up display and the rotatable reflector inside the virtual object display unit respectively, and calculate the compensation amount of the HUD display screen by detecting the relative jitter parameters between the two internal components and display the compensation.

[0055] However, this solution only addresses the problem of image jitter caused by relative jitter between components inside the virtual object display unit. When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, it cannot solve the problem of virtual object jitter caused by relative jitter between the two, nor can it solve the problem of virtual object misalignment caused by the delay in posture data transmission.

[0056] The basic principle of another existing vehicle-mounted HUD anti-shake solution is to detect the jitter between the virtual object display unit and the cockpit, and adjust the display position of the virtual object display unit for anti-shake compensation. Among them, one implementation scheme is to detect the jitter of the positioning mark on the car windshield through an image sensor installed on the virtual object display unit, thereby calculating the jitter of the HUD projection image relative to the windshield, and then offset compensating the display image. However, the above scheme only targets the relative jitter between the virtual object display unit and the cockpit (for example: windshield). When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, it cannot solve the virtual object jitter problem caused by the relative jitter between the two, nor can it solve the virtual object misalignment problem caused by the posture data transmission delay.

[0057] Generally speaking, the on-board sensors used to calculate the vehicle's posture (e.g., inertial measurement unit (IMU), wheel speed meter, camera, light detection and ranging (Lidar), global navigation satellite system (GNSS) and / or real-time kinematic (RTK)) are installed on the vehicle (e.g., vehicle shell, chassis, axle or wheel, etc.), and the AR-HUD virtual object display unit is generally installed in front of the driver's seat in the cockpit. The on-board sensors and the virtual object display unit are generally non-rigidly connected, and there will be relative jitter between the two when the vehicle is driving. When displaying virtual objects, the virtual object display unit confirms the display position of the virtual object in the projection screen based on the posture of the virtual object display unit and the posture of the virtual object in the real scene. Generally, the posture of the vehicle body is calculated by the on-board sensor, and then the posture of the virtual object display unit is inferred from the posture of the vehicle body. The vehicle-mounted sensors and vehicle posture calculation units are generally located in the chassis domain, and the virtual object display unit is generally located in the cockpit domain. There is a delay in the transmission of sensor data or posture data from the chassis domain to the cockpit domain. On the one hand, the relative jitter between the vehicle-mounted sensors and the virtual object display unit during the delay will lead to inaccurate posture. On the other hand, this will cause the posture acquired by the virtual object display unit to lag behind the actual situation. Therefore, when the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, if there is relative jitter and data transmission delay between the two, the calculated posture of the virtual object display unit will be inaccurate, resulting in the virtual object not fitting the real scene or jittering.

[0058] In addition, there is no sensor data during the period from calculating the position and posture of the virtual object display unit to actually displaying the virtual object. It is necessary to predict the position and posture of the virtual object display unit at the display time in advance based on historical sensor data.

[0059] As mentioned above, the existing technical solutions do not solve the relative jitter problem between the virtual object display unit and the vehicle-mounted sensor, the posture lag problem caused by data transmission delay, and the posture lag problem caused by virtual object rendering delay.

[0060] Based on the above problems, an embodiment of the present application provides a method for determining the posture of a virtual object display unit. When the virtual object display unit and the vehicle-mounted sensor are non-rigidly connected, the method for determining the posture of the virtual object display unit provided by the embodiment of the present application can detect the relative jitter between the two and compensate for it. When there is a delay in the transmission of data from the vehicle-mounted sensor to the virtual object display unit, the embodiment of the present application can solve the posture lag problem caused by the delay, and can also predict the posture of the virtual object display unit at the moment of virtual object display based on historical sensor data in advance, so that the virtual object displayed by the virtual object display unit is more in line with the real scene. It should be noted that in the embodiment of the present application, the posture may include position and attitude, that is, the three-dimensional translation and three-dimensional rotation relationship of a point relative to a coordinate system.

[0061] The embodiment of the present application obtains jitter parameters through a motion detection device installed on the virtual object display unit and the vehicle body, which is used to perform anti-shake compensation and advance prediction on the posture of the virtual object display unit, so that the placement of the virtual object is more consistent with the real scene.

[0062] Specifically, the embodiment of the present application combines the jitter parameters of the vehicle-mounted sensor on the vehicle body and the jitter parameters of the virtual object display unit to detect the relative movement of the two due to the non-rigid connection during data transmission, and performs anti-shake compensation and delay compensation on the calculated position and posture of the virtual object display unit. In addition, the jitter parameters during the rendering and display of the virtual object are calculated using the historical jitter parameters of the virtual object display unit and the historical jitter parameters of the vehicle-mounted sensor on the vehicle body, and the position and posture of the virtual object display unit at the time of displaying the virtual object is predicted in advance based on the position and posture of the virtual object display unit at the latest moment and the predicted jitter parameters.

[0063] The method for determining the position and posture of a virtual object display unit provided in the embodiment of the present application can be applied to the AR-HUD system of a smart car. Figure 4 A schematic diagram of a system architecture provided for an embodiment of the present application is shown in FIG. Figure 4 As shown, the above system architecture may include a chassis domain 401 , a cockpit domain 402 , a data transmission unit 403 and a clock synchronization unit 404 .

[0064] The chassis domain 401 may include a vehicle posture calculation unit 4011 and a second motion detection device 4012, and the cockpit domain 402 may include a virtual object display unit 4021 and a first motion detection device 4022. The chassis domain 401 and the cockpit domain 402 transmit posture or sensor data through a data transmission unit 403. Since the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and the data transmission between them has a time delay, a clock synchronization unit 404 is required to perform clock synchronization between the chassis domain 401 and the cockpit domain 402.

[0065] The chassis domain may also be referred to as a mobile data center (MDC) domain, and the cockpit domain may also be referred to as a cockpit domain controller (CDC) domain.

[0066] Specifically, the vehicle posture calculation unit 4011 may generally include an MDC and various types of vehicle-mounted sensors. The MDC may include a central processing unit (CPU), a graphics processing unit (GPU), a cache, a data communication bus, etc. It is mainly used to obtain vehicle-mounted sensor data and calculate the vehicle posture or the posture of the virtual object display unit. In the embodiment of the present application, the vehicle posture calculation unit 4011 may also be referred to as a vehicle-mounted sensor for calculating the vehicle posture.

[0067] The second motion detection device 4012 is generally installed on a vehicle component (such as a vehicle shell, chassis or axle) that is rigidly connected to the vehicle posture calculation unit 4011. Common second motion detection devices include accelerometers, gyroscopes, cameras, lidars, photoelectric sensors, inertial measurement units (IMUs) or inertial elements, etc., which are mainly used to detect the jitter parameters of the vehicle-mounted sensors in the chassis domain 401 for calculating the vehicle posture, and perform anti-shake compensation on the vehicle posture based on the jitter parameters.

[0068] The data transmission unit 403 generally includes a data communication line and a data transmission control unit, and is mainly used to transmit the position and posture or sensor data of the chassis domain 401 to the cockpit domain 402 .

[0069] The clock synchronization unit 404 is used to ensure that the devices in the chassis domain 401 and the devices in the cockpit domain 402 maintain clock synchronization.

[0070] The first motion detection device 4022 is generally installed on the virtual object display unit 4021, or installed on a component rigidly connected to the virtual object display unit 4021, such as a dashboard, a steering wheel, a driving recorder, or an interior rearview mirror. Common first motion detection devices 4022 include accelerometers, gyroscopes, cameras, lidars, photoelectric sensors, IMUs, or inertial elements, and are mainly used to obtain jitter parameters of the virtual object display unit 4021.

[0071] The virtual object display unit 4021, i.e., the AR-HUD virtual object display unit (e.g., AR-HUD optical machine / projector), is mainly used to render virtual objects and project them onto the windshield of the vehicle. In specific implementation, the virtual object display unit 4021 can be a HUD optical machine, a projector, or a HUD display.

[0072] Figure 5 This is a hardware diagram provided for an embodiment of the present application. For specific installation, see Figure 5 The vehicle posture calculation unit 4011, the second motion detection device 4012, the virtual object display unit 4021, the first motion detection device 4022, the data transmission unit 403 and the clock synchronization unit 404 can be installed in the chassis domain 401 and the cockpit domain 402, respectively. The vehicle posture calculation unit 4011 and the second motion detection device 4012 are installed in the chassis domain 401, and the virtual object display unit 4021 and the first motion detection device 4022 are installed in the cockpit domain 402. There is a data transmission delay between the chassis domain 401 and the cockpit domain 402, and the devices in the two areas are clock synchronized through the clock synchronization unit 404.

[0073] In specific implementation, the vehicle posture calculation unit 4011 can use Figure 6 The structure shown implements, Figure 6 A structural diagram of a vehicle posture calculation unit 4011 provided in one embodiment of the present application is shown in FIG. Figure 6 As shown, the vehicle posture calculation unit 4011 may include: a processor 410 and a communication interface 420. Optionally, the vehicle posture calculation unit 4011 may also include a memory 430. The processor 410, the communication interface 420 and the memory 430 may communicate with each other through an internal connection path to transmit control and / or data signals, the memory 430 is used to store a computer program, and the processor 410 is used to call and run the computer program from the memory 430.

[0074] The processor 410 and the memory 430 may be combined into a processing device, or more commonly, they are independent components, and the processor 410 is used to execute the program code stored in the memory 430. In specific implementation, the memory 430 may also be integrated into the processor 410, or may be independent of the processor 410.

[0075] Optionally, the vehicle body posture calculation unit 4011 may further include a power supply 450 for providing power to various devices or circuits in the vehicle body posture calculation unit 4011 .

[0076] It should be understood that Figure 6 The processor 410 in the vehicle posture calculation unit 4011 shown may be a system on chip SOC, and the processor 410 may include a CPU, and may further include other types of processors, such as a GPU, etc.

[0077] Similarly, the virtual object display unit 4021 can also use Figure 6 The structural implementation shown will not be repeated here.

[0078] The method for determining the posture of a virtual object display unit provided in an embodiment of the present application may include: determining the current posture of the virtual object display unit according to the current posture of the vehicle body, then obtaining the jitter parameter measured by the motion detection device, and finally determining the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit. Figure 4 , Figure 5 and Figure 6 , the method for determining the position and posture of the virtual object display unit provided in an embodiment of the present application is specifically explained.

[0079] Figure 7 A flowchart of a method for determining a virtual object display unit posture provided by an embodiment of the present application is shown in FIG. Figure 7 As shown, the method for determining the position and posture of the virtual object display unit may include:

[0080] Step 701 , the virtual object display unit 4021 obtains the current vehicle body posture; wherein the current vehicle body posture is transmitted to the virtual object display unit 4021 by the vehicle body posture calculation unit 4011 .

[0081] In this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit 4011 to the virtual object display unit 4021 is determined based on the data of the vehicle-mounted sensor after the vehicle posture calculation unit 4011 obtains the data of the vehicle-mounted sensor. Figure 8 In this embodiment, the current vehicle posture refers to the posture at the end of the current frame posture calculation time (i.e., time 1).

[0082] Step 702 : The virtual object display unit 4021 determines the current posture of the virtual object display unit 4021 according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0083] The current posture of the virtual object display unit 4021 is determined according to the current body posture after the virtual object display unit 4021 obtains the current body posture transmitted by the body posture calculation unit 4011. Therefore, in this embodiment, see Figure 8 , the current posture of the virtual object display unit 4021 may be the posture of the virtual object display unit 4021 at the moment when the transmission delay ends (ie, moment 2).

[0084] Step 703, the virtual object display unit 4021 obtains the first jitter parameter measured by the first motion detection device 4022 during the above-mentioned vehicle posture transmission delay period; wherein, the first motion detection device is rigidly connected to the virtual object display unit 4021, and the first jitter parameter can be the jitter parameter of the virtual object display unit 4021.

[0085] Among them, since the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and the data transmission between them has a delay, the vehicle posture transmission delay can be the time used by the vehicle posture calculation unit 4011 to transmit the current vehicle posture to the virtual object display unit 4021.

[0086] In this embodiment, the first motion detection device is rigidly connected to the virtual object display unit 4021 , which may be: the first motion detection device is installed on a component rigidly connected to the virtual object display unit 4021 .

[0087] Step 704: the virtual object display unit 4021 obtains the posture change of the virtual object display unit 4021 during the vehicle body posture transmission time delay according to the first jitter parameter; and compensates the current posture of the virtual object display unit 4021 according to the posture change.

[0088] Step 705 : The virtual object display unit 4021 predicts a third jitter parameter during virtual object rendering according to the historical jitter parameter measured by the first motion detection device 4022 .

[0089] Step 706: The virtual object display unit 4021 determines the position and posture of the virtual object display unit 4021 at the time of displaying the virtual object according to the third jitter parameter and the position and posture of the virtual object display unit 4021 after compensation.

[0090] Specifically, the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation. The posture can be: the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object according to the posture of the virtual object display unit after compensation; the posture of the virtual object display unit 4021 at the display time of the virtual object is obtained according to the third jitter parameter and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object.

[0091] In a specific implementation, the posture of the virtual object display unit 4021 at the time of display of the virtual object is obtained according to the third jitter parameter and the posture of the virtual object display unit 4021 at the time of starting rendering of the above-mentioned virtual object: the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is obtained according to the posture of the virtual object display unit 4021 at the time of starting rendering of the above-mentioned virtual object and the third jitter parameter, and the initial posture value of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object is smoothed by using the historical posture of the virtual object display unit 4021 to obtain the posture of the virtual object display unit 4021 at the time of display of the above-mentioned virtual object.

[0092] Among them, the initial value of the posture of the virtual object display unit 4021 at the time of displaying the virtual object is smoothed in order to filter out the points with large differences between the initial value of the posture at the time of displaying the virtual object and the historical posture of the virtual object display unit 4021, and reduce the difference between the posture of the virtual object display unit 4021 at the time of displaying the virtual object and the historical posture of the virtual object display unit 4021. When smoothing the initial value of the posture of the virtual object display unit 4021 at the time of displaying the virtual object, methods such as averaging or filtering can be used, but the embodiments of the present application are not limited thereto, and any method that can reduce the difference between the posture of the virtual object display unit 4021 at the time of displaying the virtual object and the historical posture of the virtual object display unit 4021 should fall within the protection scope of the embodiments of the present application.

[0093] The method provided in this embodiment can be applied in a scenario where the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are rigidly connected. In some examples, when the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are rigidly connected, the virtual object display unit 4021 only needs to obtain the first jitter parameter from the first motion detection device 4022 to perform delay compensation and advance prediction of the posture of the virtual object display unit 4021. Figure 8A working timing diagram is provided for an embodiment of the present application, such as Figure 8 As shown, in this embodiment, there is no relative jitter between the vehicle posture calculation unit 4011 and the virtual object display unit 4021, so there is no need to obtain the jitter parameters measured by the second motion detection device 4012. It is only necessary to use the first jitter parameters measured by the first motion detection device to perform transmission delay compensation and advance prediction.

[0094] In the above-mentioned method for determining the posture of the virtual object display unit, the virtual object display unit 4021 obtains the current posture of the vehicle body, determines the current posture of the virtual object display unit 4021 according to the current posture of the vehicle body, and then obtains the first jitter parameter measured by the first motion detection device 4022 during the above-mentioned vehicle body posture transmission delay period, and obtains the posture change of the virtual object display unit 4021 during the above-mentioned vehicle body posture transmission delay period according to the first jitter parameter measured during the vehicle body posture transmission delay period; and according to the above-mentioned posture change, the posture of the virtual object display unit 4021 is adjusted. The current posture is compensated, and then the virtual object display unit 4021 predicts the third jitter parameter during the virtual object rendering according to the historical jitter parameter measured by the first motion detection device 4022. Finally, the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the display moment of the above-mentioned virtual object according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation, so that the virtual object displayed by the virtual object display unit 4021 can fit the actual scene and is not jittered by the bumps of the vehicle body.

[0095] In this embodiment, when the vehicle posture calculation unit 4011 and the virtual object display unit 4021 are rigidly connected, the virtual object display unit 4021 uses the first jitter parameter measured by the first motion detection device 4022 to perform delay compensation on the posture of the virtual object display unit, which can solve the posture calculation error problem caused by the posture or sensor data transmission delay. In addition, the posture of the virtual object display unit 4021 at the time of displaying the virtual object can be predicted in advance based on the historical jitter parameters of the virtual object display unit 4021, thereby solving the problem of not being able to obtain the posture of the virtual object display unit 4021 at the time of displaying the virtual object due to the lack of sensor data during the rendering of the virtual object.

[0096] Fig. 9 A flowchart of a method for determining a virtual object display unit posture provided in another embodiment of the present application is shown in FIG. Fig. 9 As shown, the method for determining the position and posture of the virtual object display unit may include:

[0097] Step 901 , the virtual object display unit 4021 obtains the current vehicle body posture; wherein the current vehicle body posture is transmitted to the virtual object display unit 4021 by the vehicle body posture calculation unit 4011 .

[0098] In this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit 4011 to the virtual object display unit 4021 is the vehicle posture after the vehicle posture calculation unit 4011 obtains the data of the vehicle-mounted sensor, determines it according to the data of the vehicle-mounted sensor, and performs anti-shake compensation. Fig.10 In this embodiment, the current vehicle body posture obtained by the virtual object display unit 4021 may be the vehicle body posture at time 3.

[0099] Specifically, the vehicle posture after the anti-shake compensation is obtained by the vehicle posture calculation unit 4011 obtaining the second shake parameter measured by the second motion detection device 4012, and performing anti-shake compensation on the current vehicle posture according to the second shake parameter; wherein the second motion detection device 4012 is rigidly connected to the vehicle posture calculation unit 4011; and the vehicle posture calculation unit 4011 is non-rigidly connected to the virtual object display unit 4021.

[0100] Step 902 : The virtual object display unit 4021 determines the current posture of the virtual object display unit 4021 according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0101] Similarly, in this embodiment, the current posture of the virtual object display unit 4021 is determined according to the body posture after anti-shake compensation after the virtual object display unit 4021 obtains the body posture after anti-shake compensation transmitted by the body posture calculation unit 4011. Fig.10 In this embodiment, the current posture of the virtual object display unit 4021 may be the posture of the virtual object display unit 4021 at the moment when the transmission delay ends (ie, moment 4).

[0102] Step 903 : The virtual object display unit 4021 determines the relative jitter between the vehicle chassis and the virtual object display unit 4021 according to the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 .

[0103] Step 904: The virtual object display unit 4021 performs anti-shake compensation on the current position and posture of the virtual object display unit 4021 according to the above relative jitter.

[0104] Step 905 , the virtual object display unit 4021 obtains the first jitter parameter measured by the first motion detection device 4022 during the above-mentioned vehicle posture transmission delay period; wherein, the first motion detection device 4022 is rigidly connected to the virtual object display unit 4021 , and the first jitter parameter may be the jitter parameter of the virtual object display unit 4021 .

[0105] Among them, since the devices between the chassis domain 401 and the cockpit domain 402 are generally non-rigidly connected and the data transmission between them has a delay, the vehicle posture transmission delay can be the time used by the vehicle posture calculation unit 4011 to transmit the current vehicle posture to the virtual object display unit 4021.

[0106] In this embodiment, the first motion detection device is rigidly connected to the virtual object display unit 4021 , which may be: the first motion detection device is installed on a component rigidly connected to the virtual object display unit 4021 .

[0107] Step 906, the virtual object display unit 4021 obtains the posture change of the virtual object display unit 4021 during the above-mentioned vehicle body posture transmission delay period according to the first jitter parameter measured during the above-mentioned vehicle body posture transmission delay period; and according to the above-mentioned posture change, compensates the posture of the virtual object display unit 4021 after anti-shake compensation.

[0108] Step 907 : The virtual object display unit 4021 predicts a third jitter parameter during virtual object rendering according to the historical jitter parameter measured by the first motion detection device 4022 .

[0109] Step 908: The virtual object display unit 4021 determines the position and posture of the virtual object display unit 4021 at the time of displaying the virtual object according to the third jitter parameter and the position and posture of the virtual object display unit 4021 after compensation.

[0110] Specifically, according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 after compensation, the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object can be determined as follows: the virtual object display unit 4021 determines the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object according to the posture of the virtual object display unit 4021 after compensation; according to the above-mentioned third jitter parameter and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object, the posture of the virtual object display unit 4021 at the display time of the virtual object is obtained.

[0111] In a specific implementation, the posture of the virtual object display unit 4021 at the time of display of the virtual object is obtained according to the third jitter parameter and the posture of the virtual object display unit 4021 at the time of starting rendering of the virtual object: according to the posture of the virtual object display unit 4021 at the time of starting rendering of the virtual object and the third jitter parameter, the initial posture value of the virtual object display unit 4021 at the time of display of the virtual object is obtained, and the initial posture value of the virtual object display unit 4021 at the time of display of the virtual object is smoothed using the historical posture of the virtual object display unit 4021, and finally the posture of the virtual object display unit 4021 at the time of display of the virtual object is obtained.

[0112] The method provided in this embodiment can be applied in a scenario where the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are non-rigidly connected. In some examples, when the vehicle body posture calculation unit 4011 on the chassis and the virtual object display unit 4021 are non-rigidly connected, the virtual object display unit 4021 needs to combine the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and delay compensation on the posture of the virtual object display unit 4021. Fig.10 A working timing diagram is provided for another embodiment of the present application, such as Fig.10 As shown, in this embodiment, there is relative jitter between the vehicle posture calculation unit 4011 and the virtual object display unit 4021. Therefore, it is necessary to combine the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and delay compensation on the posture of the virtual object display unit 4021.

[0113] The above-mentioned method for determining the posture of the virtual object display unit can realize that the virtual object displayed by the virtual object display unit 4021 can fit the actual scene and is not affected by the bumps of the vehicle body and shakes. When the vehicle posture calculation unit 4011 on the chassis is non-rigidly connected to the virtual object display unit 4021, this embodiment combines the first jitter parameter measured by the first motion detection device 4022 and the second jitter parameter measured by the second motion detection device 4012 to perform anti-shake compensation and delay compensation on the posture of the virtual object display unit 4021, thereby solving the posture calculation error problem caused by the relative jitter and data transmission delay between the vehicle posture calculation unit 4011 and the virtual object display unit 4021. In addition, this embodiment also predicts the posture of the virtual object display unit 4021 at the time of displaying the virtual object based on the historical jitter parameters of the virtual object display unit 4021, solving the problem that the posture of the virtual object display unit 4021 at the time of displaying the virtual object cannot be obtained due to the lack of sensor data during rendering.

[0114] FIG. 11( a ) is a flow chart of a method for determining a virtual object display unit posture according to another embodiment of the present application, and FIG. 11( b ) is a working sequence diagram according to another embodiment of the present application. As shown in FIG. 11( a ) and FIG. 11( b ), the method for determining a virtual object display unit posture may include:

[0115] Step 1101, the vehicle body posture calculation unit 4011 obtains the current vehicle body posture.

[0116] Specifically, the vehicle posture calculation unit 4011 obtains the current vehicle posture as follows: the vehicle posture calculation unit 4011 obtains the data of the vehicle-mounted sensor, and determines the current vehicle posture according to the data of the vehicle-mounted sensor. Referring to FIG. 11( b ), in this embodiment, the current vehicle posture is determined by the vehicle posture calculation unit 4011 according to the data of the vehicle-mounted sensor, and therefore, the current vehicle posture may be the vehicle posture at the end of the current frame posture calculation time (i.e., time 5).

[0117] Step 1102 , the vehicle posture calculation unit 4011 determines the current posture of the virtual object display unit 4021 according to the current vehicle posture and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

[0118] In this embodiment, the current posture of the virtual object display unit 4021 is determined by the vehicle body posture calculation unit 4011 according to the current vehicle body posture. Therefore, it can be considered that after the vehicle body posture calculation unit 4011 determines the current vehicle body posture, the current posture of the virtual object display unit 4021 can be determined. Therefore, referring to Figure 11(b), in this embodiment, the current posture of the virtual object display unit 4021 can be understood as the posture of the virtual object display unit 4021 at time 5.

[0119] Step 1103 , the vehicle posture calculation unit 4011 obtains the jitter parameter currently measured by the second motion detection device 4012 ; wherein the second motion detection device 4012 is rigidly connected to the vehicle posture calculation unit 4011 , and the vehicle posture calculation unit 4011 is rigidly connected to the virtual object display unit 4021 .

[0120] Step 1104 , the vehicle posture calculation unit 4011 performs anti-shake compensation on the current posture of the virtual object display unit 4021 according to the jitter parameters currently measured by the second motion detection device 4012 .

[0121] Step 1105 , the vehicle posture calculation unit 4011 predicts the jitter parameters of the virtual object display unit 4021 during posture transmission and virtual object rendering according to the historical jitter parameters measured by the second motion detection device 4012 .

[0122] Step 1106 , the vehicle body posture calculation unit 4011 determines the posture of the virtual object display unit 4021 at the start time of rendering the virtual object according to the posture of the virtual object display unit 4021 after anti-shake compensation.

[0123] Step 1107, the vehicle body posture calculation unit 4011 obtains the posture of the virtual object display unit 4021 at the time of displaying the above virtual object according to the predicted jitter parameters and the posture of the virtual object display unit 4021 at the time of starting rendering of the above virtual object.

[0124] Specifically, the vehicle body posture calculation unit 4011 obtains the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object according to the predicted jitter parameters and the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object. The posture calculation unit 4011 obtains the initial value of the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object according to the posture of the virtual object display unit 4021 at the start time of rendering the above-mentioned virtual object and the above-mentioned predicted jitter parameters, and uses the historical posture of the virtual object display unit 4021 to smooth the initial value of the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object, and finally obtains the posture of the virtual object display unit 4021 at the display time of the above-mentioned virtual object, and sends it to the virtual object display unit 4021.

[0125] In the above-mentioned method for determining the posture of the virtual object display unit, after the vehicle body posture calculation unit 4011 obtains the current vehicle body posture, it determines the current posture of the virtual object display unit 4021 according to the current vehicle body posture, and then the vehicle body posture calculation unit 4011 obtains the jitter parameters currently measured by the second motion detection device 4012 rigidly connected to the vehicle body posture calculation unit 4011, and performs anti-shake compensation on the current posture of the virtual object display unit 4021 according to the currently measured jitter parameters, and then the vehicle body posture calculation unit 4011 predicts the jitter parameters of the virtual object display unit 4021 during the posture transmission period and the virtual object rendering period according to the historical jitter parameters measured by the second motion detection device 4012, and determines the posture of the virtual object display unit 4021 at the display moment of the above-mentioned virtual object according to the predicted jitter parameters and the posture of the virtual object display unit 4021 after anti-shake compensation. Finally, the vehicle posture calculation unit 4011 transmits the posture of the virtual object display unit 4021 at the time of displaying the above-mentioned virtual object to the virtual object display unit 4021, so that the virtual object displayed by the virtual object display unit 4021 can fit the actual scene and not shake due to the bumps of the vehicle body.

[0126] When the vehicle body posture calculation unit 4011 and the virtual object display unit 4021 are rigidly connected, this embodiment can predict in advance the posture of the virtual object display unit 4021 after the data transmission delay and the rendering time of the virtual object based on the historical jitter parameters of the chassis, thereby solving the posture jitter problem caused by the transmission delay and the lack of sensor data during rendering.

[0127] It should be noted that the method for determining the position and posture of the virtual object display unit provided in the embodiment of the present application can be applied not only to vehicle AR-HUD, but also to equipment such as aircraft, excavators or cranes with AR-HUD functions.

[0128] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0129] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0130] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0131] Fig.12 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application, in which each functional module is divided according to each function, Fig.12 A possible schematic diagram of the composition of the electronic device 1200 involved in the above embodiment is shown. Fig.12 As shown, the electronic device 1200 may include: an acquisition module 1201, a determination module 1202, a compensation module 1203 and a prediction module 1204;

[0132] The acquisition module 1201 is used to acquire the current vehicle posture; wherein the current vehicle posture is transmitted from the vehicle posture calculation unit to the virtual object display unit;

[0133] The determination module 1202 is used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body; in this embodiment, the determination module 1202 is specifically used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit;

[0134] The acquisition module 1201 is further used to acquire a first jitter parameter measured by the first motion detection device during the above-mentioned vehicle body posture transmission delay period; and obtain the posture change of the virtual object display unit during the above-mentioned vehicle body posture transmission delay period according to the first jitter parameter measured during the above-mentioned vehicle body posture transmission delay period; wherein the first motion detection device is rigidly connected to the virtual object display unit, and the first jitter parameter includes the jitter parameter of the above-mentioned virtual object display unit; the first motion detection device and the virtual object display unit are rigidly connected for example: the first motion detection device is installed on a component rigidly connected to the above-mentioned virtual object display unit;

[0135] A compensation module 1203, used to compensate the current posture of the virtual object display unit according to the above posture change;

[0136] A prediction module 1204, configured to predict a third jitter parameter during virtual object rendering according to the historical jitter parameter measured by the first motion detection device;

[0137] The determination module 1202 is further configured to determine the posture of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the virtual object display unit after compensation.

[0138] In one implementation of this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit is determined based on the data of the vehicle-mounted sensor after the vehicle posture calculation unit obtains the data of the vehicle-mounted sensor.

[0139] The determination module 1202 is specifically configured to determine the posture of the virtual object display unit at the time when the rendering of the virtual object starts according to the posture of the virtual object display unit after the compensation;

[0140] The acquisition module 1201 is specifically used to obtain the posture of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the virtual object display unit at the time of starting the rendering of the virtual object. In this embodiment, the acquisition module 1201 is specifically used to obtain the initial value of the posture of the virtual object display unit at the time of displaying the virtual object according to the posture of the virtual object display unit at the time of starting the rendering of the virtual object and the third jitter parameter, and use the historical posture of the virtual object display unit to smooth the initial value of the posture of the virtual object display unit at the time of displaying the virtual object, so as to obtain the posture of the virtual object display unit at the time of displaying the virtual object.

[0141] In another implementation of this embodiment, the current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit may be: the vehicle posture after anti-shake compensation; the above-mentioned vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining the second jitter parameter measured by the second motion detection device, and performing anti-shake compensation on the current vehicle posture according to the second jitter parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit.

[0142] In this way, the determination module 1202 is specifically used to determine the current posture of the virtual object display unit according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the above-mentioned virtual object display unit.

[0143] Further, the determination module 1202 is further configured to determine the relative jitter between the vehicle chassis and the virtual object display unit according to the first jitter parameter measured by the first motion detection device and the second jitter parameter measured by the second motion detection device after determining the current position of the virtual object display unit;

[0144] The compensation module 1203 is specifically configured to perform anti-shake compensation on the current position and posture of the virtual object display unit according to the above relative jitter.

[0145] In this implementation, the compensation module 1203 is specifically used to compensate the posture of the virtual object display unit after anti-shake compensation according to the posture change.

[0146] It should be noted that this application Figure 7 to Figure 10 All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0147] The electronic device 1200 provided in this embodiment is used to execute the present application Figure 7 to Figure 10 The method for determining the position and posture of the virtual object display unit provided in the illustrated embodiment can therefore achieve the same effect as the above method.

[0148] It should be understood that the electronic device 1200 can be used as a virtual object display unit. Figure 6 The functions of the acquisition module 1201, the determination module 1202, the compensation module 1203 and the prediction module 1204 can be implemented by Figure 6 The processor 410 in the electronic device 400 is shown implemented.

[0149] In the case of adopting an integrated unit, the electronic device 1200 may include a processing module, a storage module, and a communication module.

[0150] The processing module can be used to control and manage the actions of the electronic device 1200, for example, it can be used to support the electronic device 1200 to execute the steps executed by the acquisition module 1201, the determination module 1202, the compensation module 1203 and the prediction module 1204. The storage module can be used to support the electronic device 1200 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 1200 and other devices.

[0151] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0152] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1200 involved in this embodiment can be a Figure 6 Device of the structure shown.

[0153] Fig.13 A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, in which each functional module is divided according to each function, Fig.13 A possible schematic diagram of the composition of the electronic device 1300 involved in the above embodiment is shown. Fig.13 As shown, the electronic device 1300 may include: an acquisition module 1301, a determination module 1302, a compensation module 1303, a prediction module 1304 and a transmission module 1305;

[0154] The acquisition module 1301 is used to acquire the current vehicle posture;

[0155] The determination module 1302 is used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body; in this embodiment, the determination module 1302 is specifically used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit;

[0156] The acquisition module 1301 is also used to acquire the jitter parameter currently measured by the second motion detection device; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is rigidly connected to the virtual object display unit;

[0157] The compensation module 1303 is used to perform anti-shake compensation on the current position and posture of the virtual object display unit according to the jitter parameter currently measured by the second motion detection device;

[0158] A prediction module 1304, configured to predict jitter parameters during the posture transmission period of the virtual object display unit and during the virtual object rendering period according to the historical jitter parameters measured by the second motion detection device;

[0159] The determination module 1302 is also used to determine the posture of the virtual object display unit at the time of displaying the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit after anti-shake compensation; in this embodiment, the determination module 1302 is specifically used to determine the posture of the virtual object display unit at the time of rendering start of the virtual object according to the posture of the virtual object display unit after anti-shake compensation; and obtain the posture of the virtual object display unit at the time of displaying the virtual object according to the predicted jitter parameters and the posture of the virtual object display unit at the time of rendering start of the virtual object. In specific implementation, the determination module 1302 is specifically used to obtain the initial posture value of the virtual object display unit at the time of displaying the virtual object according to the posture of the virtual object display unit at the time of rendering start of the virtual object and the predicted jitter parameters, and use the historical posture of the virtual object display unit to smooth the initial posture value of the virtual object display unit at the time of displaying the virtual object, and finally obtain the posture of the virtual object display unit at the time of displaying the virtual object.

[0160] The transmission module 1305 is used to transmit the position and posture of the virtual object display unit at the time of displaying the virtual object to the virtual object display unit.

[0161] In this embodiment, the acquisition module 1301 is specifically used to acquire data from the vehicle-mounted sensor;

[0162] The determination module 1302 is specifically used to determine the current vehicle posture according to the data of the above-mentioned vehicle-mounted sensor.

[0163] It should be noted that this application Figure 11(a) to Figure 11(b) All relevant contents of each step involved in the illustrated method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0164] The electronic device 1300 provided in this embodiment is used to execute the present application Figure 11(a) to Figure 11(b) The method for determining the position and posture of the virtual object display unit provided in the illustrated embodiment can therefore achieve the same effect as the above method.

[0165] It should be understood that the electronic device 1300 can be used as a vehicle body posture calculation unit. Figure 6 The functions of the acquisition module 1301, the determination module 1302, the compensation module 1303, the prediction module 1304 and the transmission module 1305 can be implemented by Figure 6 The processor 410 in the electronic device 400 is shown implemented.

[0166] In the case of adopting an integrated unit, the electronic device 1300 may include a processing module, a storage module, and a communication module.

[0167] The processing module can be used to control and manage the actions of the electronic device 1300, for example, it can be used to support the electronic device 1300 to execute the steps executed by the acquisition module 1301, the determination module 1302, the compensation module 1303, the prediction module 1304 and the transmission module 1305. The storage module can be used to support the electronic device 1300 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 1300 and other devices.

[0168] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0169] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 1300 involved in this embodiment can be a Figure 6 Device of the structure shown.

[0170] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 7 to Figure 10The method provided by the illustrated embodiment.

[0171] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 11(a) to Figure 11(b) The method provided by the illustrated embodiment.

[0172] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 7 to Figure 10 The method provided by the illustrated embodiment.

[0173] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 11(a) to Figure 11(b) The method provided by the illustrated embodiment.

[0174] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0175] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0176] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0177] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0178] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining the position and posture of a virtual object display unit, characterized in that: include: Determine the current position and posture of the virtual object display unit according to the current vehicle body position and posture; Obtaining jitter parameters measured by a motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the jitter parameter and the current position and posture of the virtual object display unit.

2. The method according to claim 1, characterized in that The current vehicle posture is transmitted to the virtual object display unit by the vehicle posture calculation unit; the motion detection device includes a first motion detection device, and the first motion detection device is rigidly connected to the virtual object display unit; The obtaining of the jitter parameters measured by the motion detection device comprises: Acquire a first jitter parameter measured by the first motion detection device during the vehicle body posture transmission delay.

3. The method according to claim 2, characterized in that Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit comprises: Compensating a current posture of the virtual object display unit according to the first jitter parameter; predicting a third jitter parameter during virtual object rendering based on the historical jitter parameter measured by the first motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the third jitter parameter and the position and posture of the virtual object display unit after compensation.

4. The method according to claim 3, characterized in that The compensating the current posture of the virtual object display unit according to the first jitter parameter measured during the vehicle body posture transmission delay includes: According to the first jitter parameter, obtaining a posture change of the virtual object display unit during a delay period of the vehicle body posture transmission; According to the posture change, the current posture of the virtual object display unit is compensated.

5. The method according to claim 1, characterized in that Determining the current posture of the virtual object display unit according to the current vehicle body posture includes: The current posture of the virtual object display unit is determined according to the current posture of the vehicle body and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

6. The method according to claim 3, characterized in that: Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the third jitter parameter and the posture of the compensated virtual object display unit comprises: Determining the posture of the virtual object display unit at the start time of rendering the virtual object according to the posture of the virtual object display unit after compensation; The position and posture of the virtual object display unit at the time of displaying the virtual object is obtained according to the third jitter parameter and the position and posture of the virtual object display unit at the time of starting rendering of the virtual object.

7. The method according to claim 3, characterized in that The current vehicle posture transmitted by the vehicle posture calculation unit to the virtual object display unit includes: the vehicle posture after anti-shake compensation; the vehicle posture after anti-shake compensation is obtained by the vehicle posture calculation unit obtaining the second jitter parameter measured by the second motion detection device, and performing anti-shake compensation on the current vehicle posture according to the second jitter parameter; wherein the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is non-rigidly connected to the virtual object display unit.

8. The method according to claim 7, characterized in that Determining the current posture of the virtual object display unit according to the current vehicle body posture includes: The current posture of the virtual object display unit is determined according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit.

9. The method according to claim 8, characterized in that After determining the current posture of the virtual object display unit according to the posture of the vehicle body after anti-shake compensation and the pre-calibrated physical transformation relationship between the vehicle body and the virtual object display unit, the method further includes: determining a relative jitter between the vehicle chassis and the virtual object display unit according to a first jitter parameter measured by the first motion detection device and a second jitter parameter measured by the second motion detection device; According to the relative jitter, anti-shake compensation is performed on the current posture of the virtual object display unit.

10. The method according to claim 9, characterized in that The compensating the current posture of the virtual object display unit according to the first jitter parameter includes: According to the first jitter parameter, obtaining a posture change of the virtual object display unit during a delay period of the vehicle body posture transmission; According to the posture change, the posture of the virtual object display unit after anti-shake compensation is compensated.

11. The method according to claim 2, characterized in that The first motion detection device is rigidly connected to the virtual object display unit and comprises: The first motion detection device is mounted on a component rigidly connected to the virtual object display unit.

12. The method according to claim 1, characterized in that The motion detection device comprises a second motion detection device, the second motion detection device is rigidly connected to the vehicle posture calculation unit, and the vehicle posture calculation unit is rigidly connected to the virtual object display unit; Determining the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit comprises: Performing anti-shake compensation on the current posture of the virtual object display unit according to the jitter parameter currently measured by the second motion detection device; Predicting jitter parameters during the posture transmission period and the virtual object rendering period of the virtual object display unit according to the historical jitter parameters measured by the motion detection device; The position and posture of the virtual object display unit at the time of displaying the virtual object are determined according to the predicted jitter parameters and the position and posture of the virtual object display unit after anti-shake compensation.

13. The method according to claim 12, characterized in that The step of determining the position and posture of the virtual object display unit at the time of displaying the virtual object according to the predicted jitter parameter and the position and posture of the virtual object display unit after anti-shake compensation comprises: Determining the position and posture of the virtual object display unit at the start time of rendering the virtual object according to the position and posture of the virtual object display unit after anti-shake compensation; The position and posture of the virtual object display unit at the time of displaying the virtual object are obtained according to the predicted jitter parameter and the position and posture of the virtual object display unit at the time of starting rendering of the virtual object.

14. A device for determining the position and posture of a virtual object display unit, characterized in that: include: A determination module, used to determine the current posture of the virtual object display unit according to the current posture of the vehicle body; An acquisition module, used for acquiring jitter parameters measured by a motion detection device; The determination module is further used to determine the posture of the virtual object display unit at the time of displaying the virtual object according to the jitter parameter and the current posture of the virtual object display unit.

15. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 13.

Images