Display processing method, device and equipment for augmented reality picture

Abstract

The invention discloses a display processing method, device and equipment for an augmented reality picture, and relates to the technical field of augmented reality. The method comprises the steps that in the driving process of a target vehicle, vehicle-mounted navigation data of the target vehicle in a first time period are obtained, and the vehicle-mounted navigation data are used for indicating navigation information of the target vehicle in the driving process; based on the motion state change of the target vehicle indicated by the vehicle-mounted navigation data, a delay compensation parameter corresponding to the motion state change is obtained, and the delay compensation parameter is used for synchronizing somatosensory motion perception generated by the motion state change of the target vehicle and picture motion perception of the augmented reality picture; the augmented reality picture is a picture provided by augmented reality equipment; and rendering and displaying the augmented reality picture based on the delay compensation parameter in the first time period. According to the method, the dizziness caused by the use of the augmented reality device in the vehicle driving process of the user can be reduced, and the user experience of the augmented reality device in the vehicle-mounted scene is improved.

Description

Technical Field

[0001] The present application relates to the field of extended reality technology, and in particular to a display processing method, device and equipment for an extended reality screen. Background Art

[0002] With the development of virtual reality (VR) technology, VR technology has gradually been applied to various scenarios and fields.

[0003] In the in-vehicle scenario, the in-vehicle terminal is committed to providing users with more diverse entertainment activities. When VR technology is applied to the in-vehicle scenario, non-drivers can experience VR-related entertainment activities through VR devices in the car. Since the vehicle has movement changes during driving, such as acceleration, deceleration, turning, etc., in order to make the actual movement and brain movement match normally, the VR device will use inertial navigation technology (Inertial Navigation Technology) to make the movement of objects in the picture match the movement of the vehicle.

[0004] However, since the images displayed by the VR device are synchronized with the movement state of the vehicle, the time delay caused by the transmission between devices may make the user feel dizzy, resulting in the user being unable to experience the relevant functions provided by the VR device for a long time in the in-vehicle scenario. Summary of the invention

[0005] The embodiment of the present application provides a display processing method, device and equipment for an extended reality screen. The technical solution is as follows:

[0006] In one aspect, a method for displaying an extended reality screen is provided, the method comprising:

[0007] During the driving process of the target vehicle, obtaining the vehicle navigation data of the target vehicle in a first time period, wherein the first time period is a time period after the current time period, and the vehicle navigation data is used to indicate the navigation information of the target vehicle during the driving process;

[0008] Based on the change in the motion state of the target vehicle indicated by the on-board navigation data, obtaining a delay compensation parameter corresponding to the change in the motion state, wherein the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in the motion state of the target vehicle and the screen motion perception of the extended reality screen, wherein the extended reality screen is a screen provided by an extended reality device;

[0009] During the first time period, the extended reality screen is rendered and displayed based on the delay compensation parameters.

[0010] On the other hand, a display processing device for an extended reality screen is provided, the device comprising:

[0011] A first acquisition module is used to acquire the vehicle-mounted navigation data of the target vehicle in a first time period during the driving process of the target vehicle, wherein the first time period is a time period after the current time period, and the vehicle-mounted navigation data is used to indicate the navigation information of the target vehicle during the driving process;

[0012] A second acquisition module is used to acquire a delay compensation parameter corresponding to the change in motion state based on the change in motion state of the target vehicle indicated by the vehicle-mounted navigation data, wherein the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in motion state of the target vehicle and the screen motion perception of the extended reality screen, where the extended reality screen is a screen provided by an extended reality device;

[0013] A display module is used to render and display the extended reality screen based on the delay compensation parameters during the first time period.

[0014] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a display processing method of an extended reality screen as described in any of the above-mentioned embodiments of the present application.

[0015] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a display processing method of an extended reality screen as described in any of the above-mentioned embodiments of the present application.

[0016] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the display processing method of the extended reality screen described in any of the above embodiments.

[0017] The technical solution provided by this application includes at least the following beneficial effects:

[0018] When a user uses an extended reality device while a vehicle is driving, the on-board navigation data of the target vehicle is used to predict the motion state changes of the target vehicle in a future time period (first time period), and the corresponding delay compensation parameters are obtained according to the motion state changes of the vehicle. The extended reality device renders and displays the extended reality screen through the delay compensation parameters, so that the visual motion perception generated by the user when watching the extended reality screen can be synchronized with the somatic motion perception generated by the motion state changes of the vehicle, thereby reducing the dizziness caused by the user using the extended reality device while the vehicle is driving, and improving the user experience of the extended reality device in the vehicle-mounted scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a display processing method of an extended reality screen provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a six-degree-of-freedom model provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a flowchart of a display processing method of an extended reality screen provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a structural block diagram of a display processing device for an extended reality screen provided by an exemplary embodiment of the present application;

[0025] Figure 6 It is a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0027] First, a brief introduction is given to the terms involved in the embodiments of the present application.

[0028] Extended Reality (XR) technology: refers to the technology that combines the real and the virtual through computers to create a virtual environment for human-computer interaction. XR technology includes VR technology, augmented reality (AR) technology and mixed reality (MR). XR technology is widely used in entertainment, education, medical care, manufacturing and training, providing users with a more immersive and interactive experience. For example, in the medical field, XR can be used for surgical simulation and medical training; in the education field, XR can create an immersive learning environment and improve learning effects; in manufacturing and training, XR can be used for simulation training to reduce training costs and improve efficiency.

[0029] VR technology: a technology that generates a virtual environment through computer simulation, allowing users to immerse themselves in it through special equipment and interact with the virtual environment in real time. VR devices usually include head-mounted displays (HMD), handheld controllers and sensors, etc. Users interact with the virtual environment through these devices. The head-mounted display provides a full range of viewing angles, sensors detect the user's head and body movements, and handheld controllers are used to operate virtual objects in the virtual environment and perform specific actions. VR technology allows users to experience an immersive experience through the comprehensive stimulation of multiple senses such as vision and hearing. It is widely used in games, film and television, education, medical care, architectural design and other fields.

[0030] AR technology: is a technology that superimposes virtual information onto the real environment, allowing users to see and interact with virtual objects in the real world. AR devices usually include smart glasses, mobile phones and tablets, etc., which capture the real environment through cameras and display virtual information on the screen. Users can interact with virtual objects through a variety of interactive methods such as touch screens, gesture recognition and voice commands. AR technology is widely used in entertainment, education, medical care, industry and navigation.

[0031] MR technology: It is a technology that combines the real world with the virtual world. Through advanced computer graphics and sensor technology, virtual objects are seamlessly integrated into the user's physical environment to provide users with an immersive experience. MR devices usually include sensors and cameras that can capture and recognize the user's movements and environment to achieve accurate 3D spatial mapping. Users can interact with virtual objects through a variety of interactive methods such as gesture recognition, voice commands, eye movement detection, physical controllers, head detection, etc. MR technology is widely used in entertainment, education, medical, industrial and navigation fields.

[0032] Figure 1The structure block diagram of a computer system provided by an exemplary embodiment of the present application is shown. The computer system 100 includes: an extended reality device 110, a vehicle terminal 120 and a server 130.

[0033] The extended reality device 110 is installed and runs an application that supports a virtual environment. The application may be a virtual reality application, a three-dimensional map program, a third-person shooter (TPS) game, a first-person shooter (FPS) game, a multiplayer online tactical competitive (MOBA) game, a multiplayer gunfight survival game, a turn-based chess game, etc.

[0034] The device type of the extended reality device 110 includes at least one of a VR device, an AR device, and an MR device. The extended reality device 110 may be a device composed of a display device and a control device, wherein the display device may be implemented as a head mounted display, smart glasses, a mobile phone, a tablet computer, etc., and the control device may be implemented as a handheld controller, a sensor, etc.

[0035] Those skilled in the art will appreciate that the number of the above-mentioned extended reality devices 110 may be more or less. For example, the above-mentioned device may be only one, or the above-mentioned devices may be dozens or hundreds, or more. The embodiment of the present application does not limit the number and type of devices.

[0036] The vehicle terminal 120 is the front-end device of the vehicle monitoring and management system. It integrates a variety of advanced technologies, such as wireless communication, global positioning system (GPS) positioning, sensors, etc., and can realize functions such as vehicle remote control, vehicle condition monitoring, fault diagnosis, etc. The wireless communication realized by the vehicle terminal 120 includes vehicle network, Internet, mobile communication network, etc.

[0037] In some embodiments, the extended reality device 110 establishes a communication connection with the vehicle terminal 120. Illustratively, during the driving process of the vehicle, the extended reality device 110 obtains the vehicle navigation data of the target vehicle in the first period from the vehicle terminal 120, obtains the delay compensation parameters corresponding to the change in the motion state based on the change in the motion state of the target vehicle indicated by the vehicle navigation data, and renders and displays the extended reality screen based on the delay compensation parameters in the first period.

[0038] In some other embodiments, the computer system 100 further includes a server 130, and the server 130 includes at least one of a single server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 130 is used to provide background services for the extended reality device 110. Optionally, the server 130 undertakes the main computing work, and the extended reality device 110 undertakes the secondary computing work; or, the server 130 undertakes the secondary computing work, and the extended reality device 110 undertakes the main computing work; or, the server 130 and the extended reality device 110 adopt a distributed computing architecture for collaborative computing.

[0039] It is worth noting that the server 130 can be implemented as a physical server or a cloud server in the cloud, where cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or a local area network to achieve data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application, which can form a resource pool and be used on demand, which is flexible and convenient.

[0040] A communication connection is established between the extended reality device 110, the vehicle terminal 120, and the server 130. Schematically, during the driving process of the vehicle, the server 130 obtains the vehicle navigation data of the target vehicle in the first period from the vehicle terminal 120, obtains the delay compensation parameter corresponding to the change in the motion state based on the change in the motion state of the target vehicle indicated by the vehicle navigation data, generates the screen data corresponding to the extended reality screen based on the delay compensation parameter in the first period, and sends the screen data to the extended reality device 110. After receiving the screen data, the extended reality device 110 displays the extended reality screen based on the screen data.

[0041] In some embodiments, the method provided in the embodiments of the present application can be applied to a cloud gaming scenario, so that the calculation of data logic during the game is completed through the cloud server, while the extended reality device 110 is responsible for the display of the game interface.

[0042] In some embodiments, the server 130 may also be implemented as a node in a blockchain system.

[0043] In combination with the above-mentioned noun introduction and implementation environment, the display processing method of the extended reality screen provided by this application is explained, and the method is executed by the extended reality device as an example. Please refer to Figure 2 , which shows a flowchart of a display processing method of an extended reality screen provided by an exemplary embodiment of the present application, the method includes the following steps 210 to 230.

[0044] Step 210: Acquire the vehicle-mounted navigation data of the target vehicle in a first period of time during the driving process of the target vehicle.

[0045] Optionally, the target vehicle can be implemented as an internal combustion engine vehicle (Internal Combustion Engine Vehicles, ICEV), electric vehicles (Electric Vehicles, EV), hybrid vehicles, plug-in hybrid electric vehicles (Plug-in Hybrid Electric Vehicles, PHEV), range-extended electric vehicles (Range-Extended Electric Vehicles, REEV), fuel cell electric vehicles (Fuel Cell Electric Vehicles, FCEV), mild hybrid vehicles (Mild Hybrid Vehicles, MHV), etc., without limitation here.

[0046] Schematically, the target vehicle is equipped with an on-board terminal, which is an electronic device in the target vehicle used to provide functions and services. The on-board terminal can connect to the Internet through the Internet of Vehicles to achieve data exchange and communication between vehicles, between vehicles and infrastructure, and between vehicles and user devices.

[0047] In an embodiment of the present application, the user uses an extended reality device while the target vehicle is driving, and a communication connection is established between the extended reality device and the vehicle-mounted terminal, wherein the above-mentioned communication connection can be a direct connection, such as Bluetooth, ZigBee, mobile hotspot (Wireless Fidelity, WiFi), etc., or an indirect connection, for example, the extended reality device and the vehicle-mounted terminal are respectively connected to the Internet, and a communication connection is established through a server.

[0048] Optionally, the above-mentioned extended reality device includes at least one of a VR device, an AR device, and a MR device.

[0049] In some embodiments, when the vehicle-mounted terminal detects that the target vehicle starts to move, it sends a vehicle driving message to the connected extended reality device. The vehicle driving message is used to notify the extended reality device that the target vehicle is in a driving state. When the extended reality device receives the vehicle driving message, it starts the delay compensation algorithm.

[0050] In the embodiment of the present application, the first time period is the time period after the current time period. Optionally, the time period length of the first time period may be preset by the system, or may be determined based on the road conditions of the current driving process of the target vehicle.

[0051] Illustratively, the road condition information of the road on which the target vehicle is traveling is obtained, a road condition score of the road is determined based on the road condition information, and a corresponding time period duration is obtained according to the road condition score, wherein the road condition score is negatively correlated with the time period duration.

[0052] Optionally, the above-mentioned road condition information includes at least one of traffic flow information, road surface condition information, traffic control information, traffic signal information, and road construction information. Among them, traffic flow information is used to indicate the vehicle flow on the driving road; road surface condition information is used to indicate the condition of the road surface of the driving road, such as dry road surface, slippery road surface, icy road surface, bumpy road surface, etc.; traffic control information is used to indicate the traffic control situation of the driving road, such as lane closure, tidal lane, lane flow restriction, etc.; traffic signal information is used to indicate the traffic signal of the driving road, such as traffic lights; road construction information is used to indicate the construction section of the driving road.

[0053] In some embodiments, a road condition score is predicted by a pre-trained road condition assessment model. Schematically, road condition information is input into the road condition assessment model, and the road condition score of the target vehicle's driving road is obtained by the road condition assessment model to obtain the time period corresponding to the road condition score.

[0054] In an embodiment of the present application, the acquired vehicle-mounted navigation data is used to indicate navigation information of the target vehicle during driving.

[0055] Optionally, the above-mentioned on-vehicle navigation data includes at least one of navigation route information, road intersection information, and navigation road condition information, wherein the navigation route information is used to indicate the navigation path of the target vehicle's current trip, the road intersection information is used to indicate the intersections passed on the navigation path, and the navigation road condition information is used to indicate the road conditions on the navigation path.

[0056] In some embodiments, the in-vehicle navigation data is data obtained from the in-vehicle terminal after the extended reality device is connected to the in-vehicle terminal. Schematically, a navigation application is installed in the in-vehicle terminal. When the user sets a navigation route in the navigation application, the in-vehicle terminal sends the in-vehicle navigation data corresponding to the navigation route to the extended reality device.

[0057] In other embodiments, the in-vehicle navigation data is data obtained from the user mobile terminal after the extended reality device is connected to the user mobile terminal (for example, a mobile phone). Schematically, a navigation application is installed in the user mobile terminal. When the user sets a navigation route in the navigation application, the user mobile terminal sends the in-vehicle navigation data corresponding to the navigation route to the extended reality device.

[0058] Step 220 , based on the change in the motion state of the target vehicle indicated by the onboard navigation data, a delay compensation parameter corresponding to the change in the motion state is obtained.

[0059] In an embodiment of the present application, the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in the motion state of the target vehicle and the screen motion perception of the extended reality screen, wherein the above-mentioned extended reality screen is a screen provided by the extended reality device.

[0060] In some embodiments, multiple motion state stages of the target vehicle in the first time period are determined based on the vehicle navigation data, a target motion state stage with direction and / or speed changes is determined from the multiple motion state stages, and a delay compensation parameter corresponding to the target motion state stage is obtained. In one example, it is determined based on the vehicle navigation data that the target vehicle experiences a uniform speed driving stage and a deceleration driving stage in the first time period, and a delay compensation parameter corresponding to the deceleration driving stage is obtained.

[0061] In some embodiments, a mapping relationship table between motion state stages and delay compensation parameters is obtained, the target motion state stage is queried in the mapping relationship table, and the corresponding delay compensation parameters are determined.

[0062] In some embodiments, the generation of delay compensation parameters is achieved through a pre-trained delay prediction model. Schematically, the vehicle navigation data is input into the pre-trained delay prediction model. The delay prediction model determines the change of the motion state of the target vehicle during driving based on the vehicle navigation value, and outputs the corresponding delay compensation parameters based on the change of the motion state. Optionally, the above-mentioned delay prediction model can be implemented by neural network models such as Convolutional Neural Networks (CNN), Feedforward Neural Networks (FNN), Residual Networks (ResNet), and Transformers, which are not specifically limited here.

[0063] In some embodiments, in addition to determining the change in the motion state of the target vehicle in the first time period through the on-board navigation data, the distance prediction information between the target vehicle and the surrounding vehicles is also introduced to improve the accuracy of the determination of the change in motion state. Schematically, the distance change information between the target vehicle and the surrounding vehicles in the second time period is obtained, wherein the second time period is the time period before the first time period, and exemplarily, the second time period is a historical time period; based on the distance change information and the on-board navigation data, the distance prediction information between the target vehicle and the surrounding vehicles in the second time period is obtained; based on the on-board navigation data and the distance prediction information, the change in the motion state of the target vehicle in the first time period is determined, and the delay compensation parameter corresponding to the change in the motion state is obtained. That is, the distance change information between the target vehicle and the surrounding vehicles in the historical time period and the route conditions in the future time period indicated by the on-board navigation data are used to predict the distance prediction information between the target vehicle and the surrounding vehicles in the second time period, and the change in the motion state of the target vehicle in the first time period is determined jointly according to the distance prediction information and the on-board navigation data.

[0064] In some embodiments, the distance prediction information is predicted by a pre-trained distance prediction model. Schematically, the distance change information and the vehicle navigation data are input into the pre-trained distance prediction model. The distance prediction model determines the distance change between the target vehicle and the surrounding vehicles during driving based on the distance change information and the vehicle navigation data, and outputs the corresponding distance prediction information. Optionally, the above-mentioned delay prediction model can be implemented by a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a converter, etc., which is not specifically limited here.

[0065] Step 230: Render and display an extended reality image based on the delay compensation parameters during the first period of time.

[0066] In an embodiment of the present application, an extended reality device provides an extended reality screen corresponding to an extended reality scene to a user, where the extended reality scene includes at least one scene model. Schematically, when it is determined based on the on-board navigation data that a target vehicle has a change in motion state within a first time period, the motion of the scene model in the extended reality scene is adjusted within the first time period based on the acquired delay compensation parameters, and rendering and display of the extended reality screen are implemented based on the adjusted data.

[0067] In some embodiments, model motion data of a scene model in an extended reality scene is obtained, wherein the model motion data is used to indicate the movement of the scene model in the extended reality scene, and the model motion data includes a motion execution timestamp; the motion execution timestamp in the model motion data is adjusted based on a delay compensation parameter to obtain compensated motion data; within a first time period, the scene model is controlled to move in the extended reality scene based on the compensated motion data, and the movement process of the scene model is rendered to obtain and display an extended reality screen.

[0068] In some embodiments, the scene model is a model element built using a six-degree-of-freedom model, which can move freely in the extended reality scene. Exemplarily, the three degrees of freedom (3DoF) of the position and the three degrees of freedom (3DoF) of the orientation of the scene model can be changed arbitrarily, where the three degrees of freedom of position are the degrees of freedom of movement along the three rectangular coordinate axes of x, y, and z (X, Y, Z), and the three degrees of freedom of orientation are the degrees of freedom of rotation around the x, y, and z coordinate axes (α, β, γ). Figure 3 As shown, it shows a schematic diagram of a six-degree-of-freedom model provided by an exemplary embodiment of the present application, where the scene model 300 is a six-degree-of-freedom model. In the preset coordinate axes, the scene model 300 can move and rotate on the x-axis 301, on the y-axis 302, and on the z-axis 303. By changing the three degrees of freedom of position and the three degrees of freedom of direction of the scene model 300, it can move freely in the extended reality scene.

[0069] In some embodiments, the model motion data obtained includes the movement data of the scene model, and the movement timestamp corresponding to the movement data, wherein the movement data is used to indicate the change in the scene model's freedom of movement. Schematically, the movement timestamp corresponding to the movement data is adjusted based on the delay compensation parameter to obtain the movement compensation timestamp; within the first time period, in response to the current moment and the matching of the movement compensation timestamp, the scene model is controlled to move in the extended reality scene based on the movement data, and the movement process of the scene model is rendered to obtain and display the extended reality screen. That is, the movement timestamp corresponding to the movement data is adjusted by the delay compensation parameter, so as to realize the delay compensation of the scene model when it moves in the three rectangular coordinate axes of x, y, and z when the vehicle's motion state changes.

[0070] In some embodiments, the model motion data includes rotation data of the scene model, and a rotation timestamp corresponding to the rotation data, wherein the rotation data is used to indicate changes in the rotational freedom of the scene model. Schematically, the rotation timestamp corresponding to the rotation data is adjusted based on the delay compensation parameter to obtain a rotation compensation timestamp; within a first period, in response to the current moment matching the rotation compensation timestamp, the scene model is controlled to rotate in the extended reality scene based on the rotation data, and the rotation process of the scene model is rendered to obtain and display the extended reality screen. That is, the rotation timestamp corresponding to the rotation data is adjusted by the delay compensation parameter, so as to achieve delay compensation for the scene model rotating around the three rectangular coordinate axes of x, y, and z when the vehicle motion state changes.

[0071] In some embodiments, it is considered that the target vehicle generally moves and rotates only on two coordinate axes corresponding to the plane of the ground during movement. Therefore, when motion delay compensation is implemented through delay compensation parameters, compensation is only performed on four degrees of freedom (for example, X, Y, α, β), thereby reducing data calculation and storage, and reducing the consumption of device computing resources and storage resources.

[0072] To summarize, when a user uses an extended reality device while a vehicle is driving, the on-board navigation data of the target vehicle is used to predict the motion state changes of the target vehicle in a future period (first period), and the corresponding delay compensation parameters are obtained based on the motion state changes of the vehicle. The extended reality device uses the delay compensation parameters to render and display the extended reality screen, so that the visual motion perception generated by the user when watching the extended reality screen can be synchronized with the somatosensory motion perception generated by the motion state changes of the vehicle, thereby reducing the dizziness caused by the user using the extended reality device while the vehicle is driving, and improving the user experience of the extended reality device in the vehicle-mounted scenario.

[0073] In some optional embodiments, based on the vehicle navigation data, the event of the target vehicle's state change during driving is determined, so as to obtain the delay compensation parameter corresponding to the event to compensate for the delay of the extended reality screen displayed by the extended reality device. Figure 4 , which shows a flowchart of a display processing method of an extended reality screen provided by an exemplary embodiment of the present application, and the method includes the following steps.

[0074] Step 210: Acquire the vehicle-mounted navigation data of the target vehicle in a first period of time during the driving process of the target vehicle.

[0075] Optionally, the target vehicle may be realized as an internal combustion engine driven vehicle, an electric vehicle, a hybrid vehicle, a plug-in hybrid vehicle, an extended range electric vehicle, a fuel cell electric vehicle, a mild hybrid vehicle, etc., which is not limited here.

[0076] In the embodiment of the present application, during the driving process of the target vehicle, the extended reality device obtains the in-vehicle navigation data of the target vehicle in the first period from the in-vehicle terminal. The in-vehicle terminal is an electronic device in the target vehicle used to provide functions and services. The in-vehicle terminal can be connected to the Internet through the Internet of Vehicles to realize data exchange and communication between vehicles, between vehicles and infrastructure, and between vehicles and user devices.

[0077] Optionally, the above-mentioned extended reality device includes at least one of a VR device, an AR device, and a MR device.

[0078] In some embodiments, when the vehicle-mounted terminal detects that the target vehicle starts to move, it sends a vehicle driving message to the connected extended reality device. The vehicle driving message is used to notify the extended reality device that the target vehicle is in a driving state. When the extended reality device receives the vehicle driving message, it starts the delay compensation algorithm.

[0079] In the embodiment of the present application, the first time period is the time period after the current time period. Optionally, the time period length of the first time period may be preset by the system, or may be determined based on the road conditions of the current driving process of the target vehicle.

[0080] Optionally, the above-mentioned on-vehicle navigation data includes at least one of navigation route information, road intersection information, and navigation road condition information, wherein the navigation route information is used to indicate the navigation path of the target vehicle's current trip, the road intersection information is used to indicate the intersections passed on the navigation path, and the navigation road condition information is used to indicate the road conditions on the navigation path.

[0081] In some embodiments, the in-vehicle navigation data is data obtained from the in-vehicle terminal after the extended reality device is connected to the in-vehicle terminal. Schematically, a navigation application is installed in the in-vehicle terminal. When the user sets a navigation route in the navigation application, the in-vehicle terminal sends the in-vehicle navigation data corresponding to the navigation route to the extended reality device.

[0082] In other embodiments, the in-vehicle navigation data is data obtained from the user mobile terminal after the extended reality device is connected to the user mobile terminal (for example, a mobile phone). Schematically, a navigation application is installed in the user mobile terminal. When the user sets a navigation route in the navigation application, the user mobile terminal sends the in-vehicle navigation data corresponding to the navigation route to the extended reality device.

[0083] Step 221: determining at least one target event existing for the target vehicle at a first moment based on the vehicle-mounted navigation data.

[0084] In an embodiment of the present application, the target event is used to indicate an event to which the target vehicle needs to respond by changing its motion state during driving, that is, based on the navigation route information, road intersection information, navigation road condition information and other information indicated by the on-board navigation data, the target event is obtained by predicting the event to which the target vehicle needs to respond by changing its motion state.

[0085] Optionally, the target event includes at least one of the following:

[0086] 1. The distance to the next intersection is a meters (for example, 20 meters);

[0087] 2. The traffic light countdown has b seconds remaining (e.g., 2 seconds);

[0088] 3. The front collision warning alarm prompts the vehicle ahead to be c meters (for example, 4 meters) away from the vehicle ahead;

[0089] 4. The side collision warning did not sound an alarm;

[0090] 5. Turn left;

[0091] 6. Turn right;

[0092] 7. U-turn;

[0093] 8. There is a left tilt;

[0094] 9. There is a right tilt;

[0095] 10. There is an accelerated state;

[0096] 11. There is a deceleration state;

[0097] 12. Reverse.

[0098] Step 222: Obtain a delay compensation parameter corresponding to at least one target event.

[0099] In some embodiments, the delay compensation parameter corresponding to the target event is obtained by querying the mapping relationship table between the candidate events and the candidate parameters. Schematically, the mapping relationship table between the candidate events and the candidate parameters is obtained; at least one target event is determined from the candidate events; based on the at least one target event, at least one delay compensation parameter corresponding to the target event is obtained from the mapping relationship table.

[0100] Optionally, the extended reality device pre-stores a mapping relationship table, and in response to detecting a target event, obtains the pre-stored mapping relationship table from the device storage area; optionally, the extended reality device sends the detected target event to the server, the server obtains the delay compensation parameters corresponding to the target event based on the stored mapping relationship table query, and sends the delay compensation parameters to the extended reality device.

[0101] In some embodiments, under a target event, the change in the motion state of the target vehicle causes the user's somatosensory motion perception to include the perception of acceleration changes in displacement and the perception of steering changes in rotation. Therefore, when providing delay compensation parameters for the motion of the scene model, it is also necessary to compensate for the picture motion perception of both displacement and rotation.

[0102] Illustratively, based on at least one target event, the acceleration change information and the steering change information of the target vehicle within a first time period are determined; based on the acceleration change information, a movement compensation parameter is generated, wherein the movement compensation parameter is used to provide time delay compensation when the scene model moves on at least one coordinate axis in a preset coordinate system; based on the acceleration change information, a first rotation compensation parameter is generated, wherein the first rotation compensation parameter is used to provide time delay compensation when the scene model rotates on a first coordinate axis corresponding to the vehicle's driving direction in the preset coordinate system; based on the steering change information, a second rotation compensation parameter is generated, wherein the second rotation compensation parameter is used to provide time delay compensation when the scene model rotates on a second coordinate axis perpendicular to the vehicle's driving direction in the preset coordinate system; and a delay compensation parameter is obtained from the movement compensation parameter, the first rotation compensation parameter and the second rotation compensation parameter.

[0103] In an example, the mapping relationship between the scene model degrees of freedom, target events and delay compensation parameters is shown in Table 1. In this table, the target events determined based on the vehicle navigation data include the distance between the next intersection is a meters, the remaining b seconds of the traffic light countdown, the front collision warning alarm prompts the front vehicle following distance c meters, the left / right turn information prompt, the acceleration and deceleration status, the left tilt status and the right tilt status.

[0104] Table 1

[0105]

[0106]

[0107] Among them, k in the delay compensation parameter is the event coefficient corresponding to the event in the delay compensation parameter, and the event coefficient can be obtained by looking up the table. For example, as shown in Table 2, it shows an exemplary k a1 The correspondence between this and the actual distance a in the event "The distance to the next intersection is a meters".

[0108] Table 2

[0109] <![CDATA[k a1 ]]> 1 0.8 0.6 0.4 0.2 0.1 0 a 2m 4m 8m 15m 25m 40m ＞40m

[0110] For example, taking A = 5 ms, and the vehicle navigation data indicates that the distance between the target vehicle and the next intersection at the first candidate time in the first period is 8 meters, then k a1 A=3ms, that is, at the first candidate moment in the first time period, delay compensation is performed on the movement of the scene model in the degree of freedom X in the extended reality scene, and the compensation duration is 3ms.

[0111] Step 230: Render and display an extended reality image based on the delay compensation parameters during the first period of time.

[0112] In an embodiment of the present application, an extended reality device provides an extended reality screen corresponding to an extended reality scene to a user, where the extended reality scene includes at least one scene model. Schematically, when it is determined based on the on-board navigation data that a target vehicle has a change in motion state within a first time period, the motion of the scene model in the extended reality scene is adjusted within the first time period based on the acquired delay compensation parameters, and rendering and display of the extended reality screen are implemented based on the adjusted data.

[0113] To summarize, when a user uses an extended reality device while a vehicle is driving, the on-board navigation data of the target vehicle is used to predict the motion state changes of the target vehicle in a future period (first period), and the corresponding delay compensation parameters are obtained based on the motion state changes of the vehicle. The extended reality device uses the delay compensation parameters to render and display the extended reality screen, so that the visual motion perception generated by the user when watching the extended reality screen can be synchronized with the somatosensory motion perception generated by the motion state changes of the vehicle, thereby reducing the dizziness caused by the user using the extended reality device while the vehicle is driving, and improving the user experience of the extended reality device in the vehicle-mounted scenario.

[0114] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of relevant countries and regions.

[0115] Please refer to Figure 5 , which shows a structural block diagram of a display processing device for an extended reality screen provided by an exemplary embodiment of the present application, the device includes the following modules:

[0116] A first acquisition module 510 is used to acquire the vehicle-mounted navigation data of the target vehicle in a first time period during the driving process of the target vehicle, wherein the first time period is a time period after the current time period, and the vehicle-mounted navigation data is used to indicate the navigation information of the target vehicle during the driving process;

[0117] A second acquisition module 520 is used to acquire a delay compensation parameter corresponding to the change in motion state based on the change in motion state of the target vehicle indicated by the on-board navigation data, wherein the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in motion state of the target vehicle and the screen motion perception of the extended reality screen, where the extended reality screen is a screen provided by an extended reality device;

[0118] The display module 530 is used to render and display the extended reality screen based on the delay compensation parameters during the first time period.

[0119] In some optional embodiments, the device further comprises:

[0120] A third acquisition module (not shown in the figure) is further used to acquire model motion data of the scene model in the extended reality scene, wherein the model motion data is used to indicate the motion of the scene model in the extended reality scene, and the model motion data includes a motion execution timestamp;

[0121] The display module 530 is further configured to adjust the motion execution timestamp in the model motion data based on the delay compensation parameter to obtain compensated motion data;

[0122] The display module 530 is further used to control the scene model to move in the extended reality scene based on the compensated motion data during the first time period, and to render the movement process of the scene model to obtain and display the extended reality screen.

[0123] In some optional embodiments, the model motion data includes movement data of the scene model and a movement timestamp corresponding to the movement data, wherein the movement data is used to indicate changes in the movement freedom of the scene model;

[0124] The display module 530 is further configured to adjust the mobile timestamp corresponding to the mobile data based on the delay compensation parameter to obtain a mobile compensation timestamp;

[0125] The display module 530 is also used to control the movement of the scene model in the extended reality scene based on the movement data in response to the current moment and the match between the motion compensation timestamp during the first time period, and render the movement process of the scene model to obtain and display the extended reality screen.

[0126] In some optional embodiments, the model motion data includes rotation data of the scene model and a rotation timestamp corresponding to the rotation data, wherein the rotation data is used to indicate a change in a rotational degree of freedom of the scene model;

[0127] The display module 530 is further used to adjust the rotation timestamp corresponding to the rotation data based on the delay compensation parameter to obtain a rotation compensation timestamp;

[0128] The display module 530 is also used to control the rotation of the scene model in the extended reality scene based on the rotation data in response to the current moment matching the rotation compensation timestamp during the first time period, and render the rotation process of the scene model to obtain and display the extended reality screen.

[0129] In some optional embodiments, the second acquisition module 520 is further used to determine at least one target event existing in the target vehicle at the first moment based on the vehicle-mounted navigation data, wherein the target event is used to indicate an event to which the target vehicle needs to respond by changing its motion state during driving;

[0130] The second acquisition module 520 is further configured to acquire the delay compensation parameter corresponding to the at least one target event.

[0131] In some optional embodiments, the second acquisition module 520 is further used to acquire a mapping relationship table between candidate events and candidate parameters;

[0132] The second acquisition module 520 is further configured to determine the at least one target event from the candidate events;

[0133] The second acquisition module 520 is further configured to search the mapping relationship table based on the at least one target event to obtain at least one delay compensation parameter corresponding to the target event.

[0134] In some optional embodiments, the second acquisition module 520 is further used to determine the acceleration change information and the steering change information of the target vehicle within the first time period based on the at least one target event;

[0135] The second acquisition module 520 is further used to generate a motion compensation parameter based on the acceleration change information, wherein the motion compensation parameter is used to provide delay compensation when the scene model moves on at least one coordinate axis in a preset coordinate system;

[0136] The second acquisition module 520 is further used to generate a first rotation compensation parameter based on the acceleration change information, wherein the first rotation compensation parameter is used to provide delay compensation when the scene model rotates on a first coordinate axis corresponding to the vehicle driving direction in the preset coordinate system;

[0137] The second acquisition module 520 is further used to generate a second rotation compensation parameter based on the steering change information, and the second rotation compensation parameter is used to provide a delay compensation for the rotation of the scene model on a second coordinate axis perpendicular to the driving direction of the vehicle in the preset coordinate system;

[0138] The second acquisition module 520 is further configured to obtain the delay compensation parameter from the motion compensation parameter, the first rotation compensation parameter and the second rotation compensation parameter.

[0139] In some optional embodiments, the second acquisition module 520 is further used to acquire distance change information between the target vehicle and surrounding vehicles in a second time period, where the second time period is a time period before the first time period;

[0140] The second acquisition module 520 is further used to predict the distance between the target vehicle and the surrounding vehicles in the second time period based on the distance change information and the vehicle navigation data;

[0141] The second acquisition module 520 is further used to determine the change in the motion state of the target vehicle within the first time period based on the on-board navigation data and the distance prediction information, and to acquire the delay compensation parameter corresponding to the change in the motion state.

[0142] To summarize, when a user uses an extended reality device while a vehicle is driving, the on-board navigation data of the target vehicle is used to predict the motion state changes of the target vehicle in a future period (first period), and the corresponding delay compensation parameters are obtained based on the motion state changes of the vehicle. The extended reality device uses the delay compensation parameters to render and display the extended reality screen, so that the visual motion perception generated by the user when watching the extended reality screen can be synchronized with the somatosensory motion perception generated by the motion state changes of the vehicle, thereby reducing the dizziness caused by the user using the extended reality device while the vehicle is driving, and improving the user experience of the extended reality device in the vehicle-mounted scenario.

[0143] It should be noted that the display processing device for the extended reality screen provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the display processing device for the extended reality screen provided in the above embodiment and the display processing method embodiment of the extended reality screen belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0144] Figure 6The following is a block diagram of a terminal 600 provided by an exemplary embodiment of the present application. The terminal 600 may be a smart phone, a VR head display device, an AR display device, a tablet computer, a Moving Picture Experts Group Audio Layer III (MP3) player, a Moving Picture Experts Group Audio Layer IV (MP4) player, a laptop computer, or a desktop computer. The terminal 600 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0145] Typically, the terminal 600 includes a processor 601 and a memory 602 .

[0146] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0147] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the display processing method of the extended reality screen provided in the method embodiment of the present application.

[0148] Illustratively, the terminal 600 further includes other components 603, which can be understood by those skilled in the art. Figure 6 The structure shown in the figure does not constitute a limitation on the terminal 600, and the terminal 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0149] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, which can be a computer-readable storage medium contained in the memory in the above embodiments; or it can be a computer-readable storage medium that exists independently and is not assembled in the terminal. The computer-readable storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set is loaded and executed by the processor to implement the display processing method of the extended reality screen described in any of the above embodiments.

[0150] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0151] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0152] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A display processing method for an extended reality screen, characterized in that: The method comprises: During the driving process of the target vehicle, obtaining the vehicle navigation data of the target vehicle in a first time period, wherein the first time period is a time period after the current time period, and the vehicle navigation data is used to indicate the navigation information of the target vehicle during the driving process; Based on the change in the motion state of the target vehicle indicated by the on-board navigation data, obtaining a delay compensation parameter corresponding to the change in the motion state, wherein the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in the motion state of the target vehicle and the screen motion perception of the extended reality screen, wherein the extended reality screen is a screen provided by an extended reality device; During the first time period, the extended reality screen is rendered and displayed based on the delay compensation parameters.

2. The method according to claim 1, characterized in that The rendering and displaying of the extended reality screen based on the delay compensation parameter during the first time period includes: Acquire model motion data of a scene model in an extended reality scene, wherein the model motion data is used to indicate a motion condition of the scene model in the extended reality scene, and the model motion data includes a motion execution timestamp; Adjust the motion execution timestamp in the model motion data based on the delay compensation parameter to obtain compensated motion data; During the first time period, the scene model is controlled to move in the extended reality scene based on the compensated motion data, and the movement process of the scene model is rendered to obtain and display the extended reality screen.

3. The method according to claim 2, characterized in that The model motion data includes movement data of the scene model and a movement timestamp corresponding to the movement data, wherein the movement data is used to indicate changes in the movement freedom of the scene model; The step of adjusting the motion execution timestamp in the model motion data based on the delay compensation parameter to obtain compensated motion data includes: Adjust the mobile timestamp corresponding to the mobile data based on the delay compensation parameter to obtain a mobile compensation timestamp; The method of controlling the scene model to move in the extended reality scene based on the compensated motion data during the first time period, rendering the motion process of the scene model, and obtaining and displaying the extended reality picture includes: During the first time period, in response to a match between the current moment and the motion compensation timestamp, the scene model is controlled to move in the extended reality scene based on the motion data, and the movement process of the scene model is rendered to obtain and display the extended reality screen.

4. The method according to claim 2, characterized in that: The model motion data includes rotation data of the scene model and a rotation timestamp corresponding to the rotation data, wherein the rotation data is used to indicate a change in the rotational freedom of the scene model; The step of adjusting the motion execution timestamp in the model motion data based on the delay compensation parameter to obtain compensated motion data includes: Adjust the rotation timestamp corresponding to the rotation data based on the delay compensation parameter to obtain a rotation compensation timestamp; The method of controlling the scene model to move in the extended reality scene based on the compensated motion data during the first time period, rendering the motion process of the scene model, and obtaining and displaying the extended reality picture includes: During the first time period, in response to a match between the current moment and the rotation compensation timestamp, the scene model is controlled to rotate in the extended reality scene based on the rotation data, and the rotation process of the scene model is rendered to obtain and display the extended reality screen.

5. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring a delay compensation parameter corresponding to the change in the motion state of the target vehicle indicated by the vehicle-mounted navigation data comprises: Determine at least one target event existing for the target vehicle at the first moment based on the vehicle-mounted navigation data, wherein the target event is used to indicate an event to which the target vehicle needs to respond by changing a motion state during driving; The delay compensation parameter corresponding to the at least one target event is obtained.

6. The method according to claim 5, characterized in that The acquiring the delay compensation parameter corresponding to the at least one target event includes: Obtain a mapping relationship table between candidate events and candidate parameters; Determine the at least one target event from the candidate events; Based on the at least one target event, at least one delay compensation parameter corresponding to the target event is obtained by searching the mapping relationship table.

7. The method according to claim 5, characterized in that The acquiring the delay compensation parameter corresponding to the at least one target event includes: Determining acceleration change information and steering change information of the target vehicle within the first time period based on the at least one target event; generating a motion compensation parameter based on the acceleration change information, wherein the motion compensation parameter is used to provide delay compensation when the scene model moves on at least one coordinate axis in a preset coordinate system; generating a first rotation compensation parameter based on the acceleration change information, wherein the first rotation compensation parameter is used to provide delay compensation when the scene model rotates on a first coordinate axis corresponding to the vehicle driving direction in the preset coordinate system; generating a second rotation compensation parameter based on the steering change information, wherein the second rotation compensation parameter is used to provide delay compensation for the rotation of the scene model on a second coordinate axis perpendicular to the driving direction of the vehicle in the preset coordinate system; The delay compensation parameter is obtained from the motion compensation parameter, the first rotation compensation parameter and the second rotation compensation parameter.

8. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring a delay compensation parameter corresponding to the change in the motion state of the target vehicle indicated by the vehicle-mounted navigation data comprises: Acquire distance change information between the target vehicle and surrounding vehicles in a second time period, where the second time period is a time period before the first time period; Predicting the distance between the target vehicle and the surrounding vehicles in the second time period based on the distance change information and the vehicle-mounted navigation data; The change in the motion state of the target vehicle within the first time period is determined based on the on-board navigation data and the distance prediction information, and the delay compensation parameter corresponding to the change in the motion state is acquired.

9. A display processing device for an extended reality image, characterized in that: The device comprises: A first acquisition module is used to acquire the vehicle-mounted navigation data of the target vehicle in a first time period during the driving process of the target vehicle, wherein the first time period is a time period after the current time period, and the vehicle-mounted navigation data is used to indicate the navigation information of the target vehicle during the driving process; A second acquisition module is used to acquire a delay compensation parameter corresponding to the change in motion state based on the change in motion state of the target vehicle indicated by the vehicle-mounted navigation data, wherein the delay compensation parameter is used to synchronize the somatosensory motion perception generated by the change in motion state of the target vehicle and the screen motion perception of the extended reality screen, where the extended reality screen is a screen provided by an extended reality device; A display module is used to render and display the extended reality screen based on the delay compensation parameters during the first time period.

10. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the display processing method of the extended reality screen as described in any one of claims 1 to 8.

Images