Display processing method, device and equipment for extended reality picture

Abstract

The application discloses a display processing method, device and equipment of an extended reality picture, and relates to the technical field of extended reality. The method comprises the following steps: acquiring vehicle-mounted navigation data of a target vehicle in a first period during driving of the target vehicle, wherein the vehicle-mounted navigation data is used for indicating navigation information of the target vehicle during driving; acquiring a delay compensation parameter corresponding to a change in a motion state of the target vehicle based on the change in the motion state indicated by the vehicle-mounted navigation data, wherein the delay compensation parameter is used for synchronizing a somatic motion perception generated by the change in the motion state of the target vehicle and a picture motion perception of an extended reality picture, and the extended reality picture is a picture provided by an extended reality device; and rendering and displaying the extended reality picture based on the delay compensation parameter in the first period. The method can reduce dizziness of a user generated by using the extended reality device during driving of the vehicle, and improve user experience of the extended reality device in a vehicle-mounted scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of extended reality, and in particular to a display processing method, device and equipment for an extended reality picture. BACKGROUND

[0002] With the development of Virtual Reality (VR) technology, the VR technology is gradually applied to various scenes and fields.

[0003] In a vehicle scene, a vehicle terminal is committed to providing more diversified entertainment activities for users. When the VR technology is applied to the vehicle scene, a non-driver can experience VR-related entertainment activities in the vehicle through a VR device. Since there is a motion change in the driving process of the vehicle, such as acceleration, deceleration, turning, etc., in order to enable the actual motion and the brain motion to be normally matched, the VR device will match the motion of the objects in the picture with the motion of the vehicle through Inertial Navigation Technology.

[0004] However, when the picture displayed by the VR device is synchronized with the motion state of the vehicle, the time delay problem caused by the transmission between devices will cause the user to feel dizzy, which will cause the user to be unable to experience the related functions provided by the VR device for a long time in the vehicle scene. SUMMARY

[0005] The embodiments of the present application provide a display processing method, device and equipment for an extended reality picture. The technical solution is as follows:

[0006] On the one hand, a display processing method for an extended reality picture is provided, and the method comprises:

[0007] In the driving process of a target vehicle, vehicle navigation data of the target vehicle in a first period is acquired, the first period is a period after a current period, and the vehicle navigation data is used to indicate navigation information of the target vehicle in the driving process;

[0008] Based on the motion state change of the target vehicle indicated by the vehicle navigation data, a delay compensation parameter corresponding to the motion state change is acquired, the delay compensation parameter is used to synchronize the motion perception of the target vehicle caused by the motion state change and the picture motion perception of the extended reality picture, and the extended reality picture is a picture provided by an extended reality device;

[0009] In the first period, the extended reality picture is rendered and displayed based on the delay compensation parameter.

[0010] On the other hand, a display processing device for an extended reality picture is provided, and the device comprises:

[0011] The first obtaining module is configured to obtain vehicle-mounted navigation data of the target vehicle in a first time period during driving of the target vehicle, the first time period being a time period after a current time period, the vehicle-mounted navigation data being used to indicate navigation information of the target vehicle during driving;

[0012] The second obtaining module is configured to obtain a delay compensation parameter corresponding to a change in a motion state of the target vehicle based on the change in the motion state indicated by the vehicle-mounted navigation data, the delay compensation parameter being used to synchronize a motion perception of the target vehicle caused by the change in the motion state and a motion perception of the extended reality picture, the extended reality picture being a picture provided by an extended reality device;

[0013] The display module is configured to render and display the extended reality picture based on the delay compensation parameter in the first time period.

[0014] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the display processing method of the extended reality picture according to any one of the above embodiments of the present application.

[0015] In another aspect, a computer-readable storage medium is provided, the storage medium storing at least one instruction, at least one program, a code set or an instruction set, the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by a processor to implement the display processing method of the extended reality picture according to any one of the above embodiments of the present application.

[0016] In another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the display processing method of the extended reality picture according to any one of the above embodiments.

[0017] The technical solutions provided by the present application at least have the following beneficial effects:

[0018] When a user uses the extended reality device during vehicle driving, the motion state change of the target vehicle in a future period (a first period) is predicted through the vehicle navigation data of the target vehicle, and a corresponding delay compensation parameter is obtained according to the motion state change of the vehicle, and the extended reality device renders and displays the extended reality picture through the delay compensation parameter, so that the visual motion perception generated by the user when watching the extended reality picture can be synchronized with the somatosensory motion perception generated by the motion state change of the vehicle, the dizziness generated by the user when using the extended reality device during vehicle driving is reduced, and the user experience of the extended reality device in the vehicle-mounted scene is improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Figure 1 is a structural block diagram of a computer system provided by an exemplary embodiment of the present application;

[0021] Figure 2 is a flowchart of a display processing method of an extended reality picture provided by an exemplary embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a six-degree-of-freedom model provided by an exemplary embodiment of the present application;

[0023] Figure 4 is a flowchart of a display processing method of an extended reality picture provided by an exemplary embodiment of the present application;

[0024] Figure 5 is a structural block diagram of a display processing device of an extended reality picture provided by an exemplary embodiment of the present application;

[0025] Figure 6 is a structural block diagram of a terminal provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0027] First, the terms involved in the embodiments of the present application are briefly introduced.

[0028] Extended Reality (XR) technology: refers to the technology that combines reality and virtuality through computers to create a virtual environment that can be interacted with by humans. XR technology includes VR technology, Augmented Reality (AR) technology, and Mixed Reality (MR). XR technology is widely used in entertainment, education, medical, manufacturing, and training fields, providing users with more immersive and interactive experiences. For example, in the medical field, XR can be used for surgery simulation and medical training; in the education field, XR can create immersive learning environments to improve learning outcomes; in manufacturing and training, XR can be used for simulation training to reduce training costs and improve efficiency.

[0029] VR technology: is a technology that simulates a virtual environment through computers, allowing users to immerse themselves in it through specialized devices and interact with the virtual environment in real time. VR devices usually include head-mounted displays (HMD), handheld controllers, and sensors, etc., and users interact with the virtual environment through these devices. The head-mounted display provides a full range of visual angles, the sensor detects the user's head and body movements, and the handheld controller is used to manipulate virtual objects and perform specific actions in the virtual environment. VR technology stimulates users through multiple senses such as vision and hearing, providing an immersive experience. It is widely used in gaming, film, education, medical, architectural design, and other fields.

[0030] AR technology: is a technology that superimposes virtual information onto the real environment, allowing users to see and interact with virtual objects in the real world. AR devices usually include smart glasses, mobile phones, and tablets, etc., which capture the real environment through cameras and display virtual information on the screen. Users can interact with virtual objects through various interaction methods such as touch screens, gesture recognition, and voice commands. AR technology is widely used in entertainment, education, medical, industrial, and navigation fields.

[0031] MR technology: is a technology that combines the real world and the virtual world, seamlessly integrating virtual objects into the user's physical environment through advanced computer graphics and sensor technology, providing an immersive experience for users. MR devices usually include sensors and cameras that can capture and recognize user movements and environments, enabling precise 3D space mapping. Users can interact with virtual objects through various interaction methods such as gesture recognition, voice commands, eye tracking, physical controllers, head detection, etc. MR technology is widely used in entertainment, education, medical, industrial, and navigation fields.

[0032] Figure 1A structural block diagram of a computer system provided by an example embodiment of the present application is shown. The computer system 100 includes an extended reality device 110, a vehicle terminal 120, and a server 130.

[0033] The extended reality device 110 is installed and runs an application program supporting a virtual environment. The application program can be a virtual reality application program, a three-dimensional map program, a third-person shooter (TPS) game, a first-person shooting game (FPS), a multiplayer online battle arena (MOBA) game, a multiplayer gun battle survival game, a turn-based chess game, etc.

[0034] The device type of the extended reality device 110 includes at least one of a VR device, an AR device, and an MR device. The extended reality device 110 can be a device composed of a display device and a control device, where the display device can be implemented as a head-mounted display, smart glasses, a mobile phone, a tablet computer, etc., and the control device can be implemented as a handheld controller, a sensor, etc.

[0035] Those skilled in the art can know that the number of the above-mentioned extended reality devices 110 can be more or less. For example, the above-mentioned devices can be only one, or the above-mentioned devices can be dozens or hundreds, or more. The number of devices and the type of devices are not limited in the embodiments of the present application.

[0036] The vehicle terminal 120 is a front-end device of a vehicle monitoring and management system, which integrates various advanced technologies such as wireless communication, global positioning system (GPS) positioning, sensors, etc., and can realize functions such as vehicle remote control, vehicle condition monitoring, fault diagnosis, etc. The wireless communication realized by the vehicle terminal 120 includes vehicle network, Internet, mobile communication network, etc.

[0037] In some embodiments, the extended reality device 110 and the vehicle terminal 120 establish a communication connection. Illustratively, during the driving process of the vehicle, the extended reality device 110 obtains vehicle navigation data of a target vehicle in a first time period from the vehicle terminal 120, obtains a delay compensation parameter corresponding to a change in the motion state of the target vehicle based on the change in the motion state indicated by the vehicle navigation data, and renders and displays an extended reality picture based on the delay compensation parameter in the first time period.

[0038] In some embodiments, the computer system 100 further comprises a server 130, the server 130 comprising at least one of a server, a plurality of servers, a cloud computing platform, and a virtualization center. The server 130 is configured to provide background services for the extended reality device 110. Optionally, the server 130 undertakes the primary computing work, and the extended reality device 110 undertakes the secondary computing work; or the server 130 undertakes the secondary computing work, and the extended reality device 110 undertakes the primary computing work; or the server 130 and the extended reality device 110 cooperatively compute in a distributed computing architecture.

[0039] It is worth noting that the above-mentioned server 130 can be implemented as a physical server or a cloud server in the cloud. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network to realize data calculation, storage, processing, and sharing in a wide area network or a local area network. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on cloud computing business model application, which can form a resource pool for on-demand use and flexible convenience.

[0040] The extended reality device 110, the vehicle-mounted terminal 120, and the server 130 establish a communication connection. Illustratively, during the driving process of the vehicle, the server 130 obtains the vehicle-mounted navigation data of the target vehicle in the first time period from the vehicle-mounted terminal 120, obtains the delay compensation parameter corresponding to the change in the motion state of the target vehicle based on the change in the motion state indicated by the vehicle-mounted navigation data, generates the picture data corresponding to the extended reality picture in the first time period based on the delay compensation parameter, and sends the picture data to the extended reality device 110. After receiving the picture data, the extended reality device 110 displays the extended reality picture based on the picture data.

[0041] In some embodiments, the method provided by the present application can be applied to a cloud game scenario, so that the cloud server completes the calculation of the data logic in the game process, and the extended reality device 110 is responsible for the display of the game interface.

[0042] In some embodiments, the above-mentioned server 130 can also be implemented as a node in a blockchain system.

[0043] In combination with the above-mentioned term introduction and implementation environment, the display processing method of the extended reality picture provided by the present application is described, which is taken as an example of being executed by the extended reality device. Please refer to Figure 2 which shows the flowchart of the display processing method of the extended reality picture provided by an exemplary embodiment of the present application, which comprises the following steps 210 to 230.

[0044] At step 210, in-process vehicle navigation data of the target vehicle in a first time period is acquired.

[0045] Optionally, the target vehicle can be implemented as an Internal Combustion Engine Vehicle (ICEV), an Electric Vehicle (EV), a hybrid vehicle, a Plug-in Hybrid Electric Vehicle (PHEV), a Range-Extended Electric Vehicle (REEV), a Fuel Cell Electric Vehicle (FCEV), a Mild Hybrid Vehicle (MHV), etc., which are not limited herein.

[0046] Illustratively, the target vehicle is loaded with a vehicle terminal, which is an electronic device in the target vehicle for providing functions and services. The vehicle terminal can connect to the Internet through the Internet of Vehicles, and realize data exchange and communication between vehicles, between a vehicle and infrastructure, and between a vehicle and a user device.

[0047] In the embodiments of the present application, the user uses the extended reality device in the driving process of the target vehicle, and a communication connection is established between the extended reality device and the vehicle terminal. The communication connection can be a direct connection, such as Bluetooth, ZigBee, Wireless Fidelity (WiFi), etc., or an indirect connection, such as the extended reality device and the vehicle terminal connecting to the Internet respectively and establishing a communication connection through a server.

[0048] Optionally, the extended reality device includes at least one of a VR device, an AR device, and an MR device.

[0049] In some embodiments, when the vehicle terminal detects that the target vehicle starts driving, a vehicle driving message is sent to the connected extended reality device, the vehicle driving message being used to notify the extended reality device that the target vehicle is in a driving state, and when the extended reality device receives the vehicle driving message, a delay compensation algorithm is started.

[0050] In the embodiments of the present application, the first time period is a time period after the current time period. Optionally, the time length of the first time period can be preset by the system or determined based on the road conditions in the current driving process of the target vehicle.

[0051] Illustratively, the road condition information of the driving road of the target vehicle is acquired, the road condition score of the driving road is determined based on the road condition information, and the corresponding time period length is acquired according to the road condition score, wherein the road condition score and the time period length are in a negative correlation relationship.

[0052] Optionally, the road condition information includes at least one of traffic flow information, road surface condition information, traffic control information, traffic signal information, and road construction information. The traffic flow information is used to indicate the vehicle flow on the driving road. The road surface condition information is used to indicate the condition of the driving road, for example, dry road surface, wet and slippery road surface, icy road surface, and bumpy road surface. The traffic control information is used to indicate the traffic control condition of the driving road, for example, lane closure, tidal lane, and lane flow limitation. The traffic signal information is used to indicate the traffic signal of the driving road, for example, traffic light. The road construction information is used to indicate the construction section existing on the driving road.

[0053] In some embodiments, the road condition score is predicted by a pre-trained road condition evaluation model. Illustratively, the road condition information is input into the road condition evaluation model, and the road condition score of the driving road of the target vehicle is obtained by the road condition evaluation model. The time period length corresponding to the road condition score is acquired.

[0054] In the embodiments of the present application, the acquired vehicle navigation data is used to indicate the navigation information of the target vehicle in the driving process.

[0055] Optionally, the vehicle navigation data includes at least one of navigation route information, road intersection information, and navigation road condition information. The navigation route information is used to indicate the navigation path of the current journey of the target vehicle. The road intersection information is used to indicate the intersection passed on the navigation path. The navigation road condition information is used to indicate the road condition on the navigation path.

[0056] In some embodiments, the vehicle navigation data is acquired from the vehicle terminal after the extended reality device is connected to the vehicle terminal. Illustratively, the navigation application program is installed in the vehicle terminal. When the user sets the navigation route in the navigation application program, the vehicle terminal sends the vehicle navigation data corresponding to the navigation route to the extended reality device.

[0057] In some other embodiments, the vehicle navigation data is acquired from the user mobile terminal (for example, a mobile phone) after the extended reality device is connected to the user mobile terminal. Illustratively, the navigation application program is installed in the user mobile terminal. When the user sets the navigation route in the navigation application program, the user mobile terminal sends the vehicle navigation data corresponding to the navigation route to the extended reality device.

[0058] In step 220, the delay compensation parameter corresponding to the change of the motion state of the target vehicle is acquired based on the change of the motion state of the target vehicle indicated by the vehicle navigation data.

[0059] In the embodiments of the present application, the delay compensation parameter is used to synchronize the motion state change of the target vehicle and the motion perception of the extended reality picture, wherein the extended reality picture is a picture provided by an extended reality device.

[0060] In some embodiments, a plurality of motion state stages of the target vehicle in the first time period are determined based on the in-vehicle navigation data, a target motion state stage in which the direction and / or speed change exist is determined from the plurality of motion state stages, and the delay compensation parameter corresponding to the target motion state stage is obtained. In one example, based on the in-vehicle navigation data, it is determined that the target vehicle experiences a uniform speed driving stage and a deceleration driving stage in the first time period, and the delay compensation parameter corresponding to the deceleration driving stage is obtained.

[0061] In some embodiments, a mapping relationship table between the motion state stage and the delay compensation parameter is obtained, the target motion state stage is queried in the mapping relationship table, and the corresponding delay compensation parameter is determined.

[0062] In some embodiments, the generation of the delay compensation parameter is realized by a pre-trained delay prediction model. Illustratively, the in-vehicle navigation data is input into the pre-trained delay prediction model, the delay prediction model determines the motion state change of the target vehicle in the driving process based on the in-vehicle navigation data, and outputs the corresponding delay compensation parameter based on the change of the motion state. Optionally, the delay prediction model can be realized by a neural network model such as a convolutional neural network (CNN), a feedforward neural network (FNN), a residual network (ResNet), a transformer (Transformer), etc., which is not specifically limited here.

[0063] In some embodiments, in addition to determining the motion state change of the target vehicle in the first time period through the vehicle-mounted navigation data, distance prediction information between the target vehicle and the surrounding vehicle is introduced to improve the accuracy of the determination of the motion state change. Illustratively, distance change information between the target vehicle and the surrounding vehicle in a second time period is obtained, wherein the second time period is a time period before the first time period, and an example of the second time period is a historical time period; distance prediction information between the target vehicle and the surrounding vehicle in the second time period is predicted based on the distance change information and the vehicle-mounted navigation data; the motion state change of the target vehicle in the first time period is determined based on the vehicle-mounted navigation data and the distance prediction information, and a time delay compensation parameter corresponding to the motion state change is obtained. That is, the distance prediction information between the target vehicle and the surrounding vehicle in the second time period is predicted through the distance change information between the target vehicle and the surrounding vehicle in the historical time period and the route condition in the future time period indicated by the vehicle-mounted navigation data, and the motion state change of the target vehicle in the first time period is determined according to the distance prediction information and the vehicle-mounted navigation data.

[0064] In some embodiments, the prediction of the distance prediction information is realized through a pre-trained distance prediction model. Illustratively, the distance change information and the vehicle-mounted navigation data are input into the pre-trained distance prediction model, the distance prediction model determines the distance change between the target vehicle and the surrounding vehicle in the driving process based on the distance change information and the vehicle-mounted navigation data, and outputs corresponding distance prediction information. Optionally, the above-mentioned time delay prediction model can be realized through a neural network model such as a convolutional neural network, a feedforward neural network, a residual network, a transformer, etc., which is not specifically limited here.

[0065] In step 230, in the first time period, the extended reality picture is rendered and displayed based on the time delay compensation parameter.

[0066] In the embodiments of the present application, the extended reality device provides the user with an extended reality picture corresponding to an extended reality scene, the extended reality scene includes at least one scene model, and illustratively, when it is determined based on the vehicle-mounted navigation data that the target vehicle has a motion state change in the first time period, the motion of the scene model in the extended reality scene is adjusted based on the obtained time delay compensation parameter in the first time period, and the rendering and display of the extended reality picture are realized based on the adjusted data.

[0067] In some embodiments, model motion data of the scene model in the extended reality scene is obtained, wherein the model motion data is used to indicate the motion of the scene model in the extended reality scene, and the model motion data includes a motion execution timestamp; the motion execution timestamp in the model motion data is adjusted based on the time delay compensation parameter to obtain compensated motion data; in the first time period, the motion of the scene model in the extended reality scene is controlled based on the compensated motion data, and the motion process of the scene model is rendered to obtain and display the extended reality picture.

[0068] In some embodiments, the scene model is a model element built with a six-degree-of-freedom model, which can move freely in the extended reality scene. For example, the position three degrees of freedom (3DoF) and the orientation three degrees of freedom (3DoF) of the scene model can be changed arbitrarily, where the position three degrees of freedom is the movement freedom along the x, y, and z axes (X, Y, Z), and the orientation three degrees of freedom is the rotation freedom around the x, y, and z axes (a, b, g). Figure 3 As shown in FIG. 3, which shows a schematic diagram of a six-degree-of-freedom model according to an example embodiment of the present application, the scene model 300 is a six-degree-of-freedom model, which can move and rotate on the x-axis 301, move and rotate on the y-axis 302, and move and rotate on the z-axis 303 in the preset coordinate axes. The position three degrees of freedom and the orientation three degrees of freedom of the scene model 300 are changed to make it move freely in the extended reality scene.

[0069] In some embodiments, the obtained model motion data includes movement data of the scene model, and movement timestamps corresponding to the movement data, where the movement data is used to indicate the change of the scene model in the movement degrees of freedom. For example, the movement timestamps corresponding to the movement data are adjusted based on the delay compensation parameter to obtain movement compensation timestamps; in the first time period, in response to the current time matching the movement compensation timestamps, the movement of the scene model in the extended reality scene is controlled based on the movement data, and the movement process of the scene model is rendered to obtain and display the extended reality picture. That is, the movement timestamps corresponding to the movement data are adjusted based on the delay compensation parameter, so as to realize the delay compensation of the movement of the scene model in the x, y, and z axes when the vehicle motion state changes.

[0070] In some embodiments, the model motion data includes rotation data of the scene model, and rotation timestamps corresponding to the rotation data, where the rotation data is used to indicate the change of the scene model in the rotation degrees of freedom. For example, the rotation timestamps corresponding to the rotation data are adjusted based on the delay compensation parameter to obtain rotation compensation timestamps; in the first time period, in response to the current time matching the rotation compensation timestamps, the rotation of the scene model in the extended reality scene is controlled based on the rotation data, and the rotation process of the scene model is rendered to obtain and display the extended reality picture. That is, the rotation timestamps corresponding to the rotation data are adjusted based on the delay compensation parameter, so as to realize the delay compensation of the rotation of the scene model around the x, y, and z axes when the vehicle motion state changes.

[0071] In some embodiments, it is considered that the target vehicle generally only moves and rotates on two coordinate axes corresponding to the plane where the ground is located during the movement, and therefore, when the motion delay compensation is implemented by the delay compensation parameter, the compensation is only performed on four degrees of freedom (for example, X, Y, a, β), thereby reducing the amount of data calculation and storage, and reducing the consumption of device computing resources and storage resources.

[0072] In summary, when the user uses the extended reality device during the driving of the vehicle, the motion state change of the target vehicle in the future period (the first period) is predicted by the vehicle navigation data of the target vehicle, and the corresponding delay compensation parameter is obtained according to the motion state change of the vehicle. The extended reality device renders and displays the extended reality picture through the delay compensation parameter, so that the visual motion perception generated by the user when watching the extended reality picture can be synchronized with the somatosensory motion perception generated by the motion state change of the vehicle, the dizziness generated by the user using the extended reality device during the driving of the vehicle is reduced, and the user experience of the extended reality device in the vehicle scene is improved.

[0073] In some optional embodiments, the event of the vehicle state change of the target vehicle during the driving is determined based on the vehicle navigation data, so as to compensate the delay of the extended reality picture displayed by the extended reality device by obtaining the delay compensation parameter corresponding to the event. Please refer to Figure 4 which shows a flowchart of the display processing method of the extended reality picture provided by an example embodiment of the present application, and the method comprises the following steps.

[0074] In step 210, during the driving of the target vehicle, the vehicle navigation data of the target vehicle in the first period is obtained.

[0075] Optionally, the target vehicle can be implemented as an internal combustion engine driven car, an electric car, a hybrid car, a plug-in hybrid car, a range-extended electric car, a fuel cell electric car, a mild hybrid car, etc., which is not limited here.

[0076] In the embodiments of the present application, during the driving of the target vehicle, the extended reality device obtains the vehicle navigation data of the target vehicle in the first period from the vehicle terminal. The vehicle terminal is an electronic device in the target vehicle for providing functions and services, and the vehicle terminal can connect to the Internet through the Internet of Vehicles, and realize data exchange and communication between vehicles, between vehicles and infrastructure, and between vehicles and user devices.

[0077] Optionally, the above-mentioned extended reality device comprises at least one of a VR device, an AR device, and an MR device.

[0078] In some embodiments, when the vehicle terminal detects that the target vehicle starts driving, a vehicle driving message is sent to the connected extended reality device, the vehicle driving message being used to inform the extended reality device that the target vehicle is in a driving state, and when the extended reality device receives the vehicle driving message, a delay compensation algorithm is started.

[0079] In the embodiments of the present application, the first time period is a time period after the current time period. Optionally, the time length of the first time period can be preset by the system or determined based on the road conditions in the current driving process of the target vehicle.

[0080] Optionally, the vehicle navigation data includes at least one of navigation route information, road intersection information, and navigation road condition information, wherein the navigation route information is used to indicate a navigation path of the current journey of the target vehicle, the road intersection information is used to indicate a road intersection passed on the navigation path, and the navigation road condition information is used to indicate a road condition on the navigation path.

[0081] In some embodiments, the vehicle navigation data is obtained by the extended reality device from the vehicle terminal after the extended reality device is connected to the vehicle terminal. Illustratively, a navigation application is installed in the vehicle terminal, and after a user sets a navigation route in the navigation application, the vehicle terminal sends vehicle navigation data corresponding to the navigation route to the extended reality device.

[0082] In some other embodiments, the vehicle navigation data is obtained by the extended reality device from a user mobile terminal (e.g., a mobile phone) after the extended reality device is connected to the user mobile terminal. Illustratively, a navigation application is installed in the user mobile terminal, and after a user sets a navigation route in the navigation application, the user mobile terminal sends vehicle navigation data corresponding to the navigation route to the extended reality device.

[0083] Step 221: determining at least one target event existing at the first time based on the vehicle navigation data.

[0084] In the embodiments of the present application, the target event is used to indicate an event that the target vehicle needs to respond by changing the motion state during driving, i.e., the target event is obtained by predicting the event that the target vehicle needs to respond by changing the motion state according to the navigation route information, road intersection information, navigation road condition information, etc. indicated by the vehicle navigation data.

[0085] Optionally, the target event includes at least one of the following:

[0086] 1. The distance between the target vehicle and the next intersection is a (e.g., 20) meters;

[0087] 2. The red light countdown timer has b (e.g., 2) seconds left;

[0088] 3. A pre-collision warning alarm prompts the distance c meters (for example, 4 meters) of the front vehicle following the car;

[0089] 4. A lateral anti-collision warning is not alarmed;

[0090] 5. Left turn;

[0091] 6. Right turn;

[0092] 7. U-turn;

[0093] 8. There is a left side leaning state;

[0094] 9. There is a right side leaning state;

[0095] 10. There is an acceleration state;

[0096] 11. There is a deceleration state;

[0097] 12. Backing up.

[0098] Step 222, obtaining the time compensation parameter corresponding to the at least one target event.

[0099] In some embodiments, the time compensation parameter corresponding to the target event is obtained through a mapping relationship table between the candidate events and the candidate parameters. Illustratively, a mapping relationship table between the candidate events and the candidate parameters is obtained; at least one target event is determined from the candidate events; and at least one time compensation parameter corresponding to the target event is found from the mapping relationship table based on the at least one target event.

[0100] Optionally, the extended reality device pre-stores the mapping relationship table, and in response to detecting the target event, obtains the pre-stored mapping relationship table from the device storage area; optionally, the extended reality device sends the detected target event to the server, and the server queries the time compensation parameter corresponding to the target event based on the stored mapping relationship table and sends the time compensation parameter to the extended reality device.

[0101] In some embodiments, under the target event, the motion state change of the target vehicle affects the user's sense of motion perception, including the acceleration change perception of displacement and the steering change perception of rotation, so when providing the time compensation parameter for the motion of the scene model, it is also necessary to compensate for the picture motion perception in terms of displacement and rotation.

[0102] Illustratively, the acceleration change information and the steering change information of the target vehicle in the first time period are determined based on at least one target event; the movement compensation parameter is generated based on the acceleration change information, wherein the movement compensation parameter is used to provide time delay compensation when the scene model moves in at least one coordinate axis in the preset coordinate system; the first rotation compensation parameter is generated based on the acceleration change information, wherein the first rotation compensation parameter is used to provide time delay compensation when the scene model rotates in a first coordinate axis corresponding to the driving direction of the vehicle in the preset coordinate system; the second rotation compensation parameter is generated based on the steering change information, wherein the second rotation compensation parameter is used to provide time delay compensation when the scene model rotates in a second coordinate axis perpendicular to the driving direction of the vehicle in the preset coordinate system; and the time delay compensation parameter is obtained from the movement compensation parameter, the first rotation compensation parameter and the second rotation compensation parameter.

[0103] In one example, the mapping relationship between the scene model degrees of freedom, the target events and the time delay compensation parameters is shown in Table 1, in which the target events determined based on the vehicle navigation data include the distance between the target vehicle and the next intersection of a meters, the red light countdown time remaining of b seconds, the front collision warning alarm prompt front vehicle following distance of c meters, left / right steering information prompt, acceleration / deceleration state, left roll state and right roll state.

[0104] Table 1

[0105]

[0106]

[0107] In the time delay compensation parameter, k is an event coefficient corresponding to the event in the time delay compensation parameter, which can be obtained by looking up the table. Illustratively, as shown in Table 2, exemplary k is shown. a1 Corresponding relationship between the actual distance a in the event "the distance between the target vehicle and the next intersection is a meters".

[0108] Table 2

[0109] k a1 ]]> 1 0.8 0.6 0.4 0.2 0.1 0 a 2m 4m 8m 15m 25m 40m ＞40m

[0110] Illustratively, taking A=5ms and the vehicle navigation data indicating that the distance between the target vehicle and the next intersection is 8 meters at the first candidate time in the first time period as an example, then k a1 A=3ms, i.e., at the first candidate time in the first time period, the time delay compensation is performed on the motion of the scene model in the degree of freedom X in the extended reality scene, and the compensation duration is 3ms.

[0111] Step 230, in the first time period, rendering and displaying the extended reality picture based on the time delay compensation parameter.

[0112] In the embodiment of the present application, the extended reality device provides a user with an extended reality picture corresponding to an extended reality scene, the extended reality scene includes at least one scene model, and illustratively, in the case that the target vehicle has a motion state change in the first time period, the motion of the scene model in the extended reality scene is adjusted based on the obtained delay compensation parameter in the first time period, and the rendering and display of the extended reality picture are implemented based on the adjusted data.

[0113] In summary, when the user uses the extended reality device during the driving of the vehicle, the motion state change of the target vehicle in the future time period (the first time period) is predicted based on the vehicle navigation data of the target vehicle, the corresponding delay compensation parameter is obtained according to the motion state change of the vehicle, and the extended reality device displays the extended reality picture by using the delay compensation parameter, so that the visual motion perception generated by the user when watching the extended reality picture can be synchronized with the somatosensory motion perception generated by the motion state change of the vehicle, the dizziness generated by the user when using the extended reality device during the driving of the vehicle is reduced, and the user experience of the extended reality device in the vehicle scene is improved.

[0114] It should be noted that, before collecting the relevant data of the user and during the process of collecting the relevant data of the user, a prompt interface, a pop-up window or voice prompt information can be displayed, the prompt interface, the pop-up window or the voice prompt information is used to prompt that the relevant data of the user is currently being collected, so that the present application only starts to perform the related steps of obtaining the relevant data of the user after obtaining the confirmation operation of the user to the prompt interface or the pop-up window, otherwise (i.e. without obtaining the confirmation operation of the user to the prompt interface or the pop-up window), the related steps of obtaining the relevant data of the user are ended, that is, the relevant data of the user is not obtained. In other words, all the user data collected by the present application is collected under the condition that the user agrees and authorizes, and the collection, use and processing of the relevant user data need to comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0115] Please refer to Figure 5 which shows the structure block diagram of the display processing device of the extended reality picture provided by one exemplary embodiment of the present application, the device includes the following modules:

[0116] The first obtaining module 510 is configured to obtain vehicle navigation data of a target vehicle in a first time period during the driving of the target vehicle, the first time period is a time period after a current time period, and the vehicle navigation data is used to indicate navigation information of the target vehicle in the driving process;

[0117] The second acquisition module 520 is configured to acquire a time delay compensation parameter corresponding to a change in the motion state of the target vehicle based on the change in the motion state of the target vehicle indicated by the vehicle navigation data, the time delay compensation parameter being used to synchronize a body motion perception caused by the change in the motion state of the target vehicle and a picture motion perception of an extended reality picture, and the extended reality picture being a picture provided by an extended reality device.

[0118] The display module 530 is configured to render and display the extended reality picture based on the time delay compensation parameter within the first time period.

[0119] In some optional embodiments, the device further includes:

[0120] The third acquisition module (not shown in the figure) is further configured to acquire model motion data of a scene model in an extended reality scene, the model motion data being used to indicate a motion condition of the scene model in the extended reality scene, and the model motion data including a motion execution timestamp.

[0121] The display module 530 is further configured to adjust the motion execution timestamp in the model motion data based on the time delay compensation parameter to obtain compensated motion data.

[0122] The display module 530 is further configured to control the motion of the scene model in the extended reality scene based on the compensated motion data within the first time period, and render a motion process of the scene model to obtain and display the extended reality picture.

[0123] In some optional embodiments, the model motion data includes movement data of the scene model and a movement timestamp corresponding to the movement data, and the movement data is used to indicate a change condition of the scene model in a movement degree of freedom.

[0124] The display module 530 is further configured to adjust the movement timestamp corresponding to the movement data based on the time delay compensation parameter to obtain a movement compensation timestamp.

[0125] The display module 530 is further configured to control the movement of the scene model in the extended reality scene based on the movement data within the first time period in response to a match between a current time and the movement compensation timestamp, and render a movement process of the scene model to obtain and display the extended reality picture.

[0126] In some optional embodiments, the model motion data includes rotation data of the scene model and a rotation timestamp corresponding to the rotation data, and the rotation data is used to indicate a change condition of the scene model in a rotation degree of freedom.

[0127] The display module 530 is further configured to adjust a rotation timestamp corresponding to the rotation data based on the delay compensation parameter to obtain a rotation compensation timestamp.

[0128] The display module 530 is further configured to, in the first time period, control the scene model to rotate in the extended reality scene based on the rotation data in response to the current time and the rotation compensation timestamp matching, and render a rotation process of the scene model to obtain and display the extended reality picture.

[0129] In some optional embodiments, the second acquisition module 520 is further configured to determine at least one target event in which the target vehicle exists at the first time based on the vehicle-mounted navigation data, the target event being used to indicate an event in which the target vehicle needs to respond by changing a motion state during driving;

[0130] The second acquisition module 520 is further configured to acquire the delay compensation parameter corresponding to the at least one target event.

[0131] In some optional embodiments, the second acquisition module 520 is further configured to acquire a mapping relationship table between candidate events and candidate parameters.

[0132] The second acquisition module 520 is further configured to determine the at least one target event from the candidate events.

[0133] The second acquisition module 520 is further configured to find at least one delay compensation parameter corresponding to the target event from the mapping relationship table based on the at least one target event.

[0134] In some optional embodiments, the second acquisition module 520 is further configured to determine acceleration change information and steering change information of the target vehicle in the first time period based on the at least one target event.

[0135] The second acquisition module 520 is further configured to generate a movement compensation parameter based on the acceleration change information, the movement compensation parameter being used to provide time delay compensation when a scene model moves on at least one coordinate axis in a preset coordinate system.

[0136] The second acquisition module 520 is further configured to generate a first rotation compensation parameter based on the acceleration change information, the first rotation compensation parameter being used to provide time delay compensation when the scene model rotates on a first coordinate axis corresponding to a vehicle driving direction in the preset coordinate system.

[0137] The second acquisition module 520 is further configured to generate a second rotation compensation parameter based on the steering change information, the second rotation compensation parameter being used to provide time delay compensation for rotation of the scene model in a second coordinate axis perpendicular to the driving direction of the vehicle in the preset coordinate system.

[0138] The second acquisition module 520 is further configured to obtain the time delay compensation parameter from the movement compensation parameter, the first rotation compensation parameter and the second rotation compensation parameter.

[0139] In some optional embodiments, the second acquisition module 520 is further configured to acquire distance change information between the target vehicle and surrounding vehicles in a second time period, the second time period being a time period before the first time period.

[0140] The second acquisition module 520 is further configured to predict distance information of the target vehicle and the surrounding vehicles in the second time period based on the distance change information and the in-vehicle navigation data.

[0141] The second acquisition module 520 is further configured to determine a change in the motion state of the target vehicle in the first time period based on the in-vehicle navigation data and the distance prediction information, and acquire the time delay compensation parameter corresponding to the change in the motion state.

[0142] To sum up, when a user uses an extended reality device during vehicle driving, the motion state change of a target vehicle in a future time period (a first time period) is predicted based on in-vehicle navigation data of the target vehicle, and a corresponding time delay compensation parameter is acquired according to the change in the motion state of the vehicle. The extended reality device renders and displays an extended reality picture through the time delay compensation parameter, so that the visual motion perception generated by the user when watching the extended reality picture can be synchronized with the somatosensory motion perception generated by the change in the motion state of the vehicle, the dizziness generated by the user when using the extended reality device during vehicle driving is reduced, and the user experience of the extended reality device in a vehicle-mounted scene is improved.

[0143] It should be noted that the display processing apparatus of the extended reality picture provided in the above embodiments is only exemplified by the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the display processing apparatus of the extended reality picture and the display processing method of the extended reality picture provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0144] Figure 6A structural block diagram of a terminal 600 provided by an example embodiment of the present application is shown. The terminal 600 can be a smartphone, a VR head-mounted device, an AR display device, a tablet computer, a Moving Picture Experts Group Audio Layer III (MP3) player, a Moving Picture Experts Group Audio Layer IV (MP4) player, a notebook computer, or a desktop computer. The terminal 600 can also be referred to as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0145] Generally, the terminal 600 includes a processor 601 and a memory 602.

[0146] The processor 601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, or the like. The processor 601 can be implemented in at least one of a Digital Signal Processing (DSP), a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), or the like. The processor 601 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a Central Processing Unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 601 can be integrated with a Graphics Processing Unit (GPU) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 601 can further include an Artificial Intelligence (AI) processor for processing machine learning-related computing operations.

[0147] The memory 602 can include one or more computer-readable storage media, which can be non-transitory. The memory 602 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is configured to store at least one instruction for being executed by the processor 601 to implement the display processing method of the extended reality picture provided by the method embodiments of the present application.

[0148] Illustratively, terminal 600 also includes other components 603 that can be understood by those of skill in the art, Figure 6 The structure shown in FIG. 6 is merely illustrative of an implementation and does not limit the scope of terminal 600, which can include more or fewer components than shown or combine some components, or have a different configuration or arrangement of the components.

[0149] Those of skill in the art can understand that all or part of the steps in the methods of the above embodiments can be completed by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can be a computer readable storage medium included in the memory in the above embodiments, or can be a computer readable storage medium existing separately and not assembled into the terminal. The computer readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the display processing method of the extended reality picture in any of the above embodiments.

[0150] Optionally, the computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. The random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above embodiment numbers of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0151] Those of skill in the art can understand that all or part of the steps of the above embodiments can be completed by hardware or by a program instructing relevant hardware, and the program can be stored in a computer readable storage medium, which can be a read-only memory, a magnetic disk or an optical disk, etc.

[0152] The above description is only optional embodiments of the present application and does not limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A display processing method of an extended reality picture, characterized by, The method comprises: In the driving process of the target vehicle, vehicle-mounted navigation data of the target vehicle in a first period is acquired, the first period being a period after a current period, the vehicle-mounted navigation data being used to indicate navigation information of the target vehicle in the driving process; Based on a change of a motion state of the target vehicle indicated by the vehicle-mounted navigation data, a delay compensation parameter corresponding to the change of the motion state is acquired, the delay compensation parameter being used to synchronize a motion perception of a body sense generated by the change of the motion state of the target vehicle and a motion perception of a picture of the extended reality picture, the extended reality picture being a picture provided by an extended reality device; Model motion data of a scene model in an extended reality scene is acquired, the model motion data being used to indicate a motion condition of the scene model in the extended reality scene, the model motion data comprising a motion execution timestamp; The motion execution timestamp in the model motion data is adjusted based on the delay compensation parameter, to obtain compensated motion data; In the first period, the scene model is controlled to move in the extended reality scene based on the compensated motion data, and a motion process of the scene model is rendered, to obtain and display the extended reality picture.

2. The method of claim 1, wherein, The model motion data comprises movement data of the scene model and movement timestamps corresponding to the movement data, the movement data being used to indicate a change condition of the scene model in a movement degree of freedom; The motion execution timestamp in the model motion data is adjusted based on the delay compensation parameter, to obtain compensated motion data, comprising: The movement timestamps corresponding to the movement data are adjusted based on the delay compensation parameter, to obtain movement compensation timestamps; In the first period, the scene model is controlled to move in the extended reality scene based on the compensated motion data, and a motion process of the scene model is rendered, to obtain and display the extended reality picture, comprising: In the first period, the scene model is controlled to move in the extended reality scene based on the movement data in response to a match between a current time and the movement compensation timestamps, and a movement process of the scene model is rendered, to obtain and display the extended reality picture.

3. The method of claim 1, wherein, The model motion data comprises rotation data of the scene model and rotation timestamps corresponding to the rotation data, the rotation data being used to indicate a change condition of the scene model in a rotation degree of freedom; The motion execution timestamp in the model motion data is adjusted based on the delay compensation parameter, to obtain compensated motion data, comprising: The rotation timestamps corresponding to the rotation data are adjusted based on the delay compensation parameter, to obtain rotation compensation timestamps; In the first period, the scene model is controlled to move in the extended reality scene based on the compensated motion data, and a motion process of the scene model is rendered, to obtain and display the extended reality picture, comprising: In the first time period, in response to the current time and the rotation compensation timestamp matching, the scene model is controlled to rotate in the extended reality scene based on the rotation data, and a rotation process of the scene model is rendered to obtain and display the extended reality picture.

4. The method according to any one of claims 1 to 3, characterized in that, The delay compensation parameter corresponding to the motion state change of the target vehicle indicated by the vehicle navigation data is obtained. At least one target event of the target vehicle existing at a first time point is determined based on the vehicle navigation data, and the target event is used to indicate an event that needs to be responded to by changing the motion state of the target vehicle during driving. The delay compensation parameter corresponding to the at least one target event is obtained.

5. The method of claim 4, wherein, The delay compensation parameter corresponding to the at least one target event is obtained. A mapping relationship table between candidate events and candidate parameters is obtained. The at least one target event is determined from the candidate events. At least one delay compensation parameter corresponding to the target event is found from the mapping relationship table based on the at least one target event.

6. The method of claim 4, wherein, The delay compensation parameter corresponding to the at least one target event is obtained. Acceleration change information and steering change information of the target vehicle in the first time period are determined based on the at least one target event. A movement compensation parameter is generated based on the acceleration change information, and the movement compensation parameter is used to provide time delay compensation when a scene model moves on at least one coordinate axis in a preset coordinate system. A first rotation compensation parameter is generated based on the acceleration change information, and the first rotation compensation parameter is used to provide time delay compensation when the scene model rotates on a first coordinate axis corresponding to a vehicle driving direction in the preset coordinate system. A second rotation compensation parameter is generated based on the steering change information, and the second rotation compensation parameter is used to provide time delay compensation when the scene model rotates on a second coordinate axis perpendicular to the vehicle driving direction in the preset coordinate system. The delay compensation parameter is obtained from the movement compensation parameter, the first rotation compensation parameter, and the second rotation compensation parameter.

7. The method according to any one of claims 1 to 3, characterized in that, The delay compensation parameter corresponding to the motion state change of the target vehicle indicated by the vehicle navigation data is obtained. Distance change information between the target vehicle and surrounding vehicles in a second time period is obtained, and the second time period is a time period before the first time period. Distance prediction information of the target vehicle and the surrounding vehicles in the second time period is predicted based on the distance change information and the vehicle navigation data. The motion state change of the target vehicle in the first time period is determined based on the vehicle navigation data and the distance prediction information, and the delay compensation parameter corresponding to the motion state change is obtained.

8. A display processing apparatus of an extended reality picture, characterized by comprising: The device comprises: A first obtaining module is configured to obtain vehicle navigation data of a target vehicle in a first time period during driving of the target vehicle, the first time period being a time period after a current time period, and the vehicle navigation data being used to indicate navigation information of the target vehicle during driving. The second acquisition module is configured to acquire a time delay compensation parameter corresponding to a change in a motion state of the target vehicle based on the change in the motion state of the target vehicle indicated by the vehicle navigation data, the time delay compensation parameter being used to synchronize a body motion perception caused by the change in the motion state of the target vehicle and a picture motion perception of an extended reality picture, the extended reality picture being a picture provided by an extended reality device. The third acquisition module is configured to acquire model motion data of a scene model in an extended reality scene, the model motion data being used to indicate a motion condition of the scene model in the extended reality scene, and the model motion data including a motion execution timestamp. The display module is configured to adjust the motion execution timestamp in the model motion data based on the time delay compensation parameter to obtain compensated motion data, control the scene model to move in the extended reality scene based on the compensated motion data in the first time period, and render a motion process of the scene model to obtain and display the extended reality picture.

9. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores at least one program, which is loaded and executed by the processor to implement the display processing method of the extended reality picture according to any one of claims 1 to 7.

Images