Virtual reality display system, method and apparatus, and device and medium

Abstract

A virtual reality display system, method and apparatus, and a device and a medium. The system comprises a virtual reality device (110), a vehicle-mounted camera assembly (120) and a vehicle-mounted terminal (130), wherein the vehicle-mounted camera assembly (120) is configured to collect road condition images and send the road condition images to the vehicle-mounted terminal (130); the vehicle-mounted terminal (130) is configured to perform identification on the received road condition images to obtain an identification result, analyze the identification result to obtain loading delay compensation data, and send the loading delay compensation data and the identification result to the virtual reality device (110); and the virtual reality device (110) is configured to receive the loading delay compensation data and the identification result, acquire loading displacement data on the basis of the identification result, and display a virtual reality picture on the basis of the loading delay compensation data and the loading displacement data.

Description

Virtual reality display system, method, device, apparatus, medium

[0001] The present application claims priority from the Chinese patent application No. 202411362971.0 filed on September 27, 2024 and entitled "Virtual reality display system, method, device, apparatus, medium", the contents of which are incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of virtual reality technology, in particular to a virtual reality display system, method, device, apparatus, and medium. BACKGROUND

[0003] Virtual reality (VR) technology creates and simulates a three-dimensional virtual world, allowing users to obtain immersive experience in the virtual world through head-mounted displays, handsets, body tracking devices, and other interactive devices.

[0004] VR technology can be applied to a vehicle riding scenario, in which a vehicle-mounted VR product can provide a new way of entertainment and information acquisition for passengers. By wearing a VR device, passengers can watch multimedia content, experience games, or take virtual trips during vehicle driving.

[0005] However, the road conditions of a vehicle during driving can change at any time, and the vehicle cannot avoid bumps, which can affect the stability of the VR device when displaying pictures. SUMMARY

[0006] Embodiments of the present application provide a virtual reality display system, method, device, apparatus, and medium. The technical solution is as follows:

[0007] In one aspect, a virtual reality display system is provided, which includes a virtual reality device, a vehicle-mounted camera assembly, and a vehicle-mounted terminal.

[0008] The vehicle-mounted camera assembly is configured to collect a road condition image, the road condition image being used to indicate the road condition of a road where a first vehicle is located; and send the road condition image to the vehicle-mounted terminal.

[0009] The vehicle-mounted terminal is configured to identify the road condition image to obtain an identification result, the identification result is used to indicate a type of an obstacle contained in the road condition image; obtain loading time delay compensation data based on the identification result; wherein the loading time delay compensation data is used to indicate a time at which the virtual reality device displays at least one element in a virtual reality picture, the at least one element is used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user; and the loading time delay compensation data and the identification result are sent to the virtual reality device.

[0010] The virtual reality device is configured to obtain loading displacement data based on the identification result, the loading displacement data is used to indicate a display position of the at least one element in the virtual reality picture; and display the virtual reality picture based on the loading time delay compensation data and the loading displacement data.

[0011] In an optional embodiment, the vehicle-mounted terminal is further configured to perform target detection on the road condition image by a target detection model to obtain a target detection result, the target detection result is used to indicate a case that there is an obstacle in the road condition image; and obtain the identification result based on the target detection result in the case that the target detection result indicates that there is an obstacle in the road condition image; wherein the identification result contains a first position of the obstacle in the road condition image and an obstacle type corresponding to the obstacle.

[0012] In an optional embodiment, the vehicle-mounted terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first position and a position of the first vehicle; and determine the loading time delay compensation data based on the first distance and a vehicle speed of the first vehicle.

[0013] In an optional embodiment, the vehicle-mounted camera assembly is further configured to collect the road condition image in real time based on a preset frequency; and send the road condition image to the vehicle-mounted terminal based on the preset frequency.

[0014] The vehicle-mounted terminal is further configured to update the first distance based on the road condition image received in real time to obtain an updated first distance; obtain first delay data based on the updated first distance and a vehicle speed of the first vehicle; obtain a first unit time length consumed by the vehicle-mounted camera to collect the road condition image once and send to the vehicle-mounted terminal; and determine the loading time delay compensation data based on the first delay data and the first unit time length.

[0015] In an optional embodiment, the virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, the displacement type being used to indicate a direction of adjusting a display position of the at least one element in the virtual reality picture; acquire a loading displacement reference table and pixel proportion data of the obstacle in the road condition image; and determine the loading displacement data from the loading displacement reference table based on the pixel proportion data and the displacement type.

[0016] In an optional embodiment, the vehicle-mounted camera assembly is further configured to pre-process the road condition image to obtain a pre-processed road condition image, the pre-processed road condition image meeting preset image processing format requirements; and send the pre-processed road condition image to the vehicle-mounted terminal.

[0017] The vehicle-mounted terminal is further configured to receive the pre-processed road condition image; and identify the pre-processed road condition image to obtain the recognition result.

[0018] On the other hand, a virtual reality display method is provided, which is executed by a vehicle-mounted terminal, and the method comprises:

[0019] receiving a road condition image, the road condition image being an image collected by a vehicle-mounted camera assembly of a first vehicle, the road condition image being used to indicate a road condition of a road where the first vehicle is located;

[0020] identifying the received road condition image to obtain a recognition result, the recognition result being used to indicate a type of obstacle contained in the road condition image;

[0021] analyzing the recognition result to obtain loading latency compensation data;

[0022] sending the loading latency compensation data and the recognition result to a virtual reality device;

[0023] The loading latency compensation data is used to indicate a time when the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user. After receiving the recognition result, the virtual reality device acquires loading displacement data based on the recognition result, and displays the virtual reality picture based on the loading latency compensation data and the loading displacement data, the loading displacement data being used to indicate a display position of the at least one element in the virtual reality picture.

[0024] On the other hand, a virtual reality display device is provided, and the device comprises:

[0025] receive a road condition image, the road condition image being an image collected by a camera assembly of a first vehicle, the road condition image being used to indicate a road condition of a road where the first vehicle is located;

[0026] identify the received road condition image to obtain an identification result, the identification result being used to indicate a type of an obstacle contained in the road condition image;

[0027] analyze the identification result to obtain loading time delay compensation data;

[0028] send the loading time delay compensation data and the identification result to a virtual reality device; wherein the loading time delay compensation data is used to indicate a time at which the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user; after receiving the identification result, the virtual reality device acquires loading displacement data based on the identification result, and displays the virtual reality picture based on the loading time delay compensation data and the loading displacement data, the loading displacement data being used to indicate a display position of the at least one element in the virtual reality picture.

[0029] In another aspect, a computer device is provided, the computer device comprising a processor and a memory, the memory having stored therein at least one instruction, at least one program, a code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the virtual reality display method according to any one of the above embodiments of the present application.

[0030] In another aspect, a computer readable storage medium is provided, the storage medium having stored therein at least one instruction, at least one program, a code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by a processor to implement the virtual reality display method according to any one of the above embodiments of the present application.

[0031] In another aspect, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the virtual reality display method according to any one of the above embodiments.

[0032] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0033] When the virtual reality device is used in a riding scene, the road condition of a road where a vehicle is located is obtained by collecting road condition images in real time, and whether bumping will occur in the vehicle driving process is determined according to the existence condition and type of an obstacle in the road condition, so that the virtual reality device is compensated for delay based on the bumping prediction condition, and the problem of delay in display of a virtual reality picture caused by head movement of a user is alleviated. The road condition images are analyzed to obtain loading displacement data and loading delay compensation data, the way in which the virtual reality device displays elements in the virtual reality picture when the vehicle passes the obstacle is determined, the elements in the picture are loaded to a target position in advance, the problems of dizziness and delay of the user when the user watches the picture while wearing the virtual reality device are reduced, the stability of the virtual reality device when displaying the virtual reality picture is improved, and the fragmentation of the virtual reality picture caused by bumping is reduced. In addition, the display position and time of the elements are determined by loading the delay compensation and displacement data, the problems of out-of-picture or display position offset of the elements caused by bumping of the vehicle due to the obstacle are avoided, and the display accuracy of the virtual reality picture is improved. BRIEF DESCRIPTION OF DRAWINGS

[0034] FIG. 1 is a schematic diagram of a virtual reality display system according to an example embodiment of the present application;

[0035] FIG. 2 is a flowchart of a virtual reality display method according to an example embodiment of the present application;

[0036] FIG. 3 is a schematic diagram of a data set in a training stage according to an example embodiment of the present application;

[0037] FIG. 4 is a flowchart of a virtual reality display method according to another example embodiment of the present application;

[0038] FIG. 5 is a structural block diagram of a virtual reality display device according to an example embodiment of the present application;

[0039] FIG. 6 is a structural block diagram of a computer device according to an example embodiment of the present application. DETAILED DESCRIPTION

[0040] First, the terms involved in the embodiments of the present application are briefly introduced:

[0041] Virtual Reality (VR): a technology that generates a three-dimensional virtual environment in which a user can immerse and interact through computer technology and hardware devices. The VR technology aims to provide the user with the perception of a virtual world, so that the user can have an immersive experience in a virtual space different from the real world.

[0042] VR devices are the key to realizing VR technology. Common VR devices include head-mounted VR devices and handheld VR devices. For example, VR glasses and VR headsets. Users wear VR devices, and the VR devices display virtual reality images. Users obtain virtual reality perception by watching virtual reality images.

[0043] VR technology has a wide range of applications, including but not limited to: (1) Games: providing immersive gaming experiences, users can participate in games in a first-person perspective. (2) Three-dimensional design: for example, in car design, VR technology allows designers to build and modify car models in three-dimensional space, and to display and evaluate them. (3) In-vehicle field: VR technology can provide entertainment for passengers during travel. Passengers can experience various games and virtual world scenarios by wearing VR devices.

[0044] The virtual reality display system in this application is mainly applied to the scene of taking a vehicle. Users wear virtual reality devices when taking a vehicle and obtain virtual reality experience during vehicle travel.

[0045] Image Signal Processor (ISP): a specialized digital signal processor responsible for converting raw image data captured by a camera into high-quality digital images. This process includes a series of image processing steps to improve image quality and prepare for display or storage. ISP is usually integrated in the camera module or used as a separate chip with the camera sensor. For example, in smartphones, digital cameras, surveillance cameras and other image capture devices, image signal processors affect the quality of the final image.

[0046] The vehicle-mounted camera assembly in this application integrates an ISP. After the vehicle-mounted camera assembly captures road condition images, the ISP preprocesses the road condition images to improve their quality, so that the processed road condition images can meet the data requirements for image recognition and analysis.

[0047] Virtual Reality (VR) technology involves the use of computer graphics, sensor technology, and human-computer interaction interfaces to build and simulate three-dimensional virtual environments. VR technology uses a series of highly specialized devices, including head-mounted displays (HMDs), handheld controllers, and body motion tracking systems, to allow users to have a deep sense of immersion in virtual environments. Users interact with the virtual world through these devices, achieving a comprehensive sensory experience.

[0048] In a vehicle riding scenario, the application of a vehicle-mounted virtual reality system provides an innovative way of entertainment and information receiving for passengers. During the driving process of the vehicle, passengers can watch multimedia content, participate in game activities, or experience simulated travel by wearing a virtual reality device (i.e., a head-mounted display). Meanwhile, since the virtual reality device provides virtual reality pictures to the passengers, the display of the virtual reality pictures will not interfere with the driver, because the virtual display device is worn by the passengers. It is worth noting that the user wearing the virtual reality device can also be the driver, such as displaying the head-up display (HUD) information of the vehicle through the virtual reality device and synchronously displaying the image of the road condition. The subject wearing the virtual reality device is not limited in the embodiments of the present application.

[0049] However, the vehicle may encounter various road conditions during driving, including but not limited to road unevenness (including convex road surface, concave road surface, road barrier, deceleration, etc.), road turning, etc., which may cause the vehicle to appear to be jolted, affect the stability of the virtual reality device when loading the picture, reduce the real-time loading efficiency of the picture, cause the picture display to be mismatched with the head movement of the user wearing the device, and delay the compensation data during picture loading.

[0050] The picture loading delay phenomenon will interfere with the immersive experience of the user, reduce the fluency of the virtual reality picture, and cause the user to feel dizzy. Therefore, how to ensure that the virtual reality device provides stable and smooth pictures and experience during the driving process of the vehicle is a problem to be solved.

[0051] The present application provides a virtual reality display system, which comprises a vehicle-mounted terminal, a vehicle-mounted camera assembly, and a virtual reality device, and can perform longitudinal delay compensation on the display picture of the virtual reality device when the vehicle jolts up and down when driving through a road obstacle. Since the vehicle jolts when passing through the obstacle, the user's head will move, and the virtual reality display system can load each element in the virtual reality picture to a specified position in advance, so that the picture elements displayed in the virtual reality device are matched with the user's head movement, the influence of jolting on the stability of the picture is reduced, and the dizziness of the user when using the virtual reality device is reduced.

[0052] Illustratively, please refer to FIG. 1, which is a schematic diagram of a virtual reality display system provided by an example embodiment of the present application. The virtual reality display system 100 comprises a virtual reality device 110, a vehicle-mounted camera assembly 120, and a vehicle-mounted terminal 130.

[0053] The virtual reality display system 100 is configured to provide virtual reality services for the first vehicle, and the vehicle-mounted terminal 130 is an internal terminal of the first vehicle. The vehicle-mounted camera assembly 120 is arranged on the first vehicle, for example, is mounted on the outside of the first vehicle, or is arranged inside the first vehicle. The vehicle-mounted camera assembly 120 is configured to collect images of the environment in which the first vehicle is located, for example, collects road condition images of the road on which the first vehicle is located. The virtual reality device 110 is configured to display virtual reality images to passengers in the first vehicle. The virtual reality device 110, the vehicle-mounted camera assembly 120, and the vehicle-mounted terminal 130 are in communication connection.

[0054] During driving of the first vehicle, the vehicle-mounted camera assembly 120 is configured to collect road condition images, which are used to indicate the road conditions of the road on which the first vehicle is located, and to send the collected road condition images to the vehicle-mounted terminal 130.

[0055] For example, the vehicle-mounted camera assembly 120 is mounted at the windshield of the first vehicle and is configured to collect road conditions of the road in front of the first vehicle. During driving of the first vehicle, the vehicle-mounted camera assembly 120 collects road condition images in real time at a preset frequency and sends the collected road condition images to the vehicle-mounted terminal 130. For example, a first image is collected at a first time and is sent to the vehicle-mounted terminal 130. A second image is collected at a second time after the first time and is sent to the vehicle-mounted terminal 130. This process is repeated.

[0056] The vehicle-mounted terminal 130 is configured to identify the received road condition images and obtain an identification result, which is used to indicate the type of obstacle contained in the road condition images.

[0057] The identification process is used to determine whether the road condition images contain obstacles, and if the road condition images contain obstacles, the type of the obstacles is determined. The vehicle-mounted terminal 130 is internally arranged with a pre-trained target detection model. The target detection model is capable of analyzing and identifying the road condition images, determining whether the images contain obstacles, and determining the type of the obstacles when the road condition images contain obstacles, and generating an identification result. The identification result contains the type of the obstacles and the position information of the obstacles in the road condition images. The target detection model is a pre-trained machine learning model.

[0058] The vehicle-mounted terminal 130 analyzes the identification result and obtains loading delay compensation data. The loading delay compensation data is used to indicate the time at which the virtual reality device 110 displays at least one element in the virtual reality images, and the at least one element is used to constitute a virtual scene displayed by the virtual reality images and provide virtual reality perception to a user.

[0059] The vehicle-mounted terminal 130 can calculate the distance between the obstacle and the first vehicle according to the identification result, such as the straight-line distance between the obstacle and the first vehicle, determine the time when the first vehicle passes the obstacle based on the distance and the driving speed of the first vehicle, and further obtain the loading time delay compensation data, which is used to instruct the virtual reality device 110 to load at least one element in advance.

[0060] Since the virtual reality device 110 needs a certain time to perform steps such as calculation and rendering of the picture when loading the virtual reality picture, loading at least one element in advance can make the display time of at least one element in the virtual reality picture match the head state of the user when the first vehicle passes the obstacle.

[0061] For example, the first vehicle will bounce up and down when passing the concave road surface, causing the user wearing the virtual reality device 110 to move his head, and the position of the head changes in the longitudinal direction. The virtual reality device 110 mainly relies on monitoring the user's head movement to load the corresponding elements when presenting the virtual reality picture to the user, so that the picture and elements matched with the turning action are displayed when the user's head turns. Therefore, the loading time delay compensation data refers to the time when the virtual reality device 110 loads the corresponding elements in the virtual reality picture.

[0062] The vehicle-mounted terminal 130 sends the loading time delay compensation data and the identification result to the virtual reality device 110.

[0063] The virtual reality device 110 is configured to receive the loading time delay compensation data and the identification result, obtain loading displacement data based on the identification result, wherein the loading displacement data is used to indicate the display position of at least one element in the virtual reality picture; and display the virtual reality picture based on the loading time delay compensation data and the loading displacement data.

[0064] The displacement data is used to indicate the way in which the virtual reality device 110 adjusts the position of at least one element in the picture, so that when the first vehicle passes the obstacle, even if the user's head is affected to move, the position of the user's head changes in the longitudinal direction, the virtual reality device 110 can display at least one element at the appropriate position by matching the user's head movement, reduce the impact of the bouncing process on the displayed picture, and keep the picture stable for the user to watch.

[0065] Since different types of obstacles have different effects on the position of the first vehicle in the longitudinal direction, it is necessary to determine the loading displacement data that conforms to the actual situation according to the identification result.

[0066] The vehicle-mounted terminal 130 displays the virtual reality picture based on the loading time delay compensation data and the loading displacement data, wherein the position of at least one element in the virtual reality picture is determined according to the loading displacement data.

[0067] For example, the first vehicle passes through the obstacle and jounces at the third time, the loading time delay compensation data indicates that the virtual reality device 110 starts to load at least one element at the fourth time before the third time, and the loading displacement data is used to indicate that the virtual reality device 110 moves the position of the at least one element in the virtual reality picture to the first position. When the first vehicle passes through the obstacle, the first position of the at least one element in the virtual reality picture is loaded, and the user wearing the virtual reality device 110 is displayed.

[0068] In combination with the above-mentioned name introduction and application scenarios, the virtual reality display method provided by the present application is described. The virtual reality display method is executed by a virtual reality display system as an example. The virtual reality display system includes a vehicle-mounted camera assembly 210, a vehicle-mounted terminal 220, and a virtual reality device 230. A communication connection is established between the vehicle-mounted camera assembly 210, the vehicle-mounted terminal 220, and the virtual reality device 230. As shown in FIG. 2, FIG. 2 is a flowchart of the virtual reality display method provided by an exemplary embodiment of the present application. The method includes the following steps.

[0069] Step 211, the vehicle-mounted camera assembly collects road condition images.

[0070] The road condition image is used to indicate the road condition of the road where the first vehicle is located.

[0071] Optionally, the number of vehicle-mounted camera assemblies is at least one, and the vehicle-mounted camera assemblies are installed on the first vehicle, such as being installed on the outside or inside of the first vehicle.

[0072] For example, the first vehicle is installed with a first vehicle-mounted camera assembly at the windshield, which is used to collect road condition images of the road in front of the first vehicle. When the first vehicle is driving forward, it can be determined through the road condition images whether the front of the first vehicle is safe, thereby assisting the driver in controlling the first vehicle.

[0073] For example, the first vehicle is installed with a second vehicle-mounted camera assembly at the trunk or rear windshield, which is used to collect road condition images of the road behind the first vehicle. When the first vehicle needs to reverse, it can be determined through the road condition images whether the rear of the first vehicle is safe.

[0074] For example, the first vehicle is installed with vehicle-mounted camera assemblies on the left and right side doors, which are used to collect road condition images of the roads on the left and right sides of the first vehicle. When the first vehicle needs to turn, it can be determined through the road condition images whether the two sides of the first vehicle are safe.

[0075] The resolution of the vehicle-mounted camera assembly meets the preset resolution requirement, which can improve the quality of the road condition images and make the images clearer.

[0076] It is worth noting that the vehicle-mounted camera assembly described above can also be implemented as other sensor assemblies, such as a laser radar assembly, which identifies the road conditions of the road where the first vehicle is located through laser radar signals.

[0077] In some embodiments, the vehicle-mounted camera assembly collects road condition images in real time based on a preset frequency, and sends the road condition images to the vehicle-mounted terminal based on the preset frequency.

[0078] That is, the vehicle-mounted camera assembly collects a road condition image every preset time interval during the driving of the first vehicle, and immediately sends it to the vehicle-mounted terminal after collection.

[0079] For example, the preset frequency is 3 seconds / time, that is, a road condition image is collected every 3 seconds.

[0080] The first time is 10:00:00, a road condition image is collected and sent to the vehicle-mounted terminal; the second time is 10:00:03, a road condition image is collected and sent to the vehicle-mounted terminal; the third time is 10:00:06, a road condition image is collected and sent to the vehicle-mounted terminal, and so on.

[0081] In some embodiments, the preset frequency is related to the speed of the first vehicle. Optionally, the preset frequency is positively correlated with the speed of the first vehicle, that is, the higher the speed of the first vehicle, the higher the frequency of collecting road condition images. For example, when the speed of the first vehicle is less than 80 km / h, the collection frequency of road condition images is 3 seconds / time, and when the speed of the first vehicle is greater than 80 km / h, the collection frequency of road condition images is 1.5 seconds / time.

[0082] Collecting road condition images in real time can more accurately identify and locate possible obstacles in the road, reducing the probability of missed detection and false detection. When there are obstacles, the distance between the obstacles and the first vehicle is reflected according to the time lapse, so as to determine the possible impact of the obstacles on the picture display of the virtual reality device, and enhance the perception of the environment during the driving of the first vehicle.

[0083] Step 212, the vehicle-mounted camera assembly sends the road condition image to the vehicle-mounted terminal.

[0084] Optionally, in order to improve the image quality, after collecting the road condition image, the vehicle-mounted camera assembly pre-processes the road condition image to obtain a pre-processed road condition image, and sends the pre-processed road condition image to the vehicle-mounted terminal. The pre-processed road condition image meets the preset image processing format requirements.

[0085] For example, the vehicle-mounted camera assembly is integrated with an ISP, which pre-processes the road condition image through the ISP after the vehicle-mounted camera assembly collects the road condition image, thereby improving the quality of the road condition image.

[0086] The ISP preprocessing of the road condition image includes but is not limited to the following.

[0087] 1. Lens shading correction: Due to the optical characteristics of the lens, the edge brightness of the road condition image may not be uniform. This step compensates for the brightness of the edge of the road condition image, making the overall brightness of the road condition image more uniform.

[0088] 2. Bad pixel correction: Detect and repair bad pixels such as dead pixels, bright pixels or drifting pixels in the image to ensure image quality.

[0089] 3. Color interpolation: Estimate missing color information through algorithms to reconstruct a full-color image.

[0090] 4. Noise reduction: Remove random noise and fixed pattern noise in the image, such as noise caused by sensor readout or circuit problems, to improve the signal-to-noise ratio of the image.

[0091] 5. White balance: Correct the color temperature of the image so that white objects appear correctly under different lighting conditions.

[0092] 6. Sharpening: Enhance the edges and details of the image to improve image clarity.

[0093] 7. Auto focus: Ensure that the objects captured by the vehicle-mounted camera assembly at different distances are clear.

[0094] By performing the preprocessing steps through the ISP, the vehicle-mounted camera assembly can provide clearer, more accurate and more reliable road condition images, avoiding the problem of low accuracy of the recognition result caused by recognizing blurred road condition images in subsequent steps.

[0095] Step 221, the vehicle-mounted terminal identifies the received road condition image to obtain a recognition result.

[0096] The recognition result is used to indicate the type of obstacle contained in the road condition image.

[0097] Optionally, when the vehicle-mounted camera assembly sends the preprocessed road condition image, the vehicle-mounted terminal receives the preprocessed road condition image. The preprocessed road condition image is identified to obtain a recognition result.

[0098] The identification process of the vehicle-mounted terminal is used to determine whether the road condition image contains an obstacle, and if it contains an obstacle, the type of the obstacle is analyzed.

[0099] Optionally, the vehicle-mounted terminal is internally deployed with a pre-trained target detection model, and the vehicle-mounted terminal performs target detection on the road condition image through the target detection model to obtain a target detection result, the target detection result being used to indicate a case where an obstacle exists in the road condition image.

[0100] For example, the target detection model is a pre-trained deep neural network model, and the architecture of the deep neural network model can be one of the following model architectures.

[0101] (1) YOLO (You Only Look Once) series model architecture: YOLO is a popular single-stage target detection algorithm that converts the target detection task into a regression problem, and can predict the target position and category in the image through a single forward propagation. YOLO algorithm can process images in real time and detect targets, and directly predicts the boundary box and category probability from the image without complex region proposal or subsequent processing. YOLO series can detect multiple objects in the image, including overlapping objects. YOLO series includes YOLO_v1 to YOLO_v5 and other versions.

[0102] (2) SSD (Single Shot multibox Detector) series model architecture: SSD is also a single-stage target detection algorithm, which detects objects on feature maps of different scales and can detect objects of different sizes. By detecting on feature maps of different levels, objects of different sizes can be effectively detected. Multiple default boxes are predicted for each position, increasing the diversity of detection.

[0103] (3) FasterR-CNN (Fast Region-based Convolutional Network) series model architecture: FasterR-CNN is a two-stage target detection algorithm that first generates candidate regions through a region proposal network and then classifies and regresses the boundary box of these regions. It can generate high-quality candidate regions to provide a basis for subsequent detection, and the extracted feature maps are used for region proposal and subsequent classification regression, improving efficiency and allowing detection of objects in any direction.

[0104] (4) ResNet (Residua Network) series model architecture: ResNet is a deep convolutional neural network mainly used for image classification tasks. It solves the problem of training difficulty in deep networks by introducing residual learning. It learns residual functions to alleviate the problem of gradient vanishing, allowing the network to be deeper. The shortcut connection of identity mapping is introduced, allowing the gradient to propagate directly through the layer.

[0105] Exemplarily, the target detection model is pre-trained in the application by taking YOLO_v5 as a model architecture.

[0106] In the training stage, the target detection model is trained by using the data set 300 as shown in FIG. 3, so that the target detection model can accurately identify obstacles such as concave-convex road surface and roadblock.

[0107] The data set 300 contains various types of pictures, each of which contains a type of obstacle, such as concave road surface, concave-convex road surface, gravel obstacle, speed bump roadblock, and stone roadblock.

[0108] In the case that the target detection result indicates that there is an obstacle in the road condition image, the vehicle terminal obtains an identification result based on the target detection result.

[0109] Exemplarily, the target detection result output by the target detection model is taken as the identification result.

[0110] The identification result contains the first position of the obstacle in the road condition image and the type of the obstacle corresponding to the obstacle.

[0111] Exemplarily, referring to Table 1, different types of obstacles are marked in Table 1.

[0112] Table 1

[0113] Step 222, the vehicle terminal analyzes the identification result to obtain loading delay compensation data.

[0114] The loading delay compensation data is used to indicate the time at which the virtual reality device displays at least one element in the virtual reality picture, and the at least one element is used to constitute a virtual scene displayed by the virtual reality picture and provide virtual reality perception to the user.

[0115] The loading delay compensation data contains a first time length required to load the at least one element to a specified position in the virtual reality picture in advance, and the loading time of the at least one element is determined based on the arrival time of the first vehicle at the obstacle and the first time length. The at least one element is loaded to the specified position in the virtual reality picture at the loading time. The loading time is a time before the arrival time, and the time length between the loading time and the arrival time is the first time length.

[0116] Since the first vehicle will displace in the longitudinal direction when passing the obstacle, the head of the user in the vehicle will also displace correspondingly, and the movement of the head of the user wearing the virtual reality device will cause the virtual reality picture to change correspondingly (that is, the position of at least one element in the virtual reality picture changes), and the virtual reality device needs a certain time to load the virtual reality picture and the at least one element, therefore, in the related art, the movement track (the turning of the head) of the user's head is usually predicted, and the at least one element is loaded into the corresponding position in the virtual reality picture in advance, so as to eliminate the time delay in the loading process of the element, and the user can immediately watch the matching picture after the movement of the head.

[0117] The loading time delay compensation data in the embodiment is data determined after predicting the picture matched with the movement of the head of the user when the first vehicle passes the obstacle, and the loading time delay compensation data can instruct the virtual reality device to load the at least one element into the specified position in the virtual reality picture in advance, so that the user can watch the picture matched with the movement of the head when the vehicle passes the obstacle and bounces.

[0118] When the virtual reality device provides the user with the perception of the virtual world and displays the virtual reality picture, it is realized by building a virtual scene (that is, a VR scene) and adding different three-dimensional models of elements in the virtual scene.

[0119] All elements in the virtual scene are built by using a six-degree-of-freedom model, which is a model used to describe the movement of an object in a three-dimensional space, which includes three translational movements and three rotational movements of the object in space, that is, the three-dimensional models of all elements in the virtual scene are built by using the six-degree-of-freedom model, so that the elements have similar or the same appearance as the three-dimensional entities, and the elements can move freely in the virtual scene.

[0120] Since the first vehicle will displace in the longitudinal direction (the direction perpendicular to the ground / Z-axis direction) when passing the obstacle, for example, move upward or downward, the user riding the first vehicle will also produce head movement, in order to eliminate the influence of the user's head movement on the display of the virtual reality picture by the virtual reality device, it is necessary to make delay compensation for the Z-axis direction and γ (rotation around Y-axis, called yaw, describing the angle of left and right movement of the user's head), so that the virtual reality device can match the picture seen by the user with the movement of the user's head when the first vehicle passes the obstacle and moves longitudinally.

[0121] Optionally, the vehicle-mounted terminal determines a first distance between the obstacle and the first vehicle based on the first position (the position of the obstacle in the road condition image) and the position of the first vehicle. The loading time delay compensation data is determined based on the first distance and the vehicle speed of the first vehicle.

[0122] The vehicle-mounted camera assembly can reflect the distance between the vehicle-mounted camera assembly and the obstacle based on the pixel proportion of the obstacle in the road condition image and the perspective change principle when collecting the road condition image, and determine the first distance between the first vehicle and the obstacle. In some embodiments, the distance between the vehicle-mounted camera assembly and the obstacle is determined as the first distance between the first vehicle and the obstacle.

[0123] The vehicle speed change of the first vehicle is obtained in real time, and the time required for the first vehicle to travel the first distance is predicted. For example, when the first vehicle travels at a first constant speed, the time is directly determined by dividing the first distance by the first speed. For example, when the speed of the first vehicle changes in real time, the time is determined according to the average speed of the first vehicle within a fixed distance.

[0124] In some embodiments, the road condition image is collected in real time and transmitted to the vehicle-mounted terminal by the vehicle-mounted camera assembly. The vehicle-mounted terminal updates the first distance based on the real-time received road condition image to obtain the updated first distance.

[0125] That is, each time a road condition image is received, the newly received road condition image needs to be analyzed to update the distance between the first vehicle and the obstacle.

[0126] The vehicle-mounted terminal obtains first delay data based on the updated first distance and the vehicle speed of the first vehicle. The first delay data is the time required for the first vehicle to travel to the obstacle.

[0127] A first unit time length is obtained, which is the time consumed by the vehicle-mounted camera to collect a road condition image and send it to the vehicle-mounted terminal. The loading delay compensation data is determined based on the first delay data and the first unit time length.

[0128] In some embodiments, the virtual reality device also needs a certain calculation time to load elements into the virtual reality picture. A second time length is obtained, which is the time for the virtual reality device to move the elements in the virtual reality picture. The loading delay compensation data is determined based on the first delay data, the first unit time length and the second time length.

[0129] For example, when the first distance between the first vehicle and the obstacle is less than a preset distance threshold (e.g., 50 meters), the first delay data required for the first vehicle to reach the obstacle is determined. The delay compensation number (in milliseconds) is determined based on the first delay data, the first unit time length and the second time length.

[0130] Please refer to Table 2, which is a delay compensation number table.

[0131] Table 2

[0132] When the road condition image reflects that there are multiple obstacles, the delay compensation number corresponding to each obstacle is determined based on the above Table 2, and the loading time delay compensation data is determined based on the delay compensation number, the first delay data, the first unit time length, and the second time length.

[0133] For example, the road condition image indicates that there are two obstacles in the first vehicle driving direction, which are obstacle A and obstacle B. The road condition image is collected in real time and the distances between the first vehicle and obstacle A and obstacle B are calculated. Among them, obstacle A is closer to the first vehicle.

[0134] When the first distance between the first vehicle and obstacle A is 50 meters, the distance between the first vehicle and obstacle B exceeds 50 meters, and the delay compensation number corresponding to obstacle A is 3 and the delay compensation number corresponding to obstacle B is 4 based on the delay compensation number table.

[0135] Among them, the first delay data of the first vehicle to the obstacle includes delay data A and delay data B, delay data A refers to the time length required for the first vehicle to reach obstacle A, and delay data B refers to the time length required for the first vehicle to reach obstacle B.

[0136] The sum of delay data A, the first unit time length and the second time length is 4 seconds, and the current time is the first time, so the loading time delay compensation data sent to the virtual reality device at the current time indicates that at least one element is loaded to the first specified position starting from the time point after the current time experiences 4 seconds and 3 milliseconds.

[0137] The sum of delay data B, the first unit time length and the second time length is 6 seconds, and the current time is the first time, so the loading time delay compensation data sent to the virtual reality device at the current time also indicates that at least one element is loaded to the second specified position starting from the time point after the current time experiences 6 seconds and 4 milliseconds.

[0138] Among them, the first specified position is determined based on the type of obstacle A, and the second specified position is determined based on the type of obstacle B.

[0139] In this process, because the road condition image is collected in real time, the distances between obstacle A and obstacle B and the first vehicle will also be shortened, and the delay data A and delay data B corresponding to the first vehicle during driving will also change, therefore, the loading time delay compensation data will also be updated, and the virtual reality device will be sent once for each update of the loading time delay compensation data until the first vehicle stops sending when it reaches the obstacle.

[0140] That is, when the first vehicle reaches the obstacle A, the virtual reality device has loaded at least one element to the first designated position based on the loading time delay compensation data received at the latest time at the current time, while the first vehicle has not reached the obstacle B at this time, and the loading time delay compensation data contains data corresponding to the obstacle B at this time. During the process of the first vehicle driving from the obstacle A to the obstacle B, the loading time delay compensation data is updated in real time based on the road condition image, and the updated loading time delay compensation data is sent to the virtual reality device until the first vehicle drives to the obstacle B. At this time, the virtual reality device has loaded at least one element to the second designated position based on the loading time delay compensation data received at the latest time at the current time.

[0141] It is worth noting that the time delay compensation number table shown in Table 2 is a preset table, and the data in the table is determined based on a large amount of data after testing. In some embodiments, the time delay compensation number can also be determined in other ways, for example, based on a preset function mapping expression to represent the relationship between the time delay compensation number and the distance of the first vehicle reaching the obstacle, and the time delay compensation number is calculated based on the preset function mapping expression after collecting the distance of the first vehicle reaching the obstacle. This embodiment is not limited thereto.

[0142] It is worth noting that the loading time delay compensation data can be determined based on the sum of the time delay compensation number, the first unit time length and the second time length, or the time delay compensation number, the first unit time length and the second time length can be weighted and operated to determine the loading time delay compensation data.

[0143] Step 223, the vehicle terminal sends the loading time delay compensation data and the recognition result to the virtual reality device.

[0144] When the vehicle terminal sends the recognition result, the road condition image corresponding to the recognition result is also sent to the virtual reality device.

[0145] Step 231, the virtual reality device receives the loading time delay compensation data and the recognition result.

[0146] The virtual reality device also receives the road condition image corresponding to the recognition result.

[0147] Step 232, the virtual reality device obtains the loading displacement data based on the recognition result.

[0148] The loading displacement data is used to indicate the display position of at least one element in the virtual reality picture. That is, the loading displacement data is used to indicate the distance of moving at least one element up or down in the virtual reality picture.

[0149] The loading displacement data includes a distance and a direction required for loading the at least one element to a specified position (display position) in the virtual reality picture, and the loading displacement data is determined based on the first vehicle about to pass the obstacle type.

[0150] When the first vehicle passes the obstacle, the user's head in the vehicle also moves correspondingly. The movement of the user's head wearing the virtual reality device causes the virtual reality picture to change (i.e., the position of the at least one element in the virtual reality picture changes). The virtual reality device needs a certain time to load the virtual reality picture and the at least one element. Therefore, in the related art, the movement track (head turning) of the user's head is usually predicted, and the at least one element is loaded to the corresponding position in the virtual reality picture in advance to eliminate the time delay in the loading process, so that the user can immediately watch the matching picture after the user's head moves.

[0151] The loading displacement data in the embodiment is determined after predicting the picture matched with the movement of the user's head when the first vehicle passes the obstacle. The loading time delay compensation data can instruct the virtual reality device to load the at least one element to the specified position in the virtual reality picture in advance, so that the picture watched by the user matches the movement of the user's head when the vehicle passes the obstacle and bounces.

[0152] Optionally, the virtual reality device determines a displacement type based on the obstacle type in the recognition result, and the displacement type is used to indicate a direction for adjusting the display position of the at least one element in the virtual reality picture.

[0153] The loading displacement reference table and the pixel ratio data of the obstacle in the road condition image are obtained, and the loading displacement data is determined from the loading displacement reference table based on the pixel ratio data and the displacement type.

[0154] For illustration, please refer to Table 3, which is a loading displacement reference table.

[0155] Table 3

[0156] In the loading displacement reference table, the pixel ratio refers to the pixel ratio data of the obstacle in the road condition image collected at the moment when the distance between the first vehicle and the obstacle is a specified distance, i.e., the proportion of the pixels of the obstacle in the total pixels of the road condition image.

[0157] Since the size of the speed bump and the step in the road is fixed, it is not necessary to analyze the pixel ratio data when the obstacle is a speed bump or a step.

[0158] The speed bump in the urban road appears as a yellow and black stripe, and the speed bump in the highway appears as a multi-point obstacle.

[0159] The obstacle types include the following: stones (No. 1), pits (No. 2), speed bumps (No. 3), and steps (No. 4). Among them, the stones, speed bumps, and steps are implemented as convex obstacles on the road surface, and the pits are implemented as obstacles based on the concave road surface. Therefore, a positive offset is set for the stones, speed bumps, and steps, and a negative offset is set for the pits.

[0160] In some embodiments, when identifying the obstacle type, the concave-convex state and the concave-convex degree of the obstacle are identified, and the offset is determined in real time based on the concave-convex state and the concave-convex degree. Illustratively, a convex obstacle of 0-5% corresponds to an offset of 2, a concave obstacle of -5%-0 corresponds to an offset of -2; a convex obstacle of 5%-10% corresponds to an offset of 4, and a concave obstacle of (-10%)-(-5%) corresponds to an offset of -4.

[0161] Among them, when the obstacle type belongs to any one of the stones, speed bumps, and steps, the displacement type is determined to be positive, that is, the displacement data loaded is a positive number, and the position of the at least one element in the virtual reality picture is moved up to the display position. When the obstacle type belongs to the pit, the displacement type is determined to be negative, that is, the displacement data loaded is a negative number, and the position of the at least one element in the virtual reality picture is moved down to the display position.

[0162] The offset refers to the unit distance of movement in the loaded displacement data. For example, when the offset is 2, it means that the at least one element is moved up by 2 unit distances. The unit distance refers to the unit distance in the virtual reality picture, for example, 10 centimeters in the virtual reality picture is 1 unit distance.

[0163] Among them, the loaded displacement table shown in Table 3 is a preset table, and the data in the table is determined based on a large amount of data after testing. In some embodiments, the offset under different conditions can also be determined according to other ways.

[0164] When the obstacle is a stone, if the pixel ratio data of the stone is less than 5%, the loaded displacement data is to move up by 2 unit distances; if the pixel ratio data of the stone is 5%-10%, the loaded displacement data is to move up by 4 unit distances; and if the pixel ratio data of the stone is greater than 10%, the loaded displacement data is to move up by 6 unit distances.

[0165] When the obstacle is a pit, if the pixel ratio data of the pit is less than 10%, the loaded displacement data is to move down by 2 unit distances; if the pixel ratio data of the pit is 10%-15%, the loaded displacement data is to move down by 4 unit distances; and if the pixel ratio data of the pit is greater than 15%, the loaded displacement data is to move down by 6 unit distances.

[0166] When the obstacle is a yellow-black speed bump, the displacement data loaded is 2 unit distances up. When the obstacle is a multi-point speed bump, the displacement data loaded is 1 unit distance up. When the obstacle is a step, the displacement data loaded is 6 unit distances up.

[0167] At step 233, the virtual reality picture is displayed based on the loading time delay compensation data and the loading displacement data.

[0168] The position of the at least one element in the virtual reality picture is determined according to the loading displacement data.

[0169] For example, the first vehicle will pass the obstacle and experience a bump at time 10:00:10. The loading time delay compensation data is 2 seconds, indicating that the virtual reality device starts loading the at least one element at a time 2 seconds before reaching the obstacle, i.e., at time 10:00:08.

[0170] Loading the at least one element to the specified position in the virtual reality picture in advance can make the virtual reality picture viewed by the user remain stable when the first vehicle experiences longitudinal displacement when passing the obstacle.

[0171] The loading displacement data is used to indicate that the virtual reality device moves the position of the at least one element in the virtual reality picture to the first position, so that when the first vehicle passes the obstacle, the at least one element is loaded at the first position in the virtual reality picture, and is exhibited to the user wearing the virtual reality device.

[0172] For example, when the obstacle type is a step, a stone, a speed bump, or other objects with a certain height, the first vehicle experiences longitudinal displacement when passing the obstacle, and the direction is upward. At this time, the head of the user in the vehicle also experiences longitudinal displacement due to inertia, and the direction is upward. In order to make the virtual reality picture viewed by the user match the head movement of the user, the at least one element is loaded to the specified position in the virtual reality picture based on the time indicated by the loading time delay compensation data, wherein the at least one element is moved upward in the virtual reality picture to reach the specified position, and the moving distance is indicated by the loading displacement data.

[0173] When the first vehicle passes the obstacle, the at least one element is loaded, and the picture viewed by the user remains stable before and after the first vehicle passes the obstacle.

[0174] For example, when the obstacle type indicates that the first vehicle is about to drive through a road section with a depressed road surface or the like that is lower than the road surface level, it indicates that there is a displacement in the longitudinal direction when the first vehicle passes through the obstacle, and the direction is downward. At this time, the user's head also has a displacement in the longitudinal direction due to inertia, and the direction is downward. In order to make the virtual reality image seen by the user match the user's head movement, at least one element is loaded into a specified position in the virtual reality image based on the time indicated by the loading delay compensation data, wherein the at least one element moves downward in the virtual reality image to reach the specified position, and the moving distance is the distance indicated by the loading displacement data.

[0175] When the first vehicle passes through the obstacle, the loading of the at least one element is completed, and the image viewed by the user remains stable before and after the first vehicle passes through the obstacle.

[0176] In summary, the virtual reality display method and system provided by the present application, when the virtual reality device is applied in a vehicle riding scene, acquires the road conditions of the road where the vehicle is located by collecting images in real time, judges whether the vehicle will experience bumps during driving according to whether there are obstacles in the road conditions and the type of the obstacles, compensates for the delay of the virtual reality device, and alleviates the problem of delay of the virtual reality image display caused by the user's head movement. By analyzing the road condition image, obtaining loading displacement data and loading delay compensation data, and determining the way in which the virtual reality device displays elements in the virtual reality image when the vehicle passes through the obstacle, the elements in the image are loaded to the target position in advance, which reduces the dizziness and delay problems of the user when wearing the virtual reality device to view the image, and improves the user's experience.

[0177] FIG. 4 is a flowchart of a virtual reality display method provided by another exemplary embodiment of the present application, which is executed by a vehicle-mounted terminal of a first vehicle, and includes the following steps.

[0178] Step 410: Receive a road condition image.

[0179] The road condition image is an image collected by a vehicle-mounted camera assembly of the first vehicle, and the road condition image is used to indicate the road conditions of the road where the first vehicle is located.

[0180] Optionally, the number of vehicle-mounted camera assemblies is at least one, and the vehicle-mounted camera assemblies are installed outside the first vehicle. The vehicle-mounted camera assemblies collect road condition images in real time based on a preset frequency, and after preprocessing the collected road condition images, the vehicle-mounted camera assemblies send the preprocessed road condition images to the vehicle-mounted terminal based on the preset frequency.

[0181] The vehicle-mounted terminal receives the preprocessed road condition images based on the preset frequency, and can obtain road condition images with a clarity and image quality that meet preset requirements.

[0182] Step 420: Identify the received road condition image to obtain an identification result.

[0183] The identification results are used to indicate the types of obstacles contained in the road condition image. The identification process of the vehicle-mounted terminal is used to determine whether the road condition image contains obstacles, and if so, to analyze the type of obstacles.

[0184] The identification results include the type of obstacle and its location in the road condition image.

[0185] Optionally, a pre-trained target detection model is deployed in the vehicle terminal. The vehicle terminal uses the target detection model to perform target detection on the road condition image and obtains the target detection result. The target detection result is used to indicate the presence of obstacles in the road condition image.

[0186] For example, the target detection results output by the target detection model include the following: (1) whether there is an obstacle; (2) when there is an obstacle, the type of obstacle; and (3) when there is an obstacle, the location of the obstacle in the road condition image.

[0187] The target detection results output by the target detection model are used as the recognition results.

[0188] Step 430: Analyze the recognition results to obtain loading delay compensation data.

[0189] Among them, the loading delay compensation data is used to indicate the time when the virtual reality device displays at least one element in the virtual reality screen. The at least one element is used to constitute the virtual scene displayed in the virtual reality screen and to provide the user with virtual reality perception.

[0190] Optionally, the vehicle terminal determines a first distance between the obstacle and the first vehicle based on the first location and the position of the first vehicle; and determines loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0191] For example, the first distance is updated based on the real-time received road condition images to obtain the updated first distance. First delay data is obtained based on the updated first distance and the vehicle speed of the first vehicle. The first unit time consumed by the vehicle-mounted camera in a single acquisition of road condition images and transmission to the vehicle-mounted terminal is obtained. Loading delay compensation data is determined based on the first delay data and the first unit time.

[0192] Step 440: Load the latency compensation data and recognition results to the virtual reality device.

[0193] After receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and loading displacement data. The loading displacement data is used to indicate the display position of at least one element in the virtual reality screen.

[0194] In summary, the virtual reality display method provided in the application can analyze whether there is an obstacle in the road according to the received road condition image and the type of the obstacle, determine whether bumping will occur during vehicle driving, and compensate for the delay of the virtual reality device, thereby alleviating the problem of virtual reality picture display delay caused by head movement of the user. By analyzing the road condition image, obtaining the loading delay compensation data, determining the way in which the virtual reality device displays elements in the virtual reality picture when the vehicle passes through the obstacle, and loading the elements in the picture to the target position in advance, the problem of dizziness and delay of the user when wearing the virtual reality device to watch the picture is reduced, and the user experience is improved.

[0195] FIG. 5 is a structural block diagram of a virtual reality display device provided in an example embodiment of the application. As shown in FIG. 5, the device includes the following parts.

[0196] The receiving module 510 is configured to receive a road condition image, the road condition image being an image collected by a vehicle-mounted camera assembly of a first vehicle, and the road condition image being used to indicate the road condition of a road where the first vehicle is located.

[0197] The identification module 520 is configured to identify the received road condition image to obtain an identification result, the identification result being used to indicate the type of an obstacle contained in the road condition image.

[0198] The analysis module 530 is configured to analyze the identification result to obtain loading delay compensation data.

[0199] The sending module 540 is configured to send the loading delay compensation data and the identification result to a virtual reality device, wherein the loading delay compensation data is used to indicate the time at which the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide virtual reality perception to a user; after receiving the identification result, the virtual reality device acquires loading displacement data based on the identification result, and displays the virtual reality picture based on the loading delay compensation data and the loading displacement data, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality picture.

[0200] In an optional embodiment, the identification module 520 is further configured to perform target detection on the road condition image by a target detection model to obtain a target detection result, the target detection result being used to indicate the presence of an obstacle in the road condition image; and the identification result is acquired based on the target detection result in the case where the target detection result indicates the presence of an obstacle in the road condition image; wherein the identification result contains a first position of the obstacle in the road condition image and an obstacle type corresponding to the obstacle.

[0201] In an optional embodiment, the analysis module 530 is further configured to determine a first distance between the obstacle and the first vehicle based on the first position and the position of the first vehicle, and determine the loading time compensation data based on the first distance and a vehicle speed of the first vehicle.

[0202] In an optional embodiment, the analysis module 530 is further configured to update the first distance based on the real-time received road condition image to obtain an updated first distance, obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle, obtain a first unit time duration consumed by the vehicle-mounted camera for collecting the road condition image and sending the road condition image to the vehicle-mounted terminal, and determine the loading time compensation data based on the first delay data and the first unit time duration.

[0203] In summary, the virtual reality display device provided by the application can determine whether the vehicle will vibrate during driving according to whether there is an obstacle in the road condition and the type of the obstacle, compensate for the delay of the virtual reality device, and alleviate the problem of delay of the virtual reality picture display caused by the head movement of the user. The road condition image is analyzed to obtain loading displacement data and loading time compensation data, the way in which the virtual reality device displays elements in the virtual reality picture when the vehicle passes through the obstacle is determined, the elements in the picture are loaded to the target position in advance, the problems of dizziness and delay of the user when wearing the virtual reality device to watch the picture are reduced, and the experience of the user is improved.

[0204] It should be noted that the virtual reality display device provided in the above embodiments is only exemplified by the division of the above functional modules. In actual applications, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above described functions. In addition, the virtual reality display device and the virtual reality display method provided in the above embodiments belong to the same concept, and the specific implementation process is described in detail in the method embodiments, which will not be repeated here.

[0205] FIG. 6 shows a structural block diagram of a computer device 600 according to an example embodiment of the present application. The computer device 600 can be a smartphone, a tablet computer, an MP3 player, an MP4 player, a notebook computer or a desktop computer. The computer device 600 can also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0206] Generally, the computer device 600 includes a processor 601 and a memory 602.

[0207] The processor 601 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array), a PLA (Programmable Logic Array). The processor 601 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also referred to as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 601 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed by a display screen. In some embodiments, the processor 601 can further include an AI (Artificial Intelligence) processor for processing machine learning related computing operations.

[0208] The memory 602 can include one or more computer-readable storage media, which can be non-transitory. The memory 602 can also include a high-speed random access memory, and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction for being executed by the processor 601 to implement a virtual reality display method provided by a method embodiment of the present application.

[0209] In some embodiments, the computer device 600 further includes some other components 603, and the type and number of the other components 603 can be selected based on the functional needs of the computer device 600. Those skilled in the art can understand that the structure shown in FIG. 6 does not constitute a limitation on the computer device 600, and can include more or fewer components than illustrated, or combine certain components, or adopt a different arrangement of components.

[0210] Optionally, the computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a solid state disk (SSD), an optical disk, etc. Among them, the random access memory can include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The above application embodiment serial number is only for description, not representing the pros and cons of the embodiments.

[0211] The embodiments of the present application further provide a computer device, which comprises a processor and a memory. The memory stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the virtual reality display method according to any one of the above embodiments of the present application.

[0212] The embodiments of the present application further provide a computer readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by a processor to implement the virtual reality display method according to any one of the above embodiments of the present application.

[0213] The embodiments of the present application further provide a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions, so that the computer device executes the virtual reality display method according to any one of the above embodiments.

Claims

1. A virtual reality display system, comprising a virtual reality device, a vehicle-mounted camera assembly, and a vehicle-mounted terminal; the vehicle-mounted camera assembly is configured to collect a road condition image, the road condition image being used to indicate a road condition of a road where a first vehicle is located, and send the road condition image to the vehicle-mounted terminal; The vehicle-mounted terminal is configured to identify the road condition image to obtain an identification result, the identification result being used to indicate a type of an obstacle contained in the road condition image; load time delay compensation data is obtained based on the identification result; and the loading time delay compensation data is used to indicate a time when the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user, and the loading time delay compensation data and the recognition result are sent to the virtual reality device; the virtual reality device is configured to acquire loading displacement data based on the recognition result, the loading displacement data being used to indicate a display position of the at least one element in the virtual reality picture, and display the virtual reality picture based on the loading time delay compensation data and the loading displacement data. 2.The system of claim 1, wherein the vehicle-mounted terminal is further configured to perform target detection on the road condition image by a target detection model to obtain a target detection result, the target detection result being used to indicate a case where an obstacle exists in the road condition image; in a case where the target detection result indicates that the obstacle exists in the road condition image, the recognition result is acquired based on the target detection result; the recognition result comprises a first position of the obstacle in the road condition image and an obstacle type corresponding to the obstacle. 3.The system of claim 1 or 2, wherein the vehicle-mounted terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first position and a position of the first vehicle, and determine the loading time delay compensation data based on the first distance and a vehicle speed of the first vehicle. 4.The system of any one of claims 1 to 3, wherein the vehicle-mounted camera assembly is further configured to collect the road condition image in real time based on a preset frequency, and send the road condition image to the vehicle-mounted terminal based on the preset frequency; the vehicle-mounted terminal is further configured to update the first distance based on the road condition image received in real time to obtain an updated first distance, and acquire first delay data based on the updated first distance and the vehicle speed of the first vehicle; a first unit time length consumed by the vehicle-mounted camera assembly for collecting the road condition image once is acquired and sent to the vehicle-mounted terminal, and the loading time delay compensation data is determined based on the first delay data and the first unit time length. 5.The system of claim 2, wherein the virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, the displacement type being used to indicate a direction of adjusting the display position of the at least one element in the virtual reality picture, and acquire a loading displacement reference table and pixel proportion data of the obstacle in the road condition image. The loading displacement data is determined from the loading displacement reference table based on the pixel proportion data and the displacement type.

6. The system of any one of claims 1 to 5, wherein, The vehicle-mounted camera assembly is further configured to pre-process the road condition image to obtain a pre-processed road condition image, the pre-processed road condition image conforming to a preset image processing format requirement. The vehicle-mounted terminal is further configured to receive the pre-processed road condition image, and identify the pre-processed road condition image to obtain the identification result.

7. A virtual reality display method, executed by a vehicle-mounted terminal, the method comprising: receiving a road condition image, the road condition image being an image collected by a vehicle-mounted camera assembly of a first vehicle, the road condition image being indicative of a road condition of a road on which the first vehicle is located; identifying the received road condition image to obtain an identification result, the identification result being indicative of a type of an obstacle contained in the road condition image; analyzing the identification result to obtain loading latency compensation data; sending the loading latency compensation data and the identification result to a virtual reality device; wherein the loading latency compensation data is indicative of a time at which the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user; after receiving the identification result, the virtual reality device acquires loading displacement data based on the identification result, and displays the virtual reality picture based on the loading latency compensation data and the loading displacement data, the loading displacement data being indicative of a display position of the at least one element in the virtual reality picture.

8. A virtual reality display apparatus, the apparatus comprising: a receiving module configured to receive a road condition image, the road condition image being an image collected by a vehicle-mounted camera assembly of a first vehicle, the road condition image being indicative of a road condition of a road on which the first vehicle is located; an identifying module configured to identify the received road condition image to obtain an identification result, the identification result being indicative of a type of an obstacle contained in the road condition image; an analyzing module configured to analyze the identification result to obtain loading latency compensation data; a sending module configured to send the loading latency compensation data and the identification result to a virtual reality device; wherein the loading latency compensation data is indicative of a time at which the virtual reality device displays at least one element in a virtual reality picture, the at least one element being used to constitute a virtual scene displayed by the virtual reality picture and provide a virtual reality perception to a user; after receiving the identification result, the virtual reality device acquires loading displacement data based on the identification result, and displays the virtual reality picture based on the loading latency compensation data and the loading displacement data, the loading displacement data being indicative of a display position of the at least one element in the virtual reality picture. ​ 9. A computer device comprising a processor and a memory, wherein at least one program is stored in the memory, and the at least one program is loaded and executed by the processor to implement the virtual reality display method according to claim 7.

10. A computer readable storage medium, wherein at least one program is stored in the storage medium, and the at least one program is loaded and executed by a processor to implement the virtual reality display method according to claim 7.

Images