Virtual reality display system, method, apparatus, equipment, medium

Abstract

This application discloses a virtual reality display system, method, apparatus, device, and medium, relating to the field of virtual reality technology. The system includes a virtual reality device, an in-vehicle camera assembly, and an in-vehicle terminal. The in-vehicle camera assembly is configured to acquire road condition images and send them to the in-vehicle terminal. The in-vehicle terminal is configured to recognize the received road condition images and obtain recognition results; analyze the recognition results to obtain loading delay compensation data; and send the loading delay compensation data and recognition results to the virtual reality device. The virtual reality device is configured to receive the loading delay compensation data and recognition results; obtain loading displacement data based on the recognition results; and display a virtual reality screen based on the loading delay compensation data and loading displacement data. This system can eliminate the impact of vehicle vibrations on elements within the loading screen of the virtual reality device, reducing loading delays.

Description

Technical Field

[0001] This application relates to the field of virtual reality technology, and in particular to a virtual reality display system, method, apparatus, device, and medium. Background Technology

[0002] Virtual Reality (VR) technology creates and simulates three-dimensional virtual worlds, allowing users to have an immersive experience in these virtual worlds through interactive devices such as head-mounted displays, controllers, and body tracking devices.

[0003] VR technology can be applied to in-vehicle scenarios, where in-vehicle VR products can provide passengers with a brand-new way to access entertainment and information. By wearing VR devices, passengers can watch multimedia content, experience games, or take virtual trips while the vehicle is in motion.

[0004] However, road conditions can change at any time during a vehicle's journey, and bumps are unavoidable. This can affect the stability of the VR device's display, causing delays in loading and impacting the user's experience. Summary of the Invention

[0005] This application provides a virtual reality display system, method, apparatus, device, and medium that can eliminate the impact of vehicle vibrations on elements within the loading screen of a virtual reality device, reducing loading delays. The technical solution is as follows:

[0006] On the one hand, a virtual reality display system is provided, which includes a virtual reality device, an in-vehicle camera assembly, and an in-vehicle terminal;

[0007] The vehicle-mounted camera assembly is configured to capture road condition images, which are used to indicate the road conditions of the road where the first vehicle is located; and to send the road condition images to the vehicle-mounted terminal.

[0008] The vehicle-mounted terminal is configured to recognize the received road condition image to obtain a recognition result, the recognition result being used to indicate the type of obstacle contained in the road condition image; analyze the recognition result to obtain loading delay compensation data; wherein, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element being used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; and send the loading delay compensation data and the recognition result to the virtual reality device;

[0009] The virtual reality device is configured to receive the loading delay compensation data and the recognition result; obtain loading displacement data based on the recognition result, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality screen; and display the virtual reality screen based on the loading delay compensation data and the loading displacement data.

[0010] In an optional embodiment, the vehicle terminal is further configured to perform target detection on the road condition image using a target detection model to obtain a target detection result, the target detection result being used to indicate the presence of an obstacle in the road condition image; and, if the target detection result indicates the presence of an obstacle in the road condition image, to obtain a recognition result based on the target detection result; wherein the recognition result includes a first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

[0011] In an optional embodiment, the vehicle terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first location and the position of the first vehicle; and to determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0012] In an optional embodiment, the vehicle-mounted camera component is further configured to acquire the road condition images in real time based on a preset frequency; and to send the road condition images to the vehicle-mounted terminal based on the preset frequency.

[0013] The vehicle-mounted terminal is further configured to update the first distance based on the real-time received road condition image to obtain an updated first distance; obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle; obtain a first unit time consumed by the vehicle-mounted camera in a single acquisition of the road condition image and transmission to the vehicle-mounted terminal; and determine the loading delay compensation data based on the first delay data and the first unit time.

[0014] In an optional embodiment, the virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, the displacement type being used to indicate the direction for adjusting the display position of the at least one element in the virtual reality screen; obtain a loading displacement lookup table and pixel percentage data of the obstacle in the road condition image; and determine the loading displacement data from the loading displacement lookup table based on the pixel percentage data and the displacement type.

[0015] In an optional embodiment, the vehicle-mounted camera component is further configured to preprocess the road condition image to obtain a preprocessed road condition image, the preprocessed road condition image conforming to a preset image processing format requirement; and to send the preprocessed road condition image to the vehicle-mounted terminal.

[0016] The vehicle-mounted terminal is also configured to receive the preprocessed road condition image; to identify the preprocessed road condition image and obtain the identification result.

[0017] On the other hand, a virtual reality display method is provided, the method comprising:

[0018] Receive road condition images, which are images captured by the onboard camera assembly of the first vehicle, and are used to indicate the road conditions of the road where the first vehicle is located;

[0019] The received road condition image is identified to obtain an identification result, which is used to indicate the type of obstacle contained in the road condition image;

[0020] The identification results are analyzed to obtain loading delay compensation data;

[0021] The loading delay compensation data and the recognition result are sent to the virtual reality device;

[0022] Wherein, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

[0023] On the other hand, a virtual reality display device is provided, the device comprising:

[0024] A receiving module is used to receive road condition images, which are images captured by the vehicle-mounted camera assembly of the first vehicle, and the road condition images are used to indicate the road conditions of the road where the first vehicle is located.

[0025] The identification module is used to identify the received road condition image and obtain an identification result, which is used to indicate the type of obstacle contained in the road condition image;

[0026] The analysis module is used to analyze the identification results to obtain loading delay compensation data;

[0027] A sending module is used to send the loading latency compensation data and the recognition result to a virtual reality device; wherein, the loading latency compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading latency compensation data and the loading displacement data, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality screen.

[0028] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, the at least one instruction, the at least one program, the code set or instruction set being loaded and executed by the processor to implement the virtual reality display method as described in any of the embodiments of this application above.

[0029] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set, or instruction set is stored therein, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the virtual reality display method as described in any of the embodiments of this application above.

[0030] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the virtual reality display methods described in the above embodiments.

[0031] The beneficial effects of the technical solutions provided in this application include at least the following:

[0032] When virtual reality (VR) devices are used in in-vehicle scenarios, they acquire real-time images to understand road conditions. Based on the presence and type of obstacles, the system determines whether the vehicle will experience bumps during travel. Delay compensation is then applied to the VR device to mitigate display latency issues caused by head movements. By analyzing road images and obtaining loading displacement and latency compensation data, the system determines how elements are displayed in the VR image when the vehicle passes obstacles. Elements are pre-loaded to their target positions, reducing dizziness and latency for users and improving the overall user experience. Attached Figure Description

[0033] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 This is a schematic diagram of a virtual reality display system provided in an exemplary embodiment of this application;

[0035] Figure 2 This is a flowchart of a virtual reality display method provided in an exemplary embodiment of this application;

[0036] Figure 3 This is a schematic diagram of a dataset for the training phase provided in an exemplary embodiment of this application;

[0037] Figure 4 This is a flowchart of a virtual reality display method provided in another exemplary embodiment of this application;

[0038] Figure 5 This is a structural block diagram of a virtual reality display device provided in an exemplary embodiment of this application;

[0039] Figure 6 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0043] It should be noted that all information and data involved in this application are authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0044] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0045] First, a brief introduction to the terms used in the embodiments of this application:

[0046] Virtual Reality (VR) is a technology that uses computer technology and hardware to simulate and generate a three-dimensional virtual environment in which users can immerse themselves and interact. VR technology aims to provide users with a perception of a virtual world, allowing them to immerse themselves in a virtual space that differs from the real world.

[0047] VR devices (virtual reality devices) are key to realizing VR technology. Common VR devices include head-mounted VR devices and controller-based VR devices. For example, VR glasses and VR headsets are worn by users, who then view virtual reality images displayed on the device. Users experience virtual reality by watching these images.

[0048] VR technology has a wide range of applications, including but not limited to: (1) Games: It provides an immersive gaming experience, allowing users to participate in games as if they were there. (2) 3D design: Taking car design as an example, VR technology allows designers to build and modify car models in 3D space for display and evaluation. (3) In-vehicle area: VR technology can provide entertainment for passengers during their journey, allowing them to experience various games and virtual world scenes by wearing VR devices.

[0049] The virtual reality display system in this application is mainly used in vehicle-riding scenarios, where users wear virtual reality devices while riding in a vehicle to obtain a virtual reality experience during the vehicle's movement.

[0050] An Image Signal Processor (ISP) is a specialized digital signal processor responsible for converting raw image data captured by a camera into a high-quality digital image. This process includes a series of image processing steps to improve image quality and prepare it for display or storage. ISPs are typically integrated into camera modules or used as standalone chips in conjunction with the camera sensor. For example, in smartphones, digital cameras, surveillance cameras, and other image capture devices, the image signal processor influences the quality of the final image.

[0051] The vehicle-mounted camera component in this application integrates an ISP. After the vehicle-mounted camera component acquires road condition images, the ISP preprocesses the road condition images to improve their quality, so that the processed road condition images can meet the data requirements for image recognition and analysis.

[0052] Virtual Reality (VR) technology involves using computer graphics, sensor technology, and human-computer interfaces to construct and simulate three-dimensional virtual environments. VR technology uses a range of highly specialized devices, including head-mounted displays (HMDs), handheld controllers, and body motion tracking systems, to allow users to achieve a deep sense of immersion in the virtual environment. Users interact with the virtual world through these devices, achieving full sensory participation and experience.

[0053] In the context of vehicle travel, the application of in-vehicle virtual reality systems provides passengers with an innovative way to receive entertainment and information. While the vehicle is in motion, passengers can watch multimedia content, participate in games, or experience simulated travel by wearing virtual reality devices (head-mounted displays).

[0054] However, vehicles encounter various road conditions during operation, including but not limited to uneven roads (including raised surfaces, sunken surfaces, road obstacles, deceleration, etc.) and road turns, which may cause the vehicle to experience bumps. This can affect the stability of loading images onto virtual reality devices, reduce the real-time loading efficiency of the images, and cause a mismatch between the displayed images and the head movements of the user wearing the device, resulting in delays in image loading latency compensation data.

[0055] Lag loading can disrupt a user's immersive experience, reduce the smoothness of virtual reality visuals, and cause dizziness. Therefore, ensuring stable and smooth visuals and a positive experience for virtual reality devices while the vehicle is in motion is a pressing issue that needs to be addressed.

[0056] This application provides a virtual reality display system, which includes an in-vehicle terminal, an in-vehicle camera assembly, and a virtual reality device. The system can perform longitudinal delay compensation on the virtual reality display image when the vehicle experiences vertical bumps or jolts as it travels over road obstacles. Because the bumps caused by the vehicle passing over obstacles result in head movements, this virtual reality display system can pre-load elements from the virtual reality image to designated positions, matching the displayed elements with the user's head movements. This reduces the impact of bumps on image stability and minimizes dizziness experienced by the user when using the virtual reality device.

[0057] This is illustrative; please refer to it. Figure 1 , Figure 1 This is a schematic diagram of a virtual reality display system provided in an exemplary embodiment of this application. The virtual reality display system 100 includes a virtual reality device 110, an in-vehicle camera assembly 120, and an in-vehicle terminal 130.

[0058] The virtual reality display system 100 is used to provide virtual reality services for the first vehicle, the vehicle terminal 130 is an internal terminal of the first vehicle, the vehicle camera assembly 120 is installed on the outside of the first vehicle, the virtual reality device 110 is used to display virtual reality images to passengers inside the first vehicle, and there is a communication connection between the virtual reality device 110, the vehicle camera assembly 120, and the vehicle terminal 130.

[0059] During the first vehicle's operation, the vehicle-mounted camera assembly 120 is configured to collect road condition images, which are used to indicate the road conditions of the road where the first vehicle is located. After the images are collected, they are sent to the vehicle-mounted terminal 130.

[0060] For example, the vehicle-mounted camera assembly 120 is installed on the windshield of the first vehicle to collect road conditions on the road directly in front of the first vehicle. During the movement of the first vehicle, the vehicle-mounted camera assembly 120 collects road condition images in real time at a preset frequency and immediately sends the collected road condition images to the vehicle-mounted terminal 130. For example, a first image is collected at a first moment and sent to the vehicle-mounted terminal 130; a second image is collected at a second moment after the first moment and sent to the vehicle-mounted terminal 130, and so on.

[0061] The vehicle terminal 130 is configured to identify the received road condition image and obtain an identification result, which is used to indicate the type of obstacle contained in the road condition image.

[0062] The recognition process determines whether a road condition image contains obstacles, and if so, analyzes the type of obstacle. The vehicle-mounted terminal 130 is equipped with a pre-trained object detection model. This model analyzes and recognizes road condition images, determines whether the image contains obstacles, and if so, identifies the type of obstacle, generating a recognition result. The recognition result includes the type of obstacle and its location information within the road condition image.

[0063] The vehicle-mounted terminal 130 analyzes the recognition results to obtain loading delay compensation data. This loading delay compensation data is used to indicate the timing at which the virtual reality device 110 displays at least one element in the virtual reality screen. This at least one element constitutes the virtual scene displayed in the virtual reality screen and provides the user with virtual reality perception.

[0064] The vehicle terminal 130 can calculate the straight-line distance between the obstacle and the first vehicle based on the recognition result, determine the time when the first vehicle passes the obstacle based on the straight-line distance and the speed of the first vehicle, and further obtain loading delay compensation data. The loading delay compensation data is used to indicate the time when the virtual reality device 110 loads at least one element in advance.

[0065] Since the virtual reality device 110 needs a certain amount of time to perform calculations and render the image when loading the virtual reality image, preloading at least one element can ensure that the display time of at least one element in the virtual reality image matches the user's head state when the first vehicle passes over the obstacle.

[0066] For example, when the first vehicle travels over an uneven road surface, it will experience up-and-down bumps, causing the user wearing the virtual reality device 110 to move their head and change its position longitudinally. The virtual reality device 110 loads the virtual reality image to present to the user primarily by monitoring the user's head movements and loading corresponding elements. When the user's head turns, the device displays the image and elements matching that turning action. Therefore, the loading latency compensation data refers to the moment when the virtual reality device 110 loads the corresponding element in the virtual reality image.

[0067] The vehicle terminal 130 sends the latency compensation data and recognition results to the virtual reality device 110.

[0068] The virtual reality device 110 is configured to receive loading delay compensation data and recognition results, and to obtain loading displacement data based on the recognition results, wherein the loading displacement data is used to indicate the display position of at least one element in the virtual reality screen.

[0069] Displacement data is used to instruct the virtual reality device 110 to adjust the position of at least one element in the image so that when the first vehicle passes over an obstacle, even if the user's head is affected and its position changes longitudinally, the virtual reality device 110 can match the user's head movement and display at least one element in a suitable position, reducing the impact of the bumpy process on the displayed image and keeping the image seen by the user stable.

[0070] Since different types of obstacles have different effects on the longitudinal position of the first vehicle, it is necessary to determine the loading displacement data that conforms to the actual situation based on the identification results.

[0071] The vehicle-mounted terminal 130 displays virtual reality images based on loading delay compensation data and loading displacement data, wherein the position of at least one element in the virtual reality image is determined based on the loading displacement data.

[0072] For example, the first vehicle will experience a bumpy ride when it passes an obstacle at the third moment. The loading delay compensation data instructs the virtual reality device 110 to start loading at least one element at the fourth moment before the third moment. The loading displacement data is used to instruct the virtual reality device 110 to move the position of at least one element in the virtual reality screen to the first position. Then, when the first vehicle passes the obstacle, at least one element has been loaded at the first position in the virtual reality screen and is displayed to the user wearing the virtual reality device 110.

[0073] Based on the above-described terms and application scenarios, the virtual reality display method provided in this application will be described, taking the execution of this method by a virtual reality display system as an example. The virtual reality display system includes an in-vehicle camera assembly 210, an in-vehicle terminal 220, and a virtual reality device 230. A communication connection is established between the in-vehicle camera assembly 210, the in-vehicle terminal 220, and the virtual reality device 230, such as... Figure 2 As shown, Figure 2 This is a flowchart of a virtual reality display method provided in an exemplary embodiment of this application. The method includes the following steps.

[0074] Step 211: The vehicle-mounted camera component acquires road condition images.

[0075] Among them, the road condition image is used to indicate the road conditions of the road where the first vehicle is located.

[0076] Optionally, the number of vehicle-mounted camera components is at least one, and they are installed on the exterior of the first vehicle.

[0077] For example, a first vehicle-mounted camera assembly is installed on the windshield of the first vehicle to collect road condition images of the road ahead of the first vehicle. When the first vehicle is moving forward, the road condition images can be used to determine whether it is safe ahead of the first vehicle, thus assisting the driver in controlling the first vehicle.

[0078] For example, a second vehicle-mounted camera assembly is installed on the outside of the trunk of the first vehicle to collect road condition images of the road behind the first vehicle. When the first vehicle needs to reverse, the road condition images can be used to determine whether it is safe behind the first vehicle.

[0079] For example, vehicle-mounted camera components are installed on the left and right doors of the first vehicle to collect road condition images of the roads on the left and right sides of the first vehicle. When the first vehicle needs to turn, the road condition images can be used to determine whether the sides of the first vehicle are safe.

[0080] The vehicle-mounted camera component has a resolution that meets the preset resolution requirements, which can improve the quality of road condition images and make the images clearer.

[0081] In some embodiments, the vehicle-mounted camera component acquires road condition images in real time based on a preset frequency and sends the road condition images to the vehicle-mounted terminal based on the preset frequency.

[0082] In other words, the vehicle-mounted camera module collects road condition images at preset intervals during the first vehicle's journey, and immediately sends the collected images to the vehicle terminal.

[0083] For example, the preset frequency is 3 seconds / time, that is, a road condition image is collected once every 3 seconds.

[0084] At 10:00:00, a road condition image is captured and sent to the vehicle terminal; at 10:00:03, a road condition image is captured and sent to the vehicle terminal; at 10:00:06, a road condition image is captured and sent to the vehicle terminal, and so on.

[0085] Real-time acquisition of road condition images can more accurately identify and locate obstacles that may appear on the road, reducing the probability of missed detections and false alarms. When an obstacle is present, the distance between the obstacle and the first vehicle can be reflected over time, enhancing the first vehicle's perception of the environment during its journey.

[0086] Step 212: The vehicle-mounted camera component sends road condition images to the vehicle-mounted terminal.

[0087] Optionally, to improve image quality, after acquiring road condition images, the vehicle-mounted camera component preprocesses the images to obtain preprocessed road condition images, which are then sent to the vehicle-mounted terminal. The preprocessed road condition images conform to preset image processing format requirements.

[0088] For example, the vehicle-mounted camera assembly integrates an ISP. After the vehicle-mounted camera assembly acquires road condition images, the ISP preprocesses the road condition images to improve their quality.

[0089] The preprocessing performed by the ISP on road condition images includes, but is not limited to, the following.

[0090] 1. Lens Shadow Correction: Due to the optical characteristics of the lens, the edge brightness of the road condition image may be uneven. This step compensates for the brightness of the edge of the road condition image, making the overall brightness of the road condition image more uniform.

[0091] 2. Bad Pixel Correction: Detects and repairs bad pixels in an image, such as dead pixels, bright pixels, or drift pixels, to ensure image quality.

[0092] 3. Color interpolation: The missing color information is estimated through algorithms to reconstruct a full-color image.

[0093] 4. Noise Reduction: Removes random and fixed-pattern noise from an image, such as noise caused by sensor readout or circuit problems, to improve the signal-to-noise ratio of the image.

[0094] 5. White Balance: Corrects the color temperature of an image so that white objects can appear in the correct color under different lighting conditions.

[0095] 6. Sharpening: Enhances the edges and details of an image, improving its clarity.

[0096] 7. Auto Focus: Ensures that objects captured by the vehicle's camera components remain sharp at different distances.

[0097] By performing preprocessing steps through the ISP, the vehicle camera assembly can provide clearer, more accurate, and more reliable road condition images, avoiding the problem of low recognition accuracy caused by recognizing blurry road condition images in subsequent steps.

[0098] Step 221: The vehicle terminal identifies the received road condition image and obtains the identification result.

[0099] The identification results are used to indicate the type of obstacles contained in the road condition image.

[0100] Optionally, when the vehicle-mounted camera component sends a pre-processed road condition image, the vehicle-mounted terminal receives the pre-processed road condition image. The pre-processed road condition image is then used for recognition to obtain a recognition result.

[0101] The vehicle-mounted terminal's recognition process is used to determine whether there are obstacles in the road condition image, and if there are obstacles, to analyze the type of obstacles.

[0102] Optionally, a pre-trained target detection model is deployed in the vehicle terminal. The vehicle terminal uses the target detection model to perform target detection on the road condition image and obtains the target detection result. The target detection result is used to indicate the presence of obstacles in the road condition image.

[0103] For example, the object detection model is a pre-trained deep neural network model, and the architecture of the deep neural network model can be one of the following model architectures.

[0104] (1) YOLO Series (You Only Look Once) Model Architecture: YOLO is a popular single-stage object detection algorithm that transforms the object detection task into a regression problem, predicting the location and category of objects in an image through a single forward propagation. The YOLO algorithm can process images and detect objects in real time, directly predicting bounding boxes and category probabilities from the image without complex region proposals or subsequent processing. It can detect multiple objects in an image, including overlapping objects. The YOLO series includes multiple versions from YOLO_v1 to YOLO_v5.

[0105] (2) SSD series (Single Shot multibox Detector) model architecture: SSD is also a single-stage object detection algorithm that performs object detection on feature maps at different scales, enabling it to detect objects of different sizes. By performing detection on feature maps at different levels, it can effectively detect objects of different sizes. Multiple default boxes are predicted at each location, increasing the diversity of detection.

[0106] (3) Faster R-CNN series (Fast Region-based Convolutional Network) model architecture: Faster R-CNN is a two-stage object detection algorithm. It first generates candidate regions through a region proposal network, and then classifies and regresses bounding boxes on these regions. It can generate high-quality candidate regions, providing a foundation for subsequent detection. The extracted feature maps are used for both region proposal and subsequent classification and regression, improving efficiency. It can be used to detect objects in any orientation.

[0107] (4) ResNet series (Residual Network) model architecture: ResNet is a deep convolutional neural network mainly used for image classification tasks. It solves the problem of training difficulties in deep networks by introducing residual learning. By learning residual functions, it alleviates the gradient vanishing problem, allowing the network to go deeper. It introduces shortcut connections of identity mapping, allowing gradients to propagate directly through layers.

[0108] For example, this application uses YOLO_v5 as the model architecture to pre-train an object detection model for illustration.

[0109] During the training phase, use, for example Figure 3 The dataset 300 shown is used to train the object detection model, enabling the model to accurately identify uneven road surfaces and obstacles such as roadblocks.

[0110] The dataset contains 300 images of various types, with each image containing one type of obstacle, such as sunken roads, uneven roads, gravel obstacles, speed bumps, and stone obstacles.

[0111] When the target detection result indicates the presence of obstacles in the road condition image, the vehicle terminal obtains the recognition result based on the target detection result.

[0112] For example, the target detection result output by the target detection model can be used as the recognition result.

[0113] The identification results include the first location of the obstacle in the road condition image and the type of obstacle corresponding to the obstacle.

[0114] For example, refer to Table 1, where different types of obstacles are labeled.

[0115] Table 1

[0116] Classification pebbles pit Speed ​​bumps Steps Serial Number 1 2 3 4

[0117] Step 222: The vehicle terminal analyzes the recognition results to obtain loading delay compensation data.

[0118] Among them, the loading delay compensation data is used to indicate the time when the virtual reality device displays at least one element in the virtual reality screen. The at least one element is used to constitute the virtual scene displayed in the virtual reality screen and to provide the user with virtual reality perception.

[0119] The loading delay compensation data includes a first duration required to load at least one element into a specified position in the virtual reality screen. The loading time for loading at least one element is determined based on the arrival time of the first vehicle at the obstacle and the first duration. At the loading time, at least one element is loaded into the specified position in the virtual reality screen. The loading time is the moment before the arrival time, and the duration between the loading time and the arrival time is the first duration.

[0120] As the first vehicle moves longitudinally when it passes an obstacle, the head of the user inside the vehicle also moves accordingly. The movement of the user's head while wearing a virtual reality device will cause a corresponding change in the virtual reality image (i.e., the position of at least one element in the virtual reality image changes). Since it takes time for the virtual reality device to load the virtual reality image and at least one element, related technologies usually predict the trajectory of the user's head movement (head turning) and preload at least one element to the corresponding position in the virtual reality image to eliminate the time delay in the element loading process, so that the user can see the matching image immediately after the user's head moves.

[0121] In this embodiment, the loading delay compensation data is determined by predicting the image that matches the user's head movements when the first vehicle passes over an obstacle. The loading delay compensation data can instruct the virtual reality device to load at least one element into a specified position in the virtual reality image in advance, so that when the vehicle bumps over an obstacle, the image seen by the user matches the movement of his head.

[0122] Virtual reality devices provide users with a virtual world experience and display virtual reality images by building virtual scenes (i.e., VR scenes) and adding 3D models of different elements to the virtual scenes.

[0123] All elements in the virtual scene are built using a six-degree-of-freedom model. A six-degree-of-freedom model is a model used to describe the motion of an object in three-dimensional space. It includes three translational motions and three rotational motions of the object in space. In other words, the six-degree-of-freedom model is used to build the three-dimensional model of all elements in the virtual scene, so that the elements have a similar or identical appearance to the three-dimensional entity, and the elements can move freely in the virtual scene.

[0124] When the first vehicle passes an obstacle, it will undergo longitudinal displacement (perpendicular to the ground / Z-axis direction), such as moving upwards or downwards. As a result, the user riding in the first vehicle will also experience head movements. In order to eliminate the impact of the user's head movements on the virtual reality display of the virtual reality device, delay compensation is required for the Z-axis direction and γ (rotation around the Y-axis, called yaw, which describes the angle of the user's left and right head movements). This ensures that when the first vehicle passes an obstacle and undergoes longitudinal movement, the virtual reality device matches the image seen by the user with the movement of the user's head.

[0125] Optionally, the vehicle-mounted terminal determines a first distance between the obstacle and the first vehicle based on a first location (the location of the obstacle in the road condition image) and the location of the first vehicle. Loading delay compensation data is then determined based on the first distance and the vehicle speed of the first vehicle.

[0126] When the vehicle-mounted camera component acquires road condition images, it can reflect the distance between the vehicle-mounted camera component and the obstacle based on the pixel ratio and perspective change principle of the obstacle in the road condition image, and determine the first distance between the first vehicle and the obstacle. In some embodiments, the distance between the vehicle-mounted camera component and the obstacle is determined as the first distance between the first vehicle and the obstacle.

[0127] The system acquires real-time data on the speed changes of the first vehicle and predicts the time required for the first vehicle to travel the first distance. For example, if the first vehicle is traveling at a constant first speed, the time is determined directly by dividing the first distance by the first speed. Alternatively, if the first vehicle's speed changes in real time, the time is determined based on the average speed of the first vehicle over a fixed distance.

[0128] In some embodiments, road condition images are acquired in real time and transmitted to the vehicle terminal by the vehicle camera component. The vehicle terminal updates the first distance based on the real-time received road condition images to obtain the updated first distance.

[0129] In other words, each time a road condition image is received, the newly received road condition image needs to be analyzed to update the distance between the first vehicle and the obstacle.

[0130] The vehicle terminal obtains first delay data based on the updated first distance and the vehicle speed of the first vehicle. The first delay data is the time required for the first vehicle to travel to the obstacle.

[0131] The first unit of time consumed by the vehicle-mounted camera to collect road condition images in a single session and send them to the vehicle-mounted terminal is used to determine loading delay compensation data based on the first delay data and the first unit of time.

[0132] In some embodiments, loading elements into the virtual reality screen by the virtual reality device also requires a certain amount of computation time. The second duration for the virtual reality device to move the elements in the virtual reality screen is obtained, and loading delay compensation data is determined based on the first delay data, the first unit duration, and the second duration.

[0133] For example, when the first distance between the first vehicle and the obstacle is less than a preset distance threshold (e.g., 50 meters), the system begins to determine the first delay data required for the first vehicle to reach the obstacle. The delay compensation number (in milliseconds) is determined based on the first delay data, the first unit duration, and the second duration.

[0134] Please refer to Table 2, which is a table comparing delay compensation numbers.

[0135] Table 2

[0136]

[0137] When the road condition image shows the presence of multiple obstacles, the delay compensation number corresponding to each obstacle is determined based on Table 2 above. The loading delay compensation data is determined based on the delay compensation number, the first delay data, the first unit duration, and the second duration.

[0138] For example, the road condition image indicates the presence of two obstacles, obstacle A and obstacle B, in the direction of travel of the first vehicle. The road condition image is acquired in real time, and the distances between the first vehicle and obstacle A and obstacle B are calculated. Among them, obstacle A is closer to the first vehicle.

[0139] When the first distance between the first vehicle and obstacle A is 50 meters, and the distance between the first vehicle and obstacle B exceeds 50 meters, the delay compensation number corresponding to obstacle A is determined to be 3 based on the delay compensation number comparison table, and the delay compensation number corresponding to obstacle B is 4.

[0140] The first delay data for the first vehicle to reach the obstacle includes delay data A and delay data B. Delay data A refers to the time required for the first vehicle to reach obstacle A, and delay data B refers to the time required for the first vehicle to reach obstacle B.

[0141] If the sum of delay data A, the first unit duration, and the second duration is 4 seconds, and the current moment is the first moment, then the loading delay compensation data sent to the virtual reality device at the current moment indicates that at least one element should be loaded to the first specified position at a time 4 seconds and 3 milliseconds after the current moment.

[0142] If the sum of delay data B, the first unit duration, and the second duration is 6 seconds, and the current moment is the first moment, then the loading delay compensation data sent to the virtual reality device at the current moment also indicates that at least one element should be loaded to the second specified position starting 6 seconds and 4 milliseconds after the current moment.

[0143] The first designated location is determined based on the type of obstacle A, and the second designated location is determined based on the type of obstacle B.

[0144] During this process, because the road condition images are collected in real time, the distances between obstacle A and obstacle B and the first vehicle will also shorten, and the delay data A and delay data B corresponding to the first vehicle's movement will also change. Therefore, the loading delay compensation data will also be updated accordingly. Each time the loading delay compensation data is updated, it is sent to the virtual reality device until the first vehicle reaches the obstacle and the transmission stops.

[0145] That is, when the first vehicle reaches obstacle A, the virtual reality device has already loaded at least one element into the first designated position based on the latest received loading delay compensation data. At this time, the first vehicle has not yet reached obstacle B. The loading delay compensation data at this time contains data corresponding to obstacle B. During the process of the first vehicle traveling from obstacle A to obstacle B, road condition images are still collected in real time to update the delay compensation data and delay data B, and the updated loading delay compensation data is sent to the virtual reality device. Until the first vehicle travels to obstacle B, the virtual reality device has already loaded at least one element into the second designated position based on the latest received loading delay compensation data.

[0146] It is worth noting that the delay compensation number comparison table shown in Table 2 is a preset comparison table, and the data in the table is determined based on a large amount of data tested. In some embodiments, the delay compensation number can also be determined in other ways. For example, based on a preset function mapping expression representing the relationship between the delay compensation number and the distance of the first vehicle to the obstacle, the delay compensation number is calculated based on the preset function mapping expression after collecting the distance of the first vehicle to the obstacle. This embodiment does not limit this.

[0147] It is worth noting that the loading delay compensation data can be determined solely based on the sum of the delay compensation number, the first unit duration, and the second duration, or it can be determined by weighting the delay compensation number, the first unit duration, and the second duration using other methods.

[0148] Step 223: The vehicle terminal will send the latency compensation data and recognition results to the virtual reality device.

[0149] When the vehicle-mounted terminal sends the recognition result, it also sends the corresponding road condition image to the virtual reality device.

[0150] Step 231: The virtual reality device receives loading delay compensation data and recognition results.

[0151] The virtual reality device simultaneously receives the road condition image corresponding to the recognition result.

[0152] Step 232: The virtual reality device obtains loading displacement data based on the recognition results.

[0153] Specifically, the load displacement data is used to indicate the display position of at least one element in the virtual reality screen. That is, the load displacement data indicates the distance by which at least one element is moved up or down in the virtual reality screen.

[0154] The loading displacement data includes the distance and direction required to move at least one element to a specified position (display position) in the virtual reality screen, and the loading displacement data is determined based on the type of obstacle that the first vehicle is about to pass through.

[0155] As the first vehicle moves longitudinally when it passes an obstacle, the head of the user inside the vehicle also moves accordingly. The movement of the user's head while wearing a virtual reality device will cause a corresponding change in the virtual reality image (i.e., the position of at least one element in the virtual reality image changes). Since it takes time for the virtual reality device to load the virtual reality image and at least one element, related technologies usually predict the trajectory of the user's head movement (head turning) and preload at least one element to the corresponding position in the virtual reality image to eliminate the time delay in the element loading process, so that the user can see the matching image immediately after the user's head moves.

[0156] In this embodiment, the loading displacement data is determined by predicting the image that matches the user's head movement when the first vehicle passes over an obstacle. The loading delay compensation data can instruct the virtual reality device to load at least one element to a specified position in the virtual reality image in advance, so that when the vehicle bumps over an obstacle, the image seen by the user matches the movement of his head.

[0157] Optionally, the virtual reality device determines the displacement type based on the obstacle type in the recognition results. The displacement type is used to indicate the direction for adjusting the display position of at least one element in the virtual reality screen.

[0158] Obtain the load displacement lookup table and the pixel percentage data of obstacles in the road condition image, and determine the load displacement data from the load displacement lookup table based on the pixel percentage data and displacement type.

[0159] For illustrative purposes, please refer to Table 3, which is a loading displacement comparison table.

[0160] Table 3

[0161]

[0162] The pixel percentage in the loading displacement lookup table refers to the percentage of pixels in the road condition image acquired at the moment when the distance between the first vehicle and the obstacle is a specified distance. In other words, it is the proportion of the obstacle's pixels to the total pixels of the road condition image.

[0163] Since the size of speed bumps and steps in the road is fixed, it is not necessary to analyze the pixel ratio data when the obstacle is a speed bump or step.

[0164] Speed ​​bumps on city roads appear as yellow and black stripes, while speed bumps on highways appear as multi-point obstacles.

[0165] The types of obstacles include the following: stones (number 1), potholes (number 2), speed bumps (number 3), and steps (number 4).

[0166] Specifically, when the obstacle type is any one of pebbles, speed bumps, or steps, the displacement type is determined to be positive, meaning the displacement data is positive, and at least one element is moved upwards in the virtual reality screen to the display position. When the obstacle type is a pit, the displacement type is determined to be negative, meaning the displacement data is negative, and at least one element is moved downwards in the virtual reality screen to the display position.

[0167] Offset refers to the unit distance moved in the loaded displacement data. For example, when the offset is 2, it means that at least one element is moved up by 2 units. This unit distance refers to the unit distance in the virtual reality screen; for example, 10 centimeters in the virtual reality screen is 1 unit distance.

[0168] The loading displacement comparison table shown in Table 3 is a preset comparison table, and the data in the table is determined based on a large amount of data tested. In some embodiments, the offset under different conditions can also be determined in other ways.

[0169] When the obstacle is a pebble, if the pixel percentage of the pebble is less than 5%, the displacement data is shifted upward by 2 units; if the pixel percentage of the pebble is between 5% and 10%, the displacement data is shifted upward by 4 units; if the pixel percentage of the pebble is greater than 10%, the displacement data is shifted upward by 6 units.

[0170] When the obstacle is a pit, if the pixel percentage of the pit is less than 10%, the displacement data is shifted down by 2 units; if the pixel percentage of the pit is between 10% and 15%, the displacement data is shifted down by 4 units; if the pixel percentage of the pit is greater than 15%, the displacement data is shifted down by 6 units.

[0171] When the obstacle is a yellow and black speed bump, the applied displacement is 2 units upward. When the obstacle is a multi-point speed bump, the applied displacement is 1 unit upward. When the obstacle is a step, the applied displacement is 6 units upward.

[0172] Step 233: Display virtual reality images based on loading delay compensation data and loading displacement data.

[0173] In this case, the position of at least one element in the virtual reality scene is determined based on the loaded displacement data.

[0174] For example, the first vehicle will experience a bumpy ride when passing an obstacle at 10:00:10. The loading delay compensation data is 2 seconds, instructing the virtual reality device to start loading at least one element 2 seconds before reaching the obstacle, that is, to start loading at least one element at 10:00:08.

[0175] By preloading at least one element to a specified position in the virtual reality scene, the virtual reality scene viewed by the user remains stable when the first vehicle undergoes longitudinal displacement as it passes over an obstacle.

[0176] The displacement data is used to instruct the virtual reality device to move at least one element in the virtual reality screen to a first position. When the first vehicle passes the obstacle, at least one element is loaded at the first position in the virtual reality screen and displayed to the user wearing the virtual reality device.

[0177] For example, when the obstacle is a step, stone, speed bump, or other object of a certain height, the first vehicle experiences longitudinal displacement upwards when passing over the obstacle. Simultaneously, the user's head also experiences longitudinal displacement upwards due to inertia. To match the virtual reality image seen by the user with the user's head movement, at least one element is loaded to a designated position in the virtual reality image starting at the time indicated by the loading delay compensation data. This element moves upwards in the virtual reality image to reach the designated position, the distance of which is the distance indicated by the loading displacement data.

[0178] When the first vehicle passes the obstacle, at least one element is loaded, and the view seen by the user remains stable before and after the first vehicle passes the obstacle.

[0179] For example, when the obstacle type indicates that the first vehicle is about to pass through a road section that is lower than the road surface level, such as a recessed road surface, it means that the first vehicle will have a longitudinal displacement downwards when passing the obstacle. At this time, the user's head inside the vehicle will also have a longitudinal displacement downwards due to inertia. In order to match the virtual reality image seen by the user with the user's head movement, at least one element is loaded to a specified position in the virtual reality image based on the time indicated by the loading delay compensation data. The at least one element moves downwards in the virtual reality image to reach the specified position, and the distance moved is the distance indicated by the loading displacement data.

[0180] When the first vehicle passes the obstacle, at least one element is loaded, and the view seen by the user remains stable before and after the first vehicle passes the obstacle.

[0181] In summary, the virtual reality display method and system provided in this application, when applied in a vehicle-riding scenario, acquires real-time images of the road conditions where the vehicle is located. Based on the presence and type of obstacles, it determines whether the vehicle will experience bumps during travel and compensates for the delay in displaying the virtual reality image due to head movements. By analyzing the road condition images and obtaining loading displacement data and loading delay compensation data, it determines how the virtual reality device will display elements in the virtual reality image when the vehicle passes obstacles, pre-loading elements to the target position to reduce dizziness and delays experienced by users while wearing the virtual reality device, thus improving the user experience.

[0182] Figure 4 This is a flowchart of a virtual reality display method provided in another exemplary embodiment of this application, which is executed by an on-board terminal of a first vehicle and includes the following steps.

[0183] Step 410: Receive road condition images.

[0184] The road condition image is an image captured by the onboard camera assembly of the first vehicle, and is used to indicate the road conditions of the road where the first vehicle is located.

[0185] Optionally, at least one vehicle-mounted camera assembly is installed on the exterior of the first vehicle. The vehicle-mounted camera assembly acquires road condition images in real time based on a preset frequency, preprocesses the acquired road condition images, and then sends the preprocessed road condition images to the vehicle-mounted terminal based on the preset frequency.

[0186] The vehicle-mounted terminal receives pre-processed road condition images based on a preset frequency, and can obtain road condition images with clarity and image quality that meet preset requirements.

[0187] Step 420: Recognize the received road condition image to obtain the recognition result.

[0188] The identification results are used to indicate the types of obstacles contained in the road condition image. The identification process of the vehicle-mounted terminal is used to determine whether the road condition image contains obstacles, and if so, to analyze the type of obstacles.

[0189] The identification results include the type of obstacle and its location in the road condition image.

[0190] Optionally, a pre-trained target detection model is deployed in the vehicle terminal. The vehicle terminal uses the target detection model to perform target detection on the road condition image and obtains the target detection result. The target detection result is used to indicate the presence of obstacles in the road condition image.

[0191] For example, the target detection results output by the target detection model include the following: (1) whether there is an obstacle; (2) when there is an obstacle, the type of obstacle; and (3) when there is an obstacle, the location of the obstacle in the road condition image.

[0192] The target detection results output by the target detection model are used as the recognition results.

[0193] Step 430: Analyze the recognition results to obtain loading delay compensation data.

[0194] Among them, the loading delay compensation data is used to indicate the time when the virtual reality device displays at least one element in the virtual reality screen. The at least one element is used to constitute the virtual scene displayed in the virtual reality screen and to provide the user with virtual reality perception.

[0195] Optionally, the vehicle terminal determines a first distance between the obstacle and the first vehicle based on the first location and the position of the first vehicle; and determines loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0196] For example, the first distance is updated based on the real-time received road condition images to obtain the updated first distance. First delay data is obtained based on the updated first distance and the vehicle speed of the first vehicle. The first unit time consumed by the vehicle-mounted camera in a single acquisition of road condition images and transmission to the vehicle-mounted terminal is obtained. Loading delay compensation data is determined based on the first delay data and the first unit time.

[0197] Step 440: Load the latency compensation data and recognition results to the virtual reality device.

[0198] After receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and loading displacement data. The loading displacement data is used to indicate the display position of at least one element in the virtual reality screen.

[0199] In summary, the virtual reality display method provided in this application can analyze the received road condition images to determine whether there are obstacles in the road and the type of obstacles, and whether the vehicle will experience bumps during travel. It then compensates for the delay in the virtual reality device's display, mitigating the problem of display delay caused by the user's head movements. By analyzing the road condition images and obtaining loading delay compensation data, the method determines how the virtual reality device displays elements in the virtual reality image when the vehicle passes an obstacle, preloading the elements in the image to the target position, reducing dizziness and delay issues experienced by users when viewing the image while wearing the virtual reality device, and improving the user experience.

[0200] Figure 5 This is a structural block diagram of a virtual reality display device provided in an exemplary embodiment of this application, such as... Figure 5 As shown, the device includes the following parts.

[0201] The receiving module 510 is used to receive road condition images, which are images captured by the vehicle-mounted camera assembly of the first vehicle, and are used to indicate the road conditions of the road where the first vehicle is located.

[0202] The identification module 520 is used to identify the received road condition image and obtain an identification result, the identification result being used to indicate the type of obstacle contained in the road condition image;

[0203] Analysis module 530 is used to analyze the identification results to obtain loading delay compensation data;

[0204] The sending module 540 is used to send the loading delay compensation data and the recognition result to the virtual reality device; wherein, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality screen.

[0205] In an optional embodiment, the recognition module 520 is further configured to perform target detection on the road condition image using a target detection model to obtain a target detection result, the target detection result being used to indicate the presence of an obstacle in the road condition image; and, if the target detection result indicates the presence of an obstacle in the road condition image, to obtain the recognition result based on the target detection result; wherein the recognition result includes a first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

[0206] In an optional embodiment, the analysis module 530 is further configured to determine a first distance between the obstacle and the first vehicle based on the first location and the position of the first vehicle; and to determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0207] In an optional embodiment, the analysis module 530 is further configured to update the first distance based on the real-time received road condition image to obtain an updated first distance; obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle; obtain a first unit time consumed by the vehicle-mounted camera in a single acquisition of the road condition image and transmission to the vehicle-mounted terminal; and determine the loading delay compensation data based on the first delay data and the first unit time.

[0208] In summary, the virtual reality display device provided in this application can determine whether a vehicle will experience bumps during travel based on the presence and type of obstacles on the road, and perform latency compensation for the virtual reality device to alleviate the problem of display delay caused by the user's head movements. By analyzing road condition images and obtaining loading displacement data and loading latency compensation data, the method for displaying elements in the virtual reality image when the vehicle passes an obstacle is determined, and the elements in the image are pre-loaded to the target position, reducing dizziness and latency issues experienced by users when viewing the image while wearing the virtual reality device, thus improving the user experience.

[0209] It should be noted that the virtual reality display device provided in the above embodiments is only an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the virtual reality display device and the virtual reality display method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0210] Figure 6 This illustration shows a structural block diagram of a computer device 600 provided in an exemplary embodiment of this application. The computer device 600 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The computer device 600 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0211] Typically, computer device 600 includes a processor 601 and a memory 602.

[0212] Processor 601 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 601 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0213] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 602 are used to store at least one instruction, which is executed by the processor 601 to implement the virtual reality display method provided in the method embodiments of this application.

[0214] In some embodiments, the computer device 600 also includes other components 603, the type and number of which can be selected based on the functional needs of the computer device 600. Those skilled in the art will understand that... Figure 6 The structure shown does not constitute a limitation on the computer device 600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0215] Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM). The sequence numbers of the embodiments in this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0216] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the virtual reality display method as described in any of the above embodiments of this application.

[0217] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the virtual reality display method as described in any of the above embodiments of this application.

[0218] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the virtual reality display methods described in the above embodiments.

[0219] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0220] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A virtual reality display system, characterized in that, The virtual reality display system includes virtual reality equipment, vehicle-mounted camera components, and vehicle-mounted terminals; The vehicle-mounted camera assembly is configured to capture road condition images, which are used to indicate the road conditions of the road where the first vehicle is located. The road condition image is sent to the vehicle terminal; The vehicle-mounted terminal is configured to identify the received road condition image and obtain an identification result, the identification result being used to indicate the type of obstacle contained in the road condition image; The recognition results are analyzed to obtain loading latency compensation data; wherein, the loading latency compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; the loading latency compensation data and the recognition results are sent to the virtual reality device; The virtual reality device is configured to receive the loading delay compensation data and the recognition result; obtain loading displacement data based on the recognition result, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality screen; and display the virtual reality screen based on the loading delay compensation data and the loading displacement data.

2. The system according to claim 1, characterized in that, The vehicle-mounted terminal is also configured to perform target detection on the road condition image through a target detection model to obtain a target detection result, which is used to indicate the presence of obstacles in the road condition image. When the target detection result indicates that there is an obstacle in the road condition image, the recognition result is obtained based on the target detection result; The identification result includes the first location of the obstacle in the road condition image and the type of obstacle corresponding to the obstacle.

3. The system according to claim 2, characterized in that, The vehicle-mounted terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first location and the position of the first vehicle; and to determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

4. The system according to claim 3, characterized in that, The vehicle-mounted camera component is also configured to acquire road condition images in real time based on a preset frequency; and to send the road condition images to the vehicle-mounted terminal based on the preset frequency. The vehicle-mounted terminal is further configured to update the first distance based on the real-time received road condition image to obtain an updated first distance; and to obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle. The first unit of time consumed by the vehicle-mounted camera in a single acquisition of the road condition image and its transmission to the vehicle-mounted terminal; The loading delay compensation data is determined based on the first delay data and the first unit duration.

5. The system according to claim 2, characterized in that, The virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, the displacement type being used to indicate the direction for adjusting the display position of the at least one element in the virtual reality screen; and to obtain a loading displacement lookup table and pixel percentage data of the obstacle in the road condition image; The loading displacement data is determined from the loading displacement lookup table based on the pixel percentage data and the displacement type.

6. The system according to any one of claims 1 to 5, characterized in that, The vehicle-mounted camera assembly is further configured to preprocess the road condition image to obtain a preprocessed road condition image, wherein the preprocessed road condition image conforms to a preset image processing format requirement. The preprocessed road condition image is sent to the vehicle terminal; The vehicle-mounted terminal is also configured to receive the preprocessed road condition image; to identify the preprocessed road condition image and obtain the identification result.

7. A virtual reality display method, characterized in that, The method includes: Receive road condition images, which are images captured by the onboard camera assembly of the first vehicle, and are used to indicate the road conditions of the road where the first vehicle is located; The received road condition image is identified to obtain an identification result, which is used to indicate the type of obstacle contained in the road condition image; The identification results are analyzed to obtain loading delay compensation data; The loading delay compensation data and the recognition result are sent to the virtual reality device; Wherein, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

8. A virtual reality display device, characterized in that, The device includes: A receiving module is used to receive road condition images, which are images captured by the vehicle-mounted camera assembly of the first vehicle, and the road condition images are used to indicate the road conditions of the road where the first vehicle is located. The identification module is used to identify the received road condition image and obtain an identification result, which is used to indicate the type of obstacle contained in the road condition image; The analysis module is used to analyze the identification results to obtain loading delay compensation data; A sending module is used to send the loading latency compensation data and the recognition result to a virtual reality device; wherein, the loading latency compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading latency compensation data and the loading displacement data, the loading displacement data being used to indicate the display position of the at least one element in the virtual reality screen.

9. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one program, which is loaded and executed by the processor to implement the virtual reality display method as described in claim 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one program segment, which is loaded and executed by a processor to implement the virtual reality display method as described in claim 7.

Images