Virtual reality display system, method, device, equipment and medium

Abstract

The invention discloses a virtual reality display system, method, apparatus and device, and a medium, and relates to the technical field of virtual reality, the system comprises a virtual reality device, a vehicle-mounted camera assembly, and a vehicle-mounted terminal; the vehicle-mounted camera assembly is configured to collect a road condition image and send the road condition image to the vehicle-mounted terminal; the vehicle-mounted terminal is configured to identify the received road condition image to obtain an identification result; analyzing the identification result to obtain loading time delay compensation data; sending the loading time delay compensation data and the identification result to the virtual reality equipment; the virtual reality equipment is configured to receive and load the time delay compensation data and the identification result; acquiring loading displacement data based on an identification result; and displaying a virtual reality picture based on the loading time delay compensation data and the loading displacement data. The influence on elements in a loading picture of the virtual reality equipment can be eliminated under the condition of vehicle bumping, and the delay during picture loading is reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of virtual reality technology, and in particular to a virtual reality display system, method, apparatus, device, and medium. Background Art

[0002] Virtual Reality (VR) technology creates and simulates a three-dimensional virtual world, allowing users to gain an immersive experience in the virtual world through interactive devices such as head-mounted displays, handles, and body tracking devices.

[0003] VR technology can be applied to vehicle riding scenarios, where in-vehicle VR products can provide passengers with a new way to entertain and obtain information. By wearing VR devices, passengers can watch multimedia content, play games, or take a virtual trip while the vehicle is driving.

[0004] However, the road conditions may change at any time while the vehicle is driving, and it is difficult for the vehicle to avoid bumps, which may affect the stability of the VR device when displaying the picture, causing delays in picture loading and affecting the smoothness of the user experience. Summary of the invention

[0005] The embodiments of the present application provide a virtual reality display system, method, apparatus, device, and medium, which can eliminate the impact on the elements in the loading screen of the virtual reality device when the vehicle is bumpy, and reduce the delay when the screen is loaded. The technical solution is as follows:

[0006] In one aspect, a virtual reality display system is provided, the virtual reality display system comprising a virtual reality device, a vehicle-mounted camera assembly, and a vehicle-mounted terminal;

[0007] The vehicle-mounted camera assembly is configured to collect a road condition image, the road condition image being used to indicate the road condition of the road on which the first vehicle is located; and send the road condition image to the vehicle-mounted terminal;

[0008] The vehicle-mounted terminal is configured to identify the received road condition image to obtain an identification result, wherein the identification result is used to indicate the type of obstacles contained in the road condition image; analyze the identification result to obtain loading delay compensation data; wherein the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, wherein the at least one element is used to constitute a virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; and send the loading delay compensation data and the identification result to the virtual reality device;

[0009] The virtual reality device is configured to receive the loading delay compensation data and the recognition result; obtain loading displacement data based on the recognition result, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen; and display the virtual reality screen based on the loading delay compensation data and the loading displacement data.

[0010] In an optional embodiment, the vehicle-mounted terminal is further configured to perform target detection on the road condition image through a target detection model to obtain a target detection result, and the target detection result is used to indicate the presence of an obstacle in the road condition image; when the target detection result indicates that there is an obstacle in the road condition image, the recognition result is obtained based on the target detection result; wherein the recognition result includes the first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

[0011] In an optional embodiment, the vehicle-mounted terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first position and the position of the first vehicle; and determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0012] In an optional embodiment, the vehicle-mounted camera assembly is further configured to collect the road condition image in real time based on a preset frequency; and send the road condition image to the vehicle-mounted terminal based on the preset frequency;

[0013] The vehicle-mounted terminal is also configured to update the first distance based on the road condition image received in real time to obtain an updated first distance; obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle; obtain the first unit time consumed by the vehicle-mounted camera to capture the road condition image once and send it to the vehicle-mounted terminal; and determine the loading delay compensation data based on the first delay data and the first unit time.

[0014] In an optional embodiment, the virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, wherein the displacement type is used to indicate a direction for adjusting a display position of the at least one element in the virtual reality screen; obtain a loading displacement comparison table and pixel ratio data of the obstacle in the road condition image; and determine the loading displacement data from the loading displacement comparison table based on the pixel ratio data and the displacement type.

[0015] In an optional embodiment, the vehicle-mounted camera assembly is further configured to pre-process the road condition image to obtain a pre-processed road condition image, wherein the pre-processed road condition image meets a preset image processing format requirement; and send the pre-processed road condition image to the vehicle-mounted terminal;

[0016] The vehicle-mounted terminal is further configured to receive the pre-processed road condition image; and recognize the pre-processed road condition image to obtain the recognition result.

[0017] In another aspect, a virtual reality display method is provided, the method comprising:

[0018] Receiving a road condition image, where the road condition image is an image captured by a vehicle-mounted camera assembly of a first vehicle, and the road condition image is used to indicate a road condition of a road on which the first vehicle is located;

[0019] Recognize the received road condition image to obtain a recognition result, where the recognition result is used to indicate the type of obstacle contained in the road condition image;

[0020] Analyzing the recognition result to obtain loading delay compensation data;

[0021] Sending the loading delay compensation data and the recognition result to a virtual reality device;

[0022] Among them, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute a virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains the loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

[0023] In another aspect, a virtual reality display device is provided, the device comprising:

[0024] A receiving module, used for receiving a road condition image, wherein the road condition image is an image collected by a vehicle-mounted camera assembly of a first vehicle, and the road condition image is used for indicating a road condition of a road on which the first vehicle is located;

[0025] An identification module, used to identify the received road condition image to obtain an identification result, wherein the identification result is used to indicate the type of obstacle contained in the road condition image;

[0026] An analysis module, used for analyzing the recognition result to obtain loading delay compensation data;

[0027] A sending module, used for sending the loading delay compensation data and the recognition result to a virtual reality device; wherein the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in a virtual reality screen, and the at least one element is used to constitute a virtual scene displayed by the virtual reality screen and provide a virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

[0028] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement a virtual reality display method as described in any of the above embodiments of the present application.

[0029] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, a code set or an instruction set is stored in the storage medium, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement a virtual reality display method as described in any of the above embodiments of the present application.

[0030] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the virtual reality display method described in any of the above embodiments.

[0031] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0032] When the virtual reality device is used in a riding scenario, the road condition of the vehicle is obtained by real-time image acquisition. Whether there are obstacles in the road condition and the type of obstacles determine whether the vehicle will be bumpy during driving, and delay compensation is performed on the virtual reality device to alleviate the problem of virtual reality screen display delay caused by the user's head movement. The road condition image is analyzed to obtain loading displacement data and loading delay compensation data, determine the way the virtual reality device displays elements in the virtual reality screen when the vehicle passes an obstacle, load the elements in the screen to the target position in advance, reduce the dizziness and delay problems that occur when the user wears the virtual reality device to watch the screen, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 is a schematic diagram of a virtual reality display system provided by an exemplary embodiment of the present application;

[0035] Figure 2 is a flow chart of a virtual reality display method provided by an exemplary embodiment of the present application;

[0036] Figure 3 is a schematic diagram of a data set in a training phase provided by an exemplary embodiment of the present application;

[0037] Figure 4 is a flow chart of a virtual reality display method provided by another exemplary embodiment of the present application;

[0038] Figure 5 is a structural block diagram of a virtual reality display device provided by an exemplary embodiment of the present application;

[0039] Figure 6 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0041] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0042] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be noted that the information and data involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0044] It should be understood that, although the terms first, second, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first parameter may also be referred to as the second parameter, and similarly, the second parameter may also be referred to as the first parameter. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0045] First, a brief introduction is given to the terms involved in the embodiments of this application:

[0046] Virtual Reality (VR) is a technology that uses computer technology and hardware devices to simulate a three-dimensional virtual environment in which users can immerse and interact. VR technology aims to provide users with a sense of the virtual world, allowing users to immerse themselves in a virtual space that is different from the real world.

[0047] VR devices (virtual reality devices) are the key to realizing VR technology. Common VR devices include head-mounted VR devices and handle VR devices, such as VR glasses and VR helmets. Users wear VR devices, which display virtual reality images. Users gain the perception of virtual reality by watching virtual reality images.

[0048] VR technology has a wide range of applications, including but not limited to: (1) Games: providing an immersive gaming experience where users can participate in the game as if they were actually there. (2) Three-dimensional design: taking car design as an example, VR technology allows designers to build and modify car models in three-dimensional space for display and evaluation. (3) In-car: VR technology can provide entertainment for passengers during their journey. Passengers can experience various games and virtual world scenes by wearing VR devices.

[0049] The virtual reality display system in this application is mainly used in vehicle riding scenarios. Users wear virtual reality devices while riding in a vehicle and gain a virtual reality experience while the vehicle is driving.

[0050] Image Signal Processor (ISP): is a specialized digital signal processor responsible for converting the raw image data captured by the camera into a high-quality digital image. This process includes a series of image processing steps to improve the image quality and prepare it for display or storage. The ISP is usually integrated into the camera module or used as a standalone chip in conjunction with the camera sensor. For example, in smartphones, digital cameras, surveillance cameras, and other image capture devices, the image signal processor affects the quality of the final image.

[0051] The vehicle-mounted camera component in the present application is integrated with an ISP. After the vehicle-mounted camera component captures the road condition image, the road condition image is pre-processed by the ISP to improve the quality of the road condition image so that the processed road condition image can meet the data requirements for image recognition and analysis.

[0052] Virtual Reality (VR) technology involves the use of computer graphics, sensor technology, and human-computer interaction interfaces to build and simulate three-dimensional virtual environments. VR technology enables users to gain a deep sense of immersion in a virtual environment through a series of highly specialized devices, including head-mounted displays (HMDs), handheld controllers, and body motion tracking systems. Users interact with the virtual world through these devices, achieving full sensory participation and experience.

[0053] In vehicle riding scenarios, the application of in-vehicle virtual reality systems provides passengers with an innovative way to receive entertainment and information. Passengers can watch multimedia content, participate in game activities, or experience simulated travel by wearing virtual reality devices (head-mounted displays) while the vehicle is driving.

[0054] However, the vehicle may encounter various changes in road conditions during driving, including but not limited to uneven roads (including raised roads, sunken roads, roadblocks, deceleration, etc.), road turns, etc., which may cause the vehicle to shake, affecting the stability of the virtual reality device when loading the screen, reducing the real-time loading efficiency of the screen, causing the screen display to not match the head movement of the user wearing the device, and delaying the screen loading delay compensation data.

[0055] The phenomenon of screen loading delay will interfere with the user's immersive experience, reduce the smoothness of the virtual reality screen, and make the user feel dizzy. Therefore, how to ensure that the virtual reality device provides a stable and smooth screen and experience during vehicle driving is an urgent problem to be solved.

[0056] The present application provides a virtual reality display system, which includes a vehicle-mounted terminal, a vehicle-mounted camera assembly and a virtual reality device, and can compensate for the longitudinal delay of the display screen of the virtual reality device when the vehicle is bumped up and down when passing through a road obstacle. Since the vehicle bumps when passing through an obstacle, it will cause the user's head to move. The virtual reality display system can load various elements in the virtual reality screen to a specified position in advance, so that the screen elements displayed in the virtual reality device match the user's head movement, reduce the impact of bumps on the stability of the screen, and reduce the dizziness caused by the user when using the virtual reality device.

[0057] For illustration, please refer to Figure 1 , Figure 1 1 is a schematic diagram of a virtual reality display system provided by an exemplary embodiment of the present application. The virtual reality display system 100 includes a virtual reality device 110 , a vehicle-mounted camera assembly 120 , and a vehicle-mounted terminal 130 .

[0058] Among them, the virtual reality display system 100 is used to provide virtual reality services for the first vehicle, the vehicle-mounted terminal 130 is an internal terminal of the first vehicle, the vehicle-mounted camera assembly 120 is installed on the outside of the first vehicle, and the virtual reality device 110 is used to display virtual reality images to passengers in the first vehicle. There is a communication connection between the virtual reality device 110, the vehicle-mounted camera assembly 120, and the vehicle-mounted terminal 130.

[0059] During the driving of the first vehicle, the vehicle-mounted camera assembly 120 is configured to collect road condition images, which are used to indicate the road condition of the road on which the first vehicle is located. After the collection is completed, the road condition images are sent to the vehicle-mounted terminal 130.

[0060] Exemplarily, the vehicle-mounted camera assembly 120 is installed at the windshield of the first vehicle, and is used to collect the road condition of the road directly in front of the first vehicle. During the driving of the first vehicle, the vehicle-mounted camera assembly 120 collects road condition images in real time according to a preset frequency, and immediately sends the collected road condition images to the vehicle-mounted terminal 130. For example, a first image is collected at a first moment and sent to the vehicle-mounted terminal 130; a second image is collected at a second moment after the first moment and sent to the vehicle-mounted terminal 130, and so on.

[0061] The vehicle-mounted terminal 130 is configured to identify the received road condition image and obtain an identification result, where the identification result is used to indicate the type of obstacle contained in the road condition image.

[0062] The recognition process is used to determine whether the road condition image contains obstacles, and if so, to analyze the type of obstacles. A pre-trained target detection model is deployed in the vehicle terminal 130, which can analyze and recognize the road condition image, determine whether the image contains obstacles, and determine the type of obstacles if obstacles are contained in the road condition image, and generate recognition results. The recognition results include the type of obstacles and the location information of the obstacles in the road condition image.

[0063] The vehicle terminal 130 analyzes the recognition result to obtain loading delay compensation data, wherein the loading delay compensation data is used to indicate the moment when the virtual reality device 110 displays at least one element in the virtual reality screen, and the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user.

[0064] Among them, the vehicle-mounted terminal 130 can calculate the straight-line distance between the obstacle and the first vehicle according to the recognition result, determine the time when the first vehicle passes the obstacle based on the straight-line distance and the driving speed of the first vehicle, and further obtain loading delay compensation data. The loading delay compensation data is used to indicate the time when the virtual reality device 110 loads at least one element in advance.

[0065] Since the virtual reality device 110 needs a certain amount of time to perform steps such as calculation and rendering when loading the virtual reality screen, loading at least one element in advance can make the display moment of at least one element in the virtual reality screen match the user's head state when the first vehicle passes an obstacle.

[0066] For example, when the first vehicle passes through a concave road, it will bump up and down, causing the head of the user wearing the virtual reality device 110 to move, and the head position changes in the vertical direction. When the virtual reality device 110 loads the virtual reality screen and presents it to the user, it mainly relies on monitoring the user's head movement to load the corresponding elements, so that when the user's head turns, the screen and elements matching the turning movement are displayed. Therefore, the loading delay compensation data refers to the moment when the virtual reality device 110 loads the corresponding element in the virtual reality screen.

[0067] The vehicle-mounted terminal 130 sends the loading delay compensation data and the recognition result to the virtual reality device 110 .

[0068] The virtual reality device 110 is configured to receive loading delay compensation data and a recognition result, and obtain loading displacement data based on the recognition result, wherein the loading displacement data is used to indicate a display position of at least one element in a virtual reality screen.

[0069] The displacement data is used to indicate how the virtual reality device 110 adjusts the position of at least one element in the picture, so that when the first vehicle passes through an obstacle, even if the user's head is affected and moves and its position changes longitudinally, the virtual reality device 110 can match the movement of the user's head to display at least one element in an appropriate position, thereby reducing the impact of the bumpy process on the displayed picture and keeping the picture viewed by the user stable.

[0070] Since different types of obstacles have different effects on the longitudinal position of the first vehicle, it is necessary to determine loading displacement data that conforms to the actual situation based on the recognition result.

[0071] The vehicle terminal 130 displays a virtual reality picture based on the loading delay compensation data and the loading displacement data, wherein a position of at least one element in the virtual reality picture is determined according to the loading displacement data.

[0072] For example, the first vehicle may be jolted when passing an obstacle at a third moment. The loading delay compensation data instructs the virtual reality device 110 to start loading at least one element at a fourth moment before the third moment. The loading displacement data is used to instruct the virtual reality device 110 to move the position of the at least one element in the virtual reality screen to the first position. Then, when the first vehicle passes the obstacle, the at least one element is loaded at the first position in the virtual reality screen and displayed to the user wearing the virtual reality device 110.

[0073] In combination with the above-mentioned noun introduction and application scenarios, the virtual reality display method provided by the present application is described. The method is performed by a virtual reality display system as an example. The virtual reality display system includes a vehicle-mounted camera component 210, a vehicle-mounted terminal 220, and a virtual reality device 230. A communication connection is established between the vehicle-mounted camera component 210, the vehicle-mounted terminal 220, and the virtual reality device 230. Figure 2 As shown, Figure 2 1 is a flowchart of a virtual reality display method provided by an exemplary embodiment of the present application. The method includes the following steps.

[0074] Step 211: The vehicle-mounted camera component collects road condition images.

[0075] The road condition image is used to indicate the road condition of the road on which the first vehicle is located.

[0076] Optionally, there is at least one vehicle-mounted camera assembly installed on the outside of the first vehicle.

[0077] For example, a first vehicle-mounted camera assembly is installed on the windshield of the first vehicle, which is used to collect road condition images of the road in front of the first vehicle. When the first vehicle is moving forward, it can determine whether the road in front of the first vehicle is safe through the road condition images to assist the driver in controlling the first vehicle.

[0078] For example, a second vehicle-mounted camera assembly is installed outside the trunk of the first vehicle to collect road condition images of the road behind the first vehicle. When the first vehicle needs to reverse, it can determine whether the rear of the first vehicle is safe through the road condition images.

[0079] For example, vehicle-mounted camera assemblies are installed on the left and right doors of the first vehicle to collect road condition images on the left and right sides of the first vehicle. When the first vehicle needs to turn, it can determine whether both sides of the first vehicle are safe through the road condition images.

[0080] The resolution of the vehicle-mounted camera component reaches the preset resolution requirement, which can improve the quality of road condition images and make the images clearer.

[0081] In some embodiments, the vehicle-mounted camera component collects road condition images in real time based on a preset frequency, and sends the road condition images to the vehicle-mounted terminal based on the preset frequency.

[0082] That is, the vehicle-mounted camera component collects road condition images once every preset time period during the driving process of the first vehicle, and sends them to the vehicle-mounted terminal immediately after the collection is completed.

[0083] Exemplarily, the preset frequency is 3 seconds / time, that is, a road condition image is collected once every 3 seconds.

[0084] The first moment is 10:00:00, a road condition image is collected and sent to the vehicle terminal; the second moment is 10:00:03, a road condition image is collected and sent to the vehicle terminal; the third moment is 10:00:06, a road condition image is collected and sent to the vehicle terminal, and so on.

[0085] Real-time collection of road condition images can more accurately identify and locate obstacles that may appear on the road, reduce the probability of missed detection and false subtraction, and when there is an obstacle, reflect the distance between the obstacle and the first vehicle based on the passage of time, thereby enhancing the first vehicle's perception of the environment during driving.

[0086] Step 212: The vehicle-mounted camera assembly sends the road condition image to the vehicle-mounted terminal.

[0087] Optionally, in order to improve image quality, after acquiring the road condition image, the vehicle camera component preprocesses the road condition image to obtain a preprocessed road condition image, and sends the preprocessed road condition image to the vehicle terminal. The preprocessed road condition image meets the preset image processing format requirements.

[0088] Exemplarily, an ISP is integrated in the vehicle-mounted camera assembly. After the vehicle-mounted camera assembly captures a road condition image, the ISP pre-processes the road condition image to improve the quality of the road condition image.

[0089] The preprocessing performed by ISP on the road condition image includes but is not limited to the following.

[0090] 1. Lens shading correction: Due to the optical characteristics of the lens, the edge brightness of the road image may be uneven. This step compensates for the brightness of the edge of the road image to make the overall brightness of the road image more uniform.

[0091] 2. Bad Pixel Correction: Detect and repair bad pixels in the image, such as dead pixels, bright pixels or drifting pixels, to ensure image quality.

[0092] 3. Color interpolation: The missing color information is estimated through an algorithm to reconstruct a full-color image.

[0093] 4. Noise Reduction: Remove random noise and fixed pattern noise from the image, such as noise caused by sensor readout or circuit problems, to improve the signal-to-noise ratio of the image.

[0094] 5. White Balance: Correct the color temperature of the image so that white objects can appear in the correct color under different lighting conditions.

[0095] 6. Sharpening: Enhance the edges and details of the image and improve the clarity of the image.

[0096] 7. Auto Focus: Ensures that objects captured by the vehicle camera component at different distances remain clear.

[0097] By performing the preprocessing steps through the ISP, the vehicle camera component can provide clearer, more accurate and more reliable road images, avoiding the problem of low recognition accuracy caused by recognizing blurred road images in subsequent steps.

[0098] Step 221: the vehicle-mounted terminal recognizes the received road condition image and obtains a recognition result.

[0099] The recognition result is used to indicate the type of obstacles contained in the road condition image.

[0100] Optionally, when the vehicle-mounted camera component sends the preprocessed road condition image, the vehicle-mounted terminal receives the preprocessed road condition image and recognizes the preprocessed road condition image to obtain a recognition result.

[0101] The recognition process of the vehicle terminal is used to determine whether the road condition image contains obstacles, and if so, analyze the type of obstacles.

[0102] Optionally, a pre-trained target detection model is deployed in the vehicle terminal. The vehicle terminal performs target detection on the road condition image through the target detection model to obtain a target detection result. The target detection result is used to indicate the presence of obstacles in the road condition image.

[0103] Exemplarily, the target detection model is a pre-trained deep neural network model, and the architecture of the deep neural network model can be one of the following model architectures.

[0104] (1) YOLO series (You Only Look Once) model architecture: YOLO is a popular single-stage target detection algorithm that converts the target detection task into a regression problem and predicts the target location and category in the image through a single forward propagation. The YOLO algorithm can process images and detect targets in real time, directly predicting bounding boxes and category probabilities from images without complex region proposals or subsequent processing. It can detect multiple objects in an image, including overlapping objects. The YOLO series includes multiple versions from YOLO_v1 to YOLO_v5.

[0105] (2) SSD series (Single Shot multibox Detector) model architecture: SSD is also a single-stage target detection algorithm. It performs object detection on feature maps of different scales and can detect objects of different sizes. By performing detection on feature maps of different levels, objects of different sizes can be effectively detected. Multiple default boxes are predicted for each position, increasing the diversity of detection.

[0106] (3) FasterR-CNN series (Fast Region-based Convolutional Network) model architecture: FasterR-CNN is a two-stage object detection algorithm that first generates candidate regions through a region proposal network, and then classifies and regresses these regions. It can generate high-quality candidate regions and provide a basis for subsequent detection. The extracted feature maps are used for both region proposal and subsequent classification and regression, which improves efficiency and can be used to detect objects in any direction.

[0107] (4) ResNet series (Residua Network) model architecture: ResNet is a deep convolutional neural network mainly used for image classification tasks. It solves the problem of difficult deep network training by introducing residual learning. The problem of gradient vanishing is alleviated by learning the residual function, allowing the network to be deeper. The shortcut connection of the identity mapping is introduced to allow the gradient to propagate directly through the layer.

[0108] Exemplarily, in this application, YOLO_v5 is used as a model architecture for pre-training to obtain a target detection model as an example for explanation.

[0109] During the training phase, use Figure 3 The data set 300 shown is used to train the target detection model so that the target detection model can accurately identify obstacles such as uneven roads and roadblocks.

[0110] The data set 300 includes multiple types of images, each of which includes one type of obstacle, such as a sunken road surface, an uneven road surface, a gravel obstacle, a speed bump barrier, a stone barrier, etc.

[0111] When the target detection result indicates that there is an obstacle in the road condition image, the vehicle-mounted terminal obtains a recognition result based on the target detection result.

[0112] Exemplarily, the target detection result output by the target detection model is used as the recognition result.

[0113] The recognition result includes the first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

[0114] Exemplarily, refer to Table 1, in which different types of obstacles are marked.

[0115] Table 1

[0116] Classification Stone Pits Speed ​​bumps Stairs Serial number 1 2 3 4

[0117] Step 222: The vehicle-mounted terminal analyzes the recognition result to obtain loading delay compensation data.

[0118] Among them, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user.

[0119] The loading delay compensation data includes a first time length required to load at least one element to a specified position in the virtual reality screen in advance, and a loading time for loading at least one element is determined based on the arrival time of the first vehicle at the obstacle and the first time length, and at least one element is loaded to the specified position in the virtual reality screen at the loading time. The loading time is the time before the arrival time, and the time length between the loading time and the arrival time is the first time length.

[0120] Since the first vehicle will be displaced longitudinally when passing through an obstacle, the head of the user in the vehicle will also be displaced accordingly, and the movement of the head of the user wearing the virtual reality device will cause the virtual reality screen to change accordingly (that is, the position of at least one element in the virtual reality screen changes). It takes a certain amount of time for the virtual reality device to load the virtual reality screen and at least one element. Therefore, in the related art, the movement trajectory of the user's head (the turning of the head) is usually predicted, and at least one element is loaded in advance to the corresponding position in the virtual reality screen to eliminate the delay in the process of loading the elements, so that the user can immediately see the matching picture after the head moves.

[0121] The loading delay compensation data in this embodiment is data determined after predicting the picture that matches the user's head movement when the first vehicle passes through an obstacle. The loading delay compensation data can instruct the virtual reality device to load at least one element to a specified position in the virtual reality picture in advance, so that when the user's vehicle passes through an obstacle and bumps, the picture he sees matches the movement of his head.

[0122] When virtual reality devices provide users with virtual world perception and display virtual reality images, they do so by building a virtual scene (i.e., VR scene) and adding three-dimensional models of different elements to the virtual scene.

[0123] All elements in the virtual scene are built using a six-degree-of-freedom model. The six-degree-of-freedom model is a model used to describe the movement of objects in three-dimensional space. It includes three translational movements and three rotational movements of objects in space. That is, the three-dimensional models of all elements in the virtual scene are built using the six-degree-of-freedom model, so that the elements have a similar or identical appearance to the three-dimensional entities, and the elements can move freely in the virtual scene.

[0124] When the first vehicle passes by an obstacle, the first vehicle will be displaced in the longitudinal direction (perpendicular to the ground / Z-axis direction), for example, moving up or down, and the user riding in the first vehicle will also have head movement. In order to eliminate the influence of the user's head movement on the virtual reality image displayed by the virtual reality device, it is necessary to perform delay compensation for the Z-axis direction and γ (rotation around the Y-axis, called yaw, describing the angle of the user's head moving left and right), so that the virtual reality device can match the image seen by the user with the movement of the user's head when the first vehicle moves longitudinally when passing by the obstacle.

[0125] Optionally, the vehicle terminal determines a first distance between the obstacle and the first vehicle based on the first position (the position of the obstacle in the road condition image) and the position of the first vehicle, and determines the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0126] When the vehicle-mounted camera component collects road condition images, the distance between the vehicle-mounted camera component and the obstacle can be reflected based on the pixel ratio of the obstacle in the road condition image and the perspective change principle, and the first distance between the first vehicle and the obstacle is determined. In some embodiments, the distance between the vehicle-mounted camera component and the obstacle is determined as the first distance between the first vehicle and the obstacle.

[0127] The vehicle speed change of the first vehicle is obtained in real time, and the time required for the first vehicle to travel the first distance is predicted. For example, when the first vehicle is traveling at a constant speed at the first speed, the time is directly determined based on the first distance divided by the first speed. For example, when the vehicle speed of the first vehicle changes in real time, the time is determined based on the average vehicle speed of the first vehicle within a fixed distance.

[0128] In some embodiments, the road condition image is collected in real time and transmitted to the vehicle terminal by the vehicle camera component. The vehicle terminal updates the first distance based on the road condition image received in real time to obtain an updated first distance.

[0129] That is, each time a road condition image is received, it is necessary to analyze the newly received road condition image and update the distance between the first vehicle and the obstacle.

[0130] The vehicle terminal obtains first delay data based on the updated first distance and the vehicle speed of the first vehicle, where the first delay data is the time required for the first vehicle to travel to the obstacle.

[0131] The first unit time consumed by the vehicle-mounted camera to collect a road condition image once and send it to the vehicle-mounted terminal is obtained, and the loading delay compensation data is determined based on the first delay data and the first unit time.

[0132] In some embodiments, it also takes a certain amount of computing time for the virtual reality device to load elements into the virtual reality screen, obtains the second duration for the virtual reality device to realize the movement of elements in the virtual reality screen, and determines the loading delay compensation data based on the first delay data, the first unit duration and the second duration.

[0133] Exemplarily, when the first distance between the first vehicle and the obstacle is less than a preset distance threshold (e.g., 50 meters), the first delay data required for the first vehicle to reach the obstacle is determined, and the delay compensation number (in milliseconds) is determined according to the first delay data, the first unit time length, and the second time length.

[0134] Please refer to Table 2, which is a comparison table of delay compensation numbers.

[0135] Table 2

[0136]

[0137] When the road condition image reflects the presence of multiple obstacles, the delay compensation number corresponding to each obstacle is determined based on the above Table 2, and the loading delay compensation data is determined based on the delay compensation number, the first delay data, the first unit duration, and the second duration.

[0138] For example, the road condition image indicates that there are two obstacles in the driving direction of the first vehicle, namely obstacle A and obstacle B. The road condition image is collected in real time and the distances between the first vehicle and obstacle A and obstacle B are calculated. Among them, obstacle A is closer to the first vehicle.

[0139] When the first distance between the first vehicle and obstacle A is 50 meters, the distance between the first vehicle and obstacle B exceeds 50 meters. Based on the delay compensation number comparison table, it is determined that the delay compensation number corresponding to obstacle A is 3, and the delay compensation number corresponding to obstacle B is 4.

[0140] Among them, the first delay data of the first vehicle reaching the obstacle includes delay data A and delay data B. Delay data A refers to the time required for the first vehicle to reach obstacle A, and delay data B refers to the time required for the first vehicle to reach obstacle B.

[0141] The sum of the delay data A, the first unit duration and the second duration is 4 seconds, and the current moment is the first moment. The loading delay compensation data sent to the virtual reality device at the current moment indicates that at least one element will be loaded to the first specified position starting at 4 seconds and 3 milliseconds after the current moment.

[0142] The sum of delay data B, the first unit duration and the second duration is 6 seconds, and the current moment is the first moment. The loading delay compensation data sent to the virtual reality device at the current moment also indicates that at least one element will be loaded to the second specified position starting at 6 seconds and 4 milliseconds after the current moment.

[0143] The first designated position is determined based on the type of obstacle A, and the second designated position is determined based on the type of obstacle B.

[0144] During this process, because the road condition image is collected in real time, the distances between obstacle A and obstacle B and the first vehicle will also be shortened, and the delay data A and delay data B corresponding to the first vehicle during its driving process will also change. Therefore, the loading delay compensation data will also be updated accordingly. Each time the loading delay compensation data is updated, it will be sent to the virtual reality device once until the first vehicle reaches the obstacle and stops sending.

[0145] That is, when the first vehicle arrives at obstacle A, the virtual reality device has loaded at least one element to the first designated position based on the loading delay compensation data most recently received at the current moment, and the first vehicle has not yet arrived at obstacle B. At this time, the loading delay compensation data includes data corresponding to obstacle B. In the process of the first vehicle traveling from obstacle A to obstacle B, the road condition image is still collected in real time to update the delay compensation number and delay data B, and the updated loading delay compensation data is obtained and sent to the virtual reality device until the first vehicle travels to obstacle B. At this time, the virtual reality device has loaded at least one element to the second designated position based on the loading delay compensation data most recently received at the current moment.

[0146] It is worth noting that the delay compensation number comparison table shown in Table 2 is a preset comparison table, and the data in the table is determined after testing based on a large amount of data. In some embodiments, the delay compensation number can also be determined in other ways, for example, based on a preset function mapping expression to represent the relationship between the delay compensation number and the distance from the first vehicle to the obstacle, the delay compensation number is calculated based on the preset function mapping expression after collecting the distance from the first vehicle to the obstacle, and this embodiment does not limit this.

[0147] It is worth noting that the loading delay compensation data can be determined based only on the sum of the delay compensation number, the first unit duration, and the second duration, or by performing weighted calculation on the delay compensation number, the first unit duration, and the second duration in other ways.

[0148] In step 223, the vehicle-mounted terminal sends the loaded delay compensation data and the recognition result to the virtual reality device.

[0149] When the vehicle-mounted terminal sends the recognition result, the road condition image corresponding to the recognition result is also sent to the virtual reality device.

[0150] Step 231: The virtual reality device receives loading delay compensation data and recognition results.

[0151] The virtual reality device simultaneously receives the road condition image corresponding to the recognition result.

[0152] Step 232: The virtual reality device obtains loading displacement data based on the recognition result.

[0153] The loaded displacement data is used to indicate the display position of at least one element in the virtual reality screen, that is, the loaded displacement data is used to indicate the distance by which at least one element is moved up or down in the virtual reality screen.

[0154] The loading displacement data includes the distance and direction required to load at least one element to a specified position (display position) in the virtual reality screen, and the loading displacement data is determined based on the type of obstacle that the first vehicle is about to pass.

[0155] Since the first vehicle will be displaced longitudinally when passing through an obstacle, the head of the user in the vehicle will also be displaced accordingly, and the movement of the head of the user wearing the virtual reality device will cause the virtual reality screen to change accordingly (that is, the position of at least one element in the virtual reality screen changes). It takes a certain amount of time for the virtual reality device to load the virtual reality screen and at least one element. Therefore, in the related art, the movement trajectory of the user's head (the turning of the head) is usually predicted, and at least one element is loaded in advance to the corresponding position in the virtual reality screen to eliminate the delay in the process of loading the elements, so that the user can immediately see the matching picture after the head moves.

[0156] The loading displacement data in this embodiment is data determined after predicting the picture that matches the user's head movement when the first vehicle passes through an obstacle. The loading delay compensation data can instruct the virtual reality device to load at least one element to a specified position in the virtual reality picture in advance, so that when the user's vehicle passes through an obstacle and bumps, the picture he sees matches the movement of his head.

[0157] Optionally, the virtual reality device determines a displacement type based on the obstacle type in the recognition result, where the displacement type is used to indicate a direction for adjusting a display position of at least one element in the virtual reality screen.

[0158] Obtain a loading displacement comparison table and pixel ratio data of obstacles in the road condition image, and determine the loading displacement data from the loading displacement comparison table based on the pixel ratio data and the displacement type.

[0159] For illustration, please refer to Table 3, which is a loading displacement comparison table.

[0160] Table 3

[0161]

[0162] Among them, the pixel ratio in the loaded displacement comparison table refers to the pixel ratio data of the obstacle in the road condition image collected at the moment when the distance between the first vehicle and the obstacle is the specified distance, that is, the ratio of the obstacle's pixels to the total pixels of the road condition image.

[0163] Since the sizes of speed bumps and steps on the road are fixed, there is no need to analyze the pixel ratio data when the obstacle is a speed bump or a step.

[0164] Speed ​​bumps on urban roads appear as yellow and black stripes, while speed bumps on highways appear as multi-point obstacles.

[0165] Obstacle types include the following: stones (serial number 1), potholes (serial number 2), speed bumps (serial number 3) and steps (serial number 4).

[0166] When the obstacle type is any one of stones, speed bumps and steps, the displacement type is determined to be positive, that is, the displacement data is loaded as a positive number, and the position of at least one element in the virtual reality screen is moved up to the display position. When the obstacle type is a pit, the displacement type is determined to be negative, that is, the displacement data is loaded as a negative number, and the position of at least one element in the virtual reality screen is moved down to the display position.

[0167] The offset refers to the unit distance moved in the loaded displacement data. For example, when the offset is 2, it means that at least one element is moved up by 2 unit distances. The unit distance refers to the unit distance in the virtual reality screen. For example, 10 centimeters in the virtual reality screen is 1 unit distance.

[0168] The load displacement comparison table shown in Table 3 is a preset comparison table, and the data in the table is determined after testing a large amount of data. In some embodiments, the displacement under different conditions can also be determined in other ways.

[0169] When the obstacle is a stone, if the pixel proportion of the stone is less than 5%, the loaded displacement data is 2 unit distances upward; if the pixel proportion of the stone is 5% to 10%, the loaded displacement data is 4 unit distances upward; if the pixel proportion of the stone is greater than 10%, the loaded displacement data is 6 unit distances upward.

[0170] When the obstacle is a pit, if the pixel proportion of the pit is less than 10%, the loaded displacement data is 2 unit distances downward; if the pixel proportion of the pit is 10% to 15%, the loaded displacement data is 4 unit distances downward; if the pixel proportion of the pit is greater than 15%, the loaded displacement data is 6 unit distances downward.

[0171] When the obstacle is a yellow-black speed bump, the displacement data is loaded as 2 units of distance upward. When the obstacle is a multi-point speed bump, the displacement data is loaded as 1 unit of distance upward. When the obstacle is a step, the displacement data is loaded as 6 units of distance upward.

[0172] Step 233: Display a virtual reality screen based on the loading delay compensation data and the loading displacement data.

[0173] Wherein, the position of at least one element in the virtual reality picture is determined according to the loaded displacement data.

[0174] For example, the first vehicle will pass an obstacle and experience a bumpy phenomenon at 10:00:10, and the loading delay compensation data is 2 seconds, instructing the virtual reality device to start loading at least one element 2 seconds before reaching the obstacle, that is, to start loading at least one element at 10:00:08.

[0175] Loading at least one element to a specified position in the virtual reality picture in advance can ensure that the virtual reality picture viewed by the user remains stable when the first vehicle undergoes longitudinal displacement when passing through an obstacle.

[0176] The loaded displacement data is used to instruct the virtual reality device to move the position of at least one element in the virtual reality screen to the first position. Then, when the first vehicle passes through the obstacle, the at least one element is loaded at the first position in the virtual reality screen and displayed to the user wearing the virtual reality device.

[0177] For example, when the obstacle type is an object with a certain height such as a step, a stone, or a speed bump, the first vehicle will be displaced in the longitudinal direction upward when passing the obstacle. At this time, the head of the user in the vehicle will also be displaced in the longitudinal direction upward due to inertia. In order to match the virtual reality screen seen by the user with the user's head movement, at least one element is loaded to a specified position in the virtual reality screen based on the time indicated by the loading delay compensation data, wherein the at least one element moves upward in the virtual reality screen to the specified position, and the distance moved is the distance indicated by the loading displacement data.

[0178] When the first vehicle passes the obstacle, at least one element is loaded, and the image viewed by the user remains stable before and after the first vehicle passes the obstacle.

[0179] For example, when the obstacle type indicates that the first vehicle is about to pass through a road section that is lower than the road surface level, such as a sunken road surface, it means that the first vehicle will be displaced in the longitudinal direction downward when passing the obstacle. At this time, the head of the user in the vehicle will also be displaced in the longitudinal direction downward due to inertia. In order to match the virtual reality screen seen by the user with the user's head movement, at least one element is loaded to a specified position in the virtual reality screen based on the time indicated by the loading delay compensation data, wherein the at least one element moves downward in the virtual reality screen to reach the specified position, and the distance moved is the distance indicated by the loading displacement data.

[0180] When the first vehicle passes the obstacle, at least one element is loaded, and the image viewed by the user remains stable before and after the first vehicle passes the obstacle.

[0181] In summary, the virtual reality display method and system provided by the present application, when the virtual reality device is used in a riding scene, obtains the road condition of the road where the vehicle is located by real-time image acquisition, determines whether the vehicle will be bumpy during driving according to whether there are obstacles in the road condition and the type of obstacles, and performs delay compensation on the virtual reality device to alleviate the problem of virtual reality screen display delay caused by the user's head movement. Analyze the road condition image, obtain loading displacement data and loading delay compensation data, determine the way in which the virtual reality device displays elements in the virtual reality screen when the vehicle passes an obstacle, load the elements in the screen to the target position in advance, reduce the dizziness and delay problems that occur when the user wears the virtual reality device to watch the screen, and improve the user experience.

[0182] Figure 4 It is a flowchart of a virtual reality display method provided by another exemplary embodiment of the present application. The method is executed by an on-board terminal of a first vehicle and includes the following steps.

[0183] Step 410: Receive a road condition image.

[0184] The road condition image is an image collected by the on-board camera assembly of the first vehicle, and the road condition image is used to indicate the road condition of the road on which the first vehicle is located.

[0185] Optionally, there is at least one vehicle-mounted camera assembly installed outside the first vehicle. The vehicle-mounted camera assembly collects road condition images in real time based on a preset frequency, pre-processes the collected road condition images, and then sends the pre-processed road condition images to the vehicle-mounted terminal based on the preset frequency.

[0186] The vehicle-mounted terminal receives the pre-processed road condition images based on a preset frequency, and is able to obtain road condition images whose clarity and image quality meet preset requirements.

[0187] Step 420: Recognize the received road condition image to obtain a recognition result.

[0188] The recognition result is used to indicate the type of obstacle contained in the road condition image. The recognition process of the vehicle terminal is used to determine whether the road condition image contains an obstacle, and if so, analyze the type of obstacle.

[0189] The recognition result includes the obstacle type and the location of the obstacle in the road condition image.

[0190] Optionally, a pre-trained target detection model is deployed in the vehicle terminal. The vehicle terminal performs target detection on the road condition image through the target detection model to obtain a target detection result. The target detection result is used to indicate the presence of obstacles in the road condition image.

[0191] Exemplarily, the target detection result output by the target detection model includes the following contents: (1) whether there is an obstacle; (2) when there is an obstacle, the type of obstacle; (3) when there is an obstacle, the position of the obstacle in the road condition image.

[0192] The target detection result output by the target detection model is used as the recognition result.

[0193] Step 430: Analyze the recognition result to obtain loading delay compensation data.

[0194] Among them, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user.

[0195] Optionally, the vehicle-mounted terminal determines a first distance between the obstacle and the first vehicle based on the first position and the position of the first vehicle; and determines loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0196] Exemplarily, the first distance is updated based on the road condition image received in real time to obtain an updated first distance. The first delay data is obtained based on the updated first distance and the vehicle speed of the first vehicle. The first unit time consumed by the vehicle-mounted camera to collect the road condition image once and send it to the vehicle-mounted terminal is obtained. The loading delay compensation data is determined based on the first delay data and the first unit time.

[0197] Step 440: Send the loading delay compensation data and the recognition result to the virtual reality device.

[0198] After receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, where the loading displacement data is used to indicate the display position of at least one element in the virtual reality screen.

[0199] In summary, the virtual reality display method provided by the present application can analyze whether there are obstacles on the road and the type of obstacles based on the received road condition images, determine whether bumps will occur during the vehicle's driving, and compensate for the delay of the virtual reality device to alleviate the problem of virtual reality screen display delay caused by the user's head movement. Analyze the road condition image, obtain loading delay compensation data, determine the way in which the virtual reality device displays elements in the virtual reality screen when the vehicle passes an obstacle, load the elements in the screen to the target position in advance, reduce the dizziness and delay problems that occur when the user wears the virtual reality device to watch the screen, and improve the user's experience.

[0200] Figure 5 is a structural block diagram of a virtual reality display device provided by an exemplary embodiment of the present application, such as Figure 5 As shown, the device includes the following parts.

[0201] The receiving module 510 is used to receive a road condition image, where the road condition image is an image captured by a vehicle-mounted camera assembly of a first vehicle, and the road condition image is used to indicate the road condition of the road where the first vehicle is located;

[0202] The recognition module 520 is used to recognize the received road condition image and obtain a recognition result, wherein the recognition result is used to indicate the type of obstacle contained in the road condition image;

[0203] An analysis module 530 is used to analyze the recognition result to obtain loading delay compensation data;

[0204] The sending module 540 is used to send the loading delay compensation data and the recognition result to the virtual reality device; wherein the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide the user with virtual reality perception; after receiving the recognition result, the virtual reality device obtains the loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

[0205] In an optional embodiment, the recognition module 520 is also used to perform target detection on the road condition image through a target detection model to obtain a target detection result, and the target detection result is used to indicate the presence of an obstacle in the road condition image; when the target detection result indicates that there is an obstacle in the road condition image, the recognition result is obtained based on the target detection result; wherein the recognition result includes the first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

[0206] In an optional embodiment, the analysis module 530 is further used to determine the first distance between the obstacle and the first vehicle based on the first position and the position of the first vehicle; and determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

[0207] In an optional embodiment, the analysis module 530 is also used to update the first distance based on the road condition image received in real time to obtain an updated first distance; obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle; obtain the first unit time consumed by the on-board camera to capture the road condition image once and send it to the on-board terminal; and determine the loading delay compensation data based on the first delay data and the first unit time.

[0208] In summary, the virtual reality display device provided by the present application can determine whether the vehicle will be bumpy during driving according to whether there are obstacles in the road conditions and the types of obstacles, and perform delay compensation on the virtual reality device to alleviate the problem of virtual reality screen display delay caused by the user's head movement. The road condition image is analyzed to obtain loading displacement data and loading delay compensation data, determine the way in which the virtual reality device displays elements in the virtual reality screen when the vehicle passes an obstacle, load the elements in the screen to the target position in advance, reduce the dizziness and delay problems that occur when the user wears the virtual reality device to watch the screen, and improve the user experience.

[0209] It should be noted that the virtual reality display device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the virtual reality display device provided in the above embodiment and the virtual reality display method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0210] Figure 6 The block diagram of a computer device 600 provided by an exemplary embodiment of the present application is shown. The computer device 600 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3), an MP4 player (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4), a laptop computer or a desktop computer. The computer device 600 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0211] Typically, the computer device 600 includes a processor 601 and a memory 602 .

[0212] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0213] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the virtual reality display method provided in the method embodiment of the present application.

[0214] In some embodiments, the computer device 600 further includes some other components 603, and the type and quantity of the other components 603 can be selected based on the functional requirements of the computer device 600. Those skilled in the art will appreciate that Figure 6 The structure shown in the figure does not constitute a limitation on the computer device 600, and the computer device 600 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0215] Optionally, the computer readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid state drive (SSD), or an optical disk. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM). The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0216] An embodiment of the present application also provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the virtual reality display method as described in any of the above embodiments of the present application.

[0217] An embodiment of the present application also provides a computer-readable storage medium, wherein the storage medium stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a virtual reality display method as described in any of the above embodiments of the present application.

[0218] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the virtual reality display method described in any of the above embodiments.

[0219] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0220] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A virtual reality display system, characterized in that: The virtual reality display system includes a virtual reality device, a vehicle-mounted camera component, and a vehicle-mounted terminal; The vehicle-mounted camera assembly is configured to collect a road condition image, wherein the road condition image is used to indicate the road condition of the road on which the first vehicle is located; Sending the road condition image to the vehicle-mounted terminal; The vehicle-mounted terminal is configured to identify the received road condition image to obtain an identification result, wherein the identification result is used to indicate the type of obstacle contained in the road condition image; Analyze the recognition result to obtain loading delay compensation data; wherein the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute the virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; send the loading delay compensation data and the recognition result to the virtual reality device; The virtual reality device is configured to receive the loading delay compensation data and the recognition result; obtain loading displacement data based on the recognition result, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen; and display the virtual reality screen based on the loading delay compensation data and the loading displacement data.

2. The system according to claim 1, characterized in that The vehicle-mounted terminal is further configured to perform target detection on the road condition image through a target detection model to obtain a target detection result, wherein the target detection result is used to indicate the presence of an obstacle in the road condition image; When the target detection result indicates that an obstacle exists in the road condition image, obtaining the recognition result based on the target detection result; The recognition result includes the first position of the obstacle in the road condition image and the obstacle type corresponding to the obstacle.

3. The system according to claim 2, characterized in that The vehicle-mounted terminal is further configured to determine a first distance between the obstacle and the first vehicle based on the first position and the position of the first vehicle; and determine the loading delay compensation data based on the first distance and the vehicle speed of the first vehicle.

4. The system according to claim 3, characterized in that The vehicle-mounted camera assembly is further configured to collect the road condition image in real time based on a preset frequency; and send the road condition image to the vehicle-mounted terminal based on the preset frequency; The vehicle-mounted terminal is further configured to update the first distance based on the road condition image received in real time to obtain an updated first distance; and obtain first delay data based on the updated first distance and the vehicle speed of the first vehicle; Obtaining a first unit time length consumed by the vehicle-mounted camera for a single acquisition of the road condition image and sending the image to the vehicle-mounted terminal; The loading delay compensation data is determined based on the first delay data and the first unit time length.

5. The system according to claim 2, characterized in that The virtual reality device is further configured to determine a displacement type based on the obstacle type in the recognition result, the displacement type being used to indicate a direction for adjusting a display position of the at least one element in the virtual reality screen; and obtain a loading displacement comparison table and pixel ratio data of the obstacle in the road condition image; The loading displacement data is determined from the loading displacement comparison table based on the pixel ratio data and the displacement type.

6. The system according to any one of claims 1 to 5, characterized in that: The vehicle-mounted camera assembly is further configured to preprocess the road condition image to obtain a preprocessed road condition image, wherein the preprocessed road condition image meets a preset image processing format requirement; Sending the preprocessed road condition image to the vehicle-mounted terminal; The vehicle-mounted terminal is further configured to receive the pre-processed road condition image; and recognize the pre-processed road condition image to obtain the recognition result.

7. A virtual reality display method, characterized in that: The method comprises: Receiving a road condition image, where the road condition image is an image captured by a vehicle-mounted camera assembly of a first vehicle, and the road condition image is used to indicate a road condition of a road on which the first vehicle is located; Recognize the received road condition image to obtain a recognition result, where the recognition result is used to indicate the type of obstacle contained in the road condition image; Analyzing the recognition result to obtain loading delay compensation data; Sending the loading delay compensation data and the recognition result to a virtual reality device; Among them, the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in the virtual reality screen, and the at least one element is used to constitute a virtual scene displayed by the virtual reality screen and provide virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains the loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

8. A virtual reality display device, characterized in that: The device comprises: A receiving module, used for receiving a road condition image, wherein the road condition image is an image captured by a vehicle-mounted camera assembly of a first vehicle, and the road condition image is used for indicating a road condition of a road on which the first vehicle is located; An identification module, used to identify the received road condition image to obtain an identification result, wherein the identification result is used to indicate the type of obstacle contained in the road condition image; An analysis module, used for analyzing the recognition result to obtain loading delay compensation data; A sending module, used for sending the loading delay compensation data and the recognition result to a virtual reality device; wherein the loading delay compensation data is used to indicate the moment when the virtual reality device displays at least one element in a virtual reality screen, and the at least one element is used to constitute a virtual scene displayed by the virtual reality screen and provide a virtual reality perception to the user; after receiving the recognition result, the virtual reality device obtains loading displacement data based on the recognition result, and displays the virtual reality screen based on the loading delay compensation data and the loading displacement data, and the loading displacement data is used to indicate the display position of the at least one element in the virtual reality screen.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the virtual reality display method according to claim 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the virtual reality display method as claimed in claim 7.

Images