Virtual reality application method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a virtual reality application method, apparatus, device, and computer-readable storage medium. The method comprises the following steps: upon receiving a virtual reality driving mode switch command, obtaining a real scene of the vehicle's environment, the real scene including a real environment map and real obstacles; converting the real scene into a virtual reality scene; and displaying the virtual reality scene. This invention can prevent driver fatigue caused by long-term driving and ensure driving safety.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality, and in particular to a virtual reality application method, device, equipment and computer-readable storage medium. Background Art

[0002] At present, the research on VR (Virtual Reality) and AR (Augmented Reality) technologies is developing rapidly both at home and abroad. They have been applied to games, education, medical care, tourism, sales and other fields, playing an important role in various fields and influencing the development direction of various industries.

[0003] In the application scenario of car driving, when the driver is driving long distances on the highway, due to the monotonous road scene, he is prone to fatigue and decreased attention, which affects driving safety. Summary of the Invention

[0004] The present invention provides a virtual reality application method, device, equipment and computer-readable storage medium, aiming to solve the technical problem that drivers are prone to fatigue during long-term driving, thereby affecting driving safety.

[0005] To achieve the above object, the present invention provides a virtual reality application method, which comprises the following steps:

[0006] If a virtual reality driving mode switching command is received, the actual scene of the vehicle's environment is obtained;

[0007] Converting the actual scene into a virtual reality scene;

[0008] The virtual reality scene is displayed.

[0009] Optionally, the actual scene includes an actual environment map to obtain location information of the vehicle;

[0010] An actual environment map is created based on the location information and pre-stored map information.

[0011] Optionally, the actual scene includes actual obstacles, and obstacles in the environment where the vehicle is located are identified;

[0012] The obstacle recognition result is optimized to obtain the actual obstacle.

[0013] Optionally, the actual environment map is converted into a virtual reality map according to a preset conversion rule, and the actual obstacles are converted into virtual reality obstacles.

[0014] Optionally, acquiring image information of the environment directly in front of the vehicle;

[0015] The image information is compared with the actual environment map, and if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the driving mode is switched to the virtual reality mode.

[0016] Optionally, the image information is compared with the actual environment map at intervals of a preset time length;

[0017] If the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold, the virtual reality mode is exited.

[0018] Optionally, areas in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold are marked.

[0019] Optionally, to achieve the above objectives, the present application further proposes a virtual reality device, comprising a scene acquisition module, a scene conversion module, and a display module. The scene acquisition module is configured to acquire the actual scene of the vehicle's environment upon receiving a virtual reality driving mode switching instruction; the scene conversion module is configured to convert the actual scene into a virtual reality scene; and the display module is configured to display the virtual reality scene.

[0020] To achieve the above objectives, the present application also proposes a virtual reality device, which includes a memory, a processor, and a virtual reality application stored in the memory and executable on the processor. When the virtual reality application is executed by the processor, the virtual reality application method is implemented.

[0021] To achieve the above objectives, the present application also proposes a computer-readable storage medium, on which a virtual reality application is stored. When the virtual reality application is executed by a processor, the virtual reality application method is implemented.

[0022] In the technical solution of the present invention, upon receiving a command to switch to a virtual reality driving mode, the actual scene of the vehicle's environment is obtained, including a map of the actual environment and actual obstacles; the actual scene is converted into a virtual reality scene; and the virtual reality scene is displayed. The existing technology in this field does not provide a virtual reality driving mode. Therefore, compared with the existing technology, the present invention can convert the actual scene into a virtual reality scene and display it. This scene conversion improves driving enjoyment, avoids driver fatigue, decreased attention, and distraction caused by long-term driving, and ensures driving safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 This is a schematic diagram of the module structure of a virtual reality application method according to an embodiment of the present invention;

[0025] Figure 2 A flowchart of a virtual reality application method according to an embodiment of the present invention;

[0026] Figure 3 FIG. 4 is a module structure diagram of a virtual reality application method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0028] Please refer to Figure 1 , Figure 1 Schematic diagram of the hardware structure of the virtual reality device provided in each embodiment of the present invention. The virtual reality device includes an execution module 01, a memory 02, a processor 03, a battery system and other components. It can be understood by those skilled in the art that Figure 1 The device shown in FIG may also include more or fewer components than shown, or combine certain components, or arrange the components differently. The processor 03 is connected to the memory 02 and the execution module 01, respectively. The memory 02 stores a virtual reality application, which is simultaneously executed by the processor 03.

[0029] The execution module 01 can obtain the actual scene of the vehicle's environment; convert the actual scene into a virtual reality scene, and feed back the above information to the processor 03.

[0030] Memory 02 can be used to store software programs and various data. Memory 02 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function, while the data storage area may store data or information generated based on the use of the IoT terminal. Memory 02 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0031] Processor 03 is the control center of the processing platform. It connects the various components of the entire IoT terminal using various interfaces and lines. By running or executing software programs and / or modules stored in memory 02 and accessing data stored in memory 02, it performs various functions of the IoT terminal and processes data, thereby providing overall monitoring of the virtual reality device. Processor 03 may include one or more processing units; preferably, processor 03 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 03.

[0032] Those skilled in the art will understand that Figure 1 The virtual reality device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0033] Based on the above hardware structure, various embodiments of the method of the present invention are proposed.

[0034] Reference Figure 2 In a first embodiment of the virtual reality application method of the present invention, the virtual reality application method includes:

[0035] Step S100: If a virtual reality driving mode switching instruction is received, the actual scene of the vehicle's environment is obtained;

[0036] When driving, especially when driving long distances on highways, drivers need to concentrate highly for a long time. Because the road scenes are monotonous, drivers are prone to fatigue, decreased attention or lack of concentration, which can induce traffic accidents and affect driving safety.

[0037] To address the aforementioned issues, this application combines driving with VR (Virtual Reality) / AR (Augmented Reality) technologies. A virtual reality driving mode and a virtual reality device for implementing this virtual reality driving mode are proposed. The virtual reality device comprises at least a head-mounted VR device and an information collection system deployed on the vehicle, as well as a mapping system, a safety system, and a virtual reality conversion system deployed in the cloud. The head-mounted VR device displays a virtual reality scene for the driver when the driving mode switches to virtual reality. The information collection system comprises at least a camera, a GPS positioning system, and a millimeter-wave radar for locating the vehicle and collecting image and radar information of the vehicle's environment. The mapping system creates a map of the actual environment based on the vehicle's location information and pre-stored map information, and converts the actual environment map into a virtual reality map. The safety system determines whether the actual environment matches the map information stored in the mapping system and, based on the determination, takes appropriate measures to prevent traffic accidents caused by positioning deviations or road changes, thereby ensuring driver safety. The virtual reality conversion system projects the virtual reality map onto the head-mounted VR device and replaces identified obstacles with virtual reality images within the virtual reality map.

[0038] In this embodiment, the driver can use a mode switch button installed on the vehicle to generate a VR driving mode switch command to switch the driving mode to VR driving mode, or issue a voice command to generate a VR driving mode switch command to switch the driving mode to VR driving mode. After the driving mode is switched to VR driving mode, the vehicle's information collection system begins to operate, using cameras, radar, and other equipment to collect information about the vehicle's actual environment, and then generates a real-world scene based on this real-world environment information. The real-world scene includes a real-world environment map and real-world obstacles.

[0039] Step S200, converting the actual scene into a virtual reality scene;

[0040] In this embodiment, the virtual reality conversion system is deployed in the cloud and connected to the information collection system. After the information collection system obtains actual obstacle information about real obstacles, it transmits this information to the virtual reality conversion system in the cloud. The virtual reality conversion system converts the real obstacles into virtual reality obstacles based on preset conversion rules and the actual obstacle information. Furthermore, the mapping system creates a map of the real environment based on the vehicle's location information and pre-stored map information, and converts this map into a virtual reality map. This completes the conversion of the real scene into a virtual reality scene. The preset conversion rules are set in advance by those skilled in the art based on their needs and can be adjusted at any time to meet specific circumstances.

[0041] Step S300: displaying the virtual reality scene.

[0042] In this embodiment, the vehicle's head-mounted VR device is connected to a cloud-based VR conversion system, which is in turn connected to a mapping system. After scene conversion is complete, the cloud-based VR conversion system retrieves the VR map from the mapping system and displays it through the VR head-mounted device. Therefore, as long as the driver puts on the VR head-mounted device, they see the VR map while driving. The VR conversion system then places the converted VR obstacles on the VR map, creating a complete VR scene. Specifically, the VR scene can be space-themed, anime-themed, and so on. Because the VR environment offers a rich variety of scenes and can be adjusted to suit user needs, it can effectively reduce driver fatigue and sleepiness while driving, enhancing driving safety.

[0043] In one embodiment, the step of obtaining the actual scene of the vehicle's environment includes:

[0044] Get the vehicle's location information;

[0045] An actual environment map is created based on the location information and pre-stored map information.

[0046] The information collection system is connected to the map system, which includes a GPS positioning system (Global Positioning System), at least one millimeter-wave radar, and multiple cameras, wherein the cameras can capture 360-degree coverage of the vehicle environment. In this embodiment, the actual scene includes a map of the actual environment. The map system pre-stores pre-stored map information collected in advance by the virtual reality equipment operation and maintenance personnel. After receiving the vehicle location information obtained by the GPS positioning system, the map system can establish a map of the actual environment based on the vehicle's location information and pre-stored map information. Through the location information and pre-stored map information, an accurate map of the actual environment can be quickly established, which strongly supports the operation of the virtual reality mode.

[0047] In one embodiment, the step of obtaining the actual scene of the vehicle's environment further includes:

[0048] Identify obstacles in the vehicle's environment;

[0049] The obstacle recognition result is optimized to obtain the actual obstacle.

[0050] In this embodiment, the actual scene also includes actual obstacles. A camera can capture images of the vehicle's surroundings, and a millimeter-wave radar can transmit and receive radio waves to accurately measure radar information, such as the distance, angle, and relative speed between the vehicle and surrounding vehicles and obstacles. After acquiring image and radar information about the vehicle's surroundings, the information collection system uploads these images and radar information to a cloud-based virtual reality conversion system.

[0051] The virtual reality conversion system receives image information and radar information sent by the information collection system. Specifically, after receiving the image information of the surrounding environment captured by the camera, the virtual reality conversion system performs image recognition on the image information to identify the actual obstacles in the image. Among them, actual obstacles include dynamic obstacles and static obstacles; dynamic obstacles include passing vehicles, and static obstacles include roadblocks, potholes, etc. on the road. After identifying the obstacle information of the actual obstacles, the information collection system will further calculate the relative position, distance and relative speed of the vehicle and the surrounding actual obstacles based on the radar information sent by the millimeter-wave radar to optimize the identification results of the actual obstacles. Through multiple cameras and millimeter-wave radars, the position, speed and distance of the actual obstacles from the vehicle can be accurately determined, providing strong support for the conversion of actual scenes into virtual reality scenes.

[0052] In one embodiment, the step of converting the actual scene into a virtual reality scene includes:

[0053] The actual environment map is converted into a virtual reality map according to a preset conversion rule, and the actual obstacles are converted into virtual reality obstacles.

[0054] In this embodiment, after establishing the actual environment map and identifying the actual obstacles, the mapping system will convert the actual environment map into a virtual reality map according to the preset conversion rules. Figure 1A one-to-one match ensures that the road orientation and direction in the virtual reality map are identical to the actual environment map. However, the background style of the scene in the virtual reality map can be switched to an interesting style, such as anime or space background. The specific conversion style is pre-set by those skilled in the art and can be adjusted by the user based on actual needs. After the mapping system converts the actual environment map into a virtual reality map, the virtual reality conversion system also converts actual obstacles into virtual reality obstacles. In some embodiments, the virtual reality conversion system converts identified surrounding vehicles into running animals or spaceships of equal size, and replaces static obstacles with static animated characters, based on preset conversion rules. After the conversion, the virtual reality conversion device displays the virtual reality map on the driver's head-mounted VR device and inputs the virtual reality obstacles into the head-mounted VR device, displaying them on the virtual reality map. The preset conversion rules are pre-set by those skilled in the art based on conversion technology and can be adjusted based on the specific actual environment and obstacle type of the vehicle. By converting the actual environment map into a virtual reality map, converting actual obstacles into virtual reality obstacles, and displaying both the virtual reality map and virtual reality obstacles to the driver, the driver's visual scene can be switched during driving, preventing driver fatigue.

[0055] In one embodiment, the step of obtaining the actual scene of the vehicle's environment includes:

[0056] Acquiring image information of the environment directly in front of the vehicle;

[0057] The image information is compared with the actual environment map, and if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the driving mode is switched to the virtual reality mode.

[0058] In this embodiment, while the vehicle is in motion, the GPS positioning system on the vehicle periodically uploads vehicle location information and vehicle heading information to the safety system. In some embodiments, the GPS positioning system uploads vehicle location information and vehicle heading information every 0.5 seconds. The safety system corrects the vehicle heading information uploaded by the GPS positioning system based on the two consecutive uploads of vehicle location information to obtain more accurate vehicle heading information. After determining the vehicle's location and heading information, the safety system obtains a map of the actual environment directly in front of the vehicle from a mapping system and compares the map with the image information of the environment directly in front of the vehicle captured by a camera. The driving mode is switched to virtual reality mode only when the similarity between the map and the image information of the environment directly in front of the vehicle exceeds a preset similarity threshold; otherwise, the driving mode is not switched to virtual reality mode. Specifically, the comparison between the map and the image information of the environment directly in front of the vehicle captured by the camera includes comparing the lane line image and the highest point on the ground in the image. This method ensures that the map of the actual environment is consistent with the actual map, ensuring the driver's safety when driving in virtual reality mode.

[0059] In one embodiment, the step of displaying the virtual reality scene includes:

[0060] Comparing the image information with the actual environment map at intervals of a preset duration;

[0061] If the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold, the virtual reality mode is exited.

[0062] In this embodiment, after the vehicle's driving mode switches to VR mode, to ensure driving safety in VR mode, the safety system will compare the image information captured by the vehicle's front camera with the actual environment map at preset intervals to determine whether the similarity between the image information and the actual environment map exceeds a preset similarity threshold. If the similarity between the image information and the actual environment map exceeds the preset similarity threshold, VR mode will continue; if the similarity between the image information and the actual environment map is less than or equal to the preset similarity threshold, VR mode will be exited. By continuously monitoring the similarity between the image information and the actual environment map, driving safety can be continuously guaranteed during driving in VR mode.

[0063] In one embodiment, after the step of determining that the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold, the method further comprises:

[0064] Marking the areas in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold.

[0065] In this embodiment, the virtual reality system divides the actual environment map into multiple regions. Whenever the similarity between the monitored image information and the actual environment map is less than or equal to a preset similarity threshold, the system marks the region in the actual environment map where the image information is less than or equal to the preset similarity threshold as an abnormal region. This serves to alert the developer of the virtual reality device that the actual environment map in this area needs to be updated, and to remind other drivers using virtual reality mode to exit virtual reality mode before reaching the abnormal region to ensure driving safety. Specifically, after the abnormal region is marked as an update reminder, the operation and maintenance personnel of the virtual reality device will collect scene images on-site and update the actual environment map.

[0066] In one embodiment, after receiving the virtual reality driving mode switching instruction, the step further includes:

[0067] If it is detected that the current driving section is not a highway, the driving mode will be refused to be switched to the virtual reality mode.

[0068] In this embodiment, the VR device only creates a real-world environment map for highway scenarios, so VR mode operates only on highways. Specifically, because driving on urban or rural roads is slow and crowded, entering VR mode consumes a significant amount of computation and is difficult to achieve in accuracy. Furthermore, driving on urban or rural roads is typically short. On the other hand, driving on highways is typically longer, with a more monotonous and static environment. Therefore, creating a real-world environment map and running VR mode are more suitable.

[0069] like Figure 3 As shown, the present invention further provides a virtual reality device, the virtual reality device comprising:

[0070] The scene acquisition module A10 is used to obtain the actual scene of the vehicle's environment upon receiving a virtual reality driving mode switching instruction;

[0071] A scene conversion module A20, configured to convert the actual scene into a virtual reality scene;

[0072] The display module A30 is used to display the virtual reality scene.

[0073] Optionally, the scene acquisition module A10 may be used to:

[0074] Get the vehicle's location information;

[0075] An actual environment map is created based on the location information and pre-stored map information.

[0076] Optionally, the scene acquisition module A10 may also be used to:

[0077] Identify obstacles in the vehicle's environment;

[0078] The obstacle recognition result is optimized to obtain the actual obstacle.

[0079] Optionally, the scene conversion module A20 may be used to:

[0080] The actual environment map is converted into a virtual reality map according to a preset conversion rule, and the actual obstacles are converted into virtual reality obstacles.

[0081] Optionally, the scene acquisition module A10 may also be used to:

[0082] Acquiring image information of the environment directly in front of the vehicle;

[0083] The image information is compared with the actual environment map, and if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the driving mode is switched to the virtual reality mode.

[0084] Optionally, the display module A30 can be used to:

[0085] Comparing the image information with the actual environment map at intervals of a preset duration;

[0086] If the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold, the virtual reality mode is exited.

[0087] Optionally, the display module A30 can also be used for:

[0088] Mark the areas in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold

[0089] The present invention also provides a virtual reality device, comprising a memory, a processor, and a virtual reality application stored in the memory and executable on the processor, wherein the virtual reality application is used to execute the methods described in various embodiments of the present invention.

[0090] The present invention also provides a computer-readable storage medium on which a virtual reality application is stored. The computer-readable storage medium includes a computer-readable storage medium, and the computer-readable storage medium can be Figure 1The memory may also be at least one of a ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, and an optical disk. The computer-readable storage medium includes a number of instructions for enabling an IoT terminal device with a processor (which may be a mobile phone, a computer, a server, an IoT terminal, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0091] In the present invention, the terms "first", "second", "third", "fourth" and "fifth" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0092] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0093] Although the embodiments of the present invention have been shown and described above, the scope of protection of the present invention is not limited thereto. It should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Persons skilled in the art may make changes, modifications, and substitutions to the above embodiments within the scope of the present invention, and such changes, modifications, and substitutions should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A virtual reality application method, characterized in that: Applied to highway scenarios, the method includes the following steps: If a virtual reality driving mode switching command is received, the actual scene of the vehicle's environment is obtained; Converting the actual scene into a virtual reality scene; displaying the virtual reality scene; At preset time intervals, the image information obtained of the environment directly in front of the vehicle is compared with the actual environment map for similarity; if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the virtual reality mode continues to operate; if the similarity between the image information and the actual environment map is less than or equal to the preset similarity threshold, the area in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to the preset similarity threshold is marked as an abnormal area, so as to remind the developer of the virtual reality device that the actual environment map here needs to be updated, and to remind other drivers driving in the virtual reality mode to exit the virtual reality mode in advance before reaching the abnormal area.

2. The virtual reality application method according to claim 1, wherein: The actual scene includes an actual environment map, and the step of obtaining the actual scene of the environment where the vehicle is located includes: Get the vehicle's location information; An actual environment map is created based on the location information and pre-stored map information.

3. The virtual reality application method according to claim 1, wherein: The actual scene includes actual obstacles, and the step of obtaining the actual scene of the environment where the vehicle is located further includes: Identify obstacles in the vehicle's environment; The obstacle recognition result is optimized to obtain the actual obstacle.

4. The virtual reality application method according to claim 2, wherein: The actual scene includes actual obstacles, and the step of converting the actual scene into a virtual reality scene includes: The actual environment map is converted into a virtual reality map according to a preset conversion rule, and the actual obstacles are converted into virtual reality obstacles.

5. The virtual reality application method according to claim 1, wherein: The step of obtaining the actual scene of the vehicle's environment includes: Acquiring image information of the environment directly in front of the vehicle; The image information is compared with the actual environment map, and if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the driving mode is switched to the virtual reality mode.

6. The virtual reality application method according to claim 1, wherein: After the step of determining that the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold, the following steps are included: Marking the areas in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to a preset similarity threshold.

7. A virtual reality device, characterized in that: Applied to highway scenarios, the virtual reality device includes: A scene acquisition module is used to obtain the actual scene of the vehicle's environment upon receiving a virtual reality driving mode switching instruction; A scene conversion module, used to convert the actual scene into a virtual reality scene; A display module is configured to display the virtual reality scene; at preset time intervals, image information obtained from the environment directly in front of the vehicle is compared with an actual environment map for similarity; if the similarity between the image information and the actual environment map is greater than a preset similarity threshold, the virtual reality mode continues to operate; if the similarity between the image information and the actual environment map is less than or equal to the preset similarity threshold, the area in the actual environment map where the similarity between the image information and the actual environment map is less than or equal to the preset similarity threshold is marked as an abnormal area, so as to remind developers of the virtual reality device that the actual environment map in this area needs to be updated, and to remind other drivers driving in the virtual reality mode to exit the virtual reality mode in advance before reaching the abnormal area.

8. A virtual reality device, characterized in that: The invention comprises a memory, a processor, and a virtual reality application stored in the memory and executable on the processor, wherein the virtual reality application, when executed by the processor, implements the steps of the virtual reality application method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a virtual reality application program, and when the virtual reality application program is executed by the processor, the steps of the virtual reality application method according to any one of claims 1 to 6 are implemented.

Images