Virtual picture display method and device, vehicle-mounted virtual reality equipment and storage medium

Abstract

The invention discloses a virtual picture display method and device, vehicle-mounted virtual reality equipment and a storage medium, and the method comprises the steps: receiving the environment information, sent by a vehicle, of the vehicle, determining a target sensing object according to the environment information, generating a plurality of frames of virtual pictures according to the target sensing object and a pre-obtained target scene model, and displaying the plurality of frames of virtual pictures. Multiple frames of virtual pictures are displayed according to a current display frame rate, the current display frame rate is determined according to current speed information of the vehicle and / or biological characteristic information of a current user, and the current user is a user wearing the vehicle-mounted virtual reality equipment. The display frame rate can be determined according to the vehicle speed and / or the biological characteristic information of the user, the virtual picture is displayed according to the display frame rate, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a virtual picture display method and device, a vehicle-mounted virtual reality device, and a storage medium. BACKGROUND

[0002] With the development of automobile technology, automobiles are no longer simply a means of transportation, but have evolved into a mobile space carrier integrating multiple scenarios such as life, entertainment, and work. Vehicle-mounted games are increasingly popular among users. Vehicle-mounted games allow users to play games using devices in the vehicle, thereby enriching the entertainment needs of users.

[0003] In the existing vehicle-mounted game mode, a user can wear a VR device. The device models a three-dimensional virtual scene according to environmental information collected around the vehicle and displays the three-dimensional virtual scene through the VR device, thereby improving the user's ride experience.

[0004] However, some users may feel dizzy when wearing a VR device during vehicle travel, which affects the user experience. SUMMARY

[0005] The application provides a virtual picture display method and device, a vehicle-mounted virtual reality device, and a storage medium, which can make dynamic pictures resonate with the external environment, link game pictures with the ride process, and make the pictures more rich.

[0006] In a first aspect, the application provides a virtual picture display method. The method is applied to a vehicle-mounted virtual reality device, which is communicatively connected to a vehicle. The method comprises the following steps.

[0007] Receiving environmental information of the vehicle sent by the vehicle;

[0008] Determining a target perception object according to the environmental information;

[0009] Generating a plurality of virtual pictures according to the target perception object and a pre-acquired target scene mode;

[0010] Displaying the plurality of virtual pictures according to a current display frame rate. The current display frame rate is determined according to current speed information of the vehicle and / or biological feature information of a current user. The current user is a user wearing the vehicle-mounted virtual reality device.

[0011] In an embodiment of the application, the environmental information includes a plurality of images collected in the driving direction of the vehicle.

[0012] The determining of the target perception object according to the environmental information comprises the following steps.

[0013] perform target detection on each of the plurality of images to obtain a first result of each of the plurality of images, the first result including at least one perceived object, or the first result including a first indication indicating that no perceived object is included in the corresponding image;

[0014] delete the first result including the first indication from the first results of the plurality of images to obtain first results of N images, N being a positive integer;

[0015] determine the perceived objects in the first results of the N images as the target perceived objects.

[0016] In an embodiment of the present application, the generating of the plurality of virtual pictures according to the target perceived objects and the pre-acquired target scene mode comprises:

[0017] acquiring an identifier of each of the target perceived objects in the N images;

[0018] for each of the target perceived objects, acquiring a pre-configured first mapping relationship corresponding to the target scene mode, the first mapping relationship including an identifier of at least one perceived object and a virtual object corresponding to each identifier;

[0019] taking the virtual object corresponding to the identifier of the target perceived object in the first mapping relationship as a target virtual object matching the target perceived object;

[0020] acquiring N consecutive base pictures corresponding to the target scene mode;

[0021] rendering the plurality of target virtual objects in the N base pictures to generate the plurality of virtual pictures.

[0022] In an embodiment of the present application, the generating of the plurality of virtual pictures according to the rendering of the plurality of target virtual objects in the N base pictures comprises:

[0023] sorting the N images according to timestamps to obtain sorted N images;

[0024] for each of the target perceived objects in the sorted N images, determining second coordinate information of the target virtual object matching the target perceived object according to first coordinate information of the target perceived object in the corresponding image, the second coordinate information being coordinate information of the target virtual object in a base picture corresponding to the target virtual object in the N base pictures;

[0025] According to the second coordinate information of the target virtual object, a preset zooming ratio, and a pose of the target virtual object, a corresponding virtual picture in the image is rendered in a corresponding base picture, to generate N frames of the virtual picture corresponding to each frame of the image.

[0026] In an embodiment of the present application, before the step of generating the plurality of frames of the virtual picture according to the target perception object and the pre-acquired target scene mode, the method further comprises:

[0027] displaying an initialization picture, the initialization picture comprising a plurality of scene modes;

[0028] receiving a first input;

[0029] in response to the first input, determining the target scene mode.

[0030] In an embodiment of the present application, the step of displaying the plurality of frames of the virtual picture according to the current display frame rate comprises:

[0031] determining the current display frame rate according to current speed information of the vehicle, the speed information comprising at least one of a current speed and a current acceleration;

[0032] displaying the plurality of frames of the virtual picture according to the current display frame rate.

[0033] In an embodiment of the present application, the step of displaying the plurality of frames of the virtual picture according to the current display frame rate comprises:

[0034] determining the current display frame rate according to biological feature information of the current user;

[0035] displaying the plurality of frames of the virtual picture according to the current display frame rate.

[0036] In a second aspect, embodiments of the present application provide a virtual picture display device, the device comprising:

[0037] a transceiving module configured to receive environment information of a vehicle sent by the vehicle;

[0038] a determining module configured to determine a target perception object according to the environment information;

[0039] a processing module configured to generate a plurality of frames of a virtual picture according to the target perception object and a pre-acquired target scene mode;

[0040] a display module configured to display the plurality of frames of the virtual picture.

[0041] In a third aspect, embodiments of the present application provide a vehicle-mounted virtual reality device, comprising a processor and a memory storing computer program instructions;

[0042] The processor implements the virtual picture display method of the first aspect when executing the computer program instructions.

[0043] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores computer program instructions, and the computer program instructions are executed by a processor to implement the virtual picture display method of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, and instructions in the computer program product are executed by a processor of a vehicle-mounted virtual reality device to enable the vehicle-mounted virtual reality device to execute the virtual picture display method of the first aspect.

[0045] The virtual picture display method, device, vehicle-mounted virtual reality device and storage medium provided by the embodiment combine the target perception object in the environment information of the vehicle with the scene model to generate a corresponding virtual picture, and the virtual picture dynamically changes with the change of the environment information. The vehicle driving itself is accompanied by the dynamic change of the environment outside the vehicle, and the dynamic picture can be in harmony with the external environment, so that the game picture is linked with the process of riding the vehicle, the picture is more rich, the vehicle-mounted entertainment is endowed with more rich scenes, and the use feeling of the user is improved. Moreover, the display frame rate can be determined according to the vehicle speed and / or the biological feature information of the user, the virtual picture is displayed according to the display frame rate, and the user experience is improved. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0047] Figure 1 is a flowchart of the virtual picture display method provided by the embodiment of the present application;

[0048] Figure 2 is a structural schematic diagram of the virtual picture display device provided by the embodiment of the present application;

[0049] Figure 3 is a structural schematic diagram of the vehicle-mounted virtual reality device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0050] The features and exemplary embodiments of the various aspects of the present application will be described in detail below with reference to the drawings. For the purpose of clarity, technical terms in the description of the present application are to be interpreted as commonly understood by one of ordinary skill in the art. Commonly used reference books and handbooks, such as the Merriam-Webster Dictionary of English Words, the Oxford Dictionary of Biochemistry and Molecular Biology, and the Oxford Dictionary of Physics, can be referred to for general terms used in the present application. The specific embodiments described herein are intended to explain the present application by providing a description of the above-mentioned technical solutions. The present application is not limited to the specific embodiments described herein.

[0051] It should be noted that, in this document, the terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between such entities or operations. Also, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. Without more limitations, the elements defined by the statement "comprise" do not exclude the presence of additional identical elements in the process, method, article or device that includes the elements.

[0052] In various specific embodiments of the present application, when relevant processing is required according to user information such as user biometric information, user behavior data, user historical data, and user location information, etc. related to user identity or characteristics, the user's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the separate permission or separate consent of the user will be obtained through a pop-up window or by jumping to a confirmation page, and after obtaining the separate permission or separate consent of the user, the necessary user-related data for the normal operation of the embodiments of the present disclosure will be obtained.

[0053] The vehicle can be a private car, such as a sedan, an SUV, an MPV, or a pickup truck, etc. The vehicle can also be an operating vehicle, such as a van, a bus, a small truck, or a large trailer, etc. The vehicle can be a gasoline car or a new energy car. When the vehicle is a new energy car, it can be a hybrid car or a pure electric car.

[0054] To solve the problems in the prior art, the embodiments of the present application provide a virtual picture display method and device, a vehicle-mounted virtual reality device, and a storage medium. First, the virtual picture display method provided by the embodiments of the present application will be introduced.

[0055] Figure 1 A flowchart of a virtual picture display method provided by an embodiment of the application is shown. As shown in the figure, Figure 1 The virtual picture display method provided by the embodiment of the application is applied to a vehicle-mounted virtual reality device, the vehicle-mounted virtual reality device has a communication connection with a vehicle, and the method comprises the following steps 101-104, wherein:

[0056] Step 101: receiving environment information of the vehicle sent by the vehicle.

[0057] In the method of the embodiment, the vehicle-mounted virtual reality device has a communication connection with the vehicle, and the vehicle-mounted virtual reality device can be arranged in the vehicle. The vehicle-mounted virtual reality device establishes a data transmission channel with the vehicle through Bluetooth or Wi-Fi, and receives the environment information of the vehicle sent by the vehicle.

[0058] Step 102: determining a target perception object according to the environment information.

[0059] In the embodiment, the environment information comprises multiple frames of images, and the multiple frames of images are multiple frames of images collected in the driving direction of the vehicle. The target perception object is determined from the multiple frames of images.

[0060] Step 103: generating multiple frames of virtual pictures according to the target perception object and a pre-acquired target scene mode.

[0061] In the embodiment, the pre-acquired target scene mode can be specified by a user or be a pre-set initial scene mode. The initial scene mode is taken as the pre-acquired target scene mode, and the multiple frames of virtual pictures are generated according to the multiple target perception objects and the pre-acquired target scene mode.

[0062] Step 104: displaying the multiple frames of virtual pictures according to a current display frame rate, the current display frame rate being determined according to current speed information of the vehicle and / or biological feature information of a current user, the current user being a user wearing the vehicle-mounted virtual reality device.

[0063] In the embodiment, the current display frame rate is determined according to the current speed information of the vehicle and / or the biological feature information of the current user. The vehicle-mounted virtual reality device displays the multiple frames of virtual pictures according to the current display frame rate. The vehicle-mounted virtual reality device comprises a display, and the multiple frames of virtual pictures are displayed through the display. The user wears the vehicle-mounted virtual reality device, and watches the virtual pictures through the display.

[0064] In this embodiment, the vehicle sends the environment information in which the vehicle is located, the target perception object is determined according to the environment information, the target scene model is pre-acquired, and the multi-frame virtual picture is generated according to the target perception object and the target scene model. The target perception object in the environment information in which the vehicle is located is combined with the scene mode to generate a corresponding virtual picture. The virtual picture dynamically changes with the change of the environment information. The vehicle driving itself is accompanied by the dynamic change of the environment outside the window. The dynamic picture can be in harmony with the external environment. The game picture is linked with the process of riding the vehicle. The picture is more abundant. The vehicle-mounted entertainment is endowed with a more abundant scene. The use of the user is improved. Moreover, the display frame rate can be determined according to the vehicle speed and / or biological feature information of the user. The virtual picture is displayed according to the display frame rate. The user experience is improved.

[0065] In an embodiment of the present application, the environment information includes a plurality of images collected by the vehicle in the driving direction;

[0066] The target perception object is determined according to the environment information, including:

[0067] Target detection is performed on each of the plurality of images to obtain a first result of each image. The first result includes at least one detected perception object, or the first result includes a first indication indicating that the corresponding image does not include a perception object.

[0068] The first result including the first indication in the first result of the plurality of images is deleted to obtain a first result of N images. N is a positive integer.

[0069] The perception object in the first result of the N images is determined as the target perception object.

[0070] In this embodiment, the environment information includes a plurality of images collected by the vehicle in the driving direction. The vehicle-mounted virtual reality device performs target detection on each of the plurality of images to obtain a first result of each image. The first result includes at least one detected perception object, or the first result includes a first indication indicating that the corresponding image does not include a perception object. A plurality of perception objects are obtained. The perception objects can be traffic lights, buildings and the like around the vehicle.

[0071] Optionally, the target detection can use the YOLO detection algorithm to obtain the first result of each image. The first result includes at least one detected perception object, and can also include a detection box of the perception object, a category of the perception object and coordinate information of the perception object. The coordinate information can be pixel coordinate information. Alternatively, there is no person, vehicle or obstacle in the driving direction of the vehicle. The first result includes a first indication indicating that the corresponding image does not include a perception object.

[0072] Further, the first detection result including the first indication in the first result of the plurality of images is deleted to obtain the first result of N frames of the images, each frame of image corresponding to a first result, and further, the perception object in the first result of the N frames of images is determined as a target perception object.

[0073] In combination with the perception object outside the vehicle recognized in the plurality of images, the game picture dynamically changes with the real journey, so that the game is closely related to the journey, and the user experience is improved.

[0074] In an embodiment of the present application, the generation of the plurality of virtual pictures according to the target perception object and the pre-acquired target scene mode includes:

[0075] An identifier of each target perception object in N frames of images is acquired.

[0076] For each target perception object, a pre-configured first mapping relationship corresponding to the target scene mode is acquired, and the first mapping relationship includes an identifier of at least one perception object and a virtual object corresponding to each identifier.

[0077] The virtual object corresponding to the identifier of the target perception object in the first mapping relationship is taken as a target virtual object matched with the target perception object.

[0078] In the plurality of base pictures corresponding to the target scene mode, N frames of continuous base pictures are acquired.

[0079] According to the rendering of the plurality of target virtual objects in the N frames of base pictures, the plurality of virtual pictures are generated.

[0080] In the present embodiment, the identifier of each target perception object, such as the name of the target perception object, is acquired, and the corresponding first mapping relationship is pre-configured for each scene mode, and the virtual object corresponding to the same target perception object in different scene modes can be different or the same.

[0081] For each target perception object, a pre-configured first mapping relationship corresponding to the target scene mode is acquired, and the first mapping relationship includes an identifier of at least one perception object and a virtual object corresponding to the identifier, and the virtual object corresponding to the identifier of the target perception object in the first mapping relationship is taken as a target virtual object matched with the perception object.

[0082] Further, in the plurality of base pictures corresponding to the target scene mode, N frames of continuous base pictures are acquired, and according to the rendering of the plurality of target virtual objects in the N frames of base pictures, the plurality of virtual pictures are generated.

[0083] In combination with multi-frame image recognition of an out-of-vehicle scene, when a vehicle passes through a forest road, the game picture synchronously displays trees, the virtual picture can be more abundant by combining the perceived object in the actual scene with the target scene mode.

[0084] In an embodiment of the present application, the rendering of the target virtual objects in the N frames of the base pictures to generate the multi-frame virtual pictures comprises:

[0085] The N frames of the images are sorted according to the time stamps to obtain the sorted N frames of the images;

[0086] For each target perceived object corresponding to each frame of the sorted N frames of the images, the second coordinate information of the target virtual object matched by the target perceived object is determined according to the first coordinate information of the target perceived object in the corresponding image, and the second coordinate information is the coordinate information of the target virtual object in the corresponding base picture in the N frames of the base pictures;

[0087] According to the second coordinate information of the target virtual object, the preset scaling ratio and the posture of the target virtual object, the virtual picture corresponding to each frame of the N frames of the images is rendered in the corresponding base picture.

[0088] In the embodiment, in order to make the generated virtual picture more consistent with the appearance order of the objects in the actual environment, the target virtual object matched by the target perceived object can be inserted into the N frames of the base pictures according to the appearance order, and the N frames of the images are sorted according to the time stamps to obtain the sorted N frames of the images.

[0089] Further, each target perceived object corresponding to each frame of the sorted N frames of the images is processed as follows: according to the first coordinate information of the target perceived object in the corresponding image, the first coordinate information is pixel coordinate information, the first coordinate information is converted to obtain converted coordinate information, and the converted coordinate information is used as the second coordinate information of the target virtual object matched by the target perceived object.

[0090] Alternatively, the coordinate information in the image and the coordinate information in the VR display screen are determined, and the correspondence between the two is established, the coordinate information of the target virtual object matched by the target perceived object in the VR display screen is determined according to the first coordinate information of the target perceived object in the corresponding image and the correspondence, and the coordinate information is used as the second coordinate information of the target virtual object matched by the target perceived object.

[0091] The second coordinate information is coordinate information of the target virtual object in a corresponding base picture in N frames of base pictures. One frame of N frames of images corresponds to one frame of N frames of base pictures. A first frame of images in the N frames of images corresponds to a first frame of base pictures in the N frames of base pictures. The second coordinate information of the target virtual object can be coordinate information of a center point of the target virtual object.

[0092] The virtual pictures corresponding to each frame of the N frames of images are generated by rendering, according to the second coordinate information of the target virtual object, a preset scaling ratio, and a pose of the target virtual object in a corresponding base picture. Optionally, when rendering, information such as a building floor and a river in the navigation information can also be added to the virtual picture.

[0093] Based on the actual perception object, the virtual object is obtained, and based on the virtual object, the virtual picture is rendered. The virtual picture is changed from a relatively single game picture in the past, and the virtual experience of a user can be improved by displaying a more abundant virtual picture.

[0094] In an embodiment of the present application, before the step of generating a plurality of frames of virtual pictures according to the target perception object and a pre-acquired target scene mode, the method further includes:

[0095] displaying an initialization picture, the initialization picture including a plurality of scene modes;

[0096] receiving a first input;

[0097] in response to the first input, determining the target scene mode.

[0098] In the embodiment, the vehicle-mounted virtual reality device displays an initialization picture, the initialization picture including a plurality of scene modes. The scene modes can be game scene modes. The vehicle-mounted virtual reality device includes a handle, and the scene modes are selected by the handle. The vehicle-mounted virtual reality device receives a first input, and in response to the first input, determines the target scene mode. Alternatively, the user inputs by language, the vehicle-mounted virtual reality device receives a first input, and in response to the first input, determines the target scene mode.

[0099] A single scene mode is easy to cause fatigue of a user. By setting different scene modes for the user to select, each scene mode can be combined with a perception object, and the problem of single experience is solved by using multiple scene modes.

[0100] In an embodiment of the present application, the step of displaying a plurality of frames of virtual pictures according to a current display frame rate includes:

[0101] determining the current display frame rate according to current speed information of the vehicle, the speed information including at least one of a current speed and a current acceleration;

[0102] displaying a plurality of frames of the virtual picture according to the current display frame rate.

[0103] The frame rate of the vehicle-mounted virtual reality device refers to the number of pictures refreshed per second on the display screen, that is, the number of virtual pictures. The current speed information of the vehicle includes at least one of the current speed and the current acceleration. According to the current speed information of the vehicle, the current display frame rate is determined. For example, if the current speed is greater than a preset speed, a first preset frame rate is taken as the current display frame rate; if the current speed is less than or equal to the preset speed, a second preset frame rate is taken as the current display frame rate, and the second preset frame rate is less than the first preset frame rate. The speed is proportional to the display frame rate. When the total number of frames is fixed, the higher the playback frame rate, the faster the content presented by the vehicle-mounted virtual reality device appears.

[0104] Optionally, a vibration feedback device can be added to the vehicle seat. The device closely cooperates with the target scene mode. When there is a collision, a character is injured, or a specific game event occurs in the picture corresponding to the target scene mode, the vibration feedback device in the seat is controlled by the vehicle-mounted virtual reality device to vibrate synchronously. Alternatively, the heating, ventilation, and air conditioning system of the vehicle is used to simulate the weather environment in the virtual picture. For example, in a hot desert scene mode, the vehicle-mounted virtual reality device controls the air conditioner to blow hot air. Alternatively, for example, in a rainy scene mode, the vehicle-mounted virtual reality device controls a water spraying device to simulate the feeling of raindrops falling on the body, thereby enhancing the immersion of the user.

[0105] Adjusting the playback frame rate of the plurality of virtual pictures according to the vehicle speed dynamically adjusts the VR experience to adapt to changes in the vehicle speed, thereby improving the virtual experience of the user.

[0106] In an embodiment of the present application, the displaying a plurality of frames of the virtual picture according to the current display frame rate comprises:

[0107] The current display frame rate is determined according to the current biological feature information of the user.

[0108] The plurality of frames of the virtual picture are displayed according to the current display frame rate.

[0109] In the embodiment, the biological feature information of the current user is obtained, and the biological feature information includes at least one of the heart rate, the eye movement rate, and the brain wave. The current display frame rate is determined according to the biological feature information of the current user. The current user is a user wearing the vehicle-mounted virtual reality device. For example, the vehicle-mounted virtual reality device is provided with a heart rate monitoring device and an eye movement rate monitoring device. The heart rate monitoring device is used to monitor the heart rate of the current user, and the current display frame rate is determined by the heart rate of the current user. The eye movement rate monitoring device is used to monitor the eye movement rate of the current user, and the current display frame rate is determined by the eye movement rate of the current user. The plurality of virtual pictures are further displayed according to the current display frame rate.

[0110] Determine the display frame rate through the biological feature information of the user, so that the VR experience is more suitable for the real-time physiological state of the user, and the comfort is improved.

[0111] In an embodiment of the present application, the biological feature information includes heart rate, and determining the current display frame rate according to the heart rate of the current user includes:

[0112] If the heart rate of the current user is greater than a preset heart rate, a first preset frame rate is taken as the current display frame rate.

[0113] If the heart rate of the current user is less than or equal to the preset heart rate, a second preset frame rate is taken as the current display frame rate, and the second preset frame rate is greater than the first preset frame rate.

[0114] In the embodiment, the preset heart rate is obtained, the heart rate of the current user is compared with the preset heart rate, if the heart rate of the current user is greater than the preset heart rate, it indicates that the heart rate is too fast, which may be because the current user has dizziness or the like, and it is not suitable to be too fast when displaying multiple virtual pictures, a third preset frame rate is taken as the current display frame rate, if the heart rate of the current user is less than or equal to the preset heart rate, it indicates that the heart rate is normal, a fourth preset frame rate is taken as the current display frame rate, and the fourth preset frame rate is greater than the third preset frame rate.

[0115] Adapt different display frame rates through the heart rate of the user, reduce the stimulation to the visual system of the passenger, thereby relieving dizziness and improving the user experience.

[0116] In an embodiment of the present application, displaying multiple virtual pictures according to the current display frame rate includes:

[0117] Determine the current display frame rate according to the current speed information of the vehicle and the biological feature information of the current user, the speed information includes at least one of the current speed and the current acceleration;

[0118] Display multiple virtual pictures according to the current display frame rate.

[0119] In the embodiment, the current display frame rate is determined according to the current speed information of the vehicle and the biological feature information of the current user, and multiple virtual pictures are displayed according to the current display frame rate, for example, the biological feature information is taken as the heart rate, a second mapping relationship is obtained, the second mapping relationship includes multiple speed information, multiple heart rates and a display frame rate corresponding to each speed information and each heart rate, the display frame rate corresponding to the current speed information of the vehicle and the heart rate of the current user is obtained from the second mapping relationship according to the current speed information of the vehicle and the heart rate of the current user, and the display frame rate is taken as the current display frame rate.

[0120] The frame rate is determined by combining the speed information and the biological feature information of the user, so that the VR experience is more suitable for the real-time physiological state of the user and the vehicle speed, and the comfort is improved.

[0121] The embodiment of the application can provide a virtual reality system, comprising: a vehicle-mounted virtual reality device and a vehicle, the vehicle-mounted virtual reality device being in communication connection with the vehicle;

[0122] The vehicle is configured to send environment information of the vehicle to the vehicle-mounted virtual reality device.

[0123] The vehicle-mounted virtual reality device is configured to receive the environment information of the vehicle sent by the vehicle.

[0124] The vehicle-mounted virtual reality device is configured to determine a target perception object according to the environment information.

[0125] The vehicle-mounted virtual reality device is configured to generate a plurality of frames of virtual pictures according to the target perception object and a pre-acquired target scene mode.

[0126] The vehicle-mounted virtual reality device is configured to display the plurality of frames of virtual pictures according to a current display frame rate, the current display frame rate being determined according to current speed information of the vehicle and / or biological feature information of a current user, the current user being a user wearing the vehicle-mounted virtual reality device.

[0127] In the embodiment, the virtual reality system comprises a vehicle-mounted virtual reality device and a vehicle, the vehicle-mounted virtual reality device being in communication connection with the vehicle, and the vehicle-mounted virtual reality device establishing a data transmission channel with the vehicle through Bluetooth or Wi-Fi. The vehicle sends environment information of the vehicle to the vehicle-mounted virtual reality device.

[0128] The vehicle-mounted virtual reality device is configured to receive the environment information of the vehicle sent by the vehicle, the environment information comprising a plurality of frames of images, the plurality of frames of images being a plurality of frames of images collected in a driving direction of the vehicle, and a plurality of target perception objects being determined from the plurality of frames of images.

[0129] The pre-acquired target scene mode can be specified by a user or be a pre-set initial scene mode. The initial scene mode is taken as the pre-acquired target scene mode, and the vehicle-mounted virtual reality device is configured to generate a plurality of frames of virtual pictures according to the plurality of target perception objects and the pre-acquired target scene mode.

[0130] The vehicle-mounted virtual reality device is used for displaying multiple frames of the virtual picture according to a current display frame rate, the vehicle-mounted virtual reality device comprises a display, the multiple frames of the virtual picture are displayed through the display, a user wears the vehicle-mounted virtual reality device, and the user watches the virtual picture through the display, wherein the current display frame rate is determined according to current speed information of the vehicle and / or biological feature information of a current user, and the current user is the user wearing the vehicle-mounted virtual reality device.

[0131] In the system in the embodiment, the vehicle-mounted virtual reality device combines the dynamic picture with the external environment through the environmental information provided by the vehicle, links the game picture with the process of taking the vehicle, makes the picture more abundant, gives the vehicle-mounted entertainment more abundant scenes, and improves the use feeling of the user.

[0132] Figure 2 A structure diagram of a virtual picture display device provided by an embodiment of the application is shown. As shown in Figure 3 The virtual picture display device 200 comprises:

[0133] A transceiving module 201 is configured to receive environmental information of a vehicle sent by the vehicle.

[0134] A determining module 202 is configured to determine a target perception object according to the environmental information.

[0135] A processing module 203 is configured to generate multiple frames of virtual pictures according to the target perception object and a pre-acquired target scene mode.

[0136] A display module 204 is configured to display the multiple frames of the virtual pictures according to a current display frame rate, wherein the current display frame rate is determined according to current speed information of the vehicle and / or biological feature information of a current user, and the current user is a user wearing the vehicle-mounted virtual reality device.

[0137] In an embodiment of the application, the determining module 202 comprises a detection sub-module, a screening sub-module and a first determining sub-module.

[0138] The detection sub-module is configured to perform target detection on each of the multiple frames of images respectively to obtain a first result of each of the multiple frames of images, wherein the first result comprises at least one perception object detected, or the first result comprises a first indication indicating that the corresponding image does not comprise a perception object.

[0139] The screening sub-module is configured to delete the first result comprising the first indication from the first results of the multiple frames of images to obtain N first results of the multiple frames of images, wherein N is a positive integer.

[0140] The first determining sub-module is configured to determine the perception objects in the N first results of the multiple frames of images as the target perception object.

[0141] In an embodiment of the present application, the processing module 203 comprises a first acquisition submodule, a processing submodule, a second determination submodule, a second acquisition submodule and a third determination submodule.

[0142] The first acquisition submodule is configured to acquire the identification of each target perceptual object in N frames of the images.

[0143] The processing submodule is configured to, for each target perceptual object, acquire a preconfigured first mapping relationship corresponding to the target scene mode, the first mapping relationship comprising the identification of at least one perceptual object and a virtual object corresponding to each identification.

[0144] The second determination submodule is configured to take the virtual object corresponding to the identification of the target perceptual object in the first mapping relationship as a target virtual object matched with the target perceptual object.

[0145] The second acquisition submodule is configured to acquire N consecutive frames of base pictures in the multiple frames of base pictures corresponding to the target scene mode.

[0146] The third determination submodule is configured to generate multiple frames of virtual pictures according to the rendering of the multiple target virtual objects in the N frames of base pictures.

[0147] In an embodiment of the present application, the third determination submodule comprises a sorting subunit, a first determination subunit and a second determination subunit.

[0148] The sorting subunit is configured to sort the N frames of images according to timestamps to obtain sorted N frames of images.

[0149] The first determination subunit is configured to, for the target perceptual object corresponding to each frame of image in the sorted N frames of images, determine second coordinate information of the target virtual object matched with the target perceptual object according to first coordinate information of the target perceptual object in the corresponding image, the second coordinate information being coordinate information of the target virtual object in a corresponding base picture in the N frames of base pictures.

[0150] The second determination subunit is configured to generate a virtual picture corresponding to each frame of image in the N frames of images according to the rendering of the second coordinate information of the target virtual object, a preset scaling ratio and a pose of the target virtual object in the corresponding base picture.

[0151] In an embodiment of the present application, the virtual picture display device comprises an input module.

[0152] The display module is further configured to display an initialization picture, the initialization picture comprising multiple scene modes.

[0153] The input module is configured to receive a first input.

[0154] The determining module is further configured to determine the target scene mode in response to the first input.

[0155] In an embodiment of the present application, the display module comprises a fourth determining sub-module and a display sub-module.

[0156] The fourth determining sub-module is configured to determine the current display frame rate according to speed information of the vehicle, wherein the speed information comprises at least one of current speed and current acceleration.

[0157] The display sub-module is configured to display a plurality of frames of the virtual picture according to the current display frame rate.

[0158] In an embodiment of the present application, the display module comprises a fifth determining sub-module.

[0159] The fifth determining sub-module is configured to determine the current display frame rate according to the biological feature information of the current user.

[0160] The display sub-module is configured to display a plurality of frames of the virtual picture according to the current display frame rate.

[0161] The virtual picture display device provided by the embodiments of the present application can realize the various processes realized by the virtual picture display method embodiments and achieve the same technical effects. To avoid repetition, details are not described herein.

[0162] Figure 3 A hardware structure schematic diagram of a vehicle-mounted virtual reality device provided by an embodiment of the present application is shown.

[0163] The vehicle-mounted virtual reality device can comprise a processor 301 and a memory 302 storing computer program instructions.

[0164] Specifically, the processor 301 can comprise a central processing unit (CPU), or an application specific integrated circuit (ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0165] The memory 302 can include mass storage for data or instructions. As an example and not by way of limitation, the memory 302 can include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disc (e.g., a compact disc (CD) or a digital versatile disc (DVD)), a solid-state drive (SSD), a USB drive, or a combination of two or more of these. Where appropriate, the memory 302 can include removable or non-removable (or fixed) media. Where appropriate, the memory 302 can be internal or external to the integrated gateway disaster recovery appliance. In particular embodiments, the memory 302 is non-volatile, solid-state memory.

[0166] The memory can include read-only memory (ROM), random-access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices, electrical, optical, or other physical / tangible memory storage devices. Thus, in general, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software that, when executed (by one or more processors), is operable to access the data and / or instructions that enable the operations described with reference to the methods according to the first or second aspects of the present disclosure.

[0167] The processor 301 implements a method as described above in the embodiments by reading and executing computer program instructions stored in the memory 302.

[0168] In one example, the vehicle-mounted virtual reality device can further include a communication interface 303 and a bus 310. As shown, the processor 301, the memory 302, and the communication interface 303 are connected through the bus 310 and complete communication with each other. Figure 3

[0169] The communication interface 303 is mainly used to realize the communication between the modules, devices, units and / or equipment in the embodiments of the present application.

[0170] ​Bus 310 includes hardware, software, or both, to couple components such as those described above with one another in a communication manner, for instance, as described herein. For example, but not by way of limitation, such communication can be global among components or local, between two components. Communication can be achieved through the use of appropriate devices and can utilize one or more bus types, which are described below. Therefore, bus 310 can include a system bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport (HTX) bus, Industry Standard Architecture (ISA) bus, a low pin count (LPC) bus, a memory bus, a video bus, a wireless bus, a serial bus, a Firewire bus, a Fiber Channel bus, a Universal Serial Bus (USB), etc., or a combination of multiple buses. Bus 310 can be implemented using any suitable bus technology, including, but not limited to, a direct copper link (DCL), a direct fiber link (DFL), a fiber channel, a serial bus, a parallel bus, a wireless bus, a serial advanced technology attachment (SATA) bus, a universal serial bus (USB) bus, a Firewire bus, a Fibre Channel bus, a Peripheral Component Interconnect (PCI) bus, a HyperTransport (HTX) bus, an Industry Standard Architecture (ISA) bus, a low pin count (LPC) bus, a memory bus, a video bus, a wireless bus, a serial bus, a Firewire bus, a Fiber Channel bus, a Universal Serial Bus (USB) bus, etc., or a combination of two or more of these. Bus 310 can include one or more buses, as appropriate. Although specific bus implementations have been described and illustrated, the application is not limited to these. The application contemplates any suitable bus or interconnect.

[0171] In addition, an embodiment of the present application can be implemented as a computer storage medium having stored thereon computer program instructions. The computer program instructions, when executed by a processor, implement any of the virtual picture display methods described above.

[0172] In addition, an embodiment of the present application can be implemented as a computer program product. The instructions in the computer program product, when executed by a processor of a vehicle virtual reality device, cause the vehicle virtual reality device to implement any of the virtual picture display methods described above.

[0173] It is to be understood that the application is not limited to the particular configurations, processing, and methodologies described herein, as these can vary. The detailed description is not to be regarded as limiting, but rather as a description of the foregoing. The specific steps of the methods described in the foregoing embodiments are described as examples. However, the method processes of the application are not limited to the specific steps described, and one of ordinary skill in the art can make various changes, modifications, and additions, or change the order of the steps, after understanding the spirit of the application.

[0174] The functions noted in the description of the structural block diagrams above can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application specific integrated circuits (ASICs), appropriate firmware, plug-ins, functional cards, and the like. When implemented in software, the elements of the present application are program or code segments that are used to perform the required tasks. The program or code segments can be stored in a machine-readable medium or transmitted through a data signal carried in a carrier wave over a transmission medium or communication link. A "machine-readable medium" includes any medium that can store or transport information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, and the like. The code segments can be downloaded via computer networks such as the Internet, intranets, and the like.

[0175] It is also important to note that the examples mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the steps mentioned above, that is, the steps can be performed in the order mentioned in the examples, or in an order different from the examples, or several steps can be performed simultaneously.

[0176] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks. The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other processing device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer program instructions can also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other processing device to operate in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0177] The above merely describes a specific implementation of the present application. Those skilled in the art can clearly understand the specific working processes of the system, modules and units described above for the convenience and brevity of description, and can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein again. It should be understood that the protection scope of the present application is not limited to this, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application.

Claims

1. A method for displaying a virtual image, characterized in that, The method is applied to an in-vehicle virtual reality device, wherein the in-vehicle virtual reality device has a communication connection with the vehicle, and the method includes: Receive environmental information about the vehicle's location sent by the vehicle; Based on the environmental information, the target sensing object is determined; Based on the target perception object and the pre-acquired target scene pattern, generate multiple frames of virtual images; Multiple frames of the virtual image are displayed according to the current display frame rate, which is determined based on the vehicle's current speed information and / or the current user's biometric information, where the current user is the user wearing the in-vehicle virtual reality device.

2. The virtual screen display method according to claim 1, characterized in that, The environmental information includes multiple frames of images captured by the vehicle in the direction of travel; The step of determining the target sensing object based on the environmental information includes: Target detection is performed on each of the multiple frames of the images to obtain a first result for each frame of the images. The first result includes at least one detected object, or the first result includes a first indication, which is used to indicate that the corresponding image does not contain a detected object. The first result including the first indication is deleted from the first result of the first result of the multiple frames of the image to obtain the first result of N frames of the image, where N is a positive integer; The perceived object in the first result of the N frames of the image is identified as the target perceived object.

3. The virtual screen display method according to claim 2, characterized in that, The step of generating multiple frames of virtual images based on the target perception object and the pre-acquired target scene pattern includes: Obtain the identifier of each target perceived object in the N frames of the image; For each of the target perception objects, a pre-configured first mapping relationship corresponding to the target scene mode is obtained. The first mapping relationship includes an identifier of at least one perception object and a virtual object corresponding to each identifier. The virtual object corresponding to the identifier of the target perception object in the first mapping relationship is taken as the target virtual object that matches the target perception object; In the multi-frame base images corresponding to the target scene mode, N consecutive frames of the base images are obtained; Multiple frames of virtual images are generated by rendering multiple target virtual objects in N frames of the base image.

4. The virtual screen display method according to claim 3, characterized in that, The step of rendering multiple virtual frames based on multiple target virtual objects in N frames of the base frame to generate multiple virtual frames includes: Sort the N frames of images according to their timestamps to obtain the sorted N frames of images; For the target perception object corresponding to each of the N frames of the sorted images, the second coordinate information of the target virtual object matched by the target perception object is determined according to the first coordinate information of the target perception object in the corresponding image. The second coordinate information is the coordinate information of the target virtual object in the basic image corresponding to the N frames of the basic image. Based on the second coordinate information of the target virtual object, the preset scaling ratio, and the posture of the target virtual object, the image is rendered in the corresponding base image to generate a virtual image corresponding to each frame of the N frames of the image.

5. The virtual screen display method according to claim 1, characterized in that, Before generating multiple frames of virtual images based on the target perception object and the pre-acquired target scene pattern, the process includes: Display an initialization screen, which includes multiple scene modes; Receive the first input; In response to the first input, the target scene mode is determined.

6. The virtual screen display method according to claim 1, characterized in that, The step of displaying multiple frames of the virtual image according to the current display frame rate includes: The current display frame rate is determined based on the vehicle's current speed information, wherein the speed information includes at least one of the current speed and the current acceleration; Display multiple frames of the virtual image based on the current display frame rate.

7. The virtual screen display method according to claim 1, characterized in that, The step of displaying multiple frames of the virtual image according to the current display frame rate includes: The current display frame rate is determined based on the current user's biometric information; Display multiple frames of the virtual image based on the current display frame rate.

8. A virtual screen display device, characterized in that, The device includes: The transceiver module is used to receive environmental information about the vehicle's location sent by the vehicle. The determination module is used to determine the target sensing object based on the environmental information; The processing module is used to generate multiple frames of virtual images based on the target perception object and the pre-acquired target scene pattern; The display module is used to display multiple frames of the virtual image.

9. A vehicle-mounted virtual reality device, characterized in that, include: Processor and memory storing computer program instructions; When the processor executes the computer program instructions, it implements the virtual screen display method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the virtual screen display method as described in any one of claims 1-7.

Images