VR video processing method, device and system, storage medium and vehicle

Abstract

The invention discloses a VR video processing method, which comprises the following steps: receiving an image of a first fixture in a vehicle shot by a camera on VR wearable equipment, receiving an accurate distance from glasses to a second fixture sensed by a close-range position sensor on the VR wearable equipment, and calculating to obtain relative position information of a wearer and the vehicle; receiving a real-time geographic position, an acceleration, a direction signal and a driving speed signal of the vehicle, and obtaining a real-time position of the current vehicle relative to the ground; according to the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground, the relative position of the wearer relative to the ground is obtained; and carrying out anti-dizziness adjustment on the current VR video picture, and outputting the current VR video picture to the VR wearable device. The invention further discloses a corresponding system and device, a storage medium and a vehicle. According to the invention, image dizziness caused by shaking of a VR glasses wearer during vehicle driving can be eliminated at low cost, and the use experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of automobile-related virtual reality (VR) glasses, and in particular to a VR video processing method, device, system, storage medium and vehicle. Background Art

[0002] VR glasses can create a new virtual scene, giving the wearer an immersive experience. VR glasses have long been used in static scenes such as living rooms, and their application in vehicles has not yet been popularized.

[0003] Currently, there are two types of VR glasses. One is split VR glasses, that is, VR glasses + VR host. VR glasses are responsible for screen display, and VR host is responsible for calculation and provides video source for VR glasses. The other is integrated VR glasses, where VR glasses and host are combined into one. The main application scenario of VR glasses is in a static external environment such as a room. After the wearer wears VR glasses, the external information perceived by the wearer's brain basically comes from the VR glasses, and the wearer is not prone to dizziness.

[0004] During the driving process, vehicles often experience road bumps, braking, acceleration, and turning. During the driving process, the wearer of VR glasses will experience shaking, but the video content of the VR glasses they see comes from the locomotive. When shaking, there will generally be a large deviation from the external position information felt by the wearer's brain. Therefore, in a moving vehicle, the wearer of VR glasses is prone to dizziness. At present, there is no good solution to reduce dizziness when using VR glasses on a moving vehicle. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a VR video processing method, device, system, storage medium and vehicle, which can eliminate the image dizziness caused by shaking of VR glasses wearers while the vehicle is driving at a low cost and improve the user experience.

[0006] In order to solve the above technical problems, as one aspect of the present invention, a VR video processing method is provided, which is applied to a smart cockpit system with integrated VR glasses function, and the method at least comprises the following steps:

[0007] The relative position information between the wearer and the vehicle is calculated based on the image of the first fixed object in the vehicle captured by the camera of the VR wearable device and the precise distance from the glasses to the second fixed object sensed by the close-range PS (Position Sensor) sensor of the VR wearable device;

[0008] According to the real-time geographic location, acceleration, direction signal and driving speed signal of the vehicle, the real-time position of the current vehicle relative to the ground is obtained;

[0009] Obtain the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground;

[0010] According to the relative position of the wearer with respect to the ground, the current VR video picture is adjusted to prevent dizziness and output to the VR wearable device.

[0011] The method further includes: calculating and obtaining relative position information of the wearer and the vehicle based on an image of a first fixed object in the vehicle captured by a camera of the VR wearable device and a precise distance from the glasses to the second fixed object sensed by a close-range position sensor of the VR wearable device;

[0012] Receiving images of a first fixed object in the vehicle taken by at least two cameras arranged on the VR wearable device;

[0013] Extracting and matching features of the image, and obtaining a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points;

[0014] Receiving the precise distance between the glasses and the second fixed object sensed by the close-range position sensor on the VR wearable device;

[0015] The relative position information between the wearer and the center point of the vehicle is obtained according to the approximate relative position and the precise distance.

[0016] Among them, further include:

[0017] The real-time positioning information of the vehicle is obtained through the GNSS module installed on the cockpit host;

[0018] The IMU sensor is used to obtain the acceleration and direction signals of the vehicle, wherein the direction signals include: the pitch, roll and yaw angles of the vehicle;

[0019] The vehicle's driving speed signal is obtained through the MCU of the cockpit host using the CAN bus;

[0020] The positioning information, acceleration and direction signals and vehicle speed signals are integrated to obtain real-time position information of the vehicle center point relative to the ground coordinate system.

[0021] Wherein, obtaining the relative position of the wearer relative to the ground according to the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground further includes:

[0022] The relative position information between the wearer and the center point of the vehicle is combined with the real-time position information of the center point of the vehicle relative to the ground coordinates to obtain the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system and the orientation information.

[0023] Among them, according to the relative position of the wearer relative to the ground, the current VR video picture is adjusted to prevent dizziness, and output to the VR wearable device, further comprising:

[0024] According to the relative position of the wearer with respect to the ground, at least one of perspective correction, visual distortion correction, motion compensation and scene depth adjustment is performed on the current VR video picture, and the adjusted VR video picture is output to the VR wearable device, so that what the wearer sees on the VR wearable device is consistent with the environmental position felt by the body.

[0025] Accordingly, another aspect of the present invention further provides a VR video processing device, which is applied to a smart cockpit system with integrated VR glasses function, and which at least includes:

[0026] A person-vehicle position information acquisition unit, configured to calculate and obtain relative position information of the wearer and the vehicle based on an image of a first fixed object in the vehicle captured by a camera of the VR wearable device and an accurate distance from the glasses to a second fixed object sensed by a short-range position sensor of the VR wearable device;

[0027] The vehicle-ground position information acquisition unit is used to receive the vehicle's real-time geographical location, acceleration, direction signal and driving speed signal to obtain the current real-time position of the vehicle relative to the ground;

[0028] A person-ground position information acquisition unit, used to obtain the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground;

[0029] The VR picture adjustment unit is used to perform anti-vertigo adjustment on the current VR video picture according to the relative position of the wearer relative to the ground, and output it to the VR wearable device.

[0030] Wherein, the person and vehicle position information acquisition unit further comprises:

[0031] A camera image receiving unit, configured to receive images of a first fixed object in the vehicle taken by at least two cameras arranged on the VR wearable device;

[0032] A matching processing unit, used to extract and match features of the image, and obtain a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points;

[0033] A precise distance receiving unit, used to receive the precise distance from the glasses to the second fixed object sensed by a close-range position sensor on the VR wearable device;

[0034] The person-vehicle relative position calculation unit is used to obtain the relative position information between the center point of the wearer and the vehicle based on the approximate relative position and the precise distance.

[0035] Wherein, the vehicle-ground position information acquisition unit further comprises:

[0036] The sensor information acquisition unit is used to obtain the real-time positioning information of the vehicle through the GNSS module set on the cockpit host; obtain the acceleration and direction signals of the vehicle by setting the IMU sensor, and the direction signals include: the pitch, roll and yaw angles of the vehicle; and obtain the driving speed signal of the vehicle through the CAN bus through the MCU of the cockpit host;

[0037] The fusion processing unit is used to fuse the positioning information, acceleration and direction signals and vehicle speed signals to obtain real-time position information of the vehicle center point relative to the ground coordinate system.

[0038] Wherein, the person-ground location information acquisition unit is further used for:

[0039] The relative position information between the wearer and the center point of the vehicle is combined with the real-time position information of the center point of the vehicle relative to the ground coordinates to obtain the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system and the orientation information.

[0040] Wherein, the VR picture adjustment unit further includes:

[0041] An adjustment unit, configured to perform at least one of perspective correction, visual distortion correction, motion compensation, and scene depth adjustment on a current VR video image according to a relative position of a wearer relative to the ground;

[0042] The sending unit is used to output the adjusted VR video picture to the VR wearable device so that the content seen by the wearer on the VR wearable device is consistent with the environmental position felt by the body.

[0043] Accordingly, another aspect of the present invention further provides a VR video processing system, comprising a cockpit host, and a VR wearable device communicating with the cockpit host; wherein:

[0044] The cockpit host is provided with at least an APU module, and a microprocessor, a first IMU sensor, a GPU and a memory connected to the APU module; wherein the APU module is provided with the aforementioned VR video processing device;

[0045] At least two LCD display units and a 3D bridge are provided in the VR wearable device, and a second IMU sensor, a PS sensor and at least two cameras are connected to the 3D bridge.

[0046] Correspondingly, another aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0047] Correspondingly, another aspect of the present invention further provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the aforementioned method when executing the computer program.

[0048] Correspondingly, another aspect of the present invention further provides a vehicle, on which a cockpit host is provided, wherein the cockpit host is deployed with the aforementioned device.

[0049] The implementation of the embodiments of the present invention has the following beneficial effects:

[0050] The present invention provides a VR video processing method, device, system, storage medium and vehicle, which saves the host cost of VR glasses by integrating the VR glasses function in the traditional cockpit host; and the VR wearable device used by the wearer is an independent VR glasses, which only serves as the display function of the VR glasses, is light in weight and easy to wear.

[0051] At the same time, in this embodiment, by setting at least two cameras on the VR wearable device to take pictures of fixed objects in the car, combined with the precise distance from the glasses to the second fixed object sensed by the close-range position sensor on the VR wearable device, the relative position information of the wearer and the vehicle can be calculated; combined with the resources of the entire vehicle (vehicle speed, acceleration and angle, positioning information), the relative position of the wearer relative to the ground can be calculated; according to the relative position of the wearer relative to the ground, the current VR video screen is adjusted to prevent dizziness and output to the VR wearable device. In this way, the inconsistency between the vision of the VR glasses and the wearer's feeling is eliminated, and the wearer is prevented from dizziness.

[0052] By implementing the present invention, a low-cost solution can be used to solve the dizziness caused by VR glasses during vehicle driving, thereby improving the user experience of using VR glasses in a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technical personnel in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0054] Figure 1 A schematic diagram of the main process of an embodiment of a VR video processing method provided by the present invention;

[0055] Figure 2 A schematic diagram of an application environment of the method provided by the present invention;

[0056] Figure 3 A structural schematic diagram of an embodiment of a VR video processing system provided by the present invention;

[0057] Figure 4 for Figure 3 A schematic diagram of the structure of a vehicle position information acquisition unit;

[0058] Figure 5 for Figure 3 A schematic diagram of the structure of the CRRC location information acquisition unit;

[0059] Figure 6 for Figure 3 Schematic diagram of the structure of the VR picture adjustment unit. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0061] like Figure 1 FIG. 1 is a schematic diagram showing the main flow of an embodiment of a VR video processing method provided by the present invention. Figure 2 As shown, in this embodiment, the VR video processing method is applied to Figure 2 In the smart cockpit system with integrated VR glasses function.

[0062] Combination Figure 2 As shown, first, the application environment schematic diagram of the present invention is described as follows:

[0063] In the present invention, the smart cockpit system with integrated VR glasses function applied by the VR video processing method mainly includes: VR wearable device, cockpit host, display screen and loudspeaker.

[0064] Among them, the VR wearable device is responsible for the display output of VR images; the cockpit host is responsible for the integration of cockpit functions and VR functions; the central control display is responsible for the central control; and the speaker is responsible for the sound output of the cockpit and VR video. It can be understood that the VR wearable device can be, for example, a split VR glasses.

[0065] The cockpit host can realize the following functions:

[0066] Boot display function: After the APU module (application processing unit) of the cockpit host is started, the boot image or animation video is called from its SSD storage chip, transmitted to the serializer chip for encryption and compression, and then transmitted to the central control display via a cable, and the display displays the picture.

[0067] Map positioning function: The GNSS positioning module (Global Navigation Satellite System module) of the cockpit host is responsible for position positioning and transmits the positioning coordinate information to the APU module. The APU module calls the map information stored in the SSD, and integrates and compares the positioning coordinate information with the map chip. Finally, the map screen is transmitted to the central control display through the serializer for display.

[0068] Sound playback function: The APU module accesses the USB flash drive, reads the audio file in the USB flash drive, decodes the audio file, and transmits it to the power amplifier chip. The power amplifier chip amplifies the audio signal and then transmits it to the speaker, which plays the sound.

[0069] In the present invention, the cockpit host is further integrated with VR function: the cockpit host generates VR video source, transmits the video to the VR wearable device through a cable, and the DP to DSI chip in the VR wearable device converts the left and right LCD lenses to present the picture to the wearer. At the same time, the cockpit host generates VR audio source and transmits it to the power amplifier module, which plays the sound through the speaker after amplifying the sound signal. The video effect of VR glasses combined with the effect of the car audio gives the wearer an immersive effect experience.

[0070] However, when the car is driving, the wearer will often shake due to inertia. The wearer wears VR glasses. Since the video images presented by the VR glasses are inconsistent with the wearer's inertia in the car, a visual contrast is generated in the VR interface, resulting in the inability to match the actual movement with the brain movement, affecting the brain's judgment and causing dizziness.

[0071] The method provided by the present invention can eliminate the dizziness caused by using a VR wearable device in a car. Specifically, in an embodiment of the present invention, the method further includes the following steps:

[0072] Step S10, calculating and obtaining the relative position information between the wearer and the vehicle based on the image of the first fixed object in the vehicle (such as the rearview mirror) captured by the camera of the VR wearable device and the precise distance from the glasses to the second fixed object (such as the front windshield) sensed by the short-range position sensor (PS sensor) of the VR wearable device;

[0073] In a specific example, the step S10 further includes:

[0074] Step S100, receiving an image of a first fixed object in the car taken by at least two cameras set on the VR wearable device; in a specific example, at least two cameras can be mounted on the VR wearable device, both arranged in front of the glasses. If two cameras are set, they are set at the upper left and upper right of the VR wearable device; the two cameras in the upper left and upper right shoot the first fixed object in the car, and transmit the captured image information to the 3D bridge of the VR wearable device, and then send the image information back to the APU of the cockpit host through a USB cable for analysis; if four cameras are set, then on the basis of the above two cameras, two more cameras are further set at the lower left and lower right of the VR wearable device. The principle of the lower left and lower right cameras is the same as that of the two cameras in the upper part, except that the four cameras can shoot at a wider angle and have more precise positioning.

[0075] Step S101, extracting and matching features of the image, and obtaining a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points; the APU module can calculate the real-time approximate relative position of the glasses (i.e., the wearer) and the vehicle;

[0076] Step S102, receiving the precise distance from the glasses to the second fixed object sensed by the short-range PS sensor on the VR wearable device; in a specific example, the short-range PS sensor on the VR wearable device can calculate the distance by measuring the light transmission time, thereby sensing the precise distance from the glasses to the second fixed object, and transmitting the result to the APU module of the cockpit host through the 3D bridge. The APU module combines the real-time relative position of the glasses (i.e., the wearer) and the vehicle obtained previously to obtain the precise relative position of the wearer and the vehicle.

[0077] Step S103, obtaining relative position information between the center point of the wearer and the vehicle according to the approximate relative position and the precise distance.

[0078] Step S11, receiving the real-time geographic location, acceleration, direction signal and driving speed signal of the vehicle to obtain the real-time position of the current vehicle relative to the ground;

[0079] In a specific example, the step S11 further includes:

[0080] The real-time positioning information of the vehicle is obtained through the GNSS module installed on the cockpit host;

[0081] The IMU sensor is used to obtain the acceleration and direction signals of the vehicle, wherein the direction signals include: the pitch, roll and yaw angles of the vehicle;

[0082] The vehicle's driving speed signal is obtained through the MCU of the cockpit host using the CAN bus;

[0083] The positioning information, acceleration and direction signals, and vehicle speed signals are integrated to obtain the real-time position information of the center point of the vehicle relative to the ground coordinate system. In this step, the APU integrates the positioning information, acceleration and direction signals, and vehicle speed signals to accurately calculate the real-time position of the vehicle relative to the ground.

[0084] Step S12, obtaining the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground;

[0085] Specifically, it includes: combining the relative position information between the wearer and the center point of the vehicle with the real-time position information of the center point of the vehicle relative to the ground coordinates, and obtaining the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system, and orientation and other information.

[0086] Step S13, according to the relative position of the wearer with respect to the ground, the current VR video picture is adjusted to prevent dizziness, and output to the VR wearable device.

[0087] In a specific example, the step S13 further includes:

[0088] According to the relative position of the wearer with respect to the ground, at least one of perspective correction, visual distortion correction, motion compensation and scene depth adjustment is performed on the current VR video picture, and the adjusted VR video picture is output to the VR wearable device, so that what the wearer sees on the VR wearable device is consistent with the environmental position felt by the body.

[0089] Specifically, in some examples, perspective correction refers to adjusting the perspective in the video image based on the wearer's relative position to the ground, so that the perspective seen by the wearer in the VR glasses corresponds to their actual body position. For example, if the wearer is tilted to the left relative to the ground, the APU module can fine-tune the perspective of the video image to match the wearer's actual tilt angle.

[0090] Visual distortion correction means that the image may be distorted due to the shape and curvature of the lenses of VR glasses. The APU can perform distortion correction based on the wearer's relative position to the ground, so that the image presents a normal geometric shape in the glasses and avoids causing dizziness. This can be achieved by applying appropriate image processing algorithms, such as distortion correction algorithms or perspective correction algorithms.

[0091] Motion compensation means that when the wearer moves or rotates relative to the ground, the APU module can perform motion compensation on the video image to maintain the stability of the image. For example, if the wearer walks forward, the APU module can perform corresponding translation and rotation operations on the image based on the speed and direction information to eliminate the instability of the image caused by the wearer's movement.

[0092] Scene depth adjustment means that according to the relative position of the wearer and the ground, the APU module can adjust the depth perception of objects in the video screen so that the position of objects observed in the VR glasses is consistent with the position actually felt by the wearer. For example, if the wearer is tilted downward relative to the ground, the APU module can move the position of the object in the video screen upward accordingly to compensate for the difference between visual and physical perception.

[0093] Through this step, the video images generated by the APU module in the cockpit host only need to be properly processed according to the relative position of the wearer to the ground to ensure that the content observed in the VR glasses is consistent with the wearer's physical feelings and environmental position, thereby reducing the occurrence of dizziness.

[0094] like Figure 3 FIG. 1 is a block diagram of an embodiment of a VR video processing device provided by the present invention, and FIG. Figure 4-6 As shown, in this embodiment, the VR video processing device 1 is applied to a smart cockpit system with integrated VR glasses function. More specifically, it is arranged at Figure 2 In the APU shown, the VR video processing device 1 at least includes:

[0095] The person-vehicle position information acquisition unit 10 is used to calculate and obtain the relative position information of the wearer and the vehicle based on the image of the first fixed object in the vehicle taken by the camera of the VR wearable device and the precise distance from the glasses to the second fixed object sensed by the close-range PS sensor of the VR wearable device;

[0096] The vehicle-ground position information acquisition unit 11 is used to obtain the current real-time position of the vehicle relative to the ground according to the real-time geographical position, acceleration, direction signal and driving speed signal of the vehicle;

[0097] A person-ground position information acquisition unit 12 is used to obtain the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground;

[0098] The VR picture adjustment unit 13 is used to perform anti-vertigo adjustment on the current VR video picture according to the relative position of the wearer relative to the ground, and output it to the VR wearable device.

[0099] like Figure 4 As shown, in a specific example, the person-vehicle position information acquisition unit 10 further includes:

[0100] The camera image receiving unit 100 is used to receive images of a first fixed object in the vehicle taken by at least two cameras arranged on the VR wearable device;

[0101] A matching processing unit 101 is used to extract and match features of the image, and obtain a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points;

[0102] The precise distance receiving unit 102 is used to receive the precise distance from the glasses to the second fixed object sensed by the short-range PS sensor on the VR wearable device;

[0103] The person-vehicle relative position calculation unit 103 is used to obtain the relative position information between the center point of the wearer and the vehicle according to the approximate relative position and the precise distance.

[0104] like Figure 5 As shown, in a specific example, the vehicle-ground position information acquiring unit 11 further includes:

[0105] The sensor information acquisition unit 110 is used to obtain the real-time positioning information of the vehicle through the GNSS module set on the cockpit host; obtain the acceleration and direction signal of the vehicle by setting the IMU sensor, and the direction signal includes: the pitch, roll and yaw angle of the vehicle; obtain the driving speed signal of the vehicle through the MCU of the cockpit host using the CAN bus;

[0106] The fusion processing unit 111 is used to fuse the positioning information, acceleration and direction signals and vehicle speed signals to obtain real-time position information of the vehicle center point relative to the ground coordinate system.

[0107] In a specific example, the person-ground location information acquisition unit 12 is further used to:

[0108] The relative position information between the wearer and the center point of the vehicle is combined with the real-time position information of the center point of the vehicle relative to the ground coordinates to obtain the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system and the orientation information.

[0109] like Figure 6 As shown, in a specific example, the VR picture adjustment unit 12 further includes:

[0110] The adjustment unit 120 is used to perform at least one of perspective correction, visual distortion correction, motion compensation and scene depth adjustment on the current VR video image according to the relative position of the wearer relative to the ground, so that the content seen by the wearer on the VR wearable device is consistent with the environmental position felt by the body;

[0111] The sending unit 121 is used to output the adjusted VR video picture to the VR wearable device.

[0112] Accordingly, another aspect of the present invention further provides a VR video processing system, comprising a cockpit host, and a VR wearable device communicating with the cockpit host; wherein:

[0113] The cockpit host is provided with at least an APU module, a microprocessor connected to the APU module, a first IMU sensor, a GPU, a memory (such as DDR6, DDR4, SSD), a serializer, a power amplifier and a BT / WIFI module; wherein the APU module is deployed with Figures 3 to 6 VR video processing device described;

[0114] At least two LCD display units, a 3D bridge, and a DP to DSI chip are provided in the VR wearable device, and a second IMU sensor, a PS sensor, and at least two cameras are connected to the 3D bridge.

[0115] Combined Figure 2 As shown, in the cockpit host:

[0116] The Application Processing Unit (APU) is responsible for controlling and processing the various functions and applications of the cockpit host. For example, it can provide sufficient graphics and computing power to support the rendering and other processing requirements of VR glasses; the Inertial Measurement Unit (IMU) sensor is used to measure and detect head posture, direction and acceleration; the Graphics Processing Unit (GPU) is used to process graphics and video related computing tasks; the Microcontroller Unit (MCU) is used to control and manage the communication and operation of various devices and interfaces; the GNSS module is the Global Navigation Satellite System Module, which is used to obtain the real-time geographic location of the vehicle and provide global positioning; the first IMU sensor is an inertial measurement unit, a sensor used to detect and measure acceleration and rotational motion; DDR6 (Double Data Rate 6) is the sixth generation of double data rate memory, providing high-speed memory access and data processing; DDR4 (Double Data Rate 4) is the fourth generation of double data rate memory, providing high-speed memory access and data processing; Solid State Drive (Solid State The SSD is used to store data such as the operating system, applications, and media files. The serializer is used to encrypt and compress image and video signals and transmit them to the display or VR glasses. The power amplifier is used to amplify the audio signal to provide better sound effects. The BT / WIFI modules are Bluetooth / wireless LAN modules, which support wireless communication and connection functions.

[0117] In VR wearable devices:

[0118] The PS sensor is a position sensor used to determine the position of the head in space; the second IMU sensor is an inertial measurement unit, a sensor used to detect and measure acceleration and rotational motion; the 3D bridge is used to process and manage communication and data transmission between the glasses and other devices; the DP to DSI chip is used to transmit the video signal from the cockpit host to the liquid crystal lenses of the VR glasses; the left and right LCD lenses are liquid crystal displays corresponding to each eye, used to present VR images to the wearer; multiple cameras are used to capture image information of the first fixed object in the car.

[0119] These components work together in VR wearable devices and cockpit host to realize the virtual reality experience and various cockpit functions, including real-time image display, position positioning, audio playback, etc. Through the interconnection and interaction of sensors, chips and interfaces, the wearer can wear VR glasses to watch virtual reality content and enjoy cockpit functions such as power-on display, map positioning and sound playback at the same time. As the core control unit, the cockpit host integrates various hardware and software components to drive the video and audio output of VR wearable devices to provide an immersive interactive experience and eliminate dizziness.

[0120] For more details, please refer to and combine the above Figures 3 to 6 The description is not repeated here.

[0121] Accordingly, in another aspect of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the above-mentioned Figure 1 , Figure 2 For more details, please refer to and combine the above Figure 1 to Figure 2 The description is not repeated here.

[0122] Accordingly, another aspect of the present invention further provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the aforementioned Figure 1 to Figure 2 For more details, please refer to and combine the above Figure 1 to Figure 2 The description is not repeated here.

[0123] Accordingly, another aspect of the present invention further provides a vehicle, on which a cockpit host is provided, wherein the cockpit host is deployed with the above-mentioned Figures 3 to 6 For more details, please refer to and combine with the above Figures 3 to 6 The description is not repeated here.

[0124] The implementation of the embodiments of the present invention has the following beneficial effects:

[0125] The present invention provides a VR video processing method, device, system, storage medium and vehicle, which saves the host cost of VR glasses by integrating the VR glasses function in the traditional cockpit host; and the VR wearable device used by the wearer is an independent VR glasses, which only serves as the display function of the VR glasses, is light in weight and easy to wear.

[0126] At the same time, in this embodiment, by setting at least two cameras on the VR wearable device to take pictures of fixed objects in the car, combined with the precise distance from the glasses to the second fixed object sensed by the close-range position sensor on the VR wearable device, the relative position information of the wearer and the vehicle can be calculated; combined with the resources of the entire vehicle (vehicle speed, acceleration and angle, positioning information), the relative position of the wearer relative to the ground can be calculated; according to the relative position of the wearer relative to the ground, the current VR video screen is adjusted to prevent dizziness and output to the VR wearable device. In this way, the inconsistency between the vision of the VR glasses and the wearer's feeling is eliminated, and the wearer is prevented from dizziness.

[0127] By implementing the present invention, a low-cost solution can be used to solve the dizziness caused by VR glasses during vehicle driving, thereby improving the user experience of using VR glasses in a vehicle.

[0128] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] The above disclosure is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A VR video processing method, applied to a smart cockpit system integrating VR wearable devices, characterized in that: At least the following steps are included: Calculate the relative position information of the wearer and the vehicle based on the image of the first fixed object in the vehicle captured by the camera of the VR wearable device and the precise distance from the glasses to the second fixed object sensed by the close-range position sensor of the VR wearable device; According to the real-time geographic location, acceleration, direction signal and driving speed signal of the vehicle, the real-time position of the current vehicle relative to the ground is obtained; Obtain the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground; According to the relative position of the wearer with respect to the ground, the current VR video picture is adjusted to prevent dizziness and output to the VR wearable device.

2. The method according to claim 1, characterized in that According to the image of the first fixed object in the vehicle captured by the camera of the VR wearable device and the precise distance from the glasses to the second fixed object sensed by the close-range position sensor of the VR wearable device, the relative position information of the wearer and the vehicle is calculated and obtained, further comprising: Receiving images of a first fixed object in the vehicle taken by at least two cameras arranged on the VR wearable device; Extracting and matching features of the image, and obtaining a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points; Receiving the precise distance between the glasses and the second fixed object sensed by the close-range position sensor on the VR wearable device; The relative position information between the wearer and the center point of the vehicle is obtained according to the approximate relative position and the precise distance.

3. The method according to claim 2, characterized in that Receiving the real-time geographic location, acceleration, direction signal and driving speed signal of the vehicle to obtain the real-time position of the current vehicle relative to the ground, further comprising: The real-time positioning information of the vehicle is obtained through the GNSS module installed on the cockpit host; The IMU sensor is used to obtain the acceleration and direction signals of the vehicle, wherein the direction signals include: the pitch, roll and yaw angles of the vehicle; The vehicle's driving speed signal is obtained through the MCU of the cockpit host using the CAN bus; The positioning information, acceleration and direction signals and vehicle speed signals are integrated to obtain real-time position information of the vehicle center point relative to the ground coordinate system.

4. The method according to claim 3, characterized in that: The method of obtaining the relative position of the wearer relative to the ground according to the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground further includes: The relative position information between the wearer and the center point of the vehicle is combined with the real-time position information of the center point of the vehicle relative to the ground coordinates to obtain the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system and the orientation information.

5. The method according to claim 4, characterized in that According to the relative position of the wearer relative to the ground, the current VR video picture is adjusted to prevent dizziness, and output to the VR wearable device, further comprising: According to the relative position of the wearer with respect to the ground, at least one of perspective correction, visual distortion correction, motion compensation and scene depth adjustment is performed on the current VR video picture, and the adjusted VR video picture is output to the VR wearable device, so that what the wearer sees on the VR wearable device is consistent with the environmental position felt by the body.

6. A VR video processing device, applied to a smart cockpit system integrating VR wearable devices, characterized in that: At least: A person-vehicle position information acquisition unit, configured to calculate and obtain relative position information of the wearer and the vehicle based on an image of a first fixed object in the vehicle captured by a camera of the VR wearable device and an accurate distance from the glasses to a second fixed object sensed by a short-range position sensor on the VR wearable device; The vehicle-ground position information acquisition unit is used to obtain the current real-time position of the vehicle relative to the ground according to the vehicle's real-time geographical location, acceleration, direction signal and driving speed signal; A person-ground position information acquisition unit, used to obtain the relative position of the wearer relative to the ground based on the relative position information of the wearer and the vehicle and the real-time position of the vehicle relative to the ground; The VR picture adjustment unit is used to perform anti-vertigo adjustment on the current VR video picture according to the relative position of the wearer relative to the ground, and output it to the VR wearable device.

7. The system according to claim 6, characterized in that The person-vehicle position information acquisition unit further comprises: A camera image receiving unit, configured to receive images of a first fixed object in the vehicle taken by at least two cameras arranged on the VR wearable device; A matching processing unit, used to extract and match features of the image, and obtain a real-time approximate relative position between the VR wearable device and the first fixed object according to the relative position relationship of the matched feature points; A precise distance receiving unit, used to receive the precise distance from the glasses to the second fixed object sensed by a close-range position sensor on the VR wearable device; The person-vehicle relative position calculation unit is used to obtain the relative position information between the center point of the wearer and the vehicle based on the approximate relative position and the precise distance.

8. The system according to claim 7, characterized in that The vehicle-ground position information acquisition unit further comprises: The sensor information acquisition unit is used to obtain the real-time positioning information of the vehicle through the GNSS module set on the cockpit host; obtain the acceleration and direction signals of the vehicle by setting the IMU sensor, and the direction signals include: the pitch, roll and yaw angles of the vehicle; and obtain the driving speed signal of the vehicle through the CAN bus through the MCU of the cockpit host; The fusion processing unit is used to fuse the positioning information, acceleration and direction signals and vehicle speed signals to obtain real-time position information of the vehicle center point relative to the ground coordinate system.

9. The system of claim 8, characterized in that: The person-ground location information acquisition unit is further used for: The relative position information between the wearer and the center point of the vehicle is combined with the real-time position information of the center point of the vehicle relative to the ground coordinates to obtain the relative position of the wearer relative to the ground, including the position information of the wearer in the ground coordinate system and the orientation information.

10. The system according to claim 9, characterized in that The VR picture adjustment unit further includes: An adjustment unit, configured to perform at least one of perspective correction, visual distortion correction, motion compensation, and scene depth adjustment on a current VR video image according to a relative position of a wearer relative to the ground; The sending unit is used to output the adjusted VR video picture to the VR wearable device so that the content seen by the wearer on the VR wearable device is consistent with the environmental position felt by the body.

11. A VR video processing system, comprising a cockpit host, and a VR wearable device communicating with the cockpit host; characterized in that: in: The cockpit host is provided with at least an APU module, and a microprocessor, a first IMU sensor, a GPU and a memory connected to the APU module; wherein the APU module is provided with a VR video processing device as claimed in any one of claims 6 to 10; At least two LCD display units and a 3D bridge are provided in the VR wearable device, and a second IMU sensor, a PS sensor and at least two cameras are connected to the 3D bridge.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

13. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

14. A vehicle, on which a cockpit host is arranged, characterized in that: The cockpit host is deployed with a device as described in any one of claims 6 to 10.

Images