Video perspective method based on virtual reality

Abstract

A video perspective method based on virtual reality relates to the technical field of virtual reality, and comprises: at least two cameras for capturing video information in real time, the cameras being provided with spatial positions and angles; performing geometric correction on the video information; the eye movement tracking module monitors the gazing direction of the eyes of the user in real time and adjusts the display position; the eye movement tracking module tracks eyeball movement; reconstructing a three-dimensional model of the surrounding environment of the user; based on the three-dimensional model, determining a relative position relationship between the virtual object and the real object; the head posture sensor obtains head movement information of a user; performing semantic segmentation on the video frame through a deep learning algorithm; and synthesizing the video information and the virtual content on the head-mounted display device, and outputting the synthesized video information and virtual content. The technical problems of visual discomfort and dizziness caused by position mismatching can be solved, and natural fusion of a real scene and a virtual scene is realized.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a video perspective method based on virtual reality. Background Art

[0002] At present, virtual reality technology has been widely used in entertainment, education, medical and other fields. However, traditional virtual reality systems usually rely on a completely closed virtual environment, which limits the user's interaction with the real world. In order to solve this problem, video perspective technology came into being. Video perspective technology obtains video information of the real world through a camera and combines it with computer-generated virtual information, so that users can also perceive the real world in a virtual environment. However, there are some problems in the existing technology, such as visual discomfort and dizziness caused by the mismatch between the position of the binocular camera and the position of the human eye, and the inability to analyze the three-dimensional structure of the real world and achieve a natural fusion of real scenes and virtual scenes. Summary of the invention

[0003] In view of this, the technical problem to be solved by the present invention is: to provide a video perspective method based on virtual reality, which can solve the technical problems of visual discomfort and dizziness caused by position mismatch and realize the natural fusion of real scenes and virtual scenes.

[0004] In order to solve the above technical problems, the technical solution of the present invention is:

[0005] A video perspective method based on virtual reality comprises the following steps:

[0006] S1. Using at least two cameras fixed on the head mounted display device to capture video information of the user's surrounding environment in real time, wherein the cameras are set with pre-calibrated spatial positions and angles;

[0007] S2, performing geometric correction on the video information captured by the camera, including adjusting the nonlinear distortion in the image through a preset correction matrix to compensate for the image distortion caused by the relative position difference between the camera and the user's eyes;

[0008] S3. The head-mounted display device includes an eye tracking module, which monitors the gaze direction of the user's eyes in real time and adjusts the display position of the virtual content accordingly to ensure that the virtual object is correctly positioned in the user's field of view; the eye tracking module tracks eye movements in a non-contact manner;

[0009] S4, reconstructing a three-dimensional model of the user's surrounding environment from the video information, including identifying common features between consecutive frames through a feature matching algorithm, and using these features to build an environment map;

[0010] S5. Based on the three-dimensional model, determine the relative position relationship between the virtual object and the real object, including using a depth test algorithm to determine the occlusion order between the virtual object and the real object, so as to achieve a natural occlusion effect between the virtual object and the real object;

[0011] S6. The head-mounted display device includes a head posture sensor, which obtains head movement information of the user and predictively adjusts the position and viewing angle of the virtual content to reduce discomfort caused by visual delay;

[0012] S7. Using deep learning algorithms, including using convolutional neural networks to perform semantic segmentation of video frames, identify foreground and background elements in video streams, and optimize the integration of virtual objects with the real world;

[0013] S8. On the head-mounted display device, synthesize the processed video information with the virtual content generated by the computer, and output it to the user's field of view at a high refresh rate, wherein the display device includes a liquid crystal display.

[0014] Preferably, in S2, the parameters in the geometric correction are dynamically adjusted according to the pupil distance parameters of the user, including obtaining the pupil distance parameters through user input or automatic measurement, and adjusting the internal parameter matrix in the geometric correction algorithm based on the pupil distance parameters.

[0015] Preferably, in S4, a simultaneous localization and mapping technique is used to construct a three-dimensional environment model, including using a SLAM system based on direct sparse optical flow to improve reconstruction efficiency and reduce drift.

[0016] Preferably, in S5, the occlusion relationship between the virtual object and the real object is determined by a depth map, and the depth map is obtained by analyzing image frames captured by a camera, including using structured light or time-of-flight technology to generate depth information, and generating a depth map in combination with an RGB image.

[0017] Preferably, in S6, it also includes calculating the view angle change at the next moment in advance based on the prediction algorithm according to the user's head movement information, including using a Kalman filter or other state estimation methods to predict the future view angle.

[0018] Preferably, in S7, it also includes automatically identifying key elements in the video stream through a deep learning algorithm, and adjusting the rendering priority level according to the importance of different elements to optimize the allocation of computing resources.

[0019] Preferably, in S8, it also includes adjusting the direction and intensity of the sound in real time according to the user's head posture and the position of the sound source in the virtual environment through an audio system integrated in the head-mounted display device, including using sound localization technology based on head-related transfer function to simulate the sound propagation effect in three-dimensional space.

[0020] After adopting the above technical solution, the beneficial effects of the present invention are:

[0021] In the present invention, a video perspective method based on virtual reality includes the following steps: using at least two cameras fixed on a head-mounted display device to capture video information of a user's surrounding environment in real time, the cameras being provided with pre-calibrated spatial positions and angles; performing geometric correction on the video information captured by the cameras, including adjusting nonlinear distortion in the image through a preset correction matrix to compensate for image distortion caused by a relative position difference between the camera and the user's eyes; the head-mounted display device includes an eye tracking module, the eye tracking module monitors the gaze direction of the user's eyes in real time, and adjusts the display position of the virtual content accordingly to ensure the correct positioning of the virtual object in the user's field of view; the eye tracking module tracks eye movements in a non-contact manner; reconstructing a three-dimensional model of the user's surrounding environment from the video information, including identifying common features between consecutive frames through a feature matching algorithm Features, and use these features to build an environment map; based on the three-dimensional model, determine the relative position relationship between the virtual object and the real object, including using a depth test algorithm to determine the occlusion order between the virtual object and the real object to achieve a natural occlusion effect between the virtual object and the real object; the head-mounted display device includes a head posture sensor, which obtains the user's head movement information, predictively adjusts the position and viewing angle of the virtual content, and reduces the discomfort caused by visual delay; through a deep learning algorithm, including using a convolutional neural network to semantically segment the video frame, identify the foreground and background elements in the video stream, and optimize the degree of integration between the virtual object and the real world; on the head-mounted display device, the processed video information is synthesized with the computer-generated virtual content and output to the user's field of view at a high refresh rate, and the display device includes a liquid crystal display. This application can reduce visual discomfort and dizziness, enhance immersion and reality, improve interactivity and flexibility, optimize computing resource utilization, and improve system stability and accuracy. It not only improves user experience and reduces common visual discomfort problems, but also improves the interactivity and immersion of the virtual reality system, allowing virtual reality technology to play a greater role in more application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0023] Figure 1 It is a flow chart of a video perspective method based on virtual reality according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] like Figure 1 As shown, the present invention comprises the following steps:

[0026] S1. Using at least two cameras fixed on the head mounted display device to capture video information of the user's surrounding environment in real time, the cameras are set with pre-calibrated spatial positions and angles;

[0027] S2, performing geometric correction on the video information captured by the camera, including adjusting the nonlinear distortion in the image through a preset correction matrix to compensate for the image distortion caused by the relative position difference between the camera and the user's eyes;

[0028] S3. The head-mounted display device includes an eye tracking module, which monitors the gaze direction of the user's eyes in real time and adjusts the display position of the virtual content accordingly to ensure that the virtual object is correctly positioned in the user's field of view; the eye tracking module tracks eye movements in a non-contact manner;

[0029] S4, reconstructing a three-dimensional model of the user's surrounding environment from the video information, including identifying common features between consecutive frames through a feature matching algorithm and using these features to build an environment map;

[0030] S5. Based on the three-dimensional model, determine the relative position relationship between the virtual object and the real object, including using a depth test algorithm to determine the occlusion order between the virtual object and the real object, so as to achieve a natural occlusion effect between the virtual object and the real object;

[0031] S6. The head-mounted display device includes a head posture sensor, which obtains head movement information of the user and predictively adjusts the position and viewing angle of the virtual content to reduce discomfort caused by visual delay;

[0032] S7. Using deep learning algorithms, including using convolutional neural networks to perform semantic segmentation of video frames, identify foreground and background elements in video streams, and optimize the integration of virtual objects with the real world;

[0033] S8. On a head-mounted display device, the processed video information is synthesized with the virtual content generated by the computer and output to the user's field of view at a high refresh rate, wherein the display device includes a liquid crystal display.

[0034] In the present application, in S2, the parameters in the geometric correction are dynamically adjusted according to the user's pupil distance parameter, including obtaining the pupil distance parameter through user input or automatic measurement, and adjusting the internal parameter matrix in the geometric correction algorithm based on the pupil distance parameter. S2 includes the following steps:

[0035] S21: Obtaining a pupil distance parameter of the user, and measuring the pupil distance value of the user manually input by the user or automatically by a pupil distance measurement module integrated in the head mounted display device;

[0036] S22: According to the obtained pupil distance parameter, adjust the internal parameter matrix in the geometric correction algorithm, the internal parameter matrix includes the focal length and principal point coordinate parameters of the camera, and adjusting the internal parameter matrix can make the corrected image more suitable for the user's pupil distance, thereby reducing the visual discomfort caused by the inconsistency between the camera position and the user's eye position;

[0037] S23: Apply the adjusted intrinsic parameter matrix to perform geometric correction on the video information captured by the camera. In this way, the image distortion caused by the relative position difference between the camera and the user's eyes can be more accurately compensated.

[0038] In this application, in S4, a three-dimensional environment model is constructed using a simultaneous localization and mapping technique, including using a SLAM system based on direct sparse optical flow to improve reconstruction efficiency and reduce drift. S4 includes the following steps:

[0039] S41: starting a SLAM system based on direct sparse optical flow, which directly extracts feature points from image frames captured by a camera, and performs optical flow estimation between consecutive frames based on the feature points to calculate the motion of the camera;

[0040] S42: Use the feature points to construct a map of the environment, identify common features between consecutive frames through a feature matching algorithm, and use these features to construct an environment map, thereby achieving three-dimensional reconstruction of the user's surrounding environment;

[0041] S43: Implement a 3D environment model building process based on SLAM to ensure the accuracy of the 3D model and reduce drift caused by cumulative errors by continuously updating the positions of feature points and the environment map.

[0042] In this application, in S5, the occlusion relationship between the virtual object and the real object is determined by a depth map, and the depth map is obtained by analyzing the image frames captured by the camera, including using structured light or time-of-flight technology to generate depth information, and combining the RGB image to generate the depth map. S5 includes the following steps:

[0043] S51: Use structured light or time-of-flight technology to generate depth information. Structured light technology projects a known pattern of light into the scene and calculates depth information based on changes in reflected light. Time-of-flight technology calculates distance by measuring the round-trip time of light pulses.

[0044] S52: Generate a depth map by combining the RGB image, combining the depth information with the color information in the RGB image to generate a depth map containing the depth information;

[0045] S53: Use the depth map to determine the occlusion relationship between the virtual object and the real object. Through the depth test algorithm, determine the relative position relationship between the virtual object and the real object according to the information in the depth map, and adjust the position of the virtual object accordingly to achieve a natural occlusion effect.

[0046] In the present application, in S6, it is also included to calculate the view angle change at the next moment in advance based on the prediction algorithm according to the user's head movement information, including using a Kalman filter or other state estimation method to predict the future view angle. S6 includes the following steps:

[0047] S61: Acquire the user's head movement information through a head posture sensor, where the sensor includes an accelerometer, a gyroscope, and a magnetometer, and is used to measure the acceleration, rotation angular velocity, and magnetic field direction of the head;

[0048] S62: Applying the Kalman filter state estimation method to predict future head posture changes based on the current head posture data, the Kalman filter can estimate the state at the next moment based on the current observation data and the past prediction results;

[0049] S63: Based on the predicted head posture changes, the position and viewing angle of the virtual content are adjusted in advance to reduce the visual delay caused by head movement, thereby reducing the user's discomfort.

[0050] In this application, S7 also includes automatically identifying key elements in the video stream through a deep learning algorithm, and adjusting the rendering priority level according to the importance of different elements to optimize the allocation of computing resources. S7 includes the following steps:

[0051] S71: Analyze the video stream using a deep learning algorithm. Specifically, use a convolutional neural network to perform semantic segmentation on each frame of the video to identify foreground and background elements in the video.

[0052] S72: adjusting the rendering priority according to the importance of the identified key elements, whereby elements with high importance will be given a higher rendering priority, while elements with less importance may have their rendering quality or resolution appropriately reduced to save computing resources;

[0053] S73: By optimizing rendering priorities, computing resources can be effectively allocated, thereby ensuring high-quality presentation of key elements and smooth operation of the entire system.

[0054] In the present application, in S8, it is also included to adjust the direction and intensity of the sound in real time according to the user's head posture and the sound source position in the virtual environment through the audio system integrated in the head-mounted display device, including using the sound localization technology based on the head-related transfer function to simulate the sound propagation effect in the three-dimensional space. S8 includes the following steps:

[0055] S81: Acquire the user's head posture information, monitor the user's head movement in real time through the head posture sensor, and transmit this information to the audio processing module;

[0056] S82: Analyze the position information of each sound source in the virtual environment, and determine the position of each sound source relative to the user according to the position of the virtual content;

[0057] S83: Apply the sound localization technology based on head-related transfer function, use the HRTF algorithm to adjust the direction and intensity of the sound in real time according to the user's head posture and the location of the sound source, simulate the sound propagation effect in three-dimensional space, and enhance the user's sense of immersion.

[0058] The present invention can reduce visual discomfort and dizziness, enhance immersion and reality, improve interactivity and flexibility, optimize computing resource utilization, and enhance system stability and accuracy. It not only improves user experience and reduces common visual discomfort problems, but also improves the interactivity and immersion of the virtual reality system, allowing virtual reality technology to play a greater role in more application scenarios.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video perspective method based on virtual reality, characterized in that: The following steps are involved: S1. Using at least two cameras fixed on the head mounted display device to capture video information of the user's surrounding environment in real time, wherein the cameras are set with pre-calibrated spatial positions and angles; S2, performing geometric correction on the video information captured by the camera, including adjusting the nonlinear distortion in the image through a preset correction matrix to compensate for the image distortion caused by the relative position difference between the camera and the user's eyes; S3. The head-mounted display device includes an eye tracking module, which monitors the gaze direction of the user's eyes in real time and adjusts the display position of the virtual content accordingly to ensure that the virtual object is correctly positioned in the user's field of view; the eye tracking module tracks eye movements in a non-contact manner; S4, reconstructing a three-dimensional model of the user's surrounding environment from the video information, including identifying common features between consecutive frames through a feature matching algorithm, and using these features to build an environment map; S5. Based on the three-dimensional model, determine the relative position relationship between the virtual object and the real object, including using a depth test algorithm to determine the occlusion order between the virtual object and the real object, so as to achieve a natural occlusion effect between the virtual object and the real object; S6. The head-mounted display device includes a head posture sensor, which obtains head movement information of the user and predictively adjusts the position and viewing angle of the virtual content to reduce discomfort caused by visual delay; S7. Using deep learning algorithms, including using convolutional neural networks to perform semantic segmentation of video frames, identify foreground and background elements in video streams, and optimize the integration of virtual objects with the real world; S8. On the head-mounted display device, synthesize the processed video information with the virtual content generated by the computer, and output it to the user's field of view at a high refresh rate, wherein the display device includes a liquid crystal display.

2. The video perspective method based on virtual reality according to claim 1, characterized in that: In S2, the parameters in the geometric correction are dynamically adjusted according to the pupil distance parameters of the user, including obtaining the pupil distance parameters through user input or automatic measurement, and adjusting the internal parameter matrix in the geometric correction algorithm based on the pupil distance parameters.

3. The video perspective method based on virtual reality as claimed in claim 1, characterized in that: In S4, simultaneous localization and mapping techniques are used to construct a 3D environment model, including the use of a SLAM system based on direct sparse optical flow to improve reconstruction efficiency and reduce drift.

4. The video perspective method based on virtual reality as claimed in claim 1, characterized in that: In S5, the occlusion relationship between the virtual object and the real object is determined by a depth map, and the depth map is obtained by analyzing the image frames captured by the camera, including using structured light or time-of-flight technology to generate depth information, and combining the RGB image to generate the depth map.

5. The video perspective method based on virtual reality as claimed in claim 1, characterized in that: In S6, it also includes calculating the view angle change at the next moment in advance based on the prediction algorithm according to the user's head movement information, including using a Kalman filter or other state estimation methods to predict the future view angle.

6. The method for video perspective based on virtual reality according to claim 1, characterized in that: In S7, it also includes automatic identification of key elements in the video stream through deep learning algorithms, and adjusts rendering priorities and optimizes computing resource allocation based on the importance of different elements.

7. The video perspective method based on virtual reality as claimed in claim 1, characterized in that: In S8, it also includes adjusting the direction and intensity of the sound in real time according to the user's head posture and the position of the sound source in the virtual environment through the audio system integrated in the head-mounted display device, including using the sound localization technology based on the head-related transfer function to simulate the sound propagation effect in three-dimensional space.

Images