XR virtual-real scene rendering optimization method and system

Abstract

The invention relates to the technical field of XR, in particular to an XR virtual-real scene rendering optimization method and system, and the method comprises the steps: obtaining a real visual input signal in a visual region; obtaining a virtual reality signal, rendering the virtual reality signal through a virtual reality engine, and fusing the rendered virtual reality signal with the real visual input signal to generate a fused visual input signal; collecting an eye movement signal, tracking an eye fixation area, and obtaining a target visual area; correcting the target visual area in combination with the synchronously collected head rotation signal and body vibration signal to obtain a corrected target visual area; obtaining a fused visual input signal, and displaying the fused visual input signal in the corrected target visual area; maintaining the definition of rendering in the corrected target visual area and the ambiguity of the peripheral area of the target visual area; setting one or more dynamic rendering areas, and rendering a moving object entering the target visual area from the outside of the visual area; and rendering a schematic motion graph for avoiding the moving object in the target visual area.

Description

Technical Field

[0001] The present invention relates to the field of XR technology, and in particular to an XR virtual-reality scene rendering optimization method and system. Background Art

[0002] XR, or extended reality, includes virtual reality (VR), augmented reality (AR), and mixed reality (MR). It enhances the user's immersive experience by combining or replacing computer-generated virtual content with the real world.

[0003] In existing technologies, when rendering the gaze area, the rendering area is typically divided statically. This means that high and low resolution regions are fixed based on the current frame's gaze point. This is then mirrored along the Y axis to generate a binocular frame, split into only two resolution levels, high and low, with no intermediate transition. Furthermore, while this method predicts the gaze area position by tracking head and eye rotation, it fails to consider the impact of vibration caused by the user's body movements on position prediction in dynamic applications, which can easily lead to deviations in the rendering position. Summary of the Invention

[0004] When rendering virtual and real objects in a gaze area, the present invention simultaneously tracks the user's body vibrations based on tracking head and eye movements. In dynamic application scenarios, the prediction of the body vibration on the position of the gaze area is corrected to ensure the accuracy of the rendering position.

[0005] The technical solution proposed in the present invention is: an XR virtual-reality scene rendering optimization method, the method comprising:

[0006] Obtaining real visual input signals within the visual area;

[0007] Acquire a virtual reality signal, render the virtual reality signal through a virtual reality engine, and fuse it with a real visual input signal to generate a fused visual input signal;

[0008] Collect eye movement signals, track the eye gaze area, and obtain the target visual area; and combine the synchronously collected head rotation signals and body vibration signals to correct the target visual area and obtain the corrected target visual area;

[0009] Acquire the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendering in the corrected target visual area and the blurriness of the surrounding area;

[0010] One or more dynamic rendering areas are set in the peripheral area to render moving objects that enter the target visual area from outside the visual area;

[0011] By analyzing the danger of moving objects, a schematic animation of how to avoid the moving objects is rendered in the target visual area.

[0012] Preferably, the step of obtaining a real visual input signal within the visual area includes:

[0013] The data collection area of each sensor is calibrated using the checkerboard calibration method to match the data collection area of each sensor with the preset visual area.

[0014] Unify the clocks of multiple sensors through the Precision Time Protocol (PTP). Acquire real-scene data within the data collection area from multiple sensors at a preset collection frequency, pre-process the real-scene data, and store it locally.

[0015] The sensors include RGB camera, depth camera, infrared camera, and lidar.

[0016] Preferably, the acquiring of the virtual reality signal, rendering the virtual reality signal by a virtual reality engine and fusing it with the real visual input signal to generate a fused visual input signal, comprises:

[0017] Acquire virtual reality data from a database, divide the virtual reality data into a plurality of virtual frame data, and add a time tag to each virtual frame data;

[0018] Acquire real scene data, divide the real scene data into multiple real frame data, and add a time label 2 to each real frame data;

[0019] Align the virtual frame data and the real frame data on the time axis;

[0020] The SLAM real-time positioning algorithm is used to construct the environmental point cloud map of the real scene in real time, and calibrate the coordinates of the objects in the real scene. i_true and the corresponding pose quaternion q i_ture ;

[0021] Extract the coordinates of virtual objects from the virtual reality data and match them with the coordinates and poses of real objects in the real scene. Specifically:

[0022] v i_v =R(q i_ture )·v i +p i_ture ; Among them, v i_v Represents the coordinates of the virtual object in the world coordinate system, v i Represents the virtual object coordinates, where R(q i_ture ) represents the pose quaternion q i_ture The obtained rotation matrix.

[0023] Preferably, the collecting of eye movement signals, tracking of eye gaze areas, and acquisition of target visual areas; and correcting of the target visual areas in combination with synchronously collected head rotation signals and body vibration signals to obtain the corrected target visual areas include:

[0024] The IMU and camera installed on the user's head capture the head rotation signal and obtain the head movement speed, acceleration and movement direction data;

[0025] Continuously capture the pupil position of the eye through an infrared camera;

[0026] Obtain body vibration acceleration through the IMU on the user's waist or shoulder;

[0027] Construct a world coordinate system, a head coordinate system, an evolution coordinate system, and a body coordinate system; the head coordinate system has the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system has the center of eyeball rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system has the center of the torso as its origin and is offset with body vibration;

[0028] Constructing the state vector in, Respectively represent the position vector of the head in the world coordinate system at time t, the head posture quaternion after conversion from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head movement angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system;

[0029] Use the state vector to predict head position, head pose quaternion and eye movement vector, including:

[0030] Predict head pose quaternion: in, represents quaternion multiplication, exp(·) represents exponential mapping; represents the predicted head pose quaternion; Δt represents the time interval;

[0031] Predicting head position: Considering the impact of body vibration on head position, the prediction model of head position is:

[0032] in, Represents the rotation matrix that transforms the world coordinate system obtained by the head pose quaternion to the head coordinate system, a IMU Indicates the original angular velocity of the IMU of the head;

[0033] Predict eye movement vectors:

[0034] The head position, head pose quaternion and eye position are fused, including:

[0035] The predicted eye movement vector is corrected by the predicted head position and the predicted head posture quaternion, specifically:

[0036] The eye movement vector in the head coordinate system is: in, Represents the head pose quaternion transformed from eye coordinates to head coordinate system, Represents the rotation matrix from the eye coordinate system to the head coordinate system,

[0037] The eye movement vector in the world coordinate system is:

[0038] Assume that the effect of body vibration on the head conforms to the second-order system, that is:

[0039] in, represents the head acceleration, represents the vibration transmission damping coefficient, w n Indicates the natural vibration frequency of the body, w n ∈[4,6]; s represents the vibration transmission speed;

[0040] Will Convert to the head coordinate system:

[0041] Then Convert to world coordinate system:

[0042] pass Correction Get the corrected Then, the position vector of the target visual area in the world coordinate system after adding body vibration offset compensation is Among them, d e Indicates gaze distance;

[0043] According to the 3σ principle, the synthetic error angle is obtained, specifically: Among them, σ eye , σ head , σ zd represent eye movement tracking accuracy, head movement prediction error, and vibration compensation residual, respectively;

[0044] Under the condition of covering 99% confidence interval, the coefficient k = 2.58;

[0045] According to the focal length of the eye, the radius corresponding to the synthetic error angle is the radius r of the target visual area.base .

[0046] Preferably, maintaining the clarity of rendering within the corrected target visual area and the blurriness of its surrounding area during the display process includes:

[0047] Get the predicted target visual area, and use the predicted target visual area as the center to render outward in three levels based on distance, including:

[0048] The target gaze area is divided into core area, transition area and fuzzy area in sequence;

[0049] Set the core area to 100% resolution, the transition area to 50% resolution, and the blur area to 25% resolution;

[0050] Based on head movement, construct a pixel motion vector model:

[0051] in, represents the 3D position of the pixel in the camera coordinate system, which is obtained by back-projecting the depth map; Δt′ represents the frame interval time; z(x,y) represents the pixel depth;

[0052] Perform fuzzy intensity grading, specifically:

[0053] Among them, r1 represents the viewing angle of the dynamic blur area; d represents the angular distance from the pixel to the center of the target visual area; the maximum blurred pixel value σ max ≥8 pixels;

[0054] Apply the pixel motion vector model to each pixel to obtain the pixel value after motion blur: Among them, Q represents the number of sampling points, M x (x,y) represents the x value obtained by the pixel motion vector model calculation, M y (x,y) represents the y value obtained by calculating the pixel motion vector model.

[0055] Preferably, the step of setting one or more dynamic rendering areas in the peripheral area to render moving objects entering the target visual area from outside the visual area includes:

[0056] Input the video frames of the surrounding area and perform motion tracking based on the optical flow algorithm; specifically:

[0057] Optical flow-based motion saliency tracking of pixels containing novel objects in, Represents the optical flow vector of the pixel, which is calculated by the Lucas-Kanade algorithm. represents the average optical flow, σ f The pixel frame rate representing the optical flow;

[0058] If S(x,y)>τ move , then it is judged that a new object is contained; among them, τ move Indicates the preset significance threshold;

[0059] Detect new object categories and bounding boxes using the YOLOv5 model, and then track the objects using the tracker.

[0060] Take the center C of the new object obj is the focus, and the radius η1 = 3° defines the second clear area, i.e., the dynamic rendering area;

[0061] Improve the resolution of the second clearest area from 25% to 100% through a super-resolution network;

[0062] The rendering image of the second cleaning area and the rendering image of the target visual area are displayed simultaneously.

[0063] Preferably, the rendering of a schematic animated diagram of avoiding the moving object in the target visual area by analyzing the danger of the moving object includes:

[0064] Obtain motion trajectory analysis and analyze the dangerousness of the object based on the motion trajectory;

[0065] Generate avoidance movement directions and corresponding schematic animations based on the danger level of the object;

[0066] Obtaining motion trajectory analysis and analyzing the dangerousness of the object based on the motion trajectory includes the following steps:

[0067] Get the motion trajectory of the new object and calculate the danger level L danger , specifically including:

[0068] Obtain the position and velocity vector of the new object from the tracker, and obtain the object category from YOLOv5, which includes people, vehicles, and special-shaped obstacles;

[0069] Building a motion trajectory prediction model Among them, p t represents the current position of the new object, The velocity vector of the new object, a max represents the preset maximum assumed acceleration, and Δt″ represents the prediction time interval;

[0070] Calculating collision time Among them, p t_u Indicates the position of the human body, v t_u Represents the velocity vector of the human body;

[0071] Constructing a hazard level balance model Among them, α1, α2, α3 are weight coefficients, τ represents the attenuation coefficient of collision time, v max Indicates the maximum reference speed, s d Indicates the risk factor.

[0072] Preferably, generating the avoidance movement direction and the corresponding schematic animation according to the dangerousness of the object comprises the following steps:

[0073] Adjust the radius of the target visual area and the number of secondary clear areas based on the danger level, including:

[0074] Set the radius adjustment rule, that is, the radius r of the target visual area after adjusting the danger level and the corrected head linear speed focus :

[0075] Where δ represents the scaling factor, δ∈[1,3); based on the focal length of the eye, the corresponding r is obtained by reverse calculation focus The viewing angle η′, Indicates the speed correction factor, v u_max Indicates the user's maximum walking speed;

[0076] Set the second clear area generation threshold L0 and collision time threshold T0, if L danger > L0 and T < T0, a clear area is generated for each new object independently, and the number of clear areas is not less than 3;

[0077] When generating a new second clear area, dynamically adjust the dynamic blur intensity around the second clear area, σ b ′(d)=σ max ·(1-L danger );

[0078] Calculate the avoidance direction, specifically:

[0079] Calculate the optimal avoidance vector Among them, normalize() represents the normalization function; represents the vertical axis,

[0080] Create along Schematic diagram of the direction of flow; and set the transparency of the animation l(t) = 0.5 + 0.5sin(2πft), where the frequency f = 2Hz, to achieve the breathing light effect;

[0081] Then fix the center point of the schematic image at the edge of the target visual area, that is, in represents the coordinates of the center point of the schematic animation in the head coordinate system, θ represents the offset, and θ≥2°.

[0082] An XR virtual-reality scene rendering optimization system includes: a processor, a memory and a communication module connected to the processor, and a head tracking device, an eye tracking device, a body vibration tracking device, and a head-mounted display device connected to the processor. The system is used to execute the XR virtual-reality scene rendering optimization method.

[0083] A computer-readable storage medium stores a computer program, which is executed by a processor to implement the XR virtual-reality scene rendering optimization method.

[0084] Beneficial effects of the present invention:

[0085] 1. The present invention dynamically blurs the clarity of the target gaze area step by step, divides the target gaze area into a core area, a transition area and a blurred area in sequence; and supports multi-focus generation (second clear area).

[0086] 2. The present invention tracks new objects that enter the target gaze area in real time, renders different new objects through multiple second clear areas, and adjusts the rendering strategy in real time (for example, adjusting the range of the target gaze area and the number of second clear areas) based on the prediction of the future target gaze area and the analysis of the risk factor of the new object. In addition, based on the prediction of the trajectory of the new object, a corresponding schematic animation is dynamically rendered to guide the user to avoid danger, so that the system can not only passively display but also have the function of active safety assistance, and can adapt to both static and dynamic usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 This is a flowchart of an XR virtual-reality scene rendering optimization method of the present invention. DETAILED DESCRIPTION

[0088] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0089] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the elements may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0090] refer to Figure 1The technical solution provided by the present invention is: an XR virtual-reality scene rendering optimization method and system, comprising the following steps:

[0091] Step 1: Obtain the real visual input signal in the visual area; specifically:

[0092] The data acquisition area of each sensor is calibrated using the checkerboard calibration method (calibrateCamer function in OpenCV) so that the data acquisition area of each sensor matches the preset visual area. The purpose of this step is to eliminate lens distortion and establish a mapping between the pixel coordinate system and the physical coordinate system.

[0093] Unify the clocks of multiple sensors through the Precision Time Protocol (PTP). Acquire real-scene data within the data collection area from multiple sensors at a preset collection frequency, pre-process the real-scene data, and store it locally.

[0094] The sensors include RGB camera, depth camera, infrared camera, and lidar.

[0095] Step 2: Obtain a virtual reality signal, render it through a virtual reality engine, and fuse it with the real visual input signal to generate a fused visual input signal.

[0096] Acquire virtual reality data from a database, divide the virtual reality data into a plurality of virtual frame data, and add a time tag to each virtual frame data;

[0097] Acquire real scene data, divide the real scene data into multiple real frame data, and add a time label 2 to each real frame data;

[0098] Align the virtual frame data and the real frame data on the time axis;

[0099] The SLAM real-time positioning algorithm is used to construct the environmental point cloud map of the real scene in real time, and calibrate the coordinates of the objects in the real scene. i_true and the corresponding pose quaternion q i_ture ;

[0100] Extracting the coordinates of virtual objects from VR data and matching them with the coordinates and poses of real objects in the real scene to anchor the virtual objects. This includes the following steps:

[0101] v i_v =R(q i_ture )·v i +p i_ture ; Among them, v i_v Represents the coordinates of the virtual object in the world coordinate system, v i Represents the virtual object coordinates, where R(qi_ture ) represents the pose quaternion q i_ture The obtained rotation matrix.

[0102] In addition, it also includes processing the occlusion relationship between virtual and real objects, specifically:

[0103] In this embodiment, a depth map of the real scene is acquired using ToF / LiDAR, which is then used to render a depth buffer for the virtual object. The depth values of the virtual and real objects are then compared pixel by pixel to determine the occlusion relationship. For example, if the depth of the virtual object is smaller than that of the real object, the virtual object is in front; otherwise, the real object is in front.

[0104] In some preferred embodiments, the process further includes blending pixels of unobstructed virtual objects with pixels of real objects according to transparency, and performing illumination estimation and material matching. These are all prior art and will not be described in detail here.

[0105] Step 3: Collect eye movement signals, track the eye gaze area, and obtain the target visual area; and combine the synchronously collected head rotation signal and body vibration signal to correct the target visual area and obtain the corrected target visual area; specifically:

[0106] The IMU and camera installed on the user's head capture the head rotation signal and obtain the head movement speed, acceleration and movement direction data;

[0107] Continuously capture the pupil position of the eye through an infrared camera;

[0108] Obtain body vibration acceleration through the IMU on the user's waist or shoulder;

[0109] Construct a world coordinate system, a head coordinate system, an evolution coordinate system, and a body coordinate system; the head coordinate system has the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system has the center of eyeball rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system has the center of the torso as its origin and is offset with body vibration;

[0110] Constructing the state vector in, Respectively represent the position vector of the head in the world coordinate system at time t, the head posture quaternion after conversion from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head movement angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system;

[0111] Use the state vector to predict head position, head pose quaternion and eye movement vector, including:

[0112] Predict head pose quaternion: in, represents quaternion multiplication, exp(·) represents exponential mapping; represents the predicted head pose quaternion; Δt represents the time interval;

[0113] Predicting head position: Considering the impact of body vibration on head position, the prediction model of head position is:

[0114] in, Represents the rotation matrix that transforms the world coordinate system obtained by the head pose quaternion to the head coordinate system, a IMU Indicates the original angular velocity of the IMU of the head;

[0115] Predict eye movement vectors:

[0116] The head position, head pose quaternion and eye position are fused, including:

[0117] The predicted eye movement vector is corrected by the predicted head position and the predicted head posture quaternion, specifically:

[0118] The eye movement vector in the head coordinate system is: in, Represents the head pose quaternion transformed from eye coordinates to head coordinate system, Represents the rotation matrix from the eye coordinate system to the head coordinate system,

[0119] The eye movement vector in the world coordinate system is:

[0120] Assume that the effect of body vibration on the head conforms to the second-order system, that is:

[0121] in, represents the head acceleration, represents the vibration transmission damping coefficient, w n Indicates the natural vibration frequency of the body, w n ∈[4,6]; s represents the vibration transmission speed;

[0122] Will Convert to the head coordinate system: Then Convert to world coordinate system: pass Correction Get the corrected

[0123] Finally, the position vector of the target visual area in the world coordinate system after adding body vibration offset compensation is obtained Among them, d e Indicates gaze distance;

[0124] In this embodiment, the above prediction process is based on a pre-trained LSTM network and its internal extended Kalman filter algorithm. In this embodiment, a three-layer LSTM network with a hidden layer dimension of 128 is used to output the position vector of the target view area in the future λ frame.

[0125] According to the 3σ principle, the synthetic error angle is obtained, specifically: Among them, σ eye , σ head , σ zd They represent eye movement tracking accuracy, head movement prediction error and vibration compensation residual respectively; the synthetic error perspective here can be regarded as the perspective of the target perspective area.

[0126] Under the condition of covering 99% confidence interval, the coefficient k = 2.58;

[0127] According to the focal length of the eye, the radius corresponding to the synthetic error angle is the radius r of the target visual area. base In some preferred embodiments, the radius can also be directly used

[0128] For example, in this embodiment:

[0129] σ eye =0.5°,σ head =0.3°,σ zd =0.2°, Then the target area range is (0,1.6°], where σ eye Determined by the accuracy of the eye tracking device, which depends on the device, for example, the accuracy of the Tobii4C device is 0.3 degrees. head The standard deviation of the head position error can be obtained by extracting the extended Kalman filter covariance, σ sd is the standard deviation of vibration compensation measured experimentally.

[0130] Step 4: Obtain the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendering in the corrected target visual area and the blurriness of the surrounding area; specifically:

[0131] Get the predicted target visual area, and use the predicted target visual area as the center to render outward in three levels based on distance, including:

[0132] The target gaze area is divided into core area, transition area and fuzzy area in sequence;

[0133] Set the core area to 100% resolution, the transition area to 50% resolution, and the blur area to 25% resolution;

[0134] Based on head movement, construct a pixel motion vector model:

[0135] in, represents the 3D position of the pixel in the camera coordinate system, which is obtained by back-projecting the depth map; Δt′ represents the frame interval time; z(x,y) represents the pixel depth;

[0136] Perform fuzzy intensity grading, specifically:

[0137] Among them, r1 represents the viewing angle of the dynamic blur area; d represents the angular distance from the pixel to the center of the target visual area; the maximum blurred pixel value σ max ≥8 pixels;

[0138] For example, in this embodiment, in the user's head-mounted display device, in the core area (d≤5°), clarity is maintained without blur; in the transition area (5°<d<15°), the blur intensity increases linearly; and in the blur area (d>15°), the maximum blur is constant.

[0139] Apply the pixel motion vector model to each pixel to obtain the pixel value after motion blur: Among them, Q represents the number of sampling points, M x (x,y) represents the x value obtained by the pixel motion vector model calculation, M y (x,y) represents the y value obtained by calculating the pixel motion vector model.

[0140] Step 5: Set one or more dynamic rendering areas in the surrounding area to render moving objects that enter the target visual area from outside the visual area; specifically:

[0141] Input the video frames of the surrounding area and perform motion tracking based on the optical flow algorithm;

[0142] Specifically: Motion saliency tracking of pixels containing new objects based on optical flow in, Represents the optical flow vector of the pixel, which is calculated by the Lucas-Kanade algorithm. represents the average optical flow, σ f The pixel frame rate representing the optical flow;

[0143] If S(x,y)>τ move , then it is judged that a new object is contained; among them, τ move Indicates the preset significance threshold;

[0144] Detect new object categories and bounding boxes using the YOLOv5 model, and then track the objects using the tracker KernelizedCorrelation Filter.

[0145] Take the center C of the new object obj is the focus, and the radius η1 = 3° defines the second clear area, i.e., the dynamic rendering area;

[0146] Improve the resolution of the second clearest area from 25% to 100% through a super-resolution network;

[0147] The rendering image of the second cleaning area and the rendering image of the target visual area are displayed simultaneously.

[0148] Step 6: By analyzing the danger of the moving object, a schematic diagram of avoiding the moving object is rendered in the target visual area, specifically:

[0149] Obtain motion trajectory analysis and analyze the dangerousness of the object based on the motion trajectory. This includes the following steps:

[0150] Get the motion trajectory of the new object and calculate the danger level L danger , specifically including:

[0151] Obtain the position and velocity vector of the new object from the tracker, and obtain the object category from YOLOv5, which includes people, vehicles, and special-shaped obstacles;

[0152] Building a motion trajectory prediction model Among them, p t represents the current position of the new object, The velocity vector of the new object, a max represents the preset maximum assumed acceleration, and Δt″ represents the prediction time interval;

[0153] Calculating collision time Among them, p t_u Indicates the position of the human body, v t_u Represents the velocity vector of the human body;

[0154] Constructing a hazard level balance model Among them, α1, α2, α3 are weight coefficients, τ represents the attenuation coefficient of collision time, v max Indicates the maximum reference speed, s d In this embodiment, the risk coefficient for pedestrians is 0.2, and the risk coefficient for vehicles is 0.8.

[0155] According to the danger of the object, the avoidance movement direction and the corresponding schematic animation are generated, specifically:

[0156] Adjust the radius of the target visual area and the number of secondary clear areas based on the danger level, including:

[0157] Set the radius adjustment rule, that is, the radius r of the target visual area after adjusting the danger level and the corrected head linear speed focus :

[0158] Where δ represents the scaling factor, δ∈[1,3); based on the focal length of the eye, the corresponding r is obtained by reverse calculation focus The viewing angle η′, Indicates the speed correction factor, v u_max Indicates the user's maximum walking speed. In this embodiment,

[0159] Set the second clear area generation threshold L0 and collision time threshold T0, if L danger > L0 and T < T0, a clear area is generated for each new object independently, and the number of clear areas is not less than 3;

[0160] When generating a new second clear area, dynamically adjust the dynamic blur intensity around the second clear area, σ′ b (d)=σ max ·(1-L danger );

[0161] Calculate the avoidance direction, specifically:

[0162] Calculate the optimal avoidance vector Among them, normalize() represents the normalization function; represents the vertical axis,

[0163] Create along Schematic diagram of the direction of flow; and set the transparency of the animation l(t) = 0.5 + 0.5sin(2πft), where the frequency f = 2Hz, to achieve the breathing light effect;

[0164] Then fix the center point of the schematic image at the edge of the target visual area, that is, in represents the coordinates of the center point of the schematic animation in the head coordinate system, θ represents the offset, and θ≥2°.

[0165] The present invention also provides an XR virtual-reality scene rendering optimization system, comprising: a processor, a memory and a communication module connected to the processor, and a head tracking device, an eye tracking device, a body vibration tracking device and a head-mounted display device connected to the processor. The system is used to execute the aforementioned XR virtual-reality scene rendering optimization method.

[0166] In this embodiment, the head tracking device includes an IMU and a camera (6DoF SLAM) on the head; the eye tracking device includes an eye tracker and an infrared camera; and the body vibration tracking device includes an IMU installed on the user's waist and shoulders.

[0167] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the XR virtual-reality scene rendering optimization method.

[0168] The embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. The embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical fiber cable, RF, etc., or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or portion of code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, as well as combinations of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or operations, or can be implemented using a combination of dedicated hardware and computer instructions.

[0170] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any changes or modifications.

Claims

1. An XR virtual-reality scene rendering optimization method, characterized in that: The method comprises: Obtaining real visual input signals within the visual area; Acquire a virtual reality signal, render the virtual reality signal through a virtual reality engine, and fuse it with a real visual input signal to generate a fused visual input signal; Collect eye movement signals, track the eye gaze area, and obtain the target visual area; and combine the synchronously collected head rotation signals and body vibration signals to correct the target visual area and obtain the corrected target visual area; Acquire the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendering in the corrected target visual area and the blurriness of the surrounding area; One or more dynamic rendering areas are set in the peripheral area to render moving objects that enter the target visual area from outside the visual area; By analyzing the danger of moving objects, a schematic animation of how to avoid the moving objects is rendered in the target visual area.

2. The XR virtual-reality scene rendering optimization method according to claim 1, characterized in that: The obtaining of a real visual input signal within the visual area comprises: The data collection area of each sensor is calibrated using the checkerboard calibration method to match the data collection area of each sensor with the preset visual area. Unify the clocks of multiple sensors through the Precision Time Protocol (PTP). Acquire real-scene data within the data collection area from multiple sensors at a preset collection frequency, pre-process the real-scene data, and store it locally. The sensors include RGB camera, depth camera, infrared camera, and lidar.

3. The XR virtual-reality scene rendering optimization method according to claim 2, characterized in that: The step of obtaining a virtual reality signal, rendering the virtual reality signal through a virtual reality engine, and fusing the virtual reality signal with a real visual input signal to generate a fused visual input signal includes: Acquire virtual reality data from a database, divide the virtual reality data into a plurality of virtual frame data, and add a time tag to each virtual frame data; Acquire real scene data, divide the real scene data into multiple real frame data, and add a time label 2 to each real frame data; Align the virtual frame data and the real frame data on the time axis; The SLAM real-time positioning algorithm is used to construct the environmental point cloud map of the real scene in real time, and calibrate the coordinates of the objects in the real scene. i_true and the corresponding pose quaternion q i_ture ; Extract the coordinates of virtual objects from the virtual reality data and match them with the coordinates and poses of real objects in the real scene. Specifically: v i_v =R(q i_ture )·v i +p i_ture ; Among them, v i_v Represents the coordinates of the virtual object in the world coordinate system, v i Represents the virtual object coordinates, where R(q i_ture ) represents the pose quaternion q i_ture The obtained rotation matrix.

4. The XR virtual-reality scene rendering optimization method according to claim 3, characterized in that: The eye movement signal is collected, the eye gaze area is tracked, and the target visual area is obtained; Combined with the synchronously collected head rotation signal and body vibration signal, the target visual area is corrected to obtain the corrected target visual area, including: The IMU and camera installed on the user's head capture the head rotation signal and obtain the head movement speed, acceleration and movement direction data; Continuously capture the pupil position of the eye through an infrared camera; Obtain body vibration acceleration through the IMU on the user's waist or shoulder; Construct a world coordinate system, a head coordinate system, an evolution coordinate system, and a body coordinate system; the head coordinate system has the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system has the center of eyeball rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system has the center of the torso as its origin and is offset with body vibration; Constructing the state vector in, Respectively represent the position vector of the head in the world coordinate system at time t, the head posture quaternion after conversion from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head movement angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system; Use the state vector to predict head position, head pose quaternion and eye movement vector, including: Predict head pose quaternion: in, represents quaternion multiplication, exp(·) represents exponential mapping; represents the predicted head pose quaternion; Δt represents the time interval; Predicting head position: Considering the impact of body vibration on head position, the prediction model of head position is: in, Represents the rotation matrix that transforms the world coordinate system obtained by the head pose quaternion to the head coordinate system, a IMU Indicates the original angular velocity of the IMU of the head; Predict eye movement vectors: The head position, head pose quaternion and eye position are fused, including: The predicted eye movement vector is corrected by the predicted head position and the predicted head posture quaternion, specifically: The eye movement vector in the head coordinate system is: in, Represents the head pose quaternion transformed from eye coordinates to head coordinate system, Represents the rotation matrix from the eye coordinate system to the head coordinate system, The eye movement vector in the world coordinate system is: Assume that the effect of body vibration on the head conforms to the second-order system, that is: in, represents the head acceleration, represents the vibration transmission damping coefficient, w n Indicates the natural vibration frequency of the body, w n ∈[4,6]; s represents the vibration transmission speed; Will Convert to the head coordinate system: Then Convert to world coordinate system: pass Correction Get the corrected Then, the position vector of the target visual area in the world coordinate system after adding body vibration offset compensation is Among them, d e Indicates gaze distance; According to the 3σ principle, the synthetic error perspective is obtained, specifically: the synthetic error perspective is Among them, σ eye , σ head , σ zd represent eye movement tracking accuracy, head movement prediction error, and vibration compensation residual, respectively; Under the condition of covering 99% confidence interval, the coefficient k = 2.58; According to the focal length of the eye, the radius corresponding to the synthetic error angle is the radius r of the target visual area. base .

5. The XR virtual-reality scene rendering optimization method according to claim 4, characterized in that: During the display process, maintaining the clarity of the rendering within the corrected target visual area and the blurriness of the surrounding area includes: Get the predicted target visual area, and use the predicted target visual area as the center to render outward in three levels based on distance, including: The target gaze area is divided into core area, transition area and fuzzy area in sequence; Set the core area to 100% resolution, the transition area to 50% resolution, and the blur area to 25% resolution; Based on head movement, construct a pixel motion vector model: in, represents the 3D position of the pixel in the camera coordinate system, which is obtained by back-projecting the depth map; Δt′ represents the frame interval time; z(x,y) represents the pixel depth; Perform fuzzy intensity grading, specifically: Among them, r1 represents the viewing angle of the dynamic blur area; d represents the angular distance from the pixel to the center of the target visual area; the maximum blurred pixel value σ max ≥8 pixels; Apply the pixel motion vector model to each pixel to obtain the pixel value after motion blur: Among them, Q represents the number of sampling points, M x (x,y) represents the x value obtained by the pixel motion vector model calculation, M y (x,y) represents the y value obtained by calculating the pixel motion vector model.

6. The XR virtual-reality scene rendering optimization method according to claim 5, characterized in that: The step of setting one or more dynamic rendering areas in the peripheral area to render moving objects entering the target visual area from outside the visual area includes: Input the video frames of the surrounding area and perform motion tracking based on the optical flow algorithm; specifically: Optical flow-based motion saliency tracking of pixels containing novel objects in, Represents the optical flow vector of the pixel, which is calculated by the Lucas-Kanade algorithm, f represents the average optical flow, σ f The pixel frame rate representing the optical flow; If S(x,y)>τ move , then it is judged that a new object is contained; among them, τ move Indicates the preset significance threshold; Detect new object categories and bounding boxes using the YOLOv5 model, and then track the objects using the tracker. Take the center C of the new object obj is the focus, and the radius η1 = 3° defines the second clear area, i.e., the dynamic rendering area; Improve the resolution of the second clearest area from 25% to 100% through a super-resolution network; The rendering image of the second cleaning area and the rendering image of the target visual area are displayed simultaneously.

7. The XR virtual-reality scene rendering optimization method according to claim 6, characterized in that: The method of rendering a schematic animated diagram of avoiding the moving object in the target visual area by analyzing the danger of the moving object includes: Obtain motion trajectory analysis and analyze the dangerousness of the object based on the motion trajectory; Generate avoidance movement directions and corresponding schematic animations based on the danger level of the object; Obtaining motion trajectory analysis and analyzing the dangerousness of the object based on the motion trajectory includes the following steps: Get the motion trajectory of the new object and calculate the danger level L danger , specifically including: Obtain the position and velocity vector of the new object from the tracker, and obtain the object category from YOLOv5, which includes people, vehicles, and special-shaped obstacles; Building a motion trajectory prediction model Among them, p t represents the current position of the new object, The velocity vector of the new object, a max represents the preset maximum assumed acceleration, and Δt″ represents the prediction time interval; Calculating collision time Among them, p t_u Indicates the position of the human body, v t_u Represents the velocity vector of the human body; Constructing a hazard level balance model Among them, α1, α2, and α3 are weight coefficients, τ represents the attenuation coefficient of collision time, and v max Indicates the maximum reference speed, s d Indicates the risk factor.

8. The XR virtual-reality scene rendering optimization method according to claim 7, characterized in that: The method of generating an avoidance motion direction and a corresponding schematic animation according to the dangerousness of the object comprises the following steps: Adjust the radius of the target visual area and the number of secondary clear areas based on the danger level, including: Set the radius adjustment rule, that is, the radius r of the target visual area after adjusting the danger level and the corrected head linear speed focus : Where δ represents the scaling factor, δ∈[1,3); based on the focal length of the eye, the corresponding r is obtained by reverse calculation focus The viewing angle η′, Indicates the speed correction factor, v u_max Indicates the user's maximum walking speed; Set the second clear area generation threshold L0 and collision time threshold T0, if L danger > L0 and T < T0, a clear area is generated for each new object independently, and the number of clear areas is not less than 3; When generating a new second clear area, dynamically adjust the dynamic blur intensity around the second clear area, σ b ′(d)=σ max ·(1-L danger ); Calculate the avoidance direction, specifically: Calculate the optimal avoidance vector Among them, normalize() represents the normalization function; represents the vertical axis, Create along Schematic diagram of the direction of flow; and set the transparency of the animation l(t) = 0.5 + 0.5sin(2πft), where the frequency f = 2Hz, to achieve the breathing light effect; Then fix the center point of the schematic image at the edge of the target visual area, that is, in represents the coordinates of the center point of the schematic animation in the head coordinate system, θ represents the offset, and θ≥2°.

9. An XR virtual-reality scene rendering optimization system, comprising: A processor and a memory and a communication module connected to the processor, as well as a head tracking device, an eye tracking device, a body vibration tracking device and a head-mounted display device connected to the processor, characterized in that the system is used to execute an XR virtual-reality scene rendering optimization method as described in any one of claims 1-8 above.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement an XR virtual-reality scene rendering optimization method as described in any one of claims 1 to 8.

Images