A method and system for optimizing XR virtual and real scene rendering

Abstract

This invention relates to the field of XR technology, and particularly to an XR virtual-real scene rendering optimization method and system. The method includes: acquiring a real visual input signal within a visual region; acquiring a virtual reality signal, rendering the virtual reality signal using a virtual reality engine, and then fusing it with the real visual input signal to generate a fused visual input signal; acquiring eye movement signals, tracking the eye's gaze area, and acquiring a target visual region; and combining synchronously acquired head rotation signals and body vibration signals to correct the target visual region, acquiring a corrected target visual region; acquiring the fused visual input signal and displaying it in the corrected target visual region; maintaining the clarity of the rendering within the corrected target visual region and the blurriness of its surrounding areas; setting one or more dynamic rendering regions to render moving objects entering the target visual region from outside the visual region; and rendering a schematic animation of avoiding moving objects in the target visual region.

Description

Technical Field

[0001] This invention relates to the field of XR technology, and in particular to an XR virtual-real scene rendering optimization method and system. Background Technology

[0002] XR, or Extended Reality, includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It enhances the immersive experience for users by combining or replacing computer-generated virtual content with the real world.

[0003] In existing technologies, the division of the rendering region during gaze region rendering is typically static. This means that high / low resolution areas are fixedly divided based on the gaze point of the current frame, and then binocular frames are generated via Y-axis mirroring. Furthermore, while the gaze region position is predicted by tracking head and eye rotations, the impact of vibrations caused by user body movement in dynamic application scenarios is not considered, which can easily lead to rendering position deviations. Summary of the Invention

[0004] This invention, when rendering virtual and real objects within the gaze area, simultaneously tracks the user's body vibrations in addition to tracking head and eye movements. In dynamic application scenarios, it corrects the prediction of the gaze area's position based on body vibrations, ensuring the accuracy of the rendered position.

[0005] The technical solution proposed in this invention is: an XR virtual-real scene rendering optimization method, the method comprising:

[0006] Acquire the actual visual input signal within the visual region;

[0007] The virtual reality signal is acquired, rendered by the virtual reality engine, and then fused with the real visual input signal to generate a fused visual input signal.

[0008] Collect eye movement signals, track the eye gaze area, and obtain the target visual area; and combine the synchronously collected head rotation signals and body vibration signals to correct the target visual area and obtain the corrected target visual area.

[0009] Acquire the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendered area within the corrected target visual area and the blur of its surrounding areas;

[0010] Set one or more dynamic rendering areas in the surrounding area to render moving objects that enter the target visual area from outside the visual area;

[0011] By analyzing the danger of moving objects, a schematic animation of avoiding moving objects is rendered in the target's visual area.

[0012] Preferably, acquiring the real visual input signal within the visual region includes:

[0013] The data acquisition area of ​​each sensor is calibrated using a checkerboard calibration method to match the data acquisition area of ​​each sensor with the preset visual area.

[0014] The clocks of multiple sensors are unified through the Precision Time Protocol (PTP); real scene data within the data acquisition area is acquired from multiple sensors according to a preset acquisition frequency; the real scene data is preprocessed and then stored locally.

[0015] The sensors include an RGB camera, a depth camera, an infrared camera, and a LiDAR.

[0016] Preferably, the step of acquiring the virtual reality signal, rendering the virtual reality signal through a virtual reality engine, and then fusing it with the real visual input signal to generate a fused visual input signal includes:

[0017] Virtual reality data is retrieved from the database, divided into multiple virtual frame data, and a time tag is added to each virtual frame data.

[0018] Acquire real-world scene data, divide the real-world scene data into multiple real-world frame data, and add a time tag 2 to each real-world frame data;

[0019] Align virtual frame data and real frame data on the timeline;

[0020] The SLAM real-time localization algorithm is used to construct a real-world point cloud map of the scene and to label the coordinates of objects within the scene. and the corresponding pose quaternion ;

[0021] The coordinates of virtual objects are extracted from virtual reality data and matched with the coordinates and poses of real objects within the real scene. Specifically:

[0022] ;in, This represents the coordinates of a virtual object in the world coordinate system. Represents the coordinates of a virtual object, where, Represents the quaternion of pose The obtained rotation matrix.

[0023] Preferably, the step of acquiring eye movement signals, tracking the eye's gaze area, and obtaining the target visual region; and combining synchronously acquired head rotation signals and body vibration signals to correct the target visual region, and obtaining the corrected target visual region, includes:

[0024] By capturing head rotation signals through an IMU and camera placed on the user's head, data on head movement speed, acceleration, and direction of movement can be obtained.

[0025] The position of the pupil is continuously captured using an infrared camera;

[0026] Body vibration acceleration is obtained through an IMU located in the user's waist or shoulder;

[0027] Construct a world coordinate system, a head coordinate system, an eye coordinate system, and a body coordinate system; the head coordinate system takes the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system takes the center of eye rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system takes the center of the torso as its origin and shifts with body vibration.

[0028] Constructing state vectors ;in, They represent in At any given moment, the head position vector in the world coordinate system, the head posture quaternion after transformation from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head motion angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system.

[0029] Predicting head position, head pose quaternions, and eye movement vectors using state vectors includes:

[0030] Predicting head posture quaternions: ,in, To represent quaternion multiplication, Indicates an exponential mapping; Represents the predicted head pose quaternion; Indicates a time interval;

[0031] Predicting head position: Considering the impact of body vibration on head position, the head position prediction model is as follows:

[0032] ;in, This represents the rotation matrix that transforms the world coordinate system obtained from the head pose quaternion to the head coordinate system. This represents the raw angular velocity of the head IMU;

[0033] Predicting eye movement vectors: ;

[0034] The fusion of head position, head pose quaternion, and eye position includes:

[0035] The predicted eye movement vector is corrected using the predicted head position and predicted head pose quaternions, specifically:

[0036] The eye movement vector in the head coordinate system is: ;in, This represents the head pose quaternion that transforms the coordinates from the eye coordinate system to the head coordinate system. This represents the rotation matrix used to transform from the eye coordinate system to the head coordinate system.

[0037] The eye movement vector in the world coordinate system is: ;

[0038] Assume that the effect of body vibration on the head conforms to a second-order system, that is:

[0039] ,in, Indicates head acceleration. Indicates the vibration transmission damping coefficient. ; Indicates the body's natural vibration frequency. ; Indicates the speed of vibration transmission;

[0040] Will Transform to head coordinate system: ;

[0041] Then Transform to world coordinate system: ;

[0042] pass Correction , obtained the corrected Then, the position vector of the target's visual region in the world coordinate system after adding body vibration offset compensation. ,in, Indicates the viewing distance;

[0043] according to The principle is to obtain the synthesis error perspective, specifically: the synthesis error perspective. ,in, These represent eye movement tracking accuracy, head movement prediction error, and vibration compensation residual, respectively.

[0044] Under the condition of covering a 99% confidence interval, the coefficient ;

[0045] The radius of the target visual area is determined by the radius corresponding to the synthesized error angle based on the eye's focal length. .

[0046] Preferably, maintaining the sharpness of the rendered target visual area and the blurriness of its surrounding areas during the display process includes:

[0047] Obtain the predicted target visual region, and render it in three levels outwards from the predicted target visual region in terms of distance, including:

[0048] The target gaze area is divided into a core area, a transition area, and a blurred area in sequence;

[0049] Set the core area to 100% resolution, the transition area to 50% resolution, and the blurry area to 25% resolution;

[0050] Based on head motion, construct a pixel motion vector model:

[0051] ;in, The 3D position of a pixel in the camera coordinate system is obtained by backprojection of the depth map; Indicates the frame interval time; Indicates pixel depth;

[0052] The fuzzy intensity classification is performed as follows:

[0053] ;in, The perspective representing a dynamically blurred region; Represents the angular distance from a pixel to the center of the target visual region; maximum blurred pixel value. Pixel;

[0054] Apply a pixel motion vector model to each pixel to obtain the pixel value after dynamic blurring: ,in, Indicates the number of sampling points. The pixel motion vector model is obtained from the calculation. value, The pixel motion vector model is obtained from the calculation. value.

[0055] Preferably, setting one or more dynamic rendering areas in the surrounding area to render moving objects entering the target visual area from outside the visual area includes:

[0056] Input video frames from the surrounding area and perform motion tracking based on an optical flow algorithm; specifically:

[0057] Motion saliency tracking based on optical flow includes pixels containing new objects. ,in, The optical flow vector representing a pixel is calculated using the Lucas-Kanade algorithm. Indicates average optical flow. The pixel frame rate represents the optical flow.

[0058] if If it contains a new object, then it is determined that the object contains a new object; where, This indicates the preset significance threshold;

[0059] Using the YOLOv5 model, new object categories and bounding boxes are detected, and then the objects are tracked by a tracker.

[0060] Center of the new object With focus, radius Define the second clear area, namely the dynamic rendering area;

[0061] The resolution of the second sharp region was increased from 25% to 100% using a super-resolution network.

[0062] The rendered images of the second cleaning area and the target visual area are displayed simultaneously.

[0063] Preferably, the step of rendering a schematic animation of avoiding the moving object in the target's visual area by analyzing the danger of the moving object includes:

[0064] Acquire motion trajectory analysis and analyze the danger of objects based on the motion trajectory;

[0065] Based on the danger level of the object, generate the evasive movement direction and corresponding animated diagram;

[0066] The process of acquiring motion trajectory analysis and analyzing the hazard of an object based on that trajectory includes the following steps:

[0067] Obtain the trajectory of the new object and calculate its hazard level. Specifically, it includes:

[0068] The position and velocity vectors of new objects are obtained from the tracker, and the object category is obtained from YOLOv5, which includes people, vehicles, and irregular obstacles.

[0069] Constructing a motion trajectory prediction model ,in, Indicates the current position of the new object. The velocity vector of the new object, This represents the preset maximum assumed acceleration. Indicates the prediction time interval;

[0070] Calculate collision time ,in, Indicates the location of the human body. This represents the velocity vector of the human body.

[0071] Constructing a hazard level balance model ,in, These are the weighting coefficients. The decay coefficient representing the collision time. Indicates the maximum reference speed. Indicates the risk level.

[0072] Preferably, generating the evasive movement direction and corresponding animated diagram based on the danger of the object includes the following steps:

[0073] Adjust the radius of the target's visual area and the number of second clear areas according to the hazard level, specifically including:

[0074] Set the radius adjustment rule, which is the radius of the target visual area adjusted based on the danger level and the corrected head linear velocity. :

[0075] ;in, Indicates the scaling factor. Based on the eye's focal length, the corresponding... perspective , This represents the speed correction factor. Indicates the user's maximum walking speed;

[0076] Set the threshold for generating the second clear area. and collision time threshold ,if, and Then, a clear region is generated independently for each new object, and the number of clear regions is no less than 3;

[0077] When generating a new second sharp region, the dynamic blur intensity around the second sharp region is dynamically adjusted. ;

[0078] Calculate the evasion direction as follows:

[0079] Calculate the optimal avoidance vector ,in, Represents the normalization function; Indicates the vertical axis. .

[0080] Create along Animated illustration illustrating directional flow; and the animation's transparency can be set. , where frequency , used to achieve a breathing light effect;

[0081] Then fix the center point of the illustrative animation at the edge of the target visual area, that is ,in This indicates the coordinates of the center point of the illustrative animation in the head coordinate system. Indicates the offset. .

[0082] An XR virtual-real scene rendering optimization system includes: a processor and a memory and communication module connected to the processor, as well as a head tracking device, an eye tracking device, a body vibration tracking device and a head-mounted display device connected to the processor. The system is used to execute the XR virtual-real scene rendering optimization method.

[0083] A computer-readable storage medium storing a computer program that is executed by a processor to implement the XR virtual-real scene rendering optimization method.

[0084] The beneficial effects of this invention are:

[0085] 1. This invention performs progressive dynamic blurring on the sharpness of the target gaze region, dividing the target gaze region into a core region, a transition region, and a blurred region in sequence; and supports multi-focal generation (second sharp region).

[0086] 2. This invention tracks new objects entering outside the target's gaze area in real time. It renders different new objects through multiple secondary clear regions. Based on predictions of the future target gaze area and analysis of the new object's hazard factor, the rendering strategy is adjusted in real time (e.g., adjusting the range of the target gaze area and the number of secondary clear regions). Furthermore, based on the predicted trajectory of the new object, a corresponding illustrative animation is dynamically rendered to guide the user in avoiding danger. This allows the system to not only passively display information but also actively assist in safety, adapting to both static and dynamic usage scenarios. Attached Figure Description

[0087] Figure 1 This is a flowchart of an XR virtual-real scene rendering optimization method according to the present invention. Detailed Implementation

[0088] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious modifications will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0089] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0090] refer to Figure 1 The technical solution provided by this invention is: an XR virtual-real scene rendering optimization method and system, comprising the following steps:

[0091] Step 1: Obtain the actual visual input signal within the visual region; specifically:

[0092] The data acquisition area of ​​each sensor is calibrated using the checkerboard calibration method (calibrarteCamer function in OpenCV) to match the data acquisition area of ​​each sensor with the preset visual area. The purpose of this step is to eliminate lens distortion and establish a mapping between the pixel coordinate system and the physical coordinate system.

[0093] The clocks of multiple sensors are unified through the Precision Time Protocol (PTP); real scene data within the data acquisition area is acquired from multiple sensors according to a preset acquisition frequency; the real scene data is preprocessed and then stored locally.

[0094] The sensors include an RGB camera, a depth camera, an infrared camera, and a LiDAR.

[0095] Step 2: Acquire the virtual reality signal, render it using the virtual reality engine, and then fuse it with the real visual input signal to generate a fused visual input signal. Specifically:

[0096] Virtual reality data is retrieved from the database, divided into multiple virtual frame data, and a time tag is added to each virtual frame data.

[0097] Acquire real-world scene data, divide the real-world scene data into multiple real-world frame data, and add a time tag 2 to each real-world frame data;

[0098] Align virtual frame data and real frame data on the timeline;

[0099] The SLAM real-time localization algorithm is used to construct a real-world point cloud map of the scene and to label the coordinates of objects within the scene. and the corresponding pose quaternion ;

[0100] The coordinates of virtual objects are extracted from virtual reality data and matched with the coordinates and poses of real objects within the real scene to achieve anchoring of the virtual objects. This process includes the following steps:

[0101] ;in, This represents the coordinates of a virtual object in the world coordinate system. Represents the coordinates of a virtual object, where, Represents the quaternion of pose The obtained rotation matrix.

[0102] In addition, it also includes handling the occlusion relationship between virtual and real objects, specifically:

[0103] In this embodiment, a ToF / LiDAR image of the real scene is obtained, and a depth buffer for the virtual object is rendered. Then, the depth values ​​of the virtual and real objects are compared pixel by pixel to determine the occlusion relationship. For example, if the depth value of the virtual object is less than that of the real object, the virtual object is in front; otherwise, the real object is in front.

[0104] In some preferred embodiments, the method further includes blending the pixels of the unoccluded virtual object with the pixels of the real object according to transparency, and performing lighting estimation and material matching. These are all prior art techniques and will not be described in detail here.

[0105] Step 3: Acquire eye movement signals, track the eye's gaze area, and obtain the target visual region; then, combine this with synchronously acquired head rotation and body vibration signals to correct the target visual region, obtaining the corrected target visual region; specifically:

[0106] By capturing head rotation signals through an IMU and camera placed on the user's head, data on head movement speed, acceleration, and direction of movement can be obtained.

[0107] The position of the pupil is continuously captured using an infrared camera;

[0108] Body vibration acceleration is obtained through an IMU located in the user's waist or shoulder;

[0109] Construct a world coordinate system, a head coordinate system, an eye coordinate system, and a body coordinate system; the head coordinate system takes the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system takes the center of eye rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system takes the center of the torso as its origin and shifts with body vibration.

[0110] Constructing state vectors ;in, They represent in At any given moment, the head position vector in the world coordinate system, the head posture quaternion after transformation from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head motion angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system.

[0111] Predicting head position, head pose quaternions, and eye movement vectors using state vectors includes:

[0112] Predicting head posture quaternions: ,in, To represent quaternion multiplication, Indicates an exponential mapping; Represents the predicted head pose quaternion; Indicates a time interval;

[0113] Predicting head position: Considering the impact of body vibration on head position, the head position prediction model is as follows:

[0114] ;in, This represents the rotation matrix that transforms the world coordinate system obtained from the head pose quaternion to the head coordinate system. This represents the raw angular velocity of the head IMU;

[0115] Predicting eye movement vectors: ;

[0116] The fusion of head position, head pose quaternion, and eye position includes:

[0117] The predicted eye movement vector is corrected using the predicted head position and predicted head pose quaternions, specifically:

[0118] The eye movement vector in the head coordinate system is: ;in, This represents the head pose quaternion that transforms the coordinates from the eye coordinate system to the head coordinate system. This represents the rotation matrix used to transform from the eye coordinate system to the head coordinate system.

[0119] The eye movement vector in the world coordinate system is: ;

[0120] Assume that the effect of body vibration on the head conforms to a second-order system, that is:

[0121] ,in, Indicates head acceleration. Indicates the vibration transmission damping coefficient. ; Indicates the body's natural vibration frequency. ; Indicates the speed of vibration transmission;

[0122] Will Transform to head coordinate system: Then Transform to world coordinate system: ;pass Correction , obtained the corrected .

[0123] Finally, the position vector of the target's visual region in the world coordinate system after adding body vibration offset compensation is obtained. ,in, Indicates the viewing distance;

[0124] In this embodiment, the prediction process described above is based on a pre-trained LSTM network and its internal extended Kalman filter algorithm. This embodiment uses a 3-layer LSTM network with 128 hidden layers, capable of outputting future... The position vector of the target view area in the frame.

[0125] according to The principle is to obtain the synthesis error perspective, specifically: the synthesis error perspective. ,in, These represent the eye movement tracking accuracy, head movement prediction error, and vibration compensation residual, respectively; the synthetic error perspective here can be regarded as the perspective of the target visual region.

[0126] Under the condition of covering a 99% confidence interval, the coefficient ;

[0127] The radius of the target visual area is determined by the radius corresponding to the synthesized error angle based on the eye's focal length. In some preferred embodiments, the radius can also be directly used.

[0128] For example, in this embodiment:

[0129] , The target area range is ,in The accuracy is determined by the eye-tracking device; it depends on the device, for example, the accuracy of the Tobii4C device is 0.3 degrees. The standard deviation of the head position error can be obtained by extracting it from the covariance of the extended Kalman filter. This represents the standard deviation of vibration compensation measured experimentally.

[0130] Step 4: Acquire the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendered area within the corrected target visual area and the blurriness of its surrounding areas; specifically:

[0131] Obtain the predicted target visual region, and render it in three levels outwards from the predicted target visual region in terms of distance, including:

[0132] The target gaze area is divided into a core area, a transition area, and a blurred area in sequence;

[0133] Set the core area to 100% resolution, the transition area to 50% resolution, and the blurry area to 25% resolution;

[0134] Based on head motion, construct a pixel motion vector model:

[0135] ;in, The 3D position of a pixel in the camera coordinate system is obtained by backprojection of the depth map; Indicates the frame interval time; Indicates pixel depth;

[0136] The fuzzy intensity classification is performed as follows:

[0137] ;in, The perspective representing a dynamically blurred region; Represents the angular distance from a pixel to the center of the target visual region; maximum blurred pixel value. Pixel;

[0138] For example, in this embodiment, in the user's head-mounted display device, the core area ( Maintain clarity and avoid blurring; transition areas ( The fuzziness intensity increases linearly; the fuzzy region ( ), constant maximum fuzziness.

[0139] Apply a pixel motion vector model to each pixel to obtain the pixel value after dynamic blurring: ,in, Indicates the number of sampling points. The pixel motion vector model is obtained from the calculation. value, The pixel motion vector model is obtained from the calculation. value.

[0140] Step 5: Set up one or more dynamic rendering areas within the surrounding area to render moving objects that enter the target visual area from outside the visual area; specifically:

[0141] Input video frames from the surrounding area and perform motion tracking based on optical flow algorithms;

[0142] Specifically, it involves: motion saliency tracking based on optical flow for pixels containing new objects. ;in, The optical flow vector representing a pixel is calculated using the Lucas-Kanade algorithm. Indicates average optical flow. The pixel frame rate represents the optical flow.

[0143] if If it contains a new object, then it is determined that the object contains a new object; where, This indicates the preset significance threshold;

[0144] Using the YOLOv5 model, new object categories and bounding boxes are detected, and then the objects are tracked using the KernelizedCorrelation Filter.

[0145] Center of the new object With focus, radius Define the second clear area, namely the dynamic rendering area;

[0146] The resolution of the second sharp region was increased from 25% to 100% using a super-resolution network.

[0147] The rendered images of the second cleaning area and the target visual area are displayed simultaneously.

[0148] Step 6: Based on the hazard analysis of the moving object, render a schematic animation of avoiding the moving object in the target's visual area, specifically as follows:

[0149] Obtain motion trajectory analysis and analyze the hazard of the object based on the motion trajectory. This includes the following steps:

[0150] Obtain the trajectory of the new object and calculate its hazard level. Specifically, it includes:

[0151] The position and velocity vectors of new objects are obtained from the tracker, and the object category is obtained from YOLOv5, which includes people, vehicles, and irregular obstacles.

[0152] Constructing a motion trajectory prediction model ,in, Indicates the current position of the new object. The velocity vector of the new object, This represents the preset maximum assumed acceleration. Indicates the prediction time interval;

[0153] Calculate collision time ,in, Indicates the location of the human body. This represents the velocity vector of the human body.

[0154] Constructing a hazard level balance model ,in, These are the weighting coefficients. The decay coefficient representing the collision time. Indicates the maximum reference speed. This indicates the risk factor. In this embodiment, the risk factor for pedestrians is 0.2, and the risk factor for vehicles is 0.8.

[0155] Based on the danger level of the object, generate the evasive movement direction and corresponding animated diagram, specifically:

[0156] Adjust the radius of the target's visual area and the number of second clear areas according to the hazard level, specifically including:

[0157] Set the radius adjustment rule, which is the radius of the target visual area adjusted based on the danger level and the corrected head linear velocity. :

[0158] ;in, Indicates the scaling factor. Based on the eye's focal length, the corresponding... perspective , This represents the speed correction factor. This indicates the user's maximum walking speed. In this embodiment, .

[0159] Set the threshold for generating the second clear area. and collision time threshold ,if, and Then, a clear region is generated independently for each new object, and the number of clear regions is no less than 3;

[0160] When generating a new second sharp region, the dynamic blur intensity around the second sharp region is dynamically adjusted. ;

[0161] Calculate the evasion direction as follows:

[0162] Calculate the optimal avoidance vector ,in, Represents the normalization function; Indicates the vertical axis. .

[0163] Create along Animated illustration illustrating directional flow; and the animation's transparency can be set. , where frequency , used to achieve a breathing light effect;

[0164] Then fix the center point of the illustrative animation at the edge of the target visual area, that is ,in This indicates the coordinates of the center point of the illustrative animation in the head coordinate system. Indicates the offset. .

[0165] The present invention also provides an XR virtual-real scene rendering optimization system, comprising: a processor and a memory and a communication module connected to the processor, as well as a head tracking device, an eye tracking device, a body vibration tracking device and a head-mounted display device connected to the processor, wherein the system is used to execute the aforementioned XR virtual-real scene rendering optimization method.

[0166] In this embodiment, the head tracking device includes an IMU for the head and a camera (6DoF SLAM); the eye tracking device includes an eye tracker and an infrared camera; and the body vibration tracking device includes IMUs installed on the user's waist and shoulders.

[0167] The present invention also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement the aforementioned XR virtual-real scene rendering optimization method.

[0168] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. The embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0169] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0170] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the principles described, the implementation of the present invention may have any changes or modifications.

Claims

1. An XR virtual-real scene rendering optimization method, characterized in that, The method includes: Acquire the actual visual input signal within the visual region; The virtual reality signal is acquired, rendered by the virtual reality engine, and then fused with the real visual input signal to generate a fused visual input signal. The system acquires eye movement signals, tracks the eye's gaze area, and obtains the target visual region. It then combines these with simultaneously acquired head rotation and body vibration signals to correct the target visual region, resulting in a corrected target visual region. This includes: By capturing head rotation signals through an IMU and camera placed on the user's head, data on head movement speed, acceleration, and direction of movement can be obtained. The position of the pupil is continuously captured using an infrared camera; Body vibration acceleration is obtained through an IMU located in the user's waist or shoulder; Construct a world coordinate system, a head coordinate system, an eye coordinate system, and a body coordinate system; the head coordinate system takes the center of the user's head-mounted display device as its origin and rotates or translates with head movement; the eye coordinate system takes the center of eye rotation as its origin and is offset by a fixed amount H relative to the head coordinate system; the body coordinate system takes the center of the torso as its origin and shifts with body vibration. Constructing state vectors ;in, They represent in At any given moment, the head position vector in the world coordinate system, the head posture quaternion after transformation from the world coordinate system to the head coordinate system, the head linear velocity in the world coordinate system, the head motion angular velocity in the head coordinate system, the eye movement vector in the eye coordinate system, and the body vibration acceleration in the body coordinate system. Predicting head position, head pose quaternions, and eye movement vectors using state vectors includes: Predicting head posture quaternions: ,in, To represent quaternion multiplication, Indicates an exponential mapping; Represents the predicted head pose quaternion; Indicates a time interval; Predicting head position: Considering the impact of body vibration on head position, the head position prediction model is as follows: ;in, This represents the rotation matrix that transforms the world coordinate system obtained from the head pose quaternion to the head coordinate system. This represents the raw angular velocity of the head IMU; Predicting eye movement vectors: ; The fusion of head position, head pose quaternion, and eye position includes: The predicted eye movement vector is corrected using the predicted head position and predicted head pose quaternions, specifically: The eye movement vector in the head coordinate system is: ;in, This represents the head pose quaternion that transforms the coordinates from the eye coordinate system to the head coordinate system. This represents the rotation matrix used to transform from the eye coordinate system to the head coordinate system. The eye movement vector in the world coordinate system is: ; Assume that the effect of body vibration on the head conforms to a second-order system, that is: ,in, Indicates head acceleration. Indicates the vibration transmission damping coefficient. ; Indicates the body's natural vibration frequency. ; Indicates the speed of vibration transmission; Will Transform to head coordinate system: ; Then Transform to world coordinate system: ; pass Correction , obtained the corrected Then, the position vector of the target's visual region in the world coordinate system after adding body vibration offset compensation. ,in, Indicates the viewing distance; according to The principle is to obtain the synthesis error perspective, specifically: the synthesis error perspective is... ,in, These represent eye movement tracking accuracy, head movement prediction error, and vibration compensation residual, respectively. Under the condition of covering a 99% confidence interval, the coefficient ; The radius of the target visual area is determined by the radius corresponding to the synthesized error angle based on the eye's focal length. ; Acquire the fused visual input signal and display it in the corrected target visual area; during the display process, maintain the clarity of the rendered area within the corrected target visual area and the blur of its surrounding areas; Set one or more dynamic rendering areas in the surrounding area to render moving objects that enter the target visual area from outside the visual area; By analyzing the danger of moving objects, a schematic animation of avoiding moving objects is rendered in the target's visual area.

2. The XR virtual-real scene rendering optimization method according to claim 1, characterized in that, The acquisition of the real visual input signal within the visual region includes: The data acquisition area of ​​each sensor is calibrated using a checkerboard calibration method to match the data acquisition area of ​​each sensor with the preset visual area. The clocks of multiple sensors are unified through the Precision Time Protocol (PTP); real scene data within the data acquisition area is acquired from multiple sensors according to a preset acquisition frequency; the real scene data is preprocessed and then stored locally. The sensors include an RGB camera, a depth camera, an infrared camera, and a LiDAR.

3. The XR virtual-real scene rendering optimization method according to claim 2, characterized in that, The process of acquiring virtual reality signals, rendering them using a virtual reality engine, and then fusing them with real visual input signals to generate a fused visual input signal includes: Virtual reality data is retrieved from the database, divided into multiple virtual frame data, and a time tag is added to each virtual frame data. Acquire real-world scene data, divide the real-world scene data into multiple real-world frame data, and add a time tag 2 to each real-world frame data; Align virtual frame data and real frame data on the timeline; The SLAM real-time localization algorithm is used to construct a real-world point cloud map of the scene and to label the coordinates of objects within the scene. and the corresponding pose quaternion ; The coordinates of virtual objects are extracted from virtual reality data and matched with the coordinates and poses of real objects within the real scene. Specifically: ;in, This represents the coordinates of a virtual object in the world coordinate system. Represents the coordinates of a virtual object, where, Represents the quaternion of pose The obtained rotation matrix.

4. The XR virtual-real scene rendering optimization method according to claim 3, characterized in that, Maintaining the clarity of the rendered target visual area and the blurriness of its surrounding areas during the display process includes: Obtain the predicted target visual region, and render it in three levels outwards from the predicted target visual region in terms of distance, including: The target gaze area is divided into a core area, a transition area, and a blurred area in sequence; Set the core area to 100% resolution, the transition area to 50% resolution, and the blurry area to 25% resolution; Based on head motion, construct a pixel motion vector model: ;in, The 3D position of a pixel in the camera coordinate system is obtained by backprojection of the depth map; Indicates the frame interval time; Indicates pixel depth; The fuzzy intensity classification is performed as follows: ;in, The perspective representing a dynamically blurred region; Represents the angular distance from a pixel to the center of the target visual region; maximum blurred pixel value. Pixel; Apply a pixel motion vector model to each pixel to obtain the pixel value after dynamic blurring: ,in, Indicates the number of sampling points. The pixel motion vector model is obtained from the calculation. value, The pixel motion vector model is obtained from the calculation. value.

5. The XR virtual-real scene rendering optimization method according to claim 4, characterized in that, Setting one or more dynamic rendering areas within the surrounding area to render moving objects entering the target visual area from outside the visual area includes: Input video frames from the surrounding area and perform motion tracking based on an optical flow algorithm; specifically: Motion saliency tracking based on optical flow includes pixels containing new objects. ,in, The optical flow vector representing a pixel is calculated using the Lucas-Kanade algorithm. Indicates average optical flow. The pixel frame rate represents the optical flow. if If it contains a new object, then it is determined that the object contains a new object; where, This indicates the preset significance threshold; Using the YOLOv5 model, new object categories and bounding boxes are detected, and then the objects are tracked by a tracker. Center of the new object With focus, radius Define the second clear area, namely the dynamic rendering area; The resolution of the second sharp region was increased from 25% to 100% using a super-resolution network. The rendered images of the second cleaning area and the target visual area are displayed simultaneously.

6. The XR virtual-real scene rendering optimization method according to claim 5, characterized in that, The process of analyzing the danger posed by a moving object and rendering a schematic animation of avoiding the moving object in the target's visual area includes: Acquire motion trajectory analysis and analyze the danger of objects based on the motion trajectory; Based on the danger level of the object, generate the evasive movement direction and corresponding animated diagram; The process of acquiring motion trajectory analysis and analyzing the hazard of an object based on that trajectory includes the following steps: Obtain the trajectory of the new object and calculate its hazard level. Specifically, it includes: The position and velocity vectors of new objects are obtained from the tracker, and the object category is obtained from YOLOv5, which includes people, vehicles, and irregular obstacles. Constructing a motion trajectory prediction model ,in, Indicates the current position of the new object. The velocity vector of the new object, This represents the preset maximum assumed acceleration. Indicates the prediction time interval; Calculate collision time ,in, Indicates the location of the human body. This represents the velocity vector of the human body. Constructing a hazard level balance model ,in, These are the weighting coefficients. The decay coefficient representing the collision time. Indicates the maximum reference speed. Indicates the risk level.

7. The XR virtual-real scene rendering optimization method according to claim 6, characterized in that, The process of generating an evasive movement direction and corresponding animated diagram based on the danger of the object includes the following steps: Adjust the radius of the target's visual area and the number of second clear areas according to the hazard level, specifically including: Set the radius adjustment rule, which is the radius of the target visual area adjusted based on the danger level and the corrected head linear velocity. : ;in, Indicates the scaling factor. Based on the eye's focal length, the corresponding... perspective , This represents the speed correction factor. Indicates the user's maximum walking speed; Set the threshold for generating the second clear area. and collision time threshold ,if, and Then, a clear region is generated independently for each new object, and the number of clear regions is no less than 3; When generating a new second sharp region, the dynamic blur intensity around the second sharp region is dynamically adjusted. ; Calculate the evasion direction as follows: Calculate the optimal avoidance vector ,in, Represents the normalization function; Indicates the vertical axis. ; Create along Animated illustration illustrating directional flow; and the animation's transparency can be set. , where frequency , used to achieve a breathing light effect; Then fix the center point of the illustrative animation at the edge of the target visual area, that is ,in This indicates the coordinates of the center point of the illustrative animation in the head coordinate system. Indicates the offset. .

8. An XR virtual-real scene rendering optimization system, comprising: The system comprises a processor, a memory and a communication module connected to the processor, and a head tracking device, an eye tracking device, a body vibration tracking device and a head-mounted display device connected to the processor, characterized in that the system is used to execute an XR virtual-real scene rendering optimization method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement an XR virtual-real scene rendering optimization method according to any one of claims 1-7.

Images