XR virtual positioning method, medium and system

Abstract

The present invention provides an XR virtual positioning method, medium and system, which belongs to the technical field of XR virtual positioning, including: constructing a three-dimensional point cloud model and a visual feature image and encoding them into an environmental fingerprint; extracting local features of the point cloud and image from the user's perspective in real time in a CPU and copying them to a GPU video memory; performing initial matching of the local features with the environmental fingerprint in the GPU to obtain an initial matching score; using an improved grey wolf algorithm DIGWO to take the initial matching score as a fitness value, optimizing parameters in the matching process in the GPU to obtain a maximum matching score; determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score and rendering the user's virtual avatar to the determined position and orientation; updating the position and orientation of the virtual avatar in real time according to the user's movement, and displaying the rendered scene through an XR helmet to achieve a virtual-reality fusion experience.

Description

Technical Field

[0001] The present invention belongs to the technical field of XR virtual positioning, and in particular, relates to an XR virtual positioning method, medium and system. Background Art

[0002] With the rapid development of science and technology, extended reality (XR) technologies, including virtual reality (VR), augmented reality (AR) and mixed reality (MR), have received widespread attention and application. These technologies generate virtual information through computers and integrate it with the real world to create an immersive experience for users, showing great application prospects in education, medical care, industry, entertainment and other fields.

[0003] In the field of education, XR can provide students with interactive learning experiences such as virtual laboratories and virtual scenes. In the medical field, XR can be used for surgical simulation training, remote diagnosis and treatment, etc. In industrial manufacturing, XR technology can be used for collaborative design, remote maintenance guidance, etc. The entertainment industry can use XR to create immersive gaming experiences. Therefore, XR technology will greatly improve people's work and quality of life.

[0004] The key to achieving seamless integration of virtual and real lies in virtual positioning technology, which is to accurately obtain the user's three-dimensional position and orientation in the physical world in real time and accurately align the virtual information with it. Only when the position is aligned can the virtual elements be correctly superimposed on the real environment, allowing users to have a seamless experience. However, high-precision virtual positioning has always been a technical challenge in the XR field.

[0005] Common virtual positioning technologies currently include vision-based SLAM and sensor fusion-based methods. SLAM analyzes the image sequence acquired by the camera, tracks the movement of feature points and reconstructs the three-dimensional scene, thereby determining the position of the camera in the environment. However, this method has high requirements for image quality and lighting conditions, and is prone to failure when encountering less texture or dynamic scenes. At the same time, due to the need for continuous calculations, the amount of calculation of SLAM is also large, and the power consumption is obvious.

[0006] Although sensor fusion positioning technology based on inertial sensors, depth cameras, etc. does not rely on image features, it is limited by the measurement accuracy of each sensor and the overall positioning accuracy is not high. In addition to the accuracy problem, existing methods also generally have the defects of poor real-time performance, weak robustness, and high energy consumption, which greatly limits the practical application of XR technology.

[0007] In order to improve positioning accuracy and real-time performance, some studies have tried to accelerate calculations on GPUs and utilize the powerful parallel computing capabilities of GPUs. However, although this approach can achieve certain performance improvements, it introduces new technical problems: excessive use of GPUs can cause serious system heating, a sharp increase in power consumption, and a significant reduction in the battery life of users' mobile devices. Moreover, if the complexity of the algorithm increases further, the computing resources of a single GPU may also be stretched to the limit, resulting in a computing bottleneck. At that time, not only will it be unable to meet the real-time rendering requirements, it may also cause system freezes, frame drops, and other issues that affect the user experience.

[0008] Another technical challenge is the feature matching problem. Existing methods usually rely on feature points or feature descriptors to search and match in known environment models to solve the pose. However, this matching process is computationally intensive and is limited by the low efficiency of sequence matching. If the amount of feature data is too large, the matching time will increase sharply and real-time positioning cannot be achieved. Even with GPU acceleration, due to limited resources, the data volume threshold still exists and performance cannot be expanded indefinitely.

[0009] In addition, matching based on single-mode visual or point cloud features is also difficult to adapt to the complex and changing scenes in the real world. For example, interference from weak texture areas and dynamic objects will affect the accuracy and robustness of matching. Once a mismatch occurs, it will directly lead to positioning drift or failure.

[0010] Therefore, while the existing technology improves accuracy and real-time performance, it also brings a heavy burden to system operations. The computing resources are prone to bottlenecks, and it is difficult to resist the interference of the real environment, and the matching robustness is poor. These technical defects have seriously hindered the further development of XR technology. Summary of the invention

[0011] In view of this, the present invention provides an XR virtual positioning method, medium and system, which can solve the technical problems that the existing technology for XR virtual positioning often adopts a method of matching based on visual or point cloud features of a single modality, which has low positioning accuracy and consumes a lot of GPU resources.

[0012] The present invention is achieved in that:

[0013] A first aspect of the present invention provides an XR virtual positioning method, which includes the following steps:

[0014] S10, pre-scan the scene environment and construct a three-dimensional point cloud model and visual feature image;

[0015] S20, encoding the three-dimensional point cloud model and the visual feature image into an environmental fingerprint;

[0016] S30, using the camera built into the XR helmet to obtain the point cloud and image from the user's perspective;

[0017] S40, extracting the point cloud and local features of the image from the user's perspective in real time in the CPU and copying them to the GPU memory;

[0018] S50, performing an initial matching of the local feature with the environment fingerprint in the GPU to obtain an initial matching score;

[0019] S60, using the improved grey wolf algorithm DIGWO to take the initial matching score as a fitness value, optimizing the parameters in the matching process in the GPU, and obtaining a maximum matching score;

[0020] S70, determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score;

[0021] S80, rendering the user virtual avatar to a determined position and orientation according to the user three-dimensional position and orientation data;

[0022] S90. Update the position and orientation of the virtual avatar in real time according to the user's movement, and display the rendered scene through the XR helmet to achieve a virtual-real fusion experience.

[0023] On the basis of the above technical solution, an XR virtual positioning method of the present invention can also be improved as follows:

[0024] Among them, the steps of pre-scanning the scene environment and constructing a three-dimensional point cloud model and a visual feature image are specifically: using a laser radar or RGB-D camera three-dimensional scanning device to perform an all-round scan of the scene environment to obtain three-dimensional point cloud data of the scene environment; filtering the acquired three-dimensional point cloud data using a statistical filtering or a distance-based filtering method, and removing outliers; using a Poisson reconstruction algorithm to reconstruct the pre-processed high-quality point cloud data to construct a three-dimensional grid model of the scene environment; and using SIFT to extract a visual feature image from the collected color image.

[0025] Among them, the step of encoding the three-dimensional point cloud model and the visual feature image into an environmental fingerprint is specifically: voxelizing the three-dimensional point cloud data to discretize the continuous three-dimensional space into a regular three-dimensional voxel grid; using a method based on a deep learning model to feature encode the voxelized data to obtain a three-dimensional feature vector, wherein the method based on the deep learning model uses a 3D convolutional neural network model to directly encode the three-dimensional voxel data; and using a convolutional neural network to directly jointly encode the RGB image and the depth image for the visual feature image to obtain a visual feature vector; finally, the encoded three-dimensional feature vector and the visual feature vector are spliced ​​to form the fingerprint data of the environment.

[0026] Among them, the step of extracting the local features of the point cloud and image from the user's perspective in real time in the CPU is specifically: performing SIFT feature point detection and descriptor extraction on the acquired color image to obtain local visual features; performing equidistant sampling on the point cloud data, and using the FPFH projection-based method to encode the local features of the sampled key points and their neighborhoods to obtain local point cloud features; combining the extracted local visual features and local point cloud features to form a set of key points from the current user's perspective and their corresponding feature descriptor set as local features.

[0027] Among them, the step of initially matching the local feature with the environmental fingerprint in the GPU to obtain an initial matching score is specifically: using the nearest neighbor search method to match the local feature with the environmental fingerprint, and using the RANSAC algorithm to calculate the posture transformation function of the XR helmet built-in camera corresponding to the local feature according to the matching result, and according to the posture transformation function, projecting the local feature into the fingerprint coordinate system of the environmental fingerprint, and then calculating the similarity score of the point-to-point distance, point-to-surface distance, and descriptor distance between the projected local feature and the matching result as the initial matching score.

[0028] Among them, the step of using the improved gray wolf algorithm DIGWO to use the initial matching score as the fitness value, optimizing the parameters in the matching process in the GPU, and obtaining the maximum matching score is specifically: parallelizing and porting the improved gray wolf algorithm DIGWO to the GPU, specifically including the steps of population initialization, fitness evaluation, wolf pack hierarchy division, and wolf position update are implemented in parallel on the GPU through CUDA or OpenCL, wherein the fitness evaluation process adopts a fast feature matching and similarity calculation method; the initial matching score is input as the initial fitness value of the DIGWO algorithm, and the DIGWO iterative optimization process is executed in parallel on the GPU, in each iteration, the position of each wolf is updated according to the wolf pack hierarchy structure, corresponding to the camera posture parameters built into the XR helmet, and the matching score under the current parameter combination is calculated using feature matching, posture estimation and similarity calculation methods, so that it gradually approaches the global optimal posture; when the algorithm converges, the posture parameters corresponding to the global optimal solution at this time are output, which is the final matching posture, and the corresponding matching score is the maximum matching score.

[0029] Furthermore, the step of determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score is specifically: using the camera posture parameters corresponding to the maximum matching score, transforming the point cloud or image data from the user's perspective back to the scene environment coordinate system, and using the ICP algorithm to align the transformed point cloud or image with the corresponding part in the three-dimensional point cloud model; obtaining the position and orientation of the camera built into the XR helmet in the scene environment coordinate system according to the alignment result, and converting the camera position and orientation into the three-dimensional position and orientation of the user in the scene environment in combination with the relative posture relationship between the camera and the user, and finally outputting a six-dimensional vector including the user's three-dimensional position and orientation.

[0030] Furthermore, the method for voxelizing the three-dimensional point cloud data is: an octree-based method or a hash-based method.

[0031] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned XR virtual positioning method.

[0032] A third aspect of the present invention provides an XR virtual positioning system, which includes the above-mentioned computer-readable storage medium.

[0033] Compared with the prior art, the XR virtual positioning method, medium and system provided by the present invention have the following beneficial effects:

[0034] 1. High positioning accuracy

[0035] The method of the present invention combines two complementary environmental information, spatial point cloud and visual features, and makes full use of the dual constraints of spatial structure and texture details, so the representation of the environment is more comprehensive and the positioning accuracy is higher. Even if there is texture loss or dynamic occlusion, point cloud information can be used for constraints to avoid drift divergence. At the same time, visual features can refine the location details of the point cloud and make up for the lack of point cloud accuracy. The organic combination of the two types of information achieves high-precision positioning.

[0036] 2. Good real-time performance

[0037] The method of the present invention accelerates the computationally intensive initial matching part on the GPU. First, local features are extracted in the CPU, and only a small amount of feature data needs to be transmitted instead of the entire point cloud and image, which reduces the amount of data transferred from the CPU to the GPU. Then, the initial matching link is quickly completed on the GPU with the help of large-scale parallel computing capabilities. Through this heterogeneous computing architecture, the efficiency of initial matching is greatly improved.

[0038] In the subsequent optimization phase of the improved Grey Wolf algorithm, only a small amount of parameter adjustment is required for each generation, and the computational overhead is very small. Since the initial value is very close to the optimal solution, the algorithm converges quickly and only a few iterations are required to find a high-precision position. The entire process achieves smooth real-time positioning without noticeable delay.

[0039] 3. Strong robustness

[0040] The method of the present invention uses environmental fingerprints to encode and characterize the environment, and has strong robustness. The fingerprint integrates static geometric structure and transient visual features, and can effectively resist dynamic changes, illumination changes and other factors in the environment. At the same time, the algorithm further improves the anti-interference ability through multi-frame fusion. Even if partial occlusion occurs, constraints in other areas can still be used to avoid positioning failure. This high robustness makes the algorithm applicable to complex and changeable real scenes.

[0041] 4. High computational efficiency

[0042] Two key links improve the efficiency of the algorithm:

[0043] First, the initial matching is accelerated in parallel on the GPU, making full use of the parallel computing capabilities of the modern GPU architecture and significantly reducing the matching time.

[0044] Secondly, the improved Grey Wolf optimization algorithm converges quickly and only requires a few iterations to achieve high-precision positioning results, avoiding the "slow-heating" problem of other optimization algorithms such as particle filtering and genetic algorithms.

[0045] The total amount of computing is controlled within a reasonable range, and no extremely high-performance chips are required, so it can run smoothly on mobile terminals. Higher energy efficiency extends the battery life of XR devices and improves the user experience.

[0046] In summary, the solution of the present invention solves the technical problems that the existing technology often uses a method of matching based on visual or point cloud features of a single modality when performing XR virtual positioning, which has low positioning accuracy and consumes a lot of GPU resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0048] Figure 1 A flow chart of the method provided by the present invention; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0050] like Figure 1 As shown, it is a flow chart of an XR virtual positioning method provided by the present invention, and the method comprises the following steps: wherein, the following steps are included:

[0051] S10, pre-scan the scene environment and construct a three-dimensional point cloud model and visual feature image;

[0052] S20, encoding the three-dimensional point cloud model and the visual feature image into an environment fingerprint;

[0053] S30, using the camera built into the XR helmet to obtain the point cloud and image from the user's perspective;

[0054] S40, extracting the point cloud and local features of the image from the user's perspective in real time in the CPU and copying them to the GPU memory;

[0055] S50, performing an initial match between the local feature and the environment fingerprint in the GPU to obtain an initial matching score;

[0056] S60, using the improved grey wolf algorithm DIGWO to take the initial matching score as the fitness value, optimizing the parameters in the matching process in the GPU, and obtaining the maximum matching score;

[0057] S70, determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score;

[0058] S80, rendering the user's virtual avatar to a determined position and orientation according to the user's three-dimensional position and orientation data;

[0059] S90 updates the position and orientation of the virtual avatar in real time according to the user's movements, and displays the rendered scene through the XR helmet to achieve a virtual-real fusion experience.

[0060] The specific implementation methods of the above steps are described in detail below:

[0061] The specific implementation of step S10 is:

[0062] The purpose of this step is to pre-scan the target environment, build its 3D point cloud model and visual feature image, and provide basic data for the subsequent virtual positioning process. Generally, it can be achieved by the following sub-steps:

[0063] First, it is necessary to use three-dimensional scanning equipment such as laser radar or RGB-D camera to perform a full-scale scan of the target environment to obtain the three-dimensional point cloud data of the target environment. Among them, the laser radar calculates the distance by emitting a laser beam and measuring the time difference of the reflected signal to obtain point cloud data; while the RGB-D camera obtains point cloud data by measuring depth information and color images.

[0064] Next, the acquired 3D point cloud data is preprocessed by filtering and removing outliers to improve the quality of the point cloud. Common filtering methods include statistical filtering (such as Gaussian filtering), distance-based filtering (such as radius filtering), etc. Outlier removal can be done by using methods based on K nearest neighbors or statistical learning.

[0065] The preprocessed high-quality point cloud data needs to be further reconstructed into a complete 3D model. Commonly used point cloud reconstruction algorithms include Poisson reconstruction algorithm, Ball Pivoting algorithm, etc. These algorithms construct a 3D mesh model of the target environment by fitting the local features of the point cloud data.

[0066] At the same time, it is also necessary to extract visual features from the collected color images. Visual features refer to feature vectors that can effectively describe the appearance characteristics of local areas of images, such as SIFT, SURF, ORB, etc. These feature vectors are crucial for subsequent feature matching.

[0067] Finally, the reconstructed 3D model and the extracted visual features are encoded to form the fingerprint data of the environment. The encoding method can be based on the Bag of Words model, or a deep learning model (such as PointNet++, 3D ShapeNets, etc.) can be used to directly encode the 3D data and images. The environmental fingerprint will be used in the subsequent real-time matching process.

[0068] For this step, the key parameters include the accuracy and resolution of the 3D scanning equipment; point cloud filtering parameters (such as Gaussian kernel radius, distance threshold, etc.); visual feature detector parameters (such as Gaussian scale space of SIFT, etc.). The reasonable selection of these parameters will directly affect the subsequent positioning accuracy.

[0069] The specific implementation of step S20 is:

[0070] The purpose of this step is to encode the 3D point cloud model and visual feature image obtained in the previous step into an environmental fingerprint to prepare for real-time matching. The encoding process generally includes the following sub-steps:

[0071] First, the 3D point cloud data is voxelized. Voxelization refers to discretizing the continuous 3D space into a regular 3D voxel grid, where each voxel represents a small 3D space unit. Common voxelization algorithms include octree-based methods and hash-based methods. After voxelization, the 3D model is converted into an ordered sequence of voxel values.

[0072] Secondly, feature encoding is performed on the voxelized data. Common encoding methods include methods based on the bag of words model and methods based on deep learning.

[0073] The bag-of-words model-based method first needs to build a feature dictionary (code book), which contains the code values ​​of all possible local 3D features. All local features can be clustered using the k-means clustering algorithm, with each cluster center as a dictionary item. Next, the local features of each voxel are matched with all features in the dictionary to find the feature code of the nearest neighbor, thereby encoding the 3D model into a feature code sequence.

[0074] Deep learning-based methods use models such as 3D convolutional neural networks to directly encode 3D voxel data. Commonly used models include PointNet++ and 3D ShapeNets. These models can automatically extract high-level features of 3D data and encode them into compact feature vectors.

[0075] For visual feature images, a bag-of-words model-based method can be used to encode local features (such as SIFT features), or a two-stream convolutional neural network can be used to directly jointly encode RGB images and depth images.

[0076] Finally, the encoded three-dimensional feature sequence (or vector) and visual feature sequence (or vector) are concatenated to form the fingerprint data of the environment. This fingerprint data will be used for fast retrieval in the real-time matching stage.

[0077] It should be noted that the parameter settings of the encoding process (such as voxel scale, feature dictionary size, convolutional network structure, etc.) will affect the fingerprint quality and thus the subsequent matching accuracy. It is usually necessary to adjust the parameters on the training data set to find the optimal parameter combination.

[0078] The specific implementation of step S30 is:

[0079] The purpose of this step is to use the camera built into the XR helmet to obtain the point cloud and image from the user's perspective, providing input data for subsequent real-time matching. This can generally be achieved using the following sub-steps:

[0080] First, the color image data of the user's current perspective is obtained from the RGB camera of the XR helmet. Since the movement frequency of the helmet camera is very high, it is usually necessary to perform image deblurring on the continuously acquired images to improve the image quality. Commonly used image deblurring algorithms include blind deblurring algorithms (such as Wiener filtering) and non-blind deblurring algorithms (such as Richardson-Lucy algorithm).

[0081] Secondly, the depth camera built into the XR helmet (such as ToF camera, structured light camera, etc.) is used to obtain the depth image data under the user's current perspective. The depth image records the distance from each pixel to the camera and can be directly converted into point cloud data. Considering the measurement error of the depth camera, the depth data needs to be filtered to remove outliers. Commonly used filtering algorithms include bilateral filtering, guided filtering, etc.

[0082] The acquired color image and the processed depth data are registered to make the pixels of the two correspond one to one. The registration method can use the camera intrinsic parameter and extrinsic parameter model to transform and correct the image data. It can also use the feature-based method to use the image registration algorithm (such as SIFT+RANSAC, etc.) to automatically register the image and point cloud.

[0083] Finally, the registered depth data is converted into point cloud data and used together with the color image data as input for real-time matching.

[0084] It is worth noting that the parameter settings of the helmet camera (such as resolution, frame rate, exposure time, etc.) will affect the quality of the collected data, and thus affect the subsequent matching accuracy. At the same time, the parameters of the image processing algorithm (such as the regularization parameter of the deblurring algorithm, the window size of the filtering algorithm, etc.) also need to be reasonably set to balance the processing effect and computational efficiency.

[0085] The specific implementation of step S40 is:

[0086] The purpose of this step is to extract the local features of the point cloud and image from the user's perspective in real time in the CPU, and copy them to the GPU memory to prepare for subsequent GPU accelerated matching. Generally, the following sub-steps can be used to achieve this:

[0087] First, feature point detection and descriptor extraction are performed on the acquired color image to obtain local visual features. Common feature extraction algorithms include SIFT, SURF, ORB, etc. These algorithms first detect key points (such as corner points) in the image, and then calculate the feature descriptor vector of each key point neighborhood.

[0088] Secondly, key point sampling and local feature encoding are performed on the point cloud data. Common key point sampling methods include equidistant sampling and curvature sampling, etc., the purpose is to select representative key points from the point cloud. Local feature encoding requires building a local feature descriptor of the neighborhood for each key point. The encoding methods include projection-based methods (such as FPFH, SHOT, etc.) and deep learning-based methods (such as 3DMatch, 3DFeatNet, etc.).

[0089] The extracted visual features and point cloud features are combined to form a set of key points under the current viewing angle and their corresponding feature descriptor sets.

[0090] Next, CUDA is used to accelerate the copying of the feature data from the host memory to the GPU memory. In the specific implementation, the feature data can be packaged on the CPU side first, and then the memory copy function (such as cudaMemcpy) provided by CUDA is used for efficient transmission.

[0091] Finally, the received feature data is unpacked on the GPU side to prepare for subsequent matching.

[0092] It should be noted that the real-time performance of this step is crucial to the real-time performance of the entire system. Therefore, the feature extraction algorithm on the CPU side needs to be efficient enough to avoid becoming a system bottleneck. At the same time, the data transmission speed from the CPU to the GPU cannot be ignored. A reasonable data packaging method can improve the transmission efficiency. In addition, the algorithm parameters of feature extraction and data transmission (such as the number of Gaussian scale pyramid layers of SIFT, the parallelism of CUDA streams, etc.) also need to be tuned according to the specific hardware platform to achieve the best performance.

[0093] The specific implementation of step S50 is:

[0094] The purpose of this step is to initially match the extracted local features with the environmental fingerprint in the GPU to obtain an initial matching score, which provides an initial solution for the subsequent matching optimization process. This can generally be achieved using the following sub-steps:

[0095] First, the pre-built environment fingerprint data is loaded into the GPU memory. The fingerprint data includes visual feature word bags, point cloud feature word bags, global feature descriptors, etc.

[0096] Secondly, the incoming real-time feature data is preprocessed. If the feature data volume is too large, downsampling is required to reduce the computational pressure of subsequent matching. Downsampling methods can use grid filtering, voxelization or random sampling.

[0097] Next, the GPU uses an acceleration algorithm (such as CUDA or OpenCL) to perform an initial match between the real-time features and the fingerprint features. The matching process includes two main steps: feature matching and posture estimation:

[0098] 1) Feature matching: Use the nearest neighbor search or bag-of-words model to find feature pairs that match the real-time features in the fingerprint features. CUDA can provide efficient parallel nearest neighbor search algorithms, such as the kd-tree algorithm and the locality sensitive hashing (LSH) algorithm.

[0099] 2) Pose estimation: Based on the matched feature pairs, the camera's pose transformation is estimated using algorithms such as RANSAC and epipolar geometry. Commonly used pose estimation algorithms include EPnP, PnP, epipolar geometry, etc. These algorithms can all be accelerated on GPUs.

[0100] The estimated posture transformation is applied to the real-time feature data and projected into the fingerprint coordinate system. Then the similarity score between the projected feature and the fingerprint feature is calculated as the initial matching score. The similarity calculation can be performed using distance metrics such as point-to-point distance, point-to-plane distance, and descriptor distance.

[0101] It is worth noting that the initial matching stage does not pursue completely accurate pose estimation, but only needs to give a more reasonable initial solution as the starting point of the subsequent optimization process. Therefore, the accuracy requirements can be appropriately lowered in this stage in exchange for higher computational efficiency. At the same time, it is necessary to reasonably set the similarity threshold to filter out obviously erroneous matching pairs and improve the convergence speed of subsequent optimization. In addition, the parameter settings of the GPU acceleration algorithm (such as the construction parameters of the kd-tree, the parallelism of CUDA threads, etc.) also need to be tuned for performance according to the specific hardware.

[0102] The specific implementation of step S60 is:

[0103] The purpose of this step is to use the improved grey wolf algorithm DIGWO to take the initial matching score as the fitness value, optimize the parameters in the matching process in the GPU, and obtain the maximum matching score.

[0104] First, we need to port the improved gray wolf algorithm DIGWO to the GPU for parallel acceleration. The DIGWO algorithm is a new intelligent optimization algorithm that simulates the gray wolf's predation behavior and has strong global search capabilities and convergence speed. After parallelization, it can make full use of the massive cores of the GPU and greatly improve the efficiency of the algorithm.

[0105] Specifically, the main steps in DIGWO, such as population initialization, fitness evaluation, wolf group hierarchy division, and wolf position update, need to be implemented in parallel on the GPU through CUDA or OpenCL. For example, the fitness evaluation process can use the fast feature matching and similarity calculation methods mentioned above; the position update requires parallel calculation of various values ​​in the DIGWO formula.

[0106] Secondly, the initial matching score is used as the initial fitness value input of the DIGWO algorithm. Since DIGWO hopes to minimize the fitness, the initial matching score can be negated as input.

[0107] Next, the DIGWO iterative optimization process is executed in parallel on the GPU. In each iteration, DIGWO updates the position of each wolf according to the wolf pack hierarchy to gradually approach the global optimal solution. The position here refers to the camera pose parameters, including the translation component and the rotation angle component.

[0108] Specifically, for each wolf (i.e., a set of candidate posture parameters), the matching score (fitness value) under the current parameter combination needs to be calculated using the feature matching, posture estimation, and similarity calculation methods mentioned above. Then, the position of the wolf is updated according to the matching score, so that it gradually approaches the global optimal posture.

[0109] After each iteration, it is necessary to determine whether the convergence condition has been reached. The termination condition can be set according to the difference between the fitness value and the threshold, the upper limit of the number of iterations, and other conditions. If it has not reached convergence, it will enter the next round of iteration.

[0110] Finally, when the algorithm converges, the posture parameters corresponding to the global optimal solution are output, which is the final matching posture. The matching score at this time will also be output as the maximum matching score.

[0111] It should be noted that the performance and accuracy of the DIGWO algorithm are closely related to its internal parameter settings. For example, parameters such as population size, wolf group hierarchy division strategy, and step size factor will affect the algorithm performance. It is usually necessary to tune parameters on the training data set for different application scenarios to achieve optimal performance. At the same time, the degree of GPU parallelization, such as thread bundle size and core occupancy, also needs to be tuned according to hardware characteristics.

[0112] The specific implementation of step S70 is:

[0113] The purpose of this step is to determine the 3D position and orientation of the current user in the target environment based on the maximum matching score obtained in the previous step.

[0114] First, it is necessary to determine whether the maximum matching score meets the threshold requirement. If it does not meet the requirement, it is considered that a reasonable matching result has not been obtained, and the default value or the position and orientation data of the previous moment can be directly output. The specific setting of the threshold needs to be obtained based on pre-calibration, and the 90th percentile score can usually be used as the threshold.

[0115] If the maximum matching score reaches the threshold requirement, further calculation is performed based on the matching result. Specifically, the following steps are included:

[0116] 1) According to the optimal posture parameters output in the previous step, the point cloud or image data from the user's perspective is transformed back to the environment coordinate system.

[0117] 2) Align the transformed point cloud or image with the corresponding part in the environment model to determine its precise position in the environment coordinate system. The alignment method can use ICP (Iterative Closest Point) algorithm or epipolar geometry method.

[0118] 3) Based on the registration results, the position and orientation of the user’s camera in the environment coordinate system are obtained.

[0119] 4) Convert the camera position and orientation into the user’s three-dimensional position and orientation in the environment. The parameters required for the conversion include the relative position relationship between the camera and the user, the user’s height, etc.

[0120] The final output is a six-dimensional vector, including the user's three-dimensional position (x, y, z) and orientation (yaw, pitch, roll) in the environment. This vector will be used for subsequent virtual avatar rendering.

[0121] It should be noted that the above process will be affected by the cumulative error of the previous steps, so it is very sensitive to the setting of related parameters and thresholds. At the same time, the posture conversion process also needs to ensure accuracy, otherwise obvious drift will occur. In addition, for users who move quickly, posture smoothing filtering is also necessary to avoid jitter.

[0122] The specific implementation of step S80 is:

[0123] The purpose of this step is to render the user's virtual avatar to the corresponding position and orientation based on the user's three-dimensional position and orientation data obtained in the previous step, so as to achieve a virtual-real fusion display.

[0124] First, the 3D model data of the user's virtual avatar needs to be loaded. The model data can be constructed based on multi-view 3D scanning of real people, or it can be generated based on a parametric human body model. The model data format is usually a mesh model with skeleton information and material information.

[0125] Next, the user's three-dimensional position and orientation data need to be converted into pose data relative to the model coordinate system.

[0126] The converted pose data is then applied to the virtual avatar model. A common approach is to control the pose changes of the model through skeletal animation, that is, mapping the pose data to the skeletal hierarchy of the model. The specific process includes:

[0127] 1) Forward kinematics calculation: According to the new pose, the bone transformation matrix is ​​calculated from bottom to top according to the bone hierarchy.

[0128] 2) Skinning: Apply the bone transformation matrix to the model mesh vertices to make the model mesh deform accordingly.

[0129] Next, based on the model material information, environmental lighting information, etc., rasterization or physically based rendering (PBR) methods are used to render a virtual avatar with realistic effects such as shadows and environmental reflections in the GPU.

[0130] Finally, the rendered virtual avatar image is superimposed on the real image collected from the XR helmet to achieve a display effect of virtual-real fusion. Image superposition can use imaging methods such as Alpha blending.

[0131] It is worth noting that the efficiency of the rendering stage is crucial and directly affects the real-time performance of the entire system. Optimization methods include:

[0132] 1) Simplify the mesh and reduce the level of detail of the model

[0133] 2) Material map optimization, such as using low-precision textures or texture-free PBR models

[0134] 3) Use GPU hardware to accelerate model transformation, rasterization and other steps

[0135] 4) Reduce shadows, reflections and other advanced rendering options

[0136] At the same time, the rendering quality and effect also need to be balanced. Too precise rendering may not necessarily bring a better user experience. Usually, it is necessary to set reasonable rendering parameters based on hardware conditions, usage scenarios and other factors.

[0137] The specific implementation of step S90 is:

[0138] The purpose of this step is to update the position and orientation of the virtual avatar in real time according to the user's movement, and display the rendered scene through the XR helmet to achieve an immersive virtual-reality fusion experience.

[0139] Specifically, the following sub-steps need to be performed periodically at a certain frequency:

[0140] 1) Get the user's latest exercise status

[0141] The inertial measurement unit (IMU) sensor built into the XR helmet is used to obtain the acceleration and angular acceleration data of the user's head. IMU generally includes a three-axis gyroscope, accelerometer, etc. It can also be combined with visual sensors to obtain visual measurement results, such as visual odometers.

[0142] 2) Estimating motion increment

[0143] The IMU data is fused using incremental Kalman filtering or other filtering algorithms (such as wavelet filtering) to estimate the displacement and rotation increments of the user's movement during this cycle.

[0144] 3) Update user pose

[0145] Apply the estimated motion increment to the posture at the previous moment to update the posture data at the current moment. The update method can use matrix operations or quaternion operations, etc.

[0146] 4) Rendering the avatar

[0147] The updated posture data is used as input and the virtual avatar model is rendered according to the process of step S80.

[0148] 5) Synthesize virtual and real scenes

[0149] The rendered virtual avatar image is synthesized with the real scene image collected from the XR helmet and output on the helmet display. Image alignment and occlusion relationship processing are required during synthesis.

[0150] 6) Determine the end condition

[0151] If the user terminates the operation or other termination conditions (such as timeout) are met, this step ends, otherwise it returns to step 1 to continue the cycle execution.

[0152] In addition, to improve user experience, the following optimization measures can be taken:

[0153] a) Motion smoothing filter

[0154] The estimated displacement and rotation increments are smoothed and filtered to eliminate the jitter of the virtual avatar caused by measurement noise or jitter.

[0155] b) Predicted pose

[0156] Use Kalman filtering or other filtering algorithms to predict motion and estimate the posture at the next moment in advance to reduce the sense of delay.

[0157] c) Image fusion technology

[0158] Use technologies such as depth-based image fusion to improve the visual transition effect of virtual and real boundaries.

[0159] d) Multi-sensor fusion

[0160] Combined with the helmet's color camera and depth camera, the posture is detected and corrected to improve accuracy.

[0161] It should be noted that the execution efficiency and real-time performance of step S90 are crucial to the entire system. Therefore, it is necessary to highly optimize the algorithm implementation of each sub-step, such as using GPU to accelerate calculations and making full use of pipeline parallelism. At the same time, the setting of parameters of each sub-step will also affect the final experience indicators such as latency and jitter, which can be tuned during the specific implementation process.

[0162] According to the specific implementation of the present invention, an embodiment suitable for specific calculation is provided below:

[0163] Specific implementation of step S20:

[0164] The purpose of this step is to encode the 3D point cloud model and visual feature image obtained in the previous step into an environmental fingerprint to prepare for real-time matching. The encoding process includes the following sub-steps:

[0165] Voxelization of 3D point clouds

[0166] Discretize the continuous 3D space into a regular 3D voxel grid, where each voxel represents a small 3D space unit. Common voxelization algorithms include octree-based methods and hash-based methods.

[0167] For the octree-based method, the three-dimensional space is recursively divided into 8 sub-octants to form an octree structure. Each leaf node corresponds to a voxel, which contains all the point cloud data within the voxel. The voxel size can be set by setting the maximum depth d max To control, that is, the minimum voxel side length is Among them l max is the maximum side length of the point cloud data.

[0168] For the hash-based method, each point cloud point p = (x, y, z) is mapped to an integer index through the following hash function:

[0169]

[0170] Among them l min is the desired voxel side length, n x , n y The numbers of voxels in the x and y directions of the point cloud data are respectively. The same index corresponds to the same voxel, thus achieving voxelization.

[0171] Feature Encoding

[0172] The voxelized data is feature encoded using a Bag of Words-based method or a deep learning-based method.

[0173] The bag-of-words model requires building a feature dictionary (codebook) first. Contains N feature codes c i The commonly used construction method is to perform K-means clustering on all local features and take each cluster center as a dictionary item:

[0174]

[0175] where f i is the i-th local eigenvector, with a total of M eigenvectors.

[0176] Next, the local feature f of each voxel v is v Match all features in the dictionary and find the nearest neighbor feature code c * , thereby encoding the voxels into feature codes:

[0177]

[0178] Finally, the entire 3D model is encoded as a feature code sequence.

[0179] The deep learning-based method uses models such as 3D convolutional neural networks to directly encode three-dimensional voxel data. Taking the PointNet++ model as an example, the query ball algorithm is first used to sample a set of seed points from the input point cloud, and a neighborhood sphere region is established for each seed point; then the point cloud in each region is encoded by the PointNet layer to obtain regional features; finally, all regional features are fused through the upsampling layer and the PointNet++ layer to output a global feature vector.

[0180] The encoding process of the visual feature image is similar. The RGB image and the depth image can be directly jointly encoded using a bag-of-words model or a two-stream convolutional neural network.

[0181] Finally, the encoded three-dimensional feature sequence or vector and the visual feature sequence or vector are concatenated to form the fingerprint data of the environment.

[0182] Specific implementation of step S40:

[0183] The purpose of this step is to extract the local features of the point cloud and image from the user's perspective in real time in the CPU, and copy them to the GPU memory to prepare for subsequent GPU accelerated matching.

[0184] Visual feature extraction

[0185] The acquired color image is subjected to feature point detection and descriptor extraction such as SIFT, SURF, and ORB to obtain local visual features. These algorithms first detect the key points x in the image. i =(x i ,y i ), such as the extreme points detected by the SIFT algorithm; then calculate the feature descriptor vector f of the neighborhood of each key point i :

[0186] f i =descriptor(x i , I);

[0187] Where I is the input image, and descriptor is a specific descriptor calculation function, such as the SIFT descriptor using gradient histogram statistics.

[0188] Point cloud feature extraction

[0189] Key point sampling is performed on the point cloud data, and then local feature encoding is performed on the sampling points and their neighborhoods. Common key point sampling methods include equidistant sampling and curvature sampling. Taking equidistant sampling as an example, for a given sampling radius r, a seed point p0 is first randomly selected and added to the sampling point set. Then find all points that are at least r away from p0, where r is the sampling radius of equidistant sampling, and randomly select a new seed point p1 from them. Repeat the above process until no new point that meets the conditions is found.

[0190] For each sampling point p i , construct its neighborhood sphere area Use projection-based methods such as FPFH and SHOT or deep learning-based methods such as 3DMatch and 3DFeatNet to encode local features of the region and obtain the feature descriptor g. i .

[0191] Taking FPFH as an example, the algorithm first calculates each point Calculate its SPFH (Simplified Point Feature Histogram) features:

[0192]

[0193] Where ρ is the Euclidean distance of the point pair, η is the normal angle difference of the point pair, and the weighted sum can obtain the local geometric features of point q.

[0194] Then for p i All neighborhood points of are weighted to make contributions, and the final FPFH descriptor is obtained:

[0195]

[0196] Where k is the number of neighborhood points.

[0197] Features copied to GPU

[0198] Combine the extracted visual features and point cloud features to form a set of key points at the current perspective and its corresponding feature descriptor set Where N x is the number of visual features, N g is the number of point cloud features.

[0199] The above feature data is copied from the host memory to the GPU memory using the memory copy function cudaMemcpy provided by CUDA. The received feature data is unpacked on the GPU side to prepare for subsequent matching.

[0200] Specific implementation of step S50:

[0201] This step performs an initial match between the extracted local features and the environment fingerprint in the GPU to obtain an initial matching score. It includes the following sub-steps:

[0202] Environment fingerprint loading

[0203] Load the pre-built environment fingerprint data into the GPU memory. The fingerprint data includes the visual feature word bag. Point Cloud Feature Bag of Words Global Feature Descriptor wait.

[0204] Feature downsampling (optional)

[0205] If the amount of real-time feature data N x +N g If the size is too large, downsampling is performed to reduce the computational pressure of subsequent matching. Methods such as grid filtering, voxelization or random sampling can be used.

[0206] Taking grid filtering as an example, the feature point set is first divided into several grid cells. For each cell, only one feature point (such as the one closest to the cell center) is retained and the rest are discarded to achieve downsampling. The grid size can be set to l grid .

[0207] Feature Matching

[0208] Using CUDA or OpenCL parallel acceleration, real-time features and fingerprint features are quickly matched to obtain a set of matching feature pairs.

[0209] For the visual feature f i , which can be found in the bag-of-words Find the best matching feature code by parallel nearest neighbor search or back projection Then in the fingerprint global feature Find the corresponding global descriptor d j , thus forming a pair of matching features

[0210] For the point cloud feature g i , the matching process is similar, and Find the best matching global descriptor in .

[0211] Pose Estimation

[0212] According to the matching characteristics The camera's attitude transformation (R, t) is estimated using algorithms such as RANSAC and epipolar geometry, and the real-time feature data is transformed into the fingerprint coordinate system. The attitude estimation algorithm can be accelerated in parallel on the GPU.

[0213] Similarity calculation

[0214] For the transformed real-time feature, the similarity score between it and the fingerprint feature is calculated as the initial matching score s0.

[0215]

[0216] That is, the sum of the distances between all real-time features and the nearest neighbor fingerprint features is calculated (using weighted Euclidean distance), and the descriptor distance penalty term e is added. descriptor , where α is the weight coefficient. The descriptor distance can be Chebyshev distance, etc.

[0217] This initial matching score will serve as input for subsequent optimization.

[0218] Specific implementation of step S60:

[0219] This step uses the improved grey wolf algorithm DIGWO to take the initial matching score as the fitness value, optimizes the parameters in the matching process in the GPU, and obtains the maximum matching score. It includes the following sub-steps:

[0220] DIGWO parallel implementation

[0221] The DIGWO algorithm was ported to GPU for parallel acceleration. DIGWO is a new intelligent optimization algorithm that simulates the predation behavior of gray wolves and has strong global search capabilities and convergence speed.

[0222] The main steps of implementing DIGWO in parallel on GPU include:

[0223] (1) Population initialization: N wolves are generated in parallel, each of which corresponds to a set of posture parameters θ = (R, t) to be optimized, including the rotation matrix R and the translation vector t.

[0224] (2) Fitness evaluation: The fitness value of each wolf, i.e., the matching score f(θ) at the current posture, is calculated in parallel, using a feature matching and similarity calculation method similar to that in step S50.

[0225] (3) Wolf pack hierarchy: The population is divided into four levels according to the fitness value: alpha, beta, delta, and omega, which is used to simulate the hierarchical behavior of gray wolves when hunting.

[0226] (4) Wolf position update: Update the position of each wolf in parallel, that is, optimize its posture parameter θ so that it gradually approaches the global optimal solution. The position update process is determined by the mathematical model of DIGWO and contains multiple constant parameters, such as the step size factor Weight vector Etc., reasonable values ​​need to be set according to specific problems.

[0227] Initial fitness input

[0228] The initial matching score s0 obtained in step S50 is used as the initial fitness value input of the DIGWO algorithm. Since DIGWO expects to minimize the fitness function, s0 can be inverted as input, that is, f(θ)=-s0.

[0229] In the DIGWO algorithm, the smaller the value of the fitness function f(θ), the better the quality of the candidate solution θ. Since s0 represents the initial matching score, the larger the value, the worse the matching effect. Therefore, taking its opposite as the initial fitness can ensure that the algorithm minimizes the matching score in subsequent optimization, thereby obtaining the optimal matching posture.

[0230] GPU parallel iteration optimization

[0231] The DIGWO iterative optimization process is executed in parallel on the GPU. In each iteration, according to the above steps (3) and (4), DIGWO updates the position of each wolf (i.e., a set of candidate posture parameters θ) according to the wolf pack hierarchy.

[0232] Specifically, for each wolf, the feature matching, posture estimation and similarity calculation methods mentioned in step S50 need to be used to calculate the matching score f(θ) under the current parameter combination θ. Then, based on the matching score and the hierarchical structure, the DIGWO model determines the new position of the wolf so that it gradually approaches the global optimal posture.

[0233] Convergence condition judgment

[0234] After each iteration, it is necessary to determine whether the convergence condition of the DIGWO algorithm has been reached. Common termination conditions include:

[0235] (1) The difference between the fitness value and the target threshold ∈ is less than a given value:

[0236] |f(θ best )-∈|<τ;

[0237] where θ best is the current optimal posture parameter, and τ is the accuracy threshold.

[0238] (2) The number of iterations reaches the preset maximum number of iterations t max :

[0239] t≥t max ;

[0240] (3) The standard deviation of the population fitness is lower than the given threshold, indicating that the population has converged:

[0241] σ f <σ τ

[0242] where σ f is the standard deviation of population fitness, σ τ is the variance threshold.

[0243] If any of the above termination conditions is met, the iteration is stopped, otherwise the optimization continues in the next round of iteration.

[0244] Optimal solution output

[0245] When the algorithm is completed, the posture parameter θ corresponding to the global optimal solution is output best , which is the final matching posture. At the same time, the matching score f(θ best ), which is the maximum matching score.

[0246] It should be noted that the convergence performance and accuracy of the DIGWO algorithm are closely related to its internal parameter settings, including the population size N, the hierarchical division strategy, and the step size factor. Weight vector Parameters such as . Usually, it is necessary to tune parameters on the training data set for different application scenarios to achieve optimal performance. At the same time, the degree of GPU parallelization, such as thread bundle size and core occupancy, also needs to be tuned according to hardware characteristics.

[0247] Specific implementation of step S70:

[0248] This step determines the 3D position and orientation of the current user in the target environment based on the maximum matching score obtained in the previous step. It includes the following sub-steps:

[0249] Threshold judgment

[0250] Determine the maximum matching score f(θ best ) reaches the threshold θ f Require:

[0251] f(θ best )<θ f ;

[0252] If the requirements are not met, it is considered that a reasonable matching result has not been obtained, and the default value or the position and orientation data of the previous moment can be directly output. f The specific setting needs to be obtained based on pre-calibration, and the 90th percentile score can usually be taken as the threshold.

[0253] Coordinate system transformation

[0254] If the maximum matching score reaches the threshold requirement, then based on the optimal posture parameter θ best =(R best , t best ), transform the point cloud or image data from the user's perspective back to the environment coordinate system:

[0255]

[0256] in is the transformed feature point set, is the original feature point set.

[0257] Accurate registration

[0258] The transformed point cloud or image data It is registered with the corresponding part in the environment model to determine its precise position and posture (R′, t′) in the environment coordinate system. The registration method can adopt ICP (Iterative Closest Point) algorithm or epipolar geometry method.

[0259] Taking ICP as an example, its goal is to minimize the point cloud To Model Total distance error:

[0260]

[0261] in From point x′ to the model The closest distance.

[0262] ICP estimates the optimal R′ and t′ alternately in an iterative manner until it reaches a conclusion. It includes the following two steps:

[0263] (a) For each In the model Find the closest point Construct corresponding point pairs (x', y').

[0264] (b) For all point pairs (x′, y′), find the rotation R′ and translation t′ that minimizes the total distance error.

[0265] Repeat the above two steps until convergence.

[0266] Pose output

[0267] According to the posture of the registration result (R′, t′), combined with the relative posture relationship between the camera and the user (R c , t c ) and the user's height h, etc., to convert the camera position and orientation into the user's three-dimensional position p in the environment u and towards α u :

[0268]

[0269] α u =rpy(R′R c )

[0270] Where rpy(·) is the conversion function from rotation matrix to Euler angle.

[0271] Finally, a six-dimensional vector (p u , α u ), including the user's 3D position (x, y, z) and orientation (yaw, pitch, roll) in the environment. This vector will be used for subsequent rendering of the virtual avatar.

[0272] It should be noted that the above process will be affected by the cumulative error of the previous steps, so it is very sensitive to the setting of related parameters and thresholds. The accuracy of the posture conversion process is also critical, otherwise obvious drift will occur. At the same time, for users who move quickly, posture smoothing filtering and other means are also necessary to avoid jitter in the final output posture data.

[0273] A second aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute the above-mentioned XR virtual positioning method.

[0274] A third aspect of the present invention provides an XR virtual positioning system, which includes the above-mentioned computer-readable storage medium.

[0275] Specifically, the principle of the present invention is: first construct an environmental fingerprint code of a mixture of point cloud and image features of the target environment; during operation, use the improved Gray Wolf optimization algorithm to optimize the initial pose matching results in real time on the GPU, quickly obtain high-precision pose output, and render the virtual-real fusion scene. Its main principles are as follows:

[0276] 1. Mix point cloud and feature image to build environment fingerprint

[0277] The three-dimensional point cloud can accurately reflect the geometric structure of the scene, while the image features supplement the visual texture details. The fusion of the two can fully describe the environmental information. Therefore, the method of the present invention adopts a representation method combining point cloud and image features.

[0278] Specifically, we first use a depth camera or RGB-D sensor to scan the target environment and reconstruct a high-precision 3D point cloud model. At the same time, the camera also acquires the corresponding RGB image sequence. Then, we perform accelerated robust feature (SURF) extraction on the image to obtain the image feature vector.

[0279] Point cloud data and visual feature data are combined and encoded into a compressed environmental fingerprint. The fingerprint file is relatively small and easy to store and transmit. When encoding, a multi-view fusion strategy is adopted to improve the redundancy of the fingerprint. In this way, even if some view data is missing during runtime, the most complete environmental information can still be successfully decoded.

[0280] 2. Initial pose rough matching

[0281] During operation, the depth camera built into the XR helmet obtains the point cloud and image data from the current user's perspective. First, the CPU performs the same point cloud and visual feature extraction as above on the local data to obtain a feature set.

[0282] Then, these local feature data are copied to the GPU memory. The parallel architecture of the GPU is used to accelerate the initial rough matching of the feature set with the previously saved environmental fingerprint. Specifically, the similarity scores of each local feature and the features in the fingerprint are calculated in parallel on the GPU, and then weighted fusion is performed to obtain the overall initial matching score. This process can greatly improve the matching speed.

[0283] 3. Improve the gray wolf optimization algorithm to refine the pose

[0284] Although the above initial pose matching is fast, the accuracy is not ideal. In order to obtain a high-precision pose solution, the present invention uses the initial matching score as the fitness value, inputs the Improved Grey Wolf Optimization (IGWO) algorithm, and iteratively optimizes the pose parameters on the GPU.

[0285] The IGWO algorithm is an improvement on the traditional GWO algorithm, simulating the optimization process of wolf pack predation behavior. Compared with IGWO, the algorithm process has been improved to enhance the global search capability and convergence speed of the algorithm:

[0286] (1) The wolf pack position update equation was redesigned so that the wolf positions gradually converged to the optimal position while maintaining an appropriate degree of random discreteness in each iteration to avoid falling into the local optimum too early.

[0287] (2) Using random walk to introduce additional disturbances to promote the population to jump out of the local optimal area;

[0288] (3) In the later convergence stage, the speed of weight decrease is slowed down to ensure that the algorithm reaches a balance between global and local optimality.

[0289] The above improvements enable the IGWO algorithm to efficiently search a large range in the solution space in the early stage, and converge quickly and accurately in the later stage to achieve the global optimal match.

[0290] During the specific operation, the IGWO algorithm optimizes the pose parameters (position xyz, rotation angle, etc.), and the evaluation function is the score obtained in the above initial matching process. During the iteration process, the pose parameters are continuously fine-tuned to increase the matching score until the maximum value is reached and the corresponding pose solution is output.

[0291] Since a good initial solution is already very close to the global optimal value, the IGWO algorithm only needs a small number of iterations to reach the optimal value. Therefore, the entire optimization process is very fast, ensuring real-time high-precision positioning.

[0292] 4. Rendering virtual and real fusion scenes

[0293] The user's virtual avatar is accurately rendered in the corresponding three-dimensional physical position and orientation using the high-precision pose data output by the final optimization. At the same time, virtual objects such as furniture are superimposed on the corresponding positions of the real environment to form a seamless fusion of virtual and real scenes.

[0294] The inertial measurement unit (IMU) built into the helmet tracks the user's movements in real time, and the XR engine adjusts the position and orientation of the virtual avatar and virtual objects accordingly based on these movement data. As a result, when the user moves naturally in the physical world, the virtual and real scenes in the field of vision also change naturally, creating an ultimate immersive experience.

[0295] In summary, the method of the present invention first constructs the environmental fingerprint code of the fusion of point cloud and visual features, quickly completes the initial pose matching at runtime, and then iteratively optimizes on the GPU through the improved Gray Wolf optimization algorithm, finally obtains high-precision pose output, and seamlessly superimposes virtual elements in the physical environment, achieving a stable, real-time, and high-precision virtual-real fusion experience. Compared with existing solutions, it significantly improves positioning accuracy, real-time performance, robustness, and computing efficiency, providing strong support for the application and popularization of XR technologies such as virtual reality.

[0296] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An XR virtual positioning method, characterized in that: The following steps are involved: S10, pre-scan the scene environment and construct a three-dimensional point cloud model and visual feature image; S20, encoding the three-dimensional point cloud model and the visual feature image into an environmental fingerprint; S30, using the camera built into the XR helmet to obtain the point cloud and image from the user's perspective; S40, extracting the point cloud and local features of the image from the user's perspective in real time in the CPU and copying them to the GPU memory; S50, performing an initial matching operation on the GPU for the local feature and the environment fingerprint to obtain an initial matching score; S60, using the improved grey wolf algorithm DIGWO to take the initial matching score as a fitness value, optimizing the parameters in the matching process in the GPU, and obtaining a maximum matching score; S70, determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score; S80, rendering the user virtual avatar to a determined position and orientation according to the user three-dimensional position and orientation data; S90, updating the position and orientation of the virtual avatar in real time according to the user's movement, and displaying the rendered scene through the XR helmet to achieve a virtual-real fusion experience; The step of encoding the three-dimensional point cloud model and the visual feature image into an environmental fingerprint is specifically: voxelizing the three-dimensional point cloud data to discretize the continuous three-dimensional space into a regular three-dimensional voxel grid; using a method based on a deep learning model to feature encode the voxelized data to obtain a three-dimensional feature vector, wherein the method based on the deep learning model uses a 3D convolutional neural network model to directly encode the three-dimensional voxel data; and using a convolutional neural network to directly jointly encode the RGB image and the depth image for the visual feature image to obtain a visual feature vector; finally, the encoded three-dimensional feature vector and the visual feature vector are spliced ​​to form the fingerprint data of the environment; The step of using the improved gray wolf algorithm DIGWO to use the initial matching score as the fitness value, optimizing the parameters in the matching process in the GPU, and obtaining the maximum matching score is specifically: the improved gray wolf algorithm DIGWO is parallelized and transplanted to the GPU, specifically including the steps of population initialization, fitness evaluation, wolf group hierarchy division, and wolf position update are implemented in parallel on the GPU through CUDA or OpenCL, wherein the fitness evaluation process adopts a fast feature matching and similarity calculation method; the initial matching score is input as the initial fitness value of the DIGWO algorithm, and the DIGWO iterative optimization process is executed in parallel on the GPU, in each iteration, the position of each wolf is updated according to the wolf group hierarchy structure, corresponding to the camera posture parameters built into the XR helmet, and the matching score under the current parameter combination is calculated by using the method of feature matching, posture estimation and similarity calculation, so that it gradually approaches the global optimal posture; When the algorithm converges, the posture parameters corresponding to the global optimal solution at this time are output, which is the final matching posture, and the corresponding matching score is the maximum matching score.

2. The XR virtual positioning method according to claim 1, characterized in that: The steps of pre-scanning the scene environment and constructing a three-dimensional point cloud model and a visual feature image are specifically: using a laser radar or RGB-D camera three-dimensional scanning device to perform an all-round scan of the scene environment to obtain three-dimensional point cloud data of the scene environment; filtering the obtained three-dimensional point cloud data using a statistical filtering or distance-based filtering method, and removing outliers; reconstructing the pre-processed high-quality point cloud data using a Poisson reconstruction algorithm to construct a three-dimensional grid model of the scene environment; and extracting a visual feature image from the collected color image using SIFT.

3. The XR virtual positioning method according to claim 1, characterized in that: The step of extracting the local features of the point cloud and the image from the user's perspective in real time in the CPU is specifically: performing SIFT feature point detection and descriptor extraction on the acquired color image to obtain local visual features; performing equidistant sampling on the point cloud data, and using the FPFH projection-based method to encode the local features of the sampled key points and their neighborhoods to obtain local point cloud features; combining the extracted local visual features and local point cloud features to form a set of key points from the current user's perspective and their corresponding set of feature descriptors as local features.

4. The XR virtual positioning method according to claim 1, characterized in that: The step of initially matching the local feature with the environmental fingerprint in the GPU to obtain an initial matching score is specifically: matching the local feature with the environmental fingerprint by a nearest neighbor search method, calculating the posture transformation function of the XR helmet built-in camera corresponding to the local feature by using the RANSAC algorithm according to the matching result, and projecting the local feature into the fingerprint coordinate system of the environmental fingerprint according to the posture transformation function, and then calculating the similarity scores of the point-to-point distance, point-to-surface distance, and descriptor distance between the projected local feature and the matching result as the initial matching score.

5. The XR virtual positioning method according to claim 4, characterized in that: The step of determining the three-dimensional position and orientation of the current user in the scene environment according to the maximum matching score is specifically: using the camera posture parameters corresponding to the maximum matching score, transforming the point cloud or image data from the user's perspective back to the scene environment coordinate system, and using the ICP algorithm to align the transformed point cloud or image with the corresponding part in the three-dimensional point cloud model; obtaining the position and orientation of the camera built into the XR helmet in the scene environment coordinate system according to the alignment result, and converting the camera position and orientation into the three-dimensional position and orientation of the user in the scene environment in combination with the relative posture relationship between the camera and the user, and finally outputting a six-dimensional vector including the user's three-dimensional position and orientation.

6. The XR virtual positioning method according to claim 5, characterized in that: The method for voxelizing three-dimensional point cloud data is: an octree-based method or a hash-based method.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program instructions, and when the program instructions are executed, they are used to execute an XR virtual positioning method according to any one of claims 1 to 6.

8. An XR virtual positioning system, characterized in that: A computer-readable storage medium comprising the method of claim 7.

Images