An XR technology-based digital-real fusion interaction platform and method

Abstract

The application belongs to the technical field of extended reality interaction, and particularly relates to a data-real fusion interaction platform and method based on XR technology, wherein the data acquisition module comprises a camera for collecting video frames of the real world; the modeling module is used for calculating the depth information of object objects by multi-view stereo technology and generating 3D point clouds; the 3D point cloud environment is constructed by establishing a camera coordinate system and utilizing perspective projection technology; the controller is used for interactive operation in the 3D point cloud environment and specifying the end point of the path; the display module is used for displaying the 3D point cloud environment based on perspective projection and real-time interactive display; the computer device utilizes the PRM algorithm to randomly sample and construct a probability road network in the working space of the machine, and combines a graph search algorithm to find an optimal path trajectory. The method can automatically find a feasible path in a high-dimensional and complex environment, and can balance global planning and local optimization through a sampling mode, so that the success rate and efficiency of path planning are improved.

Description

Technical Field

[0001] This invention belongs to the field of extended reality interaction technology, specifically, it relates to a digital-real fusion interaction platform and method based on XR technology. Background Technology

[0002] In the current fields of intelligent drones, embodied robots, and quadruped robots, the efficient integration of virtual and real environments has become crucial for driving innovation in XR (Extended Reality) interaction technology. Traditional interaction relies on a mapping between remote control buttons and XR motion functions, but this is no longer suitable for today's AIGC (Artificial Intelligence of Things) machine applications. Traditional digital interaction systems and machine path planning methods often face the following shortcomings:

[0003] When users specify interaction targets through a virtual interface, they usually obtain two-dimensional pixel coordinates. However, how to accurately match these pixel coordinates with real-world coordinates remains a challenge.

[0004] Furthermore, for path planning in high-dimensional and complex environments, traditional analytical methods often struggle to balance global planning and local optimization, easily getting trapped in local optima. At the same time, computational efficiency and planning success rate are also limited, failing to meet real-time requirements. Summary of the Invention

[0005] To address the technical challenges of achieving high-precision matching between pixel coordinates and real-world coordinates in XR interaction, and the solution of path planning algorithms, this invention provides a data-real-world fusion interaction platform and method based on XR technology. This completes the closed loop from real-world data acquisition, 3D modeling, virtual-real interaction to intelligent path planning; it not only provides an immersive user experience but also significantly improves the accuracy and real-time performance of robot path planning, offering reliable technical support for intelligent control in complex environments.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] A data-real fusion interactive platform based on XR technology includes a data acquisition module, a modeling module, a controller, a display module, and a computer device;

[0008] The data acquisition module includes a camera for capturing video frames from the real world;

[0009] The modeling module is used to calculate the depth information of objects and generate 3D point clouds using multi-view stereo technology; it constructs a 3D point cloud environment by establishing a camera coordinate system and using perspective projection technology.

[0010] The controller is used for interactive operations in a 3D point cloud environment, specifying the end point of the path;

[0011] The display module is used to display a 3D point cloud environment based on perspective projection and to provide real-time interactive display.

[0012] Computer equipment is used to generate path trajectories using the PRM (Probabilistic Road Network) algorithm based on the coordinate information of the machine's path start point and path end point.

[0013] Preferably, the specific process of the modeling module includes:

[0014] Using multi-view stereo technology, the depth information between the object and the camera origin is calculated from consecutive video frames of the camera, and a 3D point cloud is generated.

[0015] Based on the camera's intrinsic parameters (focal length, principal point, distortion parameters, etc.), a camera coordinate system is established with the camera as the origin. The Z-axis points in front of the camera, and the X and Y axes represent the left and right and vertical directions, respectively. This provides a reference system for subsequent coordinate calculations and projections.

[0016] The camera's intrinsic parameter matrix is ​​established using the camera's internal parameters; its expression is:

[0017] ;

[0018] In the formula: and The effective focal length in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scaling factor). and These represent the principal coordinates of the image (the pixel position of the image center); 0 and 1 indicate the extension of homogeneous coordinates.

[0019] Project the 3D point cloud in the camera coordinate system onto the camera's XY plane to obtain its pixel coordinates; the calculation formula is as follows:

[0020] ; ; ;

[0021] In the formula: These are the normalized coordinates of the camera's XY plane; These are pixel coordinates; K is the camera's intrinsic parameter matrix;

[0022] The obtained 3D point cloud is projected from the XY plane of the camera coordinate system to calculate the pixel coordinates of each point on the XY plane.

[0023] Based on the depth information of the 3D point cloud, determine the coordinates of each point on the Z-axis;

[0024] Based on the three-dimensional coordinates of the 3D point cloud in the camera coordinate system, a 3D point cloud environment based on perspective projection is constructed, i.e., extended reality.

[0025] Preferably, the controller includes a mouse, which allows the user to control a virtual cursor to specify the end point of a path in a 3D point cloud environment.

[0026] Preferably, the display module is a head-mounted display or XR glasses;

[0027] Preferably, the real-time interactive display is implemented in the following manner:

[0028] Create and display a virtual cursor and a line segment perpendicular to the XY plane that follows the virtual cursor; the length of the line segment is adjusted in real time based on the depth information of the specified path endpoint.

[0029] Preferably, the specific process of the computer device includes:

[0030] S1) Capture the pixel coordinates and depth information of the end point of the path specified by the virtual cursor; and back-project them into the camera coordinate system, and then transform them into the world coordinate system;

[0031] S2) Obtain the machine's current position, i.e., the starting point of the path in the world coordinate system;

[0032] S3) Generate the path trajectory using the PRM (Probabilistic Road Network) algorithm based on the coordinate information of the path start point and the path end point.

[0033] Preferably, the specific process of the computer device further includes:

[0034] S4) Store the coordinate information of each point on the path trajectory in a path point manager and update it dynamically.

[0035] Preferably, the specific implementation process of step S1 includes:

[0036] S11) Defines the world coordinate system, which is used as a three-dimensional reference system in the real world;

[0037] S12) Calculate the transformation matrix between the world coordinate system and the camera coordinate system. Its expression is:

[0038] ;

[0039] In the formula, Indicates the camera coordinate system; R represents the world coordinate system; R is a 3×3 rotation matrix representing the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector representing the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the extension of homogeneous coordinates.

[0040] S13) The captured pixel coordinates are back-projected to the camera coordinate system and then transformed to the world coordinate system.

[0041] Preferably, in step S3), the specific process of the PRM algorithm generating the path trajectory includes:

[0042] S31) Construction of open motion planning library: Generate sample path points in the open motion planning library that satisfy the workspace constraints and path constraints of the current machine;

[0043] S32) Random sampling and neighborhood connection: Randomly sample a set of sample path points and check whether there are collisions or constraints on the straight interpolation path (or more complex interpolation methods);

[0044] S33) If there is no collision between the connecting lines within the boundary, then add the corresponding edge to the road network; this process constitutes the entire probabilistic road network structure.

[0045] S34) Road network integration: Add the coordinate information of two points in the path point manager to the probabilistic road network structure, and attempt to establish connections with its neighboring sample path points;

[0046] S35) Path optimization through graph search: Use graph search algorithms (such as Dijkstra's algorithm) to find the shortest path from the starting point to the ending point of the path. This path is a sequence of discrete sampled states, representing the basic motion path of the machine.

[0047] The present invention also provides a data-real fusion interaction method based on XR technology, applied to the above-mentioned platform, including the specific process of the modeling module.

[0048] A data-real fusion interaction method based on XR technology is applied to the aforementioned platform, including the specific process of computer equipment.

[0049] The beneficial effects of this invention are:

[0050] 1. Perspective projection technology is used to map 3D point clouds to 2D pixel coordinates, achieving consistency and accuracy in coordinate transformation. This provides a standardized mathematical basis for the conversion between virtual and real information, ensuring that subsequent interactions and planning can be based on accurate data.

[0051] 2. The pixel coordinates obtained through the virtual cursor are transformed to the camera coordinate system through back projection, and then to the world coordinate system, which effectively solves the problem of accurate conversion between virtual interactive data and the real workspace.

[0052] 3. The Probabilistic Road Network (PRM) algorithm is used to randomly sample within the machine's workspace and construct a probabilistic road network. This is then combined with a graph search algorithm to find the optimal path trajectory. This method can automatically discover feasible paths in high-dimensional and complex environments, and by using a sampling approach, it balances global planning with local optimization, thereby improving the success rate and efficiency of path planning. Attached Figure Description

[0053] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an architecture diagram of a data-real fusion interactive platform based on XR technology according to the present invention.

[0055] Figure 2 This is a flowchart illustrating the implementation of computer equipment in a data-real fusion interactive platform based on XR technology according to the present invention.

[0056] Figure 3 This is a flowchart of the steps of the PRM algorithm in a data-real fusion interactive platform based on XR technology according to the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Please see Figures 1-3 As shown, a data-real fusion interactive platform based on XR technology includes a data acquisition module, a modeling module, a controller, a display module, and a computer device;

[0059] The data acquisition module includes a camera for capturing video frames from the real world, providing real-time raw visual data for the subsequent generation of a 3D point cloud environment.

[0060] The modeling module is used to calculate the depth information of objects and generate 3D point clouds using multi-view stereo technology; it establishes a camera coordinate system and uses perspective projection technology to construct a 3D point cloud environment; that is, it expands the digital expression of real-world scenes and reflects the geometric structure and spatial depth distribution of the real world.

[0061] The controller is used for interactive operations in a 3D point cloud environment, specifying the end point of the path;

[0062] The display module is used to display a 3D point cloud environment based on perspective projection and to provide real-time interactive display.

[0063] Computer equipment is used to generate path trajectories using the PRM (Probabilistic Road Network) algorithm, based on the coordinates of the machine's starting and ending points. This provides a solution for machine interaction that blends digital and real-world elements.

[0064] It's important to note that the principle of XR (Extended Reality) technology lies in the real-time acquisition and analysis of real-world video streams, followed by real-time image rendering technology to fuse virtual scenes with real-world video footage. This invention, through image tracking and multi-view correction of the camera, allows the system to overlay virtual content in real-time onto real-world footage, providing users with a seamless, infinite sense of space between the virtual and real worlds. The data acquisition module provides dynamic video from the real world; the modeling module uses multi-view stereo technology to extract depth information, generate 3D point clouds, and project and transform them within the camera coordinate system. This preserves the authenticity of the real-world data while building a stable data foundation for subsequent extended reality interactions. Users can directly perform interactive path planning within extended reality by specifying the endpoints of the path, and the calculation module generates motion trajectories that can be directly applied to the machine, achieving bidirectional mapping and dynamic feedback between the virtual and the physical. The entire interactive platform realizes a closed-loop data flow from digital information acquisition and digital environment modeling to real-time interaction and motion planning, achieving data-real fusion in the field of machine interaction.

[0065] In the implementation process, panoramic cameras are used to capture continuous video data of real-world scenes, ensuring multi-view images with sufficient detail, including object outlines and terrain boundaries. Multi-view stereo technology and camera intrinsic parameters are then used to establish a camera coordinate system and calculate depth information.

[0066] Furthermore, the specific process of the modeling module includes:

[0067] Using multi-view stereo technology, the depth information between the object and the camera origin is calculated from consecutive video frames of the camera, and a 3D point cloud is generated.

[0068] Based on the camera's intrinsic parameters (focal length, principal point, distortion parameters, etc.), a camera coordinate system is established with the camera as the origin. The Z-axis points in front of the camera, and the X and Y axes represent the left and right and vertical directions, respectively. This provides a reference system for subsequent coordinate calculations and projections.

[0069] The camera's intrinsic parameter matrix is ​​established using the camera's internal parameters; its expression is:

[0070] ;

[0071] In the formula: and The effective focal length in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scaling factor). and These represent the principal coordinates of the image (the pixel position of the image center); 0 and 1 indicate the extension of homogeneous coordinates.

[0072] Project the 3D point cloud in the camera coordinate system onto the camera's XY plane to obtain its pixel coordinates; the calculation formula is as follows:

[0073] ; ; ;

[0074] In the formula: These are the normalized coordinates of the camera's XY plane; These are pixel coordinates; K is the camera's intrinsic parameter matrix;

[0075] Specifically, the camera's intrinsic parameter matrix describes the camera's internal geometry, including focal length, image center point, and pixel scaling. Input is a camera coordinate system point. ,in, Based on a pinhole camera model, 3D points are projected onto the camera's imaging plane, and their coordinates are normalized. Then, the normalized coordinates are converted into pixel coordinates. Therefore, the modeling module of this invention, through the combination of camera intrinsic and extrinsic parameter matrices, can achieve projection from three-dimensional space to two-dimensional image, and also reverse projection from two-dimensional image to three-dimensional space.

[0076] The obtained 3D point cloud is projected from the XY plane of the camera coordinate system to calculate the pixel coordinates of each point on the XY plane.

[0077] Based on the depth information of the 3D point cloud, determine the coordinates of each point on the Z-axis;

[0078] Based on the three-dimensional coordinates of the 3D point cloud in the camera coordinate system, a 3D point cloud environment based on perspective projection is constructed, i.e., extended reality.

[0079] In the specific implementation process, panoramic cameras are used to capture continuous video data of real-world scenes. Preprocessing steps such as image denoising, feature point detection, and matching provide high-quality input for subsequent multi-view stereoscopic technology. Using images captured from multiple angles, multi-view stereoscopic technology is applied to reconstruct the depth information of the scene and generate a dense 3D point cloud—a 3D point cloud describing the actual scene structure. This 3D point cloud data accurately reflects the geometric features of real-world objects and provides a true foundation for depth data; specifically, the Z-axis coordinates of the camera coordinate system.

[0080] Next, the camera is calibrated to obtain parameters such as focal length, principal point, and distortion coefficients. A camera coordinate system is then established, with its origin set as the camera's optical center. This coordinate system is used not only for calculating the 3D coordinates of each pixel in the video frame but also for providing a unified reference for subsequent point cloud projection.

[0081] Then, using a perspective projection model, the 3D point cloud is converted into pixel coordinates (u,v) on the XY plane in the camera coordinate system, and all points of the 3D point cloud are mapped onto the XY plane of the camera coordinate system to generate pixel data corresponding to the camera coordinate system; in addition, the Z-axis coordinate points remain unchanged.

[0082] By mapping 3D point cloud data to two-dimensional pixel coordinates using perspective projection technology, the consistency and high precision of coordinate transformations between different data sets are ensured. This provides a standardized mathematical basis for the conversion between virtual and real information, enabling subsequent interactions and path planning to be based on accurate digital data, effectively compensating for a series of problems caused by mapping errors in existing systems. Furthermore, a 3D point cloud environment based on perspective projection, i.e., extended reality, can be built using platforms such as Unity. This process allows extended reality to maintain consistency with the real world while possessing editable digital characteristics.

[0083] Furthermore, the external parameters of the camera include position and attitude angles; used to define the camera's attitude and position in the 3D world coordinate system;

[0084] The camera's internal parameters include focal length, image center point, and pixel scaling ratio, which are used to define the geometric characteristics of the images captured by the camera.

[0085] Furthermore, the controller includes a mouse, which allows the user to control a virtual cursor to specify the end point of the path in the 3D point cloud environment. That is, the user specifies the target location in the XR environment using the virtual cursor. Without complex programming or remote control operation, the user can interactively plan the robot's motion path simply by specifying the end point of the path in the intuitive 3D point cloud environment using the mouse; this significantly reduces the barrier to entry and time cost of path planning, and improves interaction efficiency.

[0086] Furthermore, the display module is a head-mounted display or XR glasses; it renders the 3D point cloud environment based on perspective projection in real time, providing an immersive, virtual-real fusion visual experience.

[0087] Furthermore, the specific method of the real-time interactive display is as follows:

[0088] A virtual cursor and a line segment perpendicular to the XY plane that follows the virtual cursor are created and displayed. The length of the line segment is adjusted in real time based on the depth information of the specified path endpoint. To enhance interactive feedback, the display module displays a virtual cursor and a dynamic line segment perpendicular to the XY plane in real time within the XR glasses. The length of the latter changes in real time according to the endpoint selected by the user, thus intuitively displaying the target height information.

[0089] Furthermore, the specific processes of the computer device include:

[0090] S1) Capture the pixel coordinates and depth information of the end point of the path specified by the virtual cursor; and back-project them into the camera coordinate system, and then transform them into the world coordinate system;

[0091] S2) Obtain the machine's current position, i.e., the starting point of the path in the world coordinate system;

[0092] S3) Generate the path trajectory using the PRM (Probabilistic Road Network) algorithm based on the coordinate information of the path start point and the path end point.

[0093] Specifically, the computer device captures the endpoint of the path specified by the controller and is responsible for calculations such as path point resolution, trajectory generation, and dynamic data updates. In path point resolution and trajectory generation, the PRM (Probabilistic Road Network) algorithm can be used to construct a network topology graph (i.e., road network) covering the machine's workspace with randomly sampled path points, and search for feasible paths from the starting point to the end point of the path on this network. This ensures that the motion is reachable, safe, and has a relatively low motion cost. The PRM algorithm uses random sampling and does not rely on the uniform discretization of the spatial grid, thus it can handle high-dimensional state spaces well. For robots with complex kinematic constraints and high-dimensional degrees of freedom, PRM can generate state samples with broad coverage, improving the planning success rate. Since PRM relies on random sampling and graph search algorithms, the computational complexity is usually controllable, and the algorithm itself has a highly modular design. This method uses random sampling to construct a global view and optimizes through local connectivity detection, which can automatically discover feasible paths in high-dimensional and complex environments, effectively balancing the relationship between global planning and local optimization, and significantly improving the success rate and computational efficiency of path planning.

[0094] Furthermore, the specific process of the computer device also includes:

[0095] S4) Store the coordinate information of each point on the path trajectory in a path point manager and update it dynamically. The path point manager serves as the basis for data updates, maintains the continuity between the path history and the current execution trajectory, and acts as a bridge between the interactive process and the control system.

[0096] Furthermore, the specific implementation process of step S1 includes:

[0097] S11) Defines the world coordinate system, which is used as a three-dimensional reference system in the real world;

[0098] S12) Calculate the transformation matrix between the world coordinate system and the camera coordinate system. Its expression is:

[0099] ;

[0100] In the formula, Indicates the camera coordinate system; R represents the world coordinate system; R is a 3×3 rotation matrix representing the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector representing the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the expansion of homogeneous coordinates.

[0101] Specifically, calculating the transformation matrix between the world coordinate system and the camera coordinate system typically involves a coordinate transformation process.

[0102] The rotation matrix R is calculated from the camera's attitude angles (yaw, pitch, and roll) and is defined using Euler's formula.

[0103] Translation vector The coordinates of any camera point are determined by the camera's position. A transformation matrix is ​​used to transform the coordinates of any camera point... Convert to world coordinate system points This allows us to connect the real world with the camera's perspective.

[0104] S13) The captured pixel coordinates are back-projected to the camera coordinate system and then transformed to the world coordinate system.

[0105] Specifically, it is necessary to start from two-dimensional pixel coordinates. Deducing the world coordinates in three-dimensional space The depth information Zc and extrinsic parameters (rotation matrix R and translation vector T) are combined to calculate the coordinates. This allows for the transformation from 2D image to 3D world coordinates based on the captured pixel coordinates through backprojection and spatial transformation matrices. Using the pixel coordinates obtained from the virtual cursor, backprojection transforms them back to the camera coordinate system, and then further transforms them to the world coordinate system, fundamentally solving the data matching problem between virtual interactive data and the real workspace. This process ensures that the target position specified by the user in the XR environment is accurately reflected in the real workspace, significantly improving the accuracy of interactive operations and the overall system coordination.

[0106] Furthermore, in step S3), the specific process of the PRM algorithm generating the path trajectory includes:

[0107] S31) Construction of open motion planning library: Generate sample path points in the open motion planning library that satisfy the workspace constraints and path constraints of the current machine;

[0108] S32) Random sampling and neighborhood connection: Randomly sample a set of sample path points and check whether there are collisions or constraints on the straight interpolation path (or more complex interpolation methods);

[0109] S33) Connectivity detection: If there is no collision between the lines within the boundary, then add the corresponding edge to the road network; this process constitutes the entire probabilistic road network structure.

[0110] S34) Road network integration: Add the coordinate information of two points in the path point manager to the probabilistic road network structure, and attempt to establish connections with its neighboring sample path points;

[0111] S35) Path optimization through graph search: Use graph search algorithms (such as Dijkstra's algorithm) to find the shortest path from the starting point to the ending point of the path. This path is a sequence of discrete sampled states, representing the basic motion path of the machine.

[0112] Specifically, Probabilistic Road Network (PRM) has been widely implemented in many motion planning libraries (such as OMPL) and can be seamlessly integrated with other algorithm components such as robot control systems, collision detection modules, and path smoothing algorithms. This openness and modularity allows PRM to be customized and extended in different application scenarios. This invention constructs an open motion planning library for a given scenario; it randomly generates a large number of discrete path points that satisfy non-collision constraints. The sampling strategy can be purely random, target-biased, or heuristic sampling to ensure sufficient coverage of path points. Next, collision detection and constraint verification are performed on candidate connected paths to ensure that the connected paths are locally feasible (e.g., without obstacle interference). If the detection passes, an edge is added to the probabilistic road network graph, indicating that the two states are reachable. After a large number of state samplings and neighborhood connectivity checks, the entire set of discrete nodes in the workspace and the edges between them constitute an undirected graph, which is the probabilistic road network. Finally, shortest path search algorithms such as Dijkstra's algorithm and A* are used to find a sequence of feasible paths from the starting point to the ending point in the generated undirected graph. In addition, after finding a discrete path, it may be necessary to smooth or reparameterize the path to ensure that the path has continuity and dynamic feasibility in actual execution.

[0113] This invention also provides a data-real fusion interaction method based on XR technology, applied to the above-mentioned platform, comprising the following steps:

[0114] Data acquisition and modeling: Video frames are acquired by the camera, depth information is calculated and 3D point cloud is generated using multi-view stereo technology, a camera coordinate system is established, and then an XR environment is constructed using perspective projection technology.

[0115] Interaction and display: Using devices such as head-mounted displays and mice, a 3D point cloud environment and a virtual cursor are displayed in real time in the XR environment. Users can specify the end point of the path through the interaction of the virtual cursor.

[0116] Path planning: Capture the pixel coordinates of the end point of the path specified by the user, back-project the captured pixel coordinates to the camera coordinate system, and then transform them to the world coordinate system; obtain the current position of the machine, i.e. the starting point of the path; then use the PRM algorithm to randomly sample in the workspace and construct a probabilistic road network to find and generate an optimal path trajectory.

[0117] Trajectory Management and Feedback: Real-time feedback of interaction and planning results is provided through the XR environment, and the coordinate information of the path trajectory is dynamically updated by a path point manager to achieve interactive control that integrates data and reality.

[0118] In summary, this invention utilizes a camera to capture continuous video frames and combines multi-view stereo technology to calculate depth information, thereby generating 3D point cloud data. A camera coordinate system is constructed based on camera intrinsic parameters, and perspective projection technology is used to map the 3D point cloud to two-dimensional pixel coordinates, achieving consistency and accuracy in coordinate transformation. This provides a standardized mathematical basis for the conversion between virtual and real information, ensuring that subsequent interactions and planning can be based on accurate data. Furthermore, in terms of interaction, the pixel coordinates obtained by the user through a virtual cursor are transformed to the camera coordinate system via back projection, and then to the world coordinate system, effectively solving the problem of conversion between virtual interactive data and the real workspace. For path planning, the PRM algorithm from the Open Motion Planning Library is used to randomly sample and construct a probabilistic road network within the robot's workspace, combined with a graph search algorithm to find the optimal path trajectory. This method can automatically discover feasible paths in high-dimensional and complex environments, and by using a sampling method, it considers both global planning and local optimization, improving the success rate and efficiency of path planning. The rigorous logic and collaborative operation of each component enable users to intuitively plan paths within the XR environment. Furthermore, the PRM algorithm generates optimized paths, effectively improving the system's accuracy, real-time performance, and adaptability. This has significant application value in interactive fields such as intelligent drones, embodied machines, and quadrupedal machines.

[0119] In the description of this specification, the terms "specifically," "in practice," "further," etc., refer to specific features, structures, materials, or characteristics described in connection with the embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. The above description is merely an example and illustration of the structure of the present invention. Various modifications or additions to the described specific embodiments, or substitutions using similar methods, made by those skilled in the art, without departing from the structure of the invention or exceeding the scope defined by the claims, should fall within the protection scope of the present invention.

Claims

1. A data-real fusion interactive platform based on XR technology, characterized in that: It includes a data acquisition module, a modeling module, a controller, a display module, and computer equipment; The data acquisition module includes a camera for capturing video frames from the real world; The modeling module is used to calculate the depth information of objects and generate 3D point clouds using multi-view stereo technology; it constructs a 3D point cloud environment by establishing a camera coordinate system and using perspective projection technology. The controller is used for interactive operations in a 3D point cloud environment, specifying the end point of the path; The display module is used to display a 3D point cloud environment based on perspective projection and to provide real-time interactive display. Computer equipment is used to generate a path trajectory using the PRM algorithm based on the coordinate information of the machine's path start point and path end point, where the machine's current position is the path start point in the world coordinate system. The specific process of the modeling module includes: Using multi-view stereo technology, the depth information between the object and the camera origin is calculated from consecutive video frames of the camera, and a 3D point cloud is generated. Establish a camera coordinate system with the camera as the origin, where the Z-axis points forward of the camera, and the X and Y axes represent the left, right, and vertical directions, respectively; establish the camera's intrinsic parameter matrix using the camera's internal parameters; its expression is: ； In the formula: and The effective focal length in the horizontal and vertical directions; and These represent the coordinates of the principal point of the image; 0 and 1 indicate the extension of the homogeneous coordinates. Project the 3D point cloud in the camera coordinate system onto the camera's XY plane to obtain its pixel coordinates; the calculation formula is as follows: ； ； ； In the formula: Indicates the camera coordinate system. These are the normalized coordinates of the camera's XY plane; These are pixel coordinates; K is the camera's intrinsic parameter matrix; The obtained 3D point cloud is projected from the XY plane of the camera coordinate system to calculate the pixel coordinates of each point on the XY plane. Based on the depth information of the 3D point cloud, determine the coordinates of each point on the Z-axis; Based on the three-dimensional coordinates of the 3D point cloud in the camera coordinate system, a 3D point cloud environment based on perspective projection is constructed, i.e., extended reality.

2. The XR-based data-real fusion interactive platform according to claim 1, characterized in that: The implementation process of the computer device includes: S1) Capture the pixel coordinates and depth information of the end point of the path specified by the virtual cursor; and back-project them into the camera coordinate system, and then transform them into the world coordinate system; S2) Obtain the machine's current location; S3) Generate the path trajectory using the PRM algorithm based on the coordinate information of the path start point and the path end point.

3. The XR-based data-real fusion interactive platform according to claim 2, characterized in that: The specific processes of the computer equipment also include: S4) Store the coordinate information of each point on the path trajectory in a path point manager and update it dynamically.

4. The XR-based data-real fusion interactive platform according to claim 1, characterized in that: The controller includes a mouse, which allows the user to control a virtual cursor to specify the end point of a path in a 3D point cloud environment.

5. The XR-based data-real fusion interactive platform according to claim 1, characterized in that: The specific method of real-time interactive display in the display module is as follows: Create and display a virtual cursor and a line segment perpendicular to the XY plane that follows the virtual cursor; the length of the line segment is adjusted in real time based on the depth information of the specified path endpoint.

6. The XR-based data-real fusion interactive platform according to claim 2, characterized in that: The specific steps of step S1 include: S11) Defines the world coordinate system, which is used as a three-dimensional reference system in the real world; S12) Calculate the transformation matrix between the world coordinate system and the camera coordinate system. Its expression is: ； In the formula, Indicates the camera coordinate system; R represents the world coordinate system; R is a 3×3 rotation matrix representing the rotation of the camera coordinate system relative to the world coordinate system; T is a 3×1 translation vector representing the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the expansion of homogeneous coordinates. S13) The captured pixel coordinates are back-projected to the camera coordinate system and then transformed to the world coordinate system.

7. A data-real fusion interactive platform based on XR technology according to claim 2, characterized in that: In step S3), the specific steps of the PRM algorithm to generate the path trajectory include: S31) Generate sample path points in the open motion planning library that satisfy the workspace constraints and path constraints of the current machine; S32) Randomly sample a set of sample path points and check whether there are collisions or constraints on the straight interpolation path; S33) If there is no collision between the connecting lines within the boundary, then add the corresponding edge to the road network to form the entire probabilistic road network structure. S34) Add the coordinate information of the two points in the path point manager to the probabilistic road network structure, and attempt to establish a connection with the sample path points next to them; S35) Use a graph search algorithm to find the shortest path from the starting point of the path to the ending point of the path.

8. A data-real fusion interaction method based on XR technology, applied to the platform described in any one of claims 1-7, characterized in that, Includes the following steps: Video frames are captured by a camera, depth information is calculated and 3D point clouds are generated using multi-view stereo technology, a camera coordinate system is established, and then an XR environment is constructed using perspective projection technology. Using interactive devices, a 3D point cloud environment and a virtual cursor XR environment are displayed in real time, and users specify the end point of the path through the interaction of the virtual cursor. Capture the pixel coordinates of the end point of the path specified by the user, backproject the captured pixel coordinates to the camera coordinate system, and then transform them to the world coordinate system; and obtain the current position of the machine, i.e. the starting point of the path; The PRM algorithm is used to randomly sample within the workspace and construct a probabilistic road network to find and generate an optimal path trajectory.

9. A data-real fusion interaction method based on XR technology according to claim 8, characterized in that, Also includes: The XR environment provides real-time feedback on interaction and planning results, and a path point manager dynamically updates the coordinate information of the path trajectory, enabling interactive control that integrates data and reality.

Images