Digital-real fusion interaction platform and method based on XR technology

Abstract

The invention belongs to the technical field of augmented reality interaction, and particularly relates to a data-real fusion interaction platform and method based on the XR technology, and a data collection module comprises a camera which is used for collecting video frames of the real world; the modeling module is used for calculating depth information of an object by a multi-view stereo technology and generating a 3D point cloud; a 3D point cloud environment is constructed by establishing a camera coordinate system and utilizing a perspective projection technology; the controller is used for performing interactive operation in the 3D point cloud environment and appointing a path end point; the display module is used for displaying the 3D point cloud environment based on perspective projection and performing real-time interactive display; and the computer equipment randomly samples in a working space of a machine by using a PRM algorithm and constructs a probability road network, and finds an optimal path track in combination with a graph search algorithm. According to the method, a feasible path can be automatically found in a high-dimensional and complex environment, global planning and local optimization are considered in a sampling mode, and the success rate and efficiency of path planning are improved.

Description

Technical Field

[0001] This invention belongs to the field of extended reality interaction technology, specifically, it relates to a digital-real fusion interaction platform and method based on XR technology. Background Technology

[0002] In the current fields of intelligent drones, embodied robots, and quadruped robots, the efficient integration of virtual and real environments has become crucial for driving innovation in XR (Extended Reality) interaction technology. Traditional interaction relies on a mapping between remote control buttons and XR motion functions, but this is no longer suitable for today's AIGC (Artificial Intelligence of Things) machine applications. Traditional digital interaction systems and machine path planning methods often face the following shortcomings: When users specify interaction targets through a virtual interface, they usually obtain two-dimensional pixel coordinates. However, how to accurately match these pixel coordinates with real-world coordinates remains a challenge.

[0003] Furthermore, for path planning in high-dimensional and complex environments, traditional analytical methods often struggle to balance global planning and local optimization, easily getting trapped in local optima. At the same time, computational efficiency and planning success rate are also limited, failing to meet real-time requirements. Summary of the Invention

[0004] To address the technical challenges of achieving high-precision matching between pixel coordinates and real-world coordinates in XR interaction, and the solution of path planning algorithms, this invention provides a data-real-world fusion interaction platform and method based on XR technology. This completes the closed loop from real-world data acquisition, 3D modeling, virtual-real interaction to intelligent path planning; it not only provides an immersive user experience but also significantly improves the accuracy and real-time performance of robot path planning, offering reliable technical support for intelligent control in complex environments.

[0005] The objective of this invention can be achieved through the following technical solutions: A data-real fusion interactive platform based on XR technology includes a data acquisition module, a modeling module, a controller, a display module, and a computer device; The data acquisition module includes a camera for capturing video frames from the real world; The modeling module is used to calculate the depth information of objects and generate 3D point clouds using multi-view stereo technology; it constructs a 3D point cloud environment by establishing a camera coordinate system and using perspective projection technology. The controller is used for interactive operations in a 3D point cloud environment, specifying the end point of the path; The display module is used to display a 3D point cloud environment based on perspective projection and to provide real-time interactive display. The computer device is configured to generate a path trajectory by using a PRM (Probabilistic Roadmap) algorithm according to coordinate information of a path start point and a path end point of the machine.

[0006] Preferably, the specific process of the modeling module comprises: The depth information between the object and the camera origin is calculated from the continuous video frames of the camera by using the multi-view stereo technology, and a 3D point cloud is generated. According to the internal parameters (focal length, principal point, distortion parameters, etc.) of the camera, a camera coordinate system is established with the camera as the origin, wherein the Z axis points to the front of the camera, and the X and Y axes represent the left and vertical directions, respectively, thereby providing a reference system for subsequent coordinate calculation and projection. The internal parameter matrix of the camera is established through the internal parameters of the camera, and the expression is: ; In the formula: and are the effective focal lengths in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scale factor); and are the image principal point coordinates (pixel positions of the image center point); 0 and 1 represent the extension of the homogeneous coordinates; The 3D point cloud in the camera coordinate system is projected onto the XY plane of the camera to obtain its pixel coordinates, and the calculation formula is: ; ; ; In the formula: is the normalized coordinate of the XY plane of the camera; is the pixel coordinate; K is the internal parameter matrix of the camera; The 3D point cloud obtained in the XY plane of the camera coordinate system is perspective projected to calculate the pixel coordinates of each point in the XY plane; According to the depth information of the 3D point cloud, the coordinates of each point on the Z axis are determined; According to the three-dimensional coordinates of the 3D point cloud in the camera coordinate system, a 3D point cloud environment based on perspective projection is constructed, i.e., an extended reality.

[0007] Preferably, the controller includes a mouse, and the user controls the virtual cursor to specify the path end point in the 3D point cloud environment through the mouse function.

[0008] Preferably, the display module is a head-mounted display or an XR glasses; Preferably, the specific way of real-time interactive display is: A virtual cursor and a line segment perpendicular to the XY plane and following the virtual cursor are created and displayed; the length of the line segment is adjusted in real time according to the depth information of the end point of the path specified by the virtual cursor.

[0009] Preferably, the specific process of the computer device comprises: S1) capturing the pixel coordinates and depth information of the end point of the path specified by the virtual cursor; and back-projecting the same into the camera coordinate system and then converting the same into the world coordinate system; S2) obtaining the current position of the machine, i.e. the start point of the path in the world coordinate system; S3) generating the path trajectory by using the PRM (Probabilistic Roadmap) algorithm according to the coordinate information of the start point and the end point of the path.

[0010] Preferably, the specific process of the computer device further comprises: S4) storing the coordinate information of each point on the path trajectory in a path point manager and dynamically updating the same.

[0011] Preferably, the specific implementation process of the step S1 comprises: S11) defining a world coordinate system as a three-dimensional reference system for the real world; S12) calculating the conversion matrix between the world coordinate system and the camera coordinate system, which is expressed as: ; In the formula, represents the camera coordinate system; represents the world coordinate system; R is a 3x3 rotation matrix, representing the rotation of the camera coordinate system relative to the world coordinate system; T is a 3x1 translation vector, representing the displacement of the origin of the camera coordinate system relative to the world coordinate system; 0 and 1 are used to represent the extension of the homogeneous coordinates.

[0012] S13) back-projecting the captured pixel coordinates into the camera coordinate system and then converting the same into the world coordinate system.

[0013] Preferably, in the step S3, the specific process of the PRM algorithm for generating the path trajectory comprises: S31) open motion planning library construction: generating sample path points in the open motion planning library that satisfy the workspace constraints and path constraints of the current machine; S32) random sampling and neighborhood connection: randomly sampling a group of sample path points and checking whether there is a collision or a violation of constraints on the straight-line interpolation path (or a more complex interpolation method); S33) if the connection within the boundary has no collision, adding the corresponding edge in the road network; this process constitutes the entire probabilistic road network structure. S34) Road network integration: add the coordinate information of two points in the path point manager to the probabilistic road network structure, and try to establish a connection with the adjacent sample path points; S35) Path optimization through graph search: use a graph search algorithm (such as Dijkstra's algorithm) to find the shortest path from the path start point to the path end point. The path is a sequence of discrete sample states representing the basic motion path of the machine.

[0014] The application also provides a number-real fusion interaction method based on XR technology, which is applied to the platform and includes the specific process of the modeling module.

[0015] A number-real fusion interaction method based on XR technology, which is applied to the platform and includes the specific process of the computer device.

[0016] The application has the following advantages: 1. The perspective projection technology is used to map the 3D point cloud to the two-dimensional pixel coordinates, realizing the consistency and accuracy of coordinate conversion. The method provides a standard mathematical basis for virtual-real information conversion, ensuring that subsequent interaction and planning can be based on accurate data.

[0017] 2. The pixel coordinates obtained by the virtual cursor are converted to the camera coordinate system through back projection and then to the world coordinate system, effectively solving the problem of accurate conversion between virtual interaction data and real working space.

[0018] 3. The PRM algorithm is used to randomly sample and construct a probabilistic road network in the working space of the machine, and the graph search algorithm is used to find the optimal path trajectory. This method can automatically find feasible paths in high-dimensional and complex environments, and can balance global planning and local optimization through sampling, improving the success rate and efficiency of path planning. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0020] Fig. 1 The figure is the architecture of the number-real fusion interaction platform based on XR technology.

[0021] Fig. 2 The figure is the implementation flowchart of the computer device in the number-real fusion interaction platform based on XR technology.

[0022] Fig. 3A step flowchart of a PRM algorithm in a number-real fusion interaction platform based on XR technology. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0024] Please refer to Figs. 1-3 As shown in the figure, a number-real fusion interaction platform based on XR technology includes a data acquisition module, a modeling module, a controller, a display module, and a computer device. The data acquisition module includes a camera for acquiring video frames of the real world; and provides real-time visual raw data for subsequent generation of a 3D point cloud environment.

[0025] The modeling module is used to calculate the depth information of an object and generate a 3D point cloud through multi-view stereo technology; a camera coordinate system is established, and a 3D point cloud environment is constructed by using perspective projection technology; that is, the digital expression of an extended reality scene, which reflects the geometric structure and spatial depth distribution of the real world.

[0026] The controller is used for interactive operation in the 3D point cloud environment, and specifies the end point of the path. The display module is used to display the 3D point cloud environment based on perspective projection and real-time interactive display. The computer device is used to generate a path trajectory by using a PRM (Probabilistic Roadmap) algorithm according to the coordinate information of the start point and the end point of the path of the machine. It provides a digital-real fusion solution for machine interaction.

[0027] It should be noted that the principle of XR (extended reality) technology is based on real-time acquisition and analysis of real-world video streams, and real-time image rendering technology is used to fuse virtual scenes with real video pictures. The system can superimpose virtual content in real time in front of the real shooting picture through image tracking and multi-view correction of the camera, so that the user can feel the infinite space feeling of "seamless connection" between virtual and real. Among them, the data acquisition module provides dynamic video of the real world; the modeling module uses multi-view stereo technology to extract depth information, generates three-dimensional point cloud, and projects and converts in the camera coordinate system. In this way, the reality of the real data is preserved, and a stable data foundation is built for subsequent extended reality interaction. The user directly plans the interactive path in the extended reality, specifies the end points at both ends of the path, and generates a motion path trajectory that can be directly applied to the machine by the calculation module, realizing the bidirectional mapping and dynamic feedback between virtual and physical. The whole interactive platform realizes the data flow closed loop from digital information acquisition, digital environment modeling, to real-time interaction and motion planning, realizing the integration of digital and real in the field of machine interaction.

[0028] In the specific implementation process, a panoramic camera is used to capture continuous real scene video data, ensuring that the multi-view image has sufficient picture detail information, including object contour, terrain boundary and the like. And the camera coordinate system is established by using multi-view stereo technology and camera internal parameter, and the depth information is calculated.

[0029] Further, the specific process of the modeling module includes: The multi-view stereo technology is used to calculate the depth information between the object and the camera origin from the continuous video frames of the camera, and the 3D point cloud is generated; According to the internal parameters (focal length, principal point, distortion parameters, etc.) of the camera, a camera coordinate system is established with the camera as the origin, wherein the Z axis points to the front of the camera, and the X and Y axes represent the left and vertical directions respectively; thereby providing a reference system for subsequent coordinate calculation and projection; The internal parameter matrix of the camera is established through the internal parameters of the camera; the expression is: ; In the formula: And are the effective focal lengths in the horizontal and vertical directions (usually determined by the actual focal length f and the pixel scale factor); And are the image principal point coordinates (pixel position of the image center point); 0 and 1 represent the extension of the homogeneous coordinates; The 3D point cloud in the camera coordinate system is projected onto the XY plane of the camera to obtain the pixel coordinates; the calculation formula is: ; ; ; In the formula: These are the normalized coordinates of the camera's XY plane; These are pixel coordinates; K is the camera's intrinsic parameter matrix; Specifically, the camera's intrinsic parameter matrix describes the camera's internal geometry, including focal length, image center point, and pixel scaling. Input is a camera coordinate system point. ,in, Based on a pinhole camera model, 3D points are projected onto the camera's imaging plane, and their coordinates are normalized. Then, the normalized coordinates are converted into pixel coordinates. Therefore, the modeling module of this invention, through the combination of camera intrinsic and extrinsic parameter matrices, can achieve projection from three-dimensional space to two-dimensional image, and also reverse projection from two-dimensional image to three-dimensional space.

[0030] The obtained 3D point cloud is projected from the XY plane of the camera coordinate system to calculate the pixel coordinates of each point on the XY plane. Based on the depth information of the 3D point cloud, determine the coordinates of each point on the Z-axis; Based on the three-dimensional coordinates of the 3D point cloud in the camera coordinate system, a 3D point cloud environment based on perspective projection is constructed, i.e., extended reality.

[0031] In the specific implementation process, panoramic cameras are used to capture continuous video data of real-world scenes. Preprocessing steps such as image denoising, feature point detection, and matching provide high-quality input for subsequent multi-view stereoscopic technology. Using images captured from multiple angles, multi-view stereoscopic technology is applied to reconstruct the depth information of the scene and generate a dense 3D point cloud—a 3D point cloud describing the actual scene structure. This 3D point cloud data accurately reflects the geometric features of real-world objects and provides a true foundation for depth data; specifically, the Z-axis coordinates of the camera coordinate system.

[0032] Next, the camera is calibrated to obtain parameters such as focal length, principal point, and distortion coefficients. A camera coordinate system is then established, with its origin set as the camera's optical center. This coordinate system is used not only for calculating the 3D coordinates of each pixel in the video frame but also for providing a unified reference for subsequent point cloud projection.

[0033] Then, using a perspective projection model, the 3D point cloud is converted into pixel coordinates (u,v) on the XY plane in the camera coordinate system, and all points of the 3D point cloud are mapped onto the XY plane of the camera coordinate system to generate pixel data corresponding to the camera coordinate system; in addition, the Z-axis coordinate points remain unchanged.

[0034] The 3D point cloud data is mapped to two-dimensional pixel coordinates through perspective projection technology, ensuring the consistency and high precision of coordinate conversion between data. This provides a standardized mathematical basis for the conversion between virtual and real information, enabling subsequent interaction and path planning to be based on accurate digital data, effectively addressing the mapping errors that exist in current systems. Further, a 3D point cloud environment based on perspective projection can be constructed in platforms such as Unity, which is an extended reality. This process allows the extended reality to maintain consistency with the real world while also having editable digital characteristics.

[0035] Further, the external parameters of the camera include position and attitude angle; for defining the attitude and position of the camera in the 3D world coordinate system; The internal parameters of the camera include focal length, image center point, and pixel scaling ratio, which are used to define the geometric characteristics of the camera's captured images.

[0036] Further, the controller includes a mouse, and the user controls the virtual cursor to specify the end point of the path in the 3D point cloud environment through the mouse function. That is, the user specifies the target position in the XR environment through the virtual cursor. The user does not need to perform complex programming or remote control operations, but only needs to specify the end point of the path in the intuitive 3D point cloud environment through the mouse; this allows the user to interactively plan the robot's motion path, significantly reducing the threshold and time cost of path planning, and improving interaction efficiency.

[0037] Further, the display module is a head-mounted display or XR glasses; the 3D point cloud environment based on perspective projection is rendered in real time, providing an immersive and virtual-real integrated visual experience.

[0038] Further, the specific way of real-time interaction display is: A virtual cursor and a line segment perpendicular to the XY plane and following the virtual cursor are created and displayed; the length of the line segment is adjusted in real time according to the depth information of the specified end point of the path. To enhance interaction feedback, the display module displays a virtual cursor and a dynamic line segment perpendicular to the XY plane in the XR glasses in real time, and the length of the dynamic line segment changes in real time according to the user's selected end point, thereby intuitively displaying the target height information.

[0039] Further, the specific process of the computer device includes: S1) Capture the pixel coordinates and depth information of the end point of the path specified by the virtual cursor; and project it back to the camera coordinate system, and then convert it to the world coordinate system; S2) Obtain the current position of the robot, which is the start point of the path in the world coordinate system; S3) Generate the path trajectory using the PRM (Probabilistic Roadmap) algorithm based on the coordinate information of the start point and the end point of the path.

[0040] Specifically, the computer device captures the end point of the path specified by the controller, and is responsible for path point resolution, trajectory generation, dynamic data update, etc. In path point resolution and trajectory generation, a random sampling path point network topology graph (i.e. road network) covering the machine workspace can be constructed using the PRM (Probabilistic Roadmap) algorithm, and a feasible path from the path start point to the path end point can be found on the network. Ensure that the movement is reachable, safe, and has a relatively optimal movement cost. PRM algorithm uses random sampling, which does not rely on uniform discretization of space grid, so it can better handle high-dimensional state space. For robots with complex kinematic constraints and high-dimensional degrees of freedom, PRM can generate a wider coverage of state samples, improving the planning success rate. Since PRM relies on random sampling and graph search algorithm, the computational complexity is generally controllable, and the algorithm itself has a highly modular design. This method uses random sampling to build a global view, and optimizes through local connectivity detection, which can automatically find a feasible path in a high-dimensional and complex environment, effectively balancing the relationship between global planning and local optimization, and significantly improving the success rate and computational efficiency of path planning.

[0041] Further, the specific process of the computer device further comprises: S4) Store the coordinate information of each point on the path trajectory in a path point manager and dynamically update it. The path point manager serves as the basis for data update and is used to maintain the continuity of path history and current execution trajectory, and is a bridge between the interaction process and the control system.

[0042] Further, the specific implementation process of step S1 comprises: S11) Define a world coordinate system as a three-dimensional reference system for the real world; S12) Calculate the conversion matrix between the world coordinate system and the camera coordinate system, which is expressed as: ; In the formula, represents the camera coordinate system; represents the world coordinate system; R is a 3x3 rotation matrix, representing the rotation of the camera coordinate system relative to the world coordinate system; T is a 3x1 translation vector, representing the displacement of the camera coordinate system origin relative to the world coordinate system; 0 and 1 are used to represent the extension of the homogeneous coordinates; Specifically, when calculating the conversion matrix between the world coordinate system and the camera coordinate system, the coordinate transformation process is usually involved, The rotation matrix R is calculated from the camera's attitude angle (yaw angle, pitch angle, roll angle) and is defined using Euler's formula.

[0043] The translation vector is determined by the position of the camera. Through the conversion matrix, any camera coordinate system point Convert to world coordinate system point , so as to be able to associate the real world and the camera perspective.

[0044] S13) The captured pixel coordinates are back projected to the camera coordinate system and then converted to the world coordinate system.

[0045] Specifically, the two-dimensional pixel coordinates The world coordinate point in three-dimensional space is back calculated , combined with depth information Zc and extrinsic matrix (rotation matrix R and translation vector T). In this way, the conversion from two-dimensional image to three-dimensional world coordinate can be realized by back projection and space conversion matrix according to the captured pixel coordinates. Using the pixel coordinates obtained by the virtual cursor, it is converted back to the camera coordinate system by back projection technology, and further converted to the world coordinate system, which fundamentally solves the data matching problem between virtual interaction data and real working space. This process ensures that the target position specified by the user in the XR environment can be accurately reflected in the real working space, greatly improving the accuracy of the interaction operation and the overall coordination of the system.

[0046] Further, in step S3), the specific process of the PRM algorithm to generate path trajectory includes: S31) Open motion planning library construction: generate sample path points in the open motion planning library that meet the current machine's workspace constraints and path constraint conditions; S32) Random sampling and neighborhood connection: randomly sample a group of sample path points, and check whether there is a collision or a violation of constraints on the straight line interpolation path (or more complex interpolation method); S33) Connectivity detection: if the connection within the boundary has no collision, add the corresponding edge to the road network; this process constitutes the entire probability road network structure; S34) Road network integration: add the coordinate information of two points in the path point manager to the probability road network structure, and try to establish a connection with its adjacent sample path points; S35) Path optimization through graph search: use a graph search algorithm (such as Dijkstra's algorithm) to find a shortest path from the path start point to the path end point. This path is a sequence of discrete sample states, representing the basic motion path of the machine.

[0047] Specifically, PRM has been widely implemented in many motion planning libraries (e.g. OMPL) and can be seamlessly integrated with other algorithmic components such as robot control systems, collision detection modules, path smoothing algorithms, etc. This openness and modularity feature makes PRM customizable for different application scenarios. The present application builds an open motion planning library for a given scenario; randomly generates a large number of discrete path points that satisfy the non-collision constraints. The sampling strategy can be purely random, goal-biased or heuristic sampling, thereby ensuring sufficient coverage of path points. Then, collision detection and constraint verification are performed on the candidate connected paths to ensure that the connected paths are locally feasible (e.g. without obstacle interference). If the detection passes, an edge is added to the probabilistic road network graph, indicating that the two states are reachable. After a large number of state sampling and neighborhood connectivity detection, the entire discrete nodes in the workspace and the edges between them form an undirected graph, which is the probabilistic road network. Finally, shortest path search algorithms such as Dijkstra, A* are used to find a feasible path sequence from the start point to the end point in the generated undirected graph. In addition, after finding a discrete path, it may also be necessary to smooth or re-parameterize the path to ensure that the path has continuity and dynamic feasibility when actually executed.

[0048] The present application also provides a numerical-real fusion interaction method based on XR technology, applied to the above platform, comprising the following steps: Data acquisition and modeling: video frames are acquired by a camera, depth information is calculated using multi-view stereo technology and 3D point cloud is generated, a camera coordinate system is established, and an XR environment is constructed by perspective projection technology.

[0049] Interaction and display: using a head-mounted display and a mouse, etc. devices, the 3D point cloud environment and virtual cursor are displayed in real time in the XR environment, and the user specifies the path end point through the interaction of the virtual cursor.

[0050] Path planning: the pixel coordinates of the user-specified path end point are captured, the captured pixel coordinates are back-projected to the camera coordinate system and then converted to the world coordinate system; the current position of the machine, i.e. the path start point, is obtained; and the PRM algorithm is used to randomly sample and construct a probabilistic road network in the workspace to find and generate an optimal path trajectory.

[0051] Trajectory management and feedback: the interaction and planning results are fed back in real time through the XR environment, and the coordinate information of the path trajectory is dynamically updated by a path point manager to realize numerical-real fusion interaction control.

[0052] In summary, the application uses a camera to capture continuous video frames and calculates depth information in combination with multi-view stereo technology to generate 3D point cloud data. According to the camera intrinsic parameter, the camera coordinate system is constructed, and the 3D point cloud is mapped to two-dimensional pixel coordinates using perspective projection technology, realizing the consistency and accuracy of coordinate conversion. This provides a standard mathematical basis for virtual-real information conversion, ensuring that subsequent interaction and planning can be based on accurate data. In the interaction, the pixel coordinates obtained by the user through the virtual cursor are converted to the camera coordinate system through back projection and then to the world coordinate system, effectively solving the conversion problem between virtual interaction data and real working space. In path planning, the PRM algorithm in the open motion planning library is used to randomly sample and construct a probability road network in the robot workspace, and a graph search algorithm is used to find the optimal path trajectory. This method can automatically find feasible paths in high-dimensional and complex environments, and through sampling, it takes into account global planning and local optimization, improving the success rate and efficiency of path planning. The logic of each link is rigorous and mutually coordinated, enabling users to intuitively plan paths in the XR environment, while the PRM algorithm generates an optimized path, effectively improving the accuracy, real-time performance and adaptability of the system, and has significant application value in the fields of intelligent unmanned aerial vehicles, embodied machines and quadruped robots.

[0053] In the description of the specification, the description of the terms "specifically", "in the specific implementation process", "further" and the like means that the specific features, structures, materials or characteristics described in conjunction with the embodiments or examples are included in at least one embodiment or example of the application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. The above is only an example and description of the structure of the present application, and those skilled in the art can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, as long as they do not deviate from the structure of the application or exceed the scope defined by the present claims.

Claims

1. An XR technology-based digital-real fusion interaction platform, characterized in that: The computer device comprises a data acquisition module, a modeling module, a controller, a display module and a computer device. The data acquisition module comprises a camera for acquiring video frames of the real world. The modeling module is used for calculating depth information of object objects by multi-view stereo technology and generating 3D point clouds; a camera coordinate system is established, and a 3D point cloud environment is constructed by using perspective projection technology. The controller is used for interactive operation in the 3D point cloud environment and specifying an end point of a path. The display module is used for displaying the 3D point cloud environment based on perspective projection and real-time interactive display. The computer device is used for generating a path trajectory by using a PRM algorithm according to coordinate information of a start point and an end point of the path of the machine.

2. The XR technology-based digital-real fusion interaction platform according to claim 1, characterized in that: The implementation process of the computer device comprises: S1) capturing pixel coordinates and depth information of the end point of the path specified by the virtual cursor, and projecting the pixel coordinates and the depth information into the camera coordinate system and then into the world coordinate system; S2) obtaining the current position of the machine, i.e. the start point of the path in the world coordinate system; S3) generating a path trajectory by using a PRM algorithm according to coordinate information of the start point and the end point of the path.

3. The number-real fusion interaction platform based on XR technology according to claim 2, characterized in that: The specific process of the computer device further comprises: S4) storing coordinate information of each point on the path trajectory in a path point manager and dynamically updating the coordinate information.

4. The number-real fusion interaction platform based on XR technology according to claim 1, characterized in that: The specific process of the modeling module comprises: calculating depth information between object objects and the origin of the camera from continuous video frames of the camera by using multi-view stereo technology, and generating 3D point clouds; establishing a camera coordinate system with the camera as the origin, wherein the Z axis points to the front of the camera, and the X and Y axes represent the left and vertical directions respectively; an intrinsic matrix of the camera is established by using internal parameters of the camera; the expression is: ； wherein: and are the effective focal lengths in horizontal and vertical directions; and are the image principal point coordinates; 0 and 1 indicate the extension of the homogeneous coordinates. projecting the 3D point clouds in the camera coordinate system onto the XY plane of the camera to obtain pixel coordinates; the calculation formula is: ； ； ； where: is the normalized coordinate of the XY plane of the camera; is the pixel coordinate; K is the intrinsic matrix of the camera; perspective projecting the obtained 3D point clouds on the XY plane of the camera coordinate system to calculate pixel coordinates of each point on the XY plane; determining the coordinates of each point on the Z axis according to the depth information of the 3D point clouds; constructing a 3D point cloud environment based on perspective projection according to the three-dimensional coordinates of the 3D point clouds in the camera coordinate system, i.e. extended reality.

5. The number-reality fusion interaction platform based on XR technology according to claim 1, characterized in that: The controller comprises a mouse, and the user controls the virtual cursor to specify the end point of the path in the 3D point cloud environment by using the mouse function.

6. The number-reality fusion interaction platform based on XR technology according to claim 1, characterized in that: The specific way of real-time interactive display in the display module is: creating and displaying a virtual cursor and a line segment perpendicular to the XY plane and following the virtual cursor; the length of the line segment is adjusted in real time according to the depth information of the specified end point of the path.

7. The number-reality fusion interaction platform based on XR technology according to claim 2, characterized in that: The specific steps of step S1 comprise: S11) defining a world coordinate system as a three-dimensional reference system of the real world; S12) calculating a conversion matrix between the world coordinate system and the camera coordinate system, the expression is: ； wherein represents a camera coordinate system; represents a world coordinate system; R is a 3 x 3 rotation matrix representing a rotation of the camera coordinate system with respect to the world coordinate system; T is a 3 x 1 translation vector representing a displacement of the camera coordinate system origin with respect to the world coordinate system; 0 and 1 are used to represent an extension of the homogeneous coordinates; S13) projecting the captured pixel coordinates to the camera coordinate system and then to the world coordinate system.

8. The number-real fusion interaction platform based on XR technology according to claim 1, characterized in that: In step S3), the specific steps of generating a path trajectory by using a PRM algorithm comprise: S31) generating sample path points in an open motion planning library that meet the workspace constraints and path constraints of the current machine; S32) Randomly sample a set of sample path points and check whether there is a collision or constraint violation on the straight-line interpolated path; S33) If there is no collision in the connection within the boundary, add the corresponding edge in the road network, and then constitute the entire probability road network structure; S34) Add the coordinate information of two points in the path point manager to the probability road network structure, and try to establish a connection with its adjacent sample path points; S35) Use a graph search algorithm to find a shortest path from the path start point to the path end point.

9. A digital-real fusion interaction method based on XR technology, applied to the platform of any one of claims 1-8, characterized in that, Comprising the following steps: Capture video frames through the camera, calculate depth information and generate 3D point cloud using multi-view stereo technology, establish the camera coordinate system, and then construct the XR environment through perspective projection technology; Use interactive devices to display the 3D point cloud environment and the XR environment of the virtual cursor in real time, and the user specifies the path end point through the interaction of the virtual cursor; Capture the pixel coordinates of the path end point specified by the user, back project the captured pixel coordinates to the camera coordinate system, and then convert them to the world coordinate system; and obtain the current position of the machine, i.e. the path start point; Use the PRM algorithm to randomly sample and construct a probability road network in the workspace, find and generate an optimal path trajectory.

10. The XR technology-based digital-real fusion interaction method of claim 9, wherein, Also includes: Real-time feedback of interaction and planning results through the XR environment, and dynamically update the coordinate information of the path trajectory from a path point manager, realizing the interactive control of the combination of virtual and real.

Images