Inspection robot collaborative planning method and system based on digital twinning

Abstract

The application discloses a kind of based on digital twinning's inspection robot collaborative planning method and system, method is constructed by point cloud modeling technology virtual simulation environment of large component, and integrated motion capture system realizes real-time mapping of physical environment and virtual model;Definition scanning robot and the collaborative mechanism of positioning robot, global coordinates and path guide are provided by positioning robot, and motion capture system real-time correction position error;Simulate double-machine motion trajectory in digital twinning model, and identify potential collision risk by conflict prediction model;Design collaborative planning algorithm under coupling constraint, and path optimization and scanning view coverage are optimized as joint target;Based on embodied twin intelligence framework, robot path and view are dynamically adjusted using reinforcement learning combined with real-time feedback, adaptive collaborative planning is realized. Through double-machine division of labor and real-time correction, the efficiency and precision of scanning large-size component can be significantly improved, and the collision risk is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing, and particularly relates to a method and system for collaborative planning of inspection robots based on digital twinning. BACKGROUND

[0002] In the field of large component inspection, existing technologies mainly rely on traditional path planning algorithms and preliminary digital twinning applications, but still face many challenges. Traditional methods such as A Star or genetic algorithms can achieve basic path optimization in static environments, but they are difficult to cope with the complex dynamic characteristics of large components, resulting in non-smooth paths, low efficiency and increased energy consumption. Especially when dealing with the coupling constraints of multi-machine motion planning and scanning angle planning, the length of the path is ignored, which limits the coverage of the scanning angle, resulting in insufficient scanning accuracy and missing hidden areas. Although digital twinning technology has been used to simulate physical entities by synchronizing the state of the virtual model with the robot to optimize tasks, it has not been fully extended to the multi-machine collaborative scenario of large component inspection, and lacks a real-time feedback mechanism to handle environmental changes. In addition, embodied intelligence as a new paradigm has been explored in robot systems, such as frameworks based on visual language models that improve perception and decision-making, but these methods are mostly limited to single-machine tasks and have not been deeply integrated with digital twinning, making it difficult to effectively solve the dual-machine division (scanning and positioning) and coupling constraint problems in large component inspection.

[0003] Overall, existing technologies lack adaptability, real-time performance and collaboration efficiency, leading to increased resource waste and conflict risk, and there is an urgent need for new methods to achieve efficient and high-precision scanning. SUMMARY

[0004] To solve the above technical problems, the application provides a method and system for collaborative planning of inspection robots based on digital twinning.

[0005] The technical solution adopted by the application to solve its technical problems is:

[0006] A method for collaborative planning of inspection robots based on digital twinning, the method comprising the following steps:

[0007] S100: Construct a digital twin model of the inspection environment, reconstruct the geometry of the large component using point cloud modeling technology to form a virtual simulation environment, and integrate a motion capture system to obtain real-time spatial position information of the large component, scanning robot and positioning robot, achieving real-time mapping and interaction between the physical inspection environment and the virtual model;

[0008] S200: Define a dual-machine coordination mechanism, in which the scanning robot is responsible for collecting component surface data, the positioning robot provides global coordinate reference and path guidance for the scanning robot, real-time correction of position errors between the scanning robot, the positioning robot and the large component is realized through the motion capture system, and the state of each robot is jointly modeled through the digital twin model;

[0009] S300: In the digital twin model, the motion trajectory of the robot is simulated based on the state information of the robot, and the potential collision risk between the scanning robot and the positioning robot and between the robot and the large component is predicted based on a conflict prediction model;

[0010] S400: Design a collaborative planning algorithm under coupling constraints, build a coupling model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on a coupling loss function;

[0011] S500: Realize dynamic interaction learning through embodied twin intelligent driving, in the digital twin virtual and real environment, based on the coupling model and real-time feedback of the motion capture system, dynamically adjust the path of the scanning robot and the positioning robot and the scanning view through reinforcement learning, and realize collaborative planning.

[0012] Preferably, S100 includes:

[0013] S110: Scan the surface of the large component through the laser radar or structured light scanner equipped on the inspection robot or auxiliary equipment, generate high-density point cloud data, and use point cloud modeling technology for geometric reconstruction; wherein, after generating point cloud data by the scanning device, first use voxel filtering for down-sampling to reduce redundancy, then use the iterative closest point algorithm for point cloud registration, through iterative optimization, solve the optimal transformation parameter, realize real-time fusion of multiple frames of point cloud, and form a complete three-dimensional model of the component;

[0014] S120: Integrate a motion capture system to obtain the relative position information of the large component and the robot by tracking the reflective markers attached to the key points of the component and the robot;

[0015] S130: Realize synchronization of scanning sensor data and motion capture position information through timestamp alignment and data fusion algorithm, and ensure real-time mapping of physical environment data to virtual environment;

[0016] S140: Build an interactive virtual simulation environment based on a robot operating system, define and simulate the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realize interactive perception between the robot and the digital twin model through an API interface.

[0017] Preferably, S120 includes:

[0018] S121: Detect the two-dimensional image coordinates of the markers attached to the large component and the robot through the multiple cameras of the motion capture system, denoted as ;

[0019] S122: For each marker, reconstruct the three-dimensional position based on its image coordinates in at least two cameras using a direct linear transformation algorithm, specifically:

[0020] the projection matrix of the camera , the 3D point X satisfies , by constructing the equation group of the coefficient matrix A , and solving its singular value decomposition to obtain X, where is the intrinsic matrix of the camera, , the rotation matrix and translation vector of the camera, respectively;

[0021] S123: Based on the calculated three-dimensional positions of multiple markers, calculate the six-degree-of-freedom pose of the target through a rigid body pose estimation algorithm, specifically calculate the rotation matrix and translation vector by solving the following optimization problem:

[0022] ;

[0023] where, and are the rigid body rotation matrix and translation vector to be solved, is the three-dimensional coordinate of the i-th marker point measured by the camera under the current pose, is the three-dimensional coordinate of the corresponding i-th marker point in the model coordinate system, is the number of marker points used for pose estimation.

[0024] Preferably, S200 includes:

[0025] S210: Establish a cooperative division mechanism between the scanning robot and the positioning robot, where the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot;

[0026] S220: Real-time track the markers on the scanning robot, the positioning robot, and the large component through the motion capture system, and correct the position error among the three using an optimization algorithm;

[0027] S230: Model the state of each robot as a multi-dimensional vector containing position, velocity, pose, energy level, and motion capture correction error, and construct a joint state of the two robots, fuse multi-modal data, and generate a language description of the robot state in combination with a visual language model.

[0028] Preferably, the optimization algorithm in S220 is a least squares optimization algorithm based on singular value decomposition, which is used to calculate the transformation parameters to minimize the error between the motion capture observed marker positions and the reference points, specifically:

[0029] ;

[0030] wherein is the n-th marker position observed by motion capture, is the corresponding n-th reference point position on the component or robot, N is the number of marker points used for coordinate system alignment, , is the transformation parameter, i.e. the rotation matrix and translation vector of coordinate system alignment.

[0031] Preferably, the dual-machine cooperation mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the positioning robot through a value function, specifically:

[0032] ;

[0033] wherein, is the division of labor decision made at time t, is the motion capture error term, is the assignable role variable, which limits the division of labor range of the dual-machine system, denotes the scanning robot, denotes the positioning robot, denotes the value function, which is used to evaluate the expected return of the current state under the selected role , is the learning parameter.

[0034] Preferably, the conflict prediction model in S300 adopts the Monte Carlo tree search algorithm, which calculates the path collision probability through multiple simulations; in each simulation, starting from the current state, the future possible trajectory is generated through the steps of selection, expansion, simulation and backtracking, and the number of times the distance between the scanning robot and the positioning robot trajectory is below the safety distance threshold is counted; the collision probability calculation formula is specifically:

[0035] ;

[0036] wherein M is the total number of simulations, is the indicator function, is the safety distance threshold, is the collision probability, and are the predicted positions of the scanning robot and the positioning robot in the m-th simulation, respectively;

[0037] The embodiment of the present application guides the embodiment of the present application through the collision probability to optimize the strategy, and dynamically adjusts the path of the robot to avoid collision.

[0038] Preferably, in S400, the coupling model is a graph network structure, which is a coupling constraint graph, the nodes of which represent the state information of the robot, and the edges of which encode the motion path The coupling constraint relationship between the scanning angle coverage rate , wherein is the view angle visibility function, is the scanning angle, E is the set of edges in the coupling constraint graph, is the length of the edge ; and global optimization is performed by minimizing the coupling loss function, and the coupling loss function is specifically:

[0039] ;

[0040] , wherein are the weight coefficients of the path length term, the view angle coverage rate term and the collision penalty term, respectively, is the path length, is the view angle coverage rate, is the collision penalty, is the indicator function, and represent the coordinates of the i th and j th robot in the three-dimensional space, is the safety distance threshold.

[0041] Preferably, S500 is specifically:

[0042] The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint, the scanning robot and the positioning robot are used as agents, the virtual agent first simulates planning in the digital twin model to generate a candidate path, the physical agent executes the candidate path and feeds back the error, thereby updating the strategy network, and the path and the view angle are dynamically adjusted based on the reinforcement learning and the motion capture feedback interaction; wherein, the learning process uses variational inference to optimize the strategy, and is specifically:

[0043] ;

[0044] , wherein the reward function is defined as:

[0045] ;

[0046] , wherein represents the optimal strategy, i.e. the probability distribution of selecting action under state , and represents the error of the policy Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

[0047] A collaborative planning system for inspection robots based on digital twins, comprising:

[0048] The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model.

[0049] The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model;

[0050] The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model.

[0051] A cooperative planning algorithm design module is configured to design a cooperative planning algorithm under coupling constraints, construct a coupling model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on a coupling loss function;

[0052] A dynamic interactive learning module is configured to realize dynamic interactive learning through embodied twin intelligent driving, dynamically adjust the path and scanning view of the scanning robot and the positioning robot based on the real-time feedback of the coupling model and the motion capture system in the digital twin virtual-real environment through reinforcement learning, and realize cooperative planning.

[0053] The above-mentioned cooperative planning method and system for the inspection robot based on digital twinning realize efficient and high-precision scanning of large-size components through double-machine division (scanning and positioning) and real-time error correction of motion capture. Compared with traditional technologies, the scanning efficiency, accuracy and conflict rate can be improved. The embodied twin intelligent framework supports virtual-real interactive adaptive learning, ensures the generalization ability in new environments, reduces resource waste, is suitable for large and complex scenes such as wind blades, bridges, airplanes and high-speed rails, significantly reduces the inspection cost and risk, and promotes industrial intelligent maintenance. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 The flowchart of the cooperative planning method for the inspection robot based on digital twinning in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to enable personnel in the technical field to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings.

[0056] In one embodiment, as shown in Figure 1 A cooperative planning method for an inspection robot based on digital twinning, the method comprising the following steps:

[0057] S100: Construct a digital twin model of the inspection environment, reconstruct the geometry of the large component through point cloud modeling technology to form a virtual simulation environment, and integrate a motion capture system to obtain the spatial position information of the large component, the scanning robot and the positioning robot in real time, realize real-time mapping and interaction between the physical inspection environment and the virtual model;

[0058] S200: Define a double-machine cooperative mechanism, wherein the scanning robot is responsible for collecting component surface data, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the position error between the scanning robot, the positioning robot and the large component is corrected in real time through the motion capture system, and the state of each robot is jointly modeled through the digital twin model;

[0059] S300: In the digital twin model, the robot's motion trajectory is simulated based on its state information, and the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, are predicted based on the conflict prediction model.

[0060] S400: Design a collaborative planning algorithm under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function;

[0061] S500: Through embodied twin intelligence, dynamic interactive learning is achieved. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, the path and scanning perspective of the scanning robot and the positioning robot are dynamically adjusted through reinforcement learning to achieve collaborative planning.

[0062] The aforementioned collaborative planning method for inspection robots based on digital twins constructs a digital twin model of the inspection environment. This model uses point cloud modeling and a motion capture system to map the positions of large components, the scanning robot, and the positioning robot in real time. Then, state modeling is incorporated into this model, resulting in a dual-machine division of labor: the scanning robot collects data, the positioning robot provides guidance, and the motion capture system corrects errors and performs adaptive optimization. Subsequently, a coupled model is constructed, a coupling loss function is defined, and embodied twin intelligent interactive learning is embedded to achieve dynamic path-viewpoint adjustment, avoid collisions, improve efficiency, and ensure high-efficiency and high-precision inspection.

[0063] Furthermore, a digital twin model of the inspection environment is constructed. Through high-fidelity virtual simulation, the physical inspection scene is mirrored in real time, ensuring the model's real-time performance and interactivity. This provides a reliable simulation environment for subsequent dual-robot collaborative planning, enabling efficient collaboration among multiple inspection robots. For large components, this invention employs point cloud modeling technology for geometric reconstruction, while utilizing a motion capture system to acquire the positional information of the large components and the robot, improving mapping accuracy and real-time performance.

[0064] In one embodiment, S100 includes:

[0065] S110: The surface of large components is scanned by a LiDAR or structured light scanner equipped on an inspection robot or auxiliary equipment to generate high-density point cloud data, and geometric reconstruction is performed using point cloud modeling technology. Specifically, after generating point cloud data using the scanning equipment, voxel filtering is first used for downsampling to reduce redundancy, and then the iterative nearest point algorithm is used for point cloud registration. Through iterative optimization, the optimal transformation parameters are solved to achieve real-time fusion of multiple frames of point clouds to form a complete three-dimensional model of the component.

[0066] Specifically, point cloud data can be represented as a three-dimensional coordinate set. N represents the number of points, and each point contains its location coordinates and optional color or intensity information. To handle the scale of large components, voxel filtering is used to downsample the point cloud, reducing data redundancy while preserving key features. This method effectively reduces point cloud density while maintaining the surface details of the components. Iterative nearest-point algorithm is used for registration and fusion, achieving high-precision mapping.

[0067] S120: An integrated motion capture system that obtains the relative position information of large components and the robot by tracking reflective markers attached to key construction points and the robot.

[0068] Specifically, in order to obtain the relative position information of large components and robots, the present invention integrates a motion capture system, which uses an infrared camera to track reflective marks attached to key points of the components and the robot, providing sub-millimeter-level position data.

[0069] In one embodiment, S120 includes:

[0070] S121: The two-dimensional image coordinates of the markers attached to the large components and the robot are detected by multiple cameras of the motion capture system, denoted as... ;

[0071] S122: For each marker, based on its image coordinates in at least two cameras, reconstruct its 3D position using a direct linear transformation algorithm, specifically:

[0072] camera projection matrix 3D point X satisfies By constructing a system of equations with coefficient matrix A And solve its singular value decomposition to obtain X, where, This is the intrinsic parameter matrix of the camera. , These are the rotation matrix and translation vector of the camera, respectively;

[0073] Furthermore, for the two cameras, assume that the camera projection matrices are respectively and 3D points Satisfies the projection equation:

[0074] ;

[0075] Solving least squares problems using direct linear transformation (DLT):

[0076] ;

[0077] The matrix A is constructed by the projection equation, and the singular value decomposition (SVD) of X is solved. The vector corresponding to the smallest singular value is taken as the solution.

[0078] S123: Based on the calculated 3D positions of multiple markers, the six-degree-of-freedom attitude of the target is calculated using a rigid body attitude estimation algorithm. Specifically, the rotation matrix is ​​calculated by solving the following optimization problem. Translation vector :

[0079] ;

[0080] in, and Let be the rigid body rotation matrix and translation vector to be determined, respectively. Given the current pose, let's consider the 3D coordinates of the i-th marker point as measured by the camera. These are the three-dimensional coordinates of the i-th marker point in the model coordinate system. The number of markers used for attitude estimation.

[0081] Specifically, the centroids of the observation point set and the reference point set are first calculated. And centralize the point set. Calculate the covariance matrix of the centered point set; perform singular value decomposition on the matrix. Calculate the rotation matrix based on the decomposition results. ; Calculate the translation vector based on the selected matrix and centroid. This process ensures that the attitude information is represented in six degrees of freedom and integrated into the point cloud model in real time, avoiding sensor drift issues. The motion capture system can achieve a frame rate of up to 120Hz, ensuring dynamic tracking accuracy.

[0082] S130: Through timestamp alignment and data fusion algorithms, the synchronization of scanning sensor data and motion capture position information is achieved, ensuring that physical environment data is mapped to the virtual environment in real time;

[0083] Specifically, this involves synchronizing sensor data to ensure that data from the physical world is reflected in the virtual environment in real time. The scanning sensor information of the inspection robot is combined with motion capture location information, using a timestamp alignment mechanism to avoid data delays. Assume the sensor data stream is sequential. Where t is the timestamp and m is the number of sensors, including motion capture attitude data. An Extended Kalman Filter (EKF) is used to fuse multi-source data to predict and correct the state. The EKF state update formula is:

[0084] ;

[0085] in For state estimation, K is the Kalman gain, z is the measured value (including motion capture position), h is the nonlinear observation function, H is its Jacobian matrix, and P is the covariance matrix.

[0086] S140: Based on the robot operating system, an interactive virtual simulation environment is built, defining and simulating the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realizing the interactive perception between the robot and the digital twin model through the API interface.

[0087] Specifically, an interactive simulation space is created based on ROS, supporting virtual testing of robots and their environment. For dual-robot collaboration, the environment defines the robot's role: the scanning robot is responsible for high-precision data acquisition, and the positioning robot provides coordinate references. The interaction mechanism allows the robot to interact with the model via API interfaces; for example, during simulated path planning, the robot can "sense" virtual obstacles.

[0088] In one embodiment, S200 includes:

[0089] S210: Establish a collaborative division of labor mechanism between the scanning robot and the positioning robot, wherein the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot;

[0090] S220: The motion capture system tracks the markings on the scanning robot, the positioning robot, and large components in real time, and uses an optimization algorithm to correct the positional errors between the three.

[0091] S230: Perform state modeling for each robot, representing its state as a multi-dimensional vector containing position, velocity, attitude, energy level, and motion capture correction error, and construct a joint state of the two robots, integrate multimodal data, and combine a visual language model to generate a linguistic description of the robot's state.

[0092] Specifically, the S200 implements robot role division and state modeling. A dual-machine collaborative mechanism is defined, where the scanning robot is responsible for high-precision scanning, equipped with a high-resolution camera and laser sensor to acquire image / point cloud data, achieving sub-millimeter-level detection of component surfaces. The localization robot is responsible for the scanning robot's localization, providing reference coordinates and path guidance, and using its sensors to generate navigation anchor points. This division of labor is based on task coupling: the localization robot's positioning information is transmitted to the scanning robot in real time, guiding its scanning path adjustment to ensure maximum coverage. The motion capture system is responsible for real-time correction of positional errors among the scanning robot, localization robot, and component.

[0093] In one embodiment, the optimization algorithm in S220 is a least-squares optimization algorithm based on singular value decomposition, used to calculate the transformation parameters to minimize the error between the position of the motion capture observation marker and the reference point, specifically:

[0094] ;

[0095] in This represents the position of the nth marker point observed by motion capture. This represents the position of the nth reference point on the component or robot, where N is the number of marker points used for coordinate system alignment. , The transformation parameters are the rotation matrix and translation vector for coordinate system alignment. This optimization is solved using SVD, ensuring an error of less than 0.01m and improving collaborative robustness.

[0096] In one embodiment, the dual-machine collaboration mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the localization robot through a value function, specifically as follows:

[0097] ;

[0098] in, The division of labor decision made at time t. For motion capture error term, It is an assignable role variable that limits the scope of division of labor in the dual-machine system. Indicates a scanning robot. Indicates the location of the robot. This represents the value function, used to evaluate the current state. Choose a character Expected returns To learn the parameters. To ensure that the efficiency of task division is improved by more than 25%.

[0099] Further state modeling of the robots is performed, with each robot's state represented as a multi-dimensional vector. ,in For location, For speed, As a gesture, For energy levels, To correct errors in motion capture. Dual-machine combined state is... It integrates virtual mirrors and component states into a digital twin model. In addition, it innovatively adopts multimodal fusion modeling, combines visual language models to process image data, and uses motion capture data to realize semantic state descriptions, such as "the scanning robot is approaching the scanning area, and the error has been corrected".

[0100] In a virtual environment, the intersection of two robot trajectories and scanning interference are simulated, and motion capture is incorporated to predict three-way collisions. In one embodiment, the collision prediction model in S300 uses a Monte Carlo tree search algorithm to calculate the path collision probability through multiple simulations. In each simulation, starting from the current state, possible future trajectories are generated through selection, expansion, simulation, and backtracking steps, and the number of times the distance between the scanning robot and the localization robot's trajectories is lower than the safe distance threshold is counted. The specific formula for calculating the collision probability is as follows:

[0101] ;

[0102] Where M represents the total number of simulations. For indicator functions, For safe distance threshold, Let be the collision probability. and These are the predicted positions of the scanning robot and the localization robot in the m-th simulation, respectively;

[0103] By using collision probability to guide the embodied twin intelligent interactive learning mechanism to optimize strategies, the robot path is dynamically adjusted to avoid collisions.

[0104] Next, we will design a collaborative planning algorithm under coupled constraints. We will design a coupled model of multi-machine motion planning (path optimization) and scanning view planning (view coverage and accuracy constraints), and combine it with interactive learning of the digital twin virtual and real environments to achieve dynamic adjustment of path and view, avoid collisions and maximize scanning efficiency.

[0105] In one embodiment, S400, the coupling model is a graph network structure, specifically a coupling constraint graph, where nodes represent the robot's state information and edges encode motion paths. With scanning field coverage The coupling constraint relationship between them, where For visually visible functions, Let E be the scanning angle, and E be the set of edges in the coupling constraint graph. For the edge The length of the coupling loss function is determined, and global optimization is performed by minimizing the coupling loss function, which is specifically:

[0106] ;

[0107] in These are the weight coefficients for the path length term, view coverage term, and penalty collision term, respectively. For path length, For view coverage, To punish collisions, For indicator functions, and Let i and j represent the coordinates of the i-th and j-th robots in three-dimensional space, respectively. This represents the safe distance threshold. The model achieves global optimization by propagating constraints through a graph neural network.

[0108] Finally, an interactive learning model driven by embodied twin intelligence is employed. In the digital twin virtual-real environment, the scanning robot and the localization robot act as agents, interacting through reinforcement learning and motion capture feedback: the virtual agent first simulates planning in the twin model, generating candidate paths; the physical agent executes and reports errors, updating the policy network. The learning utilizes variational inference to optimize the policy.

[0109] In one embodiment, S500 specifically refers to:

[0110] The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint. The scanning robot and the localization robot act as agents. The virtual agent first simulates planning in the digital twin model to generate candidate paths. The physical agent executes the candidate paths and provides feedback on the errors, thereby updating the policy network. Based on the interaction between reinforcement learning and motion capture feedback, the path and perspective are dynamically adjusted. The learning process uses a variational inference optimization strategy, specifically:

[0111] ;

[0112] Among them, the reward function Defined as:

[0113] ;

[0114] in, This represents the optimal policy, i.e., in state . Select action The probability distribution, Indicates the policy of following Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

[0115] This mechanism allows for dynamic adjustments: if the motion capture detection error does not converge, the positioning robot will replan the scanning robot's path in real time to avoid blind spots.

[0116] The coupling loss function generated by the above coupling model The reward function is directly embedded into the embodied twin intelligence as a prior constraint to guide interactive learning. Node states and edge constraints (such as path length L and view coverage C) of the coupled graph network (CCG) are injected into the input layer of the policy network, ensuring that the learning process starts from the model's global optimization objective and avoids local biases caused by decoupling. In the digital twin environment, the virtual agents of the scanning robot (Robot_B) and the localization robot (Robot_P) first generate initial candidate paths based on the coupled model, and then iteratively update the policy through interactive learning. Among them, the reward R fusion Gradient feedback enables dynamic adjustment of the path and perspective. This bridging mechanism shifts the algorithm from static graph optimization to adaptive evolution: the collision penalty term of the coupled model calibrates the trajectory distribution of variational inference in real time, ensuring the robustness of the entire planning process and providing efficient and high-precision collaboration for the inspection of large components.

[0117] The aforementioned collaborative planning method and system for inspection robots based on digital twins has the following technical advantages:

[0118] Virtual-Real Interaction and Self-Evolution Capability: By constructing a high-fidelity digital twin model and integrating a motion capture system, real-time and accurate mapping and interaction between the physical environment and the virtual model are achieved. Building upon this, a reinforcement learning mechanism is introduced, and a loss function coupled with path length, viewpoint coverage, and collision risk is embedded as an optimization objective into the agent's policy. This drives the system to continuously self-evolve in virtual-real interaction, dynamically optimizing the scanning path and viewpoint, thereby significantly improving the adaptability and efficiency of scanning complex components.

[0119] High-precision collaboration and safety: Based on sub-millimeter-level position information provided by the motion capture system, a least-squares optimization algorithm with singular value decomposition as its core is used to calibrate the relative pose errors between the scanning robot, the positioning robot, and large components in real time. Combined with a conflict prediction model based on Monte Carlo tree search, potential collisions can be predicted and avoided in virtual space, significantly reducing the conflict risk of dual-robot collaborative operations and ensuring operational safety.

[0120] Adaptive Collaborative Division of Labor: An adaptive dual-machine collaborative mechanism based on a value function was designed. The localization robot provides global coordinate reference and path guidance, while the scanning robot focuses on high-precision data acquisition. The division of labor between the two can be dynamically adjusted according to real-time conditions such as motion capture error through a value function, breaking through the limitations of the traditional static collaboration mode. This enables efficient and blind-spot-free point cloud data acquisition on the surface of large components, with point cloud coverage quality and modeling accuracy significantly superior to traditional methods.

[0121] In one embodiment, a collaborative planning system for inspection robots based on digital twins is also provided, comprising:

[0122] The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model.

[0123] The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model;

[0124] The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model.

[0125] The collaborative planning algorithm design module is used to design collaborative planning algorithms under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function.

[0126] The dynamic interactive learning module is used to achieve dynamic interactive learning through embodied twin intelligence. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

[0127] Specific limitations regarding a collaborative planning system for inspection robots based on digital twins can be found in the limitations of a collaborative planning method for inspection robots based on digital twins mentioned above, and will not be repeated here. Each module in the aforementioned collaborative planning system for inspection robots based on digital twins can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0128] The above provides a detailed description of the collaborative planning method and system for inspection robots based on digital twins provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention, and the descriptions of the embodiments are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A collaborative planning method for inspection robots based on digital twins, characterized in that, The method includes the following steps: S100: Constructs a digital twin model of the inspection environment, uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. S200: Defines a dual-machine collaborative mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional errors between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; S300: In the digital twin model, motion trajectories are simulated based on the state information of the scanning robot and the positioning robot, and potential collision risks between the scanning robot and the positioning robot and between the robot and large components are predicted based on the conflict prediction model. S400: Design a collaborative planning algorithm under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function; S500: Through embodied twin intelligence, dynamic interactive learning is achieved. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, the path and scanning perspective of the scanning robot and the positioning robot are dynamically adjusted through reinforcement learning to achieve collaborative planning.

2. The method according to claim 1, characterized in that, S100 includes: S110: The surface of large components is scanned by a LiDAR or structured light scanner equipped on an inspection robot or auxiliary equipment to generate high-density point cloud data, and geometric reconstruction is performed using point cloud modeling technology. Specifically, after generating point cloud data using the scanning equipment, voxel filtering is first used for downsampling to reduce redundancy, and then the iterative nearest point algorithm is used for point cloud registration. Through iterative optimization, the optimal transformation parameters are solved to achieve real-time fusion of multiple frames of point clouds to form a complete three-dimensional model of the component. S120: An integrated motion capture system that obtains the relative position information of large components and the robot by tracking reflective markers attached to key construction points and the robot. S130: Through timestamp alignment and data fusion algorithms, the synchronization of scanning sensor data and motion capture position information is achieved, ensuring that physical environment data is mapped to the virtual environment in real time; S140: Based on the robot operating system, an interactive virtual simulation environment is built, defining and simulating the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realizing the interactive perception between the robot and the digital twin model through the API interface.

3. The method according to claim 2, characterized in that, S120 includes: S121: The two-dimensional image coordinates of the markers attached to the large components and the robot are detected by multiple cameras of the motion capture system, denoted as... ; S122: For each marker, its 3D position is reconstructed using a direct linear transformation algorithm based on its image coordinates in at least two cameras, specifically: camera projection matrix 3D point X satisfies By constructing a system of equations with coefficient matrix A And solve its singular value decomposition to obtain X, where, This is the intrinsic parameter matrix of the camera. , These are the rotation matrix and translation vector of the camera, respectively; S123: Based on the calculated 3D positions of multiple markers, the six-degree-of-freedom attitude of the target is calculated using a rigid body attitude estimation algorithm. Specifically, the rotation matrix is ​​calculated by solving the following optimization problem. Translation vector : ； in, and Let be the rigid body rotation matrix and translation vector to be determined, respectively. Given the current pose, let's consider the 3D coordinates of the i-th marker point as measured by the camera. These are the three-dimensional coordinates of the i-th marker point in the model coordinate system. The number of markers used for attitude estimation.

4. The method according to claim 3, characterized in that, S200 includes: S210: Establish a collaborative division of labor mechanism between the scanning robot and the positioning robot, wherein the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot; S220: The motion capture system tracks the markings on the scanning robot, the positioning robot, and large components in real time, and uses an optimization algorithm to correct the positional errors between the three. S230: Perform state modeling for each robot, representing its state as a multi-dimensional vector containing position, velocity, attitude, energy level, and motion capture correction error, and construct a joint state of the two robots, integrate multimodal data, and combine a visual language model to generate a linguistic description of the robot's state.

5. The method according to claim 4, characterized in that, The optimization algorithm in S220 is a least-squares optimization algorithm based on singular value decomposition, used to calculate transformation parameters to minimize the error between the position of the motion capture observation marker and the reference point. Specifically: ； in This represents the position of the nth marker point observed by motion capture. This represents the position of the nth reference point on the component or robot, where N is the number of marker points used for coordinate system alignment. , These are the transformation parameters, namely the rotation matrix and translation vector for coordinate system alignment.

6. The method according to claim 4, characterized in that, The dual-machine collaborative mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the localization robot through a value function, specifically: ； in, The division of labor decision made at time t. For motion capture error term, It is an assignable role variable that limits the scope of division of labor in the dual-machine system. Indicates a scanning robot. Indicates the location of the robot. This represents the value function, used to evaluate the current state. Choose a character Expected returns These are the learning parameters.

7. The method according to any one of claims 5 or 6, characterized in that, The S300 conflict prediction model uses the Monte Carlo tree search algorithm to calculate the path collision probability through multiple simulations. In each simulation, starting from the current state, possible future trajectories are generated through selection, expansion, simulation, and backtracking steps, and the number of times the distance between the trajectories of the scanning robot and the positioning robot is lower than the safe distance threshold is counted. The specific formula for calculating the collision probability is as follows: ； Where M represents the total number of simulations. For indicator functions, For safe distance threshold, Let be the collision probability. and These are the predicted positions of the scanning robot and the localization robot in the m-th simulation, respectively. To correct errors in motion capture; By using collision probability to guide the embodied twin intelligent interactive learning mechanism to optimize strategies, the robot path is dynamically adjusted to avoid collisions.

8. The method according to claim 7, characterized in that, In S400, the coupling model is a graph network structure, specifically a coupled constraint graph, where nodes represent the robot's state information and edges encode the motion path. With scanning field coverage The coupling constraint relationship between them, where For visually visible functions, Let E be the scanning angle, and E be the set of edges in the coupling constraint graph. For the edge The length of the coupling loss function is determined; and global optimization is performed by minimizing the coupling loss function. Specifically: ； in These are the weight coefficients for the path length term, view coverage term, and penalty collision term, respectively. For path length, For view coverage, To punish collisions, For indicator functions, and Let i and j represent the coordinates of the i-th and j-th robots in three-dimensional space, respectively. This is the safe distance threshold.

9. The method according to claim 8, characterized in that, The S500 specifically refers to: The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint. The scanning robot and the localization robot act as agents. The virtual agent first simulates planning in the digital twin model to generate candidate paths. The physical agent executes the candidate paths and provides feedback on the errors, thereby updating the policy network. Based on the interaction between reinforcement learning and motion capture feedback, the path and perspective are dynamically adjusted. The learning process uses a variational inference optimization strategy, specifically: ； Among them, the reward function Defined as: ； in, This represents the optimal policy, i.e., in state . Select action The probability distribution, Indicates the policy of following Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

10. A collaborative planning system for inspection robots based on digital twins, characterized in that, include: The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model. The collaborative planning algorithm design module is used to design collaborative planning algorithms under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function. The dynamic interactive learning module is used to achieve dynamic interactive learning through embodied twin intelligence. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

Images