Inspection robot collaborative planning method and system based on digital twinning

Abstract

The invention discloses an inspection robot collaborative planning method and system based on digital twinning, and the method comprises the steps: constructing a virtual simulation environment of a large-scale component through a point cloud modeling technology, and integrating a motion capture system to achieve the real-time mapping of a physical environment and a virtual model; defining a cooperative mechanism of the scanning robot and the positioning robot, providing global coordinates and path guidance by the positioning robot, and correcting position errors in real time by the motion capture system; simulating a dual-machine motion track in the digital twin model, and identifying a potential collision risk through a conflict prediction model; designing a collaborative planning algorithm under coupling constraint, and performing optimization by taking path optimization and scanning visual angle coverage as a joint target; based on a twin intelligent framework with a body, the path and the visual angle of the robot are dynamically adjusted by combining reinforcement learning with real-time feedback, and self-adaptive collaborative planning is achieved. Through double-machine division cooperation and real-time correction, the scanning efficiency and precision of the large-size component can be remarkably improved, and the collision risk is reduced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing technology, and in particular relates to a collaborative planning method and system for inspection robots based on digital twins. Background Technology

[0002] In the field of large component inspection, existing technologies mainly rely on traditional path planning algorithms and rudimentary digital twin applications, but still face many challenges. Traditional methods such as A While algorithms or genetic algorithms can achieve basic path optimization in static environments, they struggle to handle the complex dynamic characteristics of large components, leading to uneven paths, low efficiency, and increased energy consumption. This is particularly problematic when dealing with the coupling constraints between multi-machine motion planning and scanning viewpoint planning, as they neglect the limitations of path length on viewpoint coverage, resulting in insufficient scanning accuracy and missed hidden areas. Although digital twin technology has been used to simulate physical entities and synchronize robot states through virtual models to optimize tasks, it has not been fully extended to multi-machine collaborative scenarios for inspecting large components, lacking real-time feedback mechanisms to handle environmental changes. Furthermore, embodied intelligence, as an emerging paradigm, has been explored in robotic systems, such as using visual language models to improve perception and decision-making. However, these methods are mostly limited to single-machine tasks and have not been deeply integrated with digital twins, failing to effectively address the dual-machine division of labor (scanning and localization) and coupling constraint problems in the inspection of large components.

[0003] Overall, existing technologies are insufficient in terms of adaptability, real-time performance, and collaborative efficiency, leading to resource waste and increased risk of conflict. New methods are urgently needed to achieve efficient and high-precision scanning. Summary of the Invention

[0004] To address the above technical problems, this invention provides a collaborative planning method and system for inspection robots based on digital twins.

[0005] The technical solution adopted by this invention to solve its technical problem is: A collaborative planning method for inspection robots based on digital twins, the method comprising the following steps: S100: Constructs a digital twin model of the inspection environment, uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. S200: Defines a dual-machine collaborative mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional errors between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; S300: In the digital twin model, the robot's motion trajectory is simulated based on its state information, and the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, are predicted based on the conflict prediction model. S400: Design a collaborative planning algorithm under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function; S500: Driven by embodied twin intelligence, it achieves dynamic interactive learning. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

[0006] Preferably, S100 includes: S110: The surface of large components is scanned by a LiDAR or structured light scanner equipped on an inspection robot or auxiliary equipment to generate high-density point cloud data, and geometric reconstruction is performed using point cloud modeling technology. Specifically, after generating point cloud data using the scanning equipment, voxel filtering is first used for downsampling to reduce redundancy, and then the iterative nearest point algorithm is used for point cloud registration. Through iterative optimization, the optimal transformation parameters are solved to achieve real-time fusion of multiple frames of point clouds to form a complete three-dimensional model of the component. S120: An integrated motion capture system that obtains the relative position information of large components and the robot by tracking reflective markers attached to key construction points and the robot. S130: Through timestamp alignment and data fusion algorithms, the synchronization of scanning sensor data and motion capture position information is achieved, ensuring that physical environment data is mapped to the virtual environment in real time; S140: Based on the robot operating system, an interactive virtual simulation environment is built, defining and simulating the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realizing the interactive perception between the robot and the digital twin model through the API interface.

[0007] Preferably, S120 includes: S121: The two-dimensional image coordinates of the markers attached to the large components and the robot are detected by multiple cameras of the motion capture system, denoted as... ; S122: For each marker, its 3D position is reconstructed using a direct linear transformation algorithm based on its image coordinates in at least two cameras, specifically: camera projection matrix 3D point X satisfies By constructing a system of equations with coefficient matrix A And solve its singular value decomposition to obtain X, where, This is the intrinsic parameter matrix of the camera. , These are the rotation matrix and translation vector of the camera, respectively; S123: Based on the calculated 3D positions of multiple markers, the six-degree-of-freedom attitude of the target is calculated using a rigid body attitude estimation algorithm. Specifically, the rotation matrix is ​​calculated by solving the following optimization problem. Translation vector : ; in, and Let be the rigid body rotation matrix and translation vector to be determined, respectively. Given the current pose, let's consider the 3D coordinates of the i-th marker point as measured by the camera. These are the three-dimensional coordinates of the i-th marker point in the model coordinate system. The number of markers used for attitude estimation.

[0008] Preferably, S200 includes: S210: Establish a collaborative division of labor mechanism between the scanning robot and the positioning robot, wherein the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot; S220: The motion capture system tracks the markings on the scanning robot, the positioning robot, and large components in real time, and uses an optimization algorithm to correct the positional errors between the three. S230: Perform state modeling for each robot, representing its state as a multi-dimensional vector containing position, velocity, attitude, energy level, and motion capture correction error, and construct a joint state of the two robots, integrate multimodal data, and combine a visual language model to generate a linguistic description of the robot's state.

[0009] Preferably, the optimization algorithm in S220 is a least-squares optimization algorithm based on singular value decomposition, used to calculate the transformation parameters to minimize the error between the position of the motion capture observation marker and the reference point, specifically: ; in This represents the position of the nth marker point observed by motion capture. This represents the position of the nth reference point on the component or robot, where N is the number of marker points used for coordinate system alignment. , These are the transformation parameters, namely the rotation matrix and translation vector for coordinate system alignment.

[0010] Preferably, the dual-machine collaborative mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the localization robot through a value function, specifically as follows: ; in, The division of labor decision made at time t. For motion capture error term, It is an assignable role variable that limits the scope of division of labor in the dual-machine system. Indicates a scanning robot. Indicates the location of the robot. This represents the value function, used to evaluate the current state. Choose a character Expected returns These are the learning parameters.

[0011] Preferably, the conflict prediction model in S300 employs a Monte Carlo tree search algorithm, calculating the path collision probability through multiple simulations. In each simulation, starting from the current state, possible future trajectories are generated through selection, expansion, simulation, and backtracking steps, and the number of times the distance between the scanning robot and the positioning robot's trajectories falls below the safe distance threshold is counted. The specific formula for calculating the collision probability is as follows: ; Where M represents the total number of simulations. For indicator functions, For safe distance threshold, Let be the collision probability. and These are the predicted positions of the scanning robot and the localization robot in the m-th simulation, respectively; By using collision probability to guide the embodied twin intelligent interactive learning mechanism to optimize strategies, the robot path is dynamically adjusted to avoid collisions.

[0012] Preferably, in S400, the coupling model is a graph network structure, which is a coupling constraint graph, where the nodes represent the robot's state information and the edges encode the motion path. With scanning field coverage The coupling constraint relationship between them, where For visually visible functions, Let E be the scanning angle, and E be the set of edges in the coupling constraint graph. For the edge The length of the coupling loss function is determined, and global optimization is performed by minimizing the coupling loss function, which is specifically: ; in These are the weight coefficients for the path length term, view coverage term, and penalty collision term, respectively. For path length, For view coverage, To punish collisions, For indicator functions, and Let i and j represent the coordinates of the i-th and j-th robots in three-dimensional space, respectively. This is the safe distance threshold.

[0013] Preferably, S500 specifically refers to: The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint. The scanning robot and the localization robot act as agents. The virtual agent first simulates planning in the digital twin model to generate candidate paths. The physical agent executes the candidate paths and provides feedback on the errors, thereby updating the policy network. Based on the interaction between reinforcement learning and motion capture feedback, the path and perspective are dynamically adjusted. The learning process uses a variational inference optimization strategy, specifically: ; Among them, the reward function Defined as: ; in, This represents the optimal policy, i.e., in state . Select action The probability distribution, Indicates the policy of following Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

[0014] A collaborative planning system for inspection robots based on digital twins, comprising: The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model. The collaborative planning algorithm design module is used to design collaborative planning algorithms under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function. The dynamic interactive learning module is used to achieve dynamic interactive learning through embodied twin intelligence. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

[0015] The aforementioned collaborative planning method and system for inspection robots based on digital twins achieves efficient and high-precision scanning of large-sized components through dual-machine division of labor (scanning and positioning) and real-time error correction via motion capture. Compared to traditional technologies, it improves scanning efficiency and accuracy while reducing collision rates. The embodied twin intelligent framework supports adaptive learning through virtual-real interaction, ensuring generalization capabilities in new environments, reducing resource waste, and is suitable for large and complex scenarios such as wind turbine blades, bridges, aircraft, and high-speed trains. It significantly reduces inspection costs and risks, promoting intelligent industrial maintenance. Attached Figure Description

[0016] Figure 1 This is a flowchart of a collaborative planning method for inspection robots based on digital twins in one embodiment of the present invention. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0018] In one embodiment, such as Figure 1As shown, a collaborative planning method for inspection robots based on digital twins is proposed, the method comprising the following steps: S100: Constructs a digital twin model of the inspection environment, uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. S200: Defines a dual-machine collaborative mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional errors between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; S300: In the digital twin model, the robot's motion trajectory is simulated based on its state information, and the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, are predicted based on the conflict prediction model. S400: Design a collaborative planning algorithm under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function; S500: Driven by embodied twin intelligence, it achieves dynamic interactive learning. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

[0019] The aforementioned collaborative planning method for inspection robots based on digital twins constructs a digital twin model of the inspection environment. This model uses point cloud modeling and a motion capture system to map the positions of large components, the scanning robot, and the positioning robot in real time. Then, state modeling is incorporated into this model, resulting in a dual-machine division of labor: the scanning robot collects data, the positioning robot provides guidance, and the motion capture system corrects errors and performs adaptive optimization. Subsequently, a coupled model is constructed, a coupling loss function is defined, and embodied twin intelligent interactive learning is embedded to achieve dynamic path-viewpoint adjustment, avoid collisions, improve efficiency, and ensure high-efficiency and high-precision inspection.

[0020] Furthermore, a digital twin model of the inspection environment is constructed. Through high-fidelity virtual simulation, the physical inspection scene is mirrored in real time, ensuring the model's real-time performance and interactivity. This provides a reliable simulation environment for subsequent dual-robot collaborative planning, enabling efficient collaboration among multiple inspection robots. For large components, this invention employs point cloud modeling technology for geometric reconstruction, while utilizing a motion capture system to acquire the positional information of the large components and the robot, improving mapping accuracy and real-time performance.

[0021] In one embodiment, S100 includes: S110: The surface of large components is scanned by a LiDAR or structured light scanner equipped on an inspection robot or auxiliary equipment to generate high-density point cloud data, and geometric reconstruction is performed using point cloud modeling technology. Specifically, after generating point cloud data using the scanning equipment, voxel filtering is first used for downsampling to reduce redundancy, and then the iterative nearest point algorithm is used for point cloud registration. Through iterative optimization, the optimal transformation parameters are solved to achieve real-time fusion of multiple frames of point clouds to form a complete three-dimensional model of the component.

[0022] Specifically, point cloud data can be represented as a three-dimensional coordinate set. N represents the number of points, and each point contains its location coordinates and optional color or intensity information. To handle the scale of large components, voxel filtering is used to downsample the point cloud, reducing data redundancy while preserving key features. This method effectively reduces point cloud density while maintaining the surface details of the components. Iterative nearest-point algorithm is used for registration and fusion, achieving high-precision mapping.

[0023] S120: An integrated motion capture system that obtains the relative position information of large components and the robot by tracking reflective markers attached to key construction points and the robot. Specifically, in order to obtain the relative position information of large components and robots, the present invention integrates a motion capture system, which uses an infrared camera to track reflective marks attached to key points of the components and the robot, providing sub-millimeter-level position data.

[0024] In one embodiment, S120 includes: S121: The two-dimensional image coordinates of the markers attached to the large components and the robot are detected by multiple cameras of the motion capture system, denoted as... ; S122: For each marker, its 3D position is reconstructed using a direct linear transformation algorithm based on its image coordinates in at least two cameras, specifically: camera projection matrix 3D point X satisfies By constructing a system of equations with coefficient matrix A And solve its singular value decomposition to obtain X, where, This is the intrinsic parameter matrix of the camera. , These are the rotation matrix and translation vector of the camera, respectively; Furthermore, for the two cameras, assume that the camera projection matrices are respectively and 3D points Satisfies the projection equation: ; Solving least squares problems using direct linear transformation (DLT): ; Matrix A is constructed from the projection equation. The singular value decomposition (SVD) of X is solved, and the vector corresponding to the smallest singular value is taken as the solution.

[0025] S123: Based on the calculated 3D positions of multiple markers, the six-degree-of-freedom attitude of the target is calculated using a rigid body attitude estimation algorithm. Specifically, the rotation matrix is ​​calculated by solving the following optimization problem. Translation vector : ; in, and Let be the rigid body rotation matrix and translation vector to be determined, respectively. Given the current pose, let's consider the 3D coordinates of the i-th marker point as measured by the camera. These are the three-dimensional coordinates of the i-th marker point in the model coordinate system. The number of markers used for attitude estimation.

[0026] Specifically, the centroids of the observation point set and the reference point set are first calculated. And centralize the point set. Calculate the covariance matrix of the centered point set; perform singular value decomposition on the matrix. Calculate the rotation matrix based on the decomposition results. ; Calculate the translation vector based on the selected matrix and centroid. This process ensures that the attitude information is represented in six degrees of freedom and integrated into the point cloud model in real time, avoiding sensor drift issues. The motion capture system can achieve a frame rate of up to 120Hz, ensuring dynamic tracking accuracy.

[0027] S130: Through timestamp alignment and data fusion algorithms, the synchronization of scanning sensor data and motion capture position information is achieved, ensuring that physical environment data is mapped to the virtual environment in real time; Specifically, this involves synchronizing sensor data to ensure that data from the physical world is reflected in the virtual environment in real time. The scanning sensor information of the inspection robot is combined with motion capture location information, using a timestamp alignment mechanism to avoid data delays. Assume the sensor data stream is sequential. Where t is the timestamp and m is the number of sensors, including motion capture attitude data. An Extended Kalman Filter (EKF) is used to fuse multi-source data to predict and correct the state. The EKF state update formula is: ; in For state estimation, K is the Kalman gain, z is the measured value (including motion capture position), h is the nonlinear observation function, H is its Jacobian matrix, and P is the covariance matrix.

[0028] S140: Based on the robot operating system, an interactive virtual simulation environment is built, defining and simulating the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realizing the interactive perception between the robot and the digital twin model through the API interface.

[0029] Specifically, an interactive simulation space is created based on ROS, supporting virtual testing of robots and their environment. For dual-robot collaboration, the environment defines the robot's role: the scanning robot is responsible for high-precision data acquisition, and the positioning robot provides coordinate references. The interaction mechanism allows the robot to interact with the model via API interfaces; for example, during simulated path planning, the robot can "sense" virtual obstacles.

[0030] In one embodiment, S200 includes: S210: Establish a collaborative division of labor mechanism between the scanning robot and the positioning robot, wherein the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot; S220: The motion capture system tracks the markings on the scanning robot, the positioning robot, and large components in real time, and uses an optimization algorithm to correct the positional errors between the three. S230: Perform state modeling for each robot, representing its state as a multi-dimensional vector containing position, velocity, attitude, energy level, and motion capture correction error, and construct a joint state of the two robots, integrate multimodal data, and combine a visual language model to generate a linguistic description of the robot's state.

[0031] Specifically, the S200 implements robot role division and state modeling. A dual-machine collaborative mechanism is defined, where the scanning robot is responsible for high-precision scanning, equipped with a high-resolution camera and laser sensor to acquire image / point cloud data, achieving sub-millimeter-level detection of component surfaces. The localization robot is responsible for the scanning robot's localization, providing reference coordinates and path guidance, and using its sensors to generate navigation anchor points. This division of labor is based on task coupling: the localization robot's positioning information is transmitted to the scanning robot in real time, guiding its scanning path adjustment to ensure maximum coverage. The motion capture system is responsible for real-time correction of positional errors among the scanning robot, localization robot, and component.

[0032] In one embodiment, the optimization algorithm in S220 is a least-squares optimization algorithm based on singular value decomposition, used to calculate the transformation parameters to minimize the error between the position of the motion capture observation marker and the reference point, specifically: ; in This represents the position of the nth marker point observed by motion capture. This represents the position of the nth reference point on the component or robot, where N is the number of marker points used for coordinate system alignment. , The transformation parameters are the rotation matrix and translation vector for coordinate system alignment. This optimization is solved using SVD, ensuring an error of less than 0.01m and improving collaborative robustness.

[0033] In one embodiment, the dual-machine collaboration mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the localization robot through a value function, specifically as follows: ; in, The division of labor decision made at time t. For motion capture error term, It is an assignable role variable that limits the scope of division of labor in the dual-machine system. Indicates a scanning robot. Indicates the location of the robot. This represents the value function, used to evaluate the current state. Choose a character Expected returns To learn the parameters. To ensure that the efficiency of task division is improved by more than 25%.

[0034] Further state modeling of the robots is performed, with each robot's state represented as a multi-dimensional vector. ,in For location, For speed, As a gesture, For energy levels, To correct errors in motion capture. Dual-machine combined state is... It integrates virtual mirrors and component states into a digital twin model. In addition, it innovatively adopts multimodal fusion modeling, combines visual language models to process image data, and uses motion capture data to realize semantic state descriptions, such as "the scanning robot is approaching the scanning area, and the error has been corrected".

[0035] In a virtual environment, the intersection of two robot trajectories and scanning interference are simulated, and motion capture is incorporated to predict three-way collisions. In one embodiment, the collision prediction model in S300 uses a Monte Carlo tree search algorithm to calculate the path collision probability through multiple simulations. In each simulation, starting from the current state, possible future trajectories are generated through selection, expansion, simulation, and backtracking steps, and the number of times the distance between the scanning robot and the localization robot's trajectories is lower than the safe distance threshold is counted. The specific formula for calculating the collision probability is as follows: ; Where M represents the total number of simulations. For indicator functions, For safe distance threshold, Let be the collision probability. and These are the predicted positions of the scanning robot and the localization robot in the m-th simulation, respectively; By using collision probability to guide the embodied twin intelligent interactive learning mechanism to optimize strategies, the robot path is dynamically adjusted to avoid collisions.

[0036] Next, we will design a collaborative planning algorithm under coupled constraints. We will design a coupled model of multi-machine motion planning (path optimization) and scanning view planning (view coverage and accuracy constraints), and combine it with interactive learning of the digital twin virtual and real environments to achieve dynamic adjustment of path and view, avoid collisions and maximize scanning efficiency.

[0037] In one embodiment, S400, the coupling model is a graph network structure, specifically a coupling constraint graph, where nodes represent the robot's state information and edges encode motion paths. With scanning field coverage The coupling constraint relationship between them, where For visually visible functions, Let E be the scanning angle, and E be the set of edges in the coupling constraint graph. For the edge The length of the coupling loss function is determined, and global optimization is performed by minimizing the coupling loss function, which is specifically: ; in These are the weight coefficients for the path length term, view coverage term, and penalty collision term, respectively. For path length, For view coverage, To punish collisions, For indicator functions, and Let i and j represent the coordinates of the i-th and j-th robots in three-dimensional space, respectively. This represents the safe distance threshold. The model achieves global optimization by propagating constraints through a graph neural network.

[0038] Finally, an interactive learning model driven by embodied twin intelligence is employed. In the digital twin virtual-real environment, the scanning robot and the localization robot act as agents, interacting through reinforcement learning and motion capture feedback: the virtual agent first simulates planning in the twin model, generating candidate paths; the physical agent executes and reports errors, updating the policy network. The learning utilizes variational inference to optimize the policy.

[0039] In one embodiment, S500 specifically refers to: The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint. The scanning robot and the localization robot act as agents. The virtual agent first simulates planning in the digital twin model to generate candidate paths. The physical agent executes the candidate paths and provides feedback on the errors, thereby updating the policy network. Based on the interaction between reinforcement learning and motion capture feedback, the path and perspective are dynamically adjusted. The learning process uses a variational inference optimization strategy, specifically: ; Among them, the reward function Defined as: ; in, This represents the optimal policy, i.e., in state . Select action The probability distribution, Indicates the policy of following Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

[0040] This mechanism allows for dynamic adjustments: if the motion capture detection error does not converge, the positioning robot will replan the scanning robot's path in real time to avoid blind spots.

[0041] The coupling loss function generated by the above coupling model The reward function is directly embedded into the embodied twin intelligence as a prior constraint to guide interactive learning. Node states and edge constraints (such as path length L and view coverage C) of the coupled graph network (CCG) are injected into the input layer of the policy network, ensuring that the learning process starts from the model's global optimization objective and avoids local biases caused by decoupling. In the digital twin environment, the virtual agents of the scanning robot (Robot_B) and the localization robot (Robot_P) first generate initial candidate paths based on the coupled model, and then iteratively update the policy through interactive learning. Among them, the reward R fusion Gradient feedback enables dynamic adjustment of the path and perspective. This bridging mechanism shifts the algorithm from static graph optimization to adaptive evolution: the collision penalty term of the coupled model calibrates the trajectory distribution of variational inference in real time, ensuring the robustness of the entire planning process and providing efficient and high-precision collaboration for the inspection of large components.

[0042] The aforementioned collaborative planning method and system for inspection robots based on digital twins has the following technical advantages: Virtual-Real Interaction and Self-Evolution Capability: By constructing a high-fidelity digital twin model and integrating a motion capture system, real-time and accurate mapping and interaction between the physical environment and the virtual model are achieved. Building upon this, a reinforcement learning mechanism is introduced, and a loss function coupled with path length, viewpoint coverage, and collision risk is embedded as an optimization objective into the agent's policy. This drives the system to continuously self-evolve in virtual-real interaction, dynamically optimizing the scanning path and viewpoint, thereby significantly improving the adaptability and efficiency of scanning complex components.

[0043] High-precision collaboration and safety: Based on sub-millimeter-level position information provided by the motion capture system, a least-squares optimization algorithm with singular value decomposition as its core is used to calibrate the relative pose errors between the scanning robot, the positioning robot, and large components in real time. Combined with a conflict prediction model based on Monte Carlo tree search, potential collisions can be predicted and avoided in virtual space, significantly reducing the conflict risk of dual-robot collaborative operations and ensuring operational safety.

[0044] Adaptive Collaborative Division of Labor: An adaptive dual-machine collaborative mechanism based on a value function was designed. The localization robot provides global coordinate reference and path guidance, while the scanning robot focuses on high-precision data acquisition. The division of labor between the two can be dynamically adjusted according to real-time conditions such as motion capture error through a value function, breaking through the limitations of the traditional static collaboration mode. This enables efficient and blind-spot-free point cloud data acquisition on the surface of large components, with point cloud coverage quality and modeling accuracy significantly superior to traditional methods.

[0045] In one embodiment, a collaborative planning system for inspection robots based on digital twins is also provided, comprising: The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model. The collaborative planning algorithm design module is used to design collaborative planning algorithms under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function. The dynamic interactive learning module is used to achieve dynamic interactive learning through embodied twin intelligence. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

[0046] Specific limitations regarding a collaborative planning system for inspection robots based on digital twins can be found in the limitations of a collaborative planning method for inspection robots based on digital twins mentioned above, and will not be repeated here. Each module in the aforementioned collaborative planning system for inspection robots based on digital twins can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0047] The above provides a detailed description of the collaborative planning method and system for inspection robots based on digital twins provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention, and the descriptions of the embodiments are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A collaborative planning method for inspection robots based on digital twins, characterized in that, The method includes the following steps: S100: Constructs a digital twin model of the inspection environment, uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. S200: Defines a dual-machine collaborative mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional errors between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; S300: In the digital twin model, motion trajectories are simulated based on the state information of the scanning robot and the positioning robot, and potential collision risks between the scanning robot and the positioning robot and between the robot and large components are predicted based on the conflict prediction model. S400: Design a collaborative planning algorithm under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function; S500: Through embodied twin intelligence, dynamic interactive learning is achieved. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, the path and scanning perspective of the scanning robot and the positioning robot are dynamically adjusted through reinforcement learning to achieve collaborative planning.

2. The method according to claim 1, characterized in that, S100 includes: S110: The surface of large components is scanned by a LiDAR or structured light scanner equipped on an inspection robot or auxiliary equipment to generate high-density point cloud data, and geometric reconstruction is performed using point cloud modeling technology. Specifically, after generating point cloud data using the scanning equipment, voxel filtering is first used for downsampling to reduce redundancy, and then the iterative nearest point algorithm is used for point cloud registration. Through iterative optimization, the optimal transformation parameters are solved to achieve real-time fusion of multiple frames of point clouds to form a complete three-dimensional model of the component. S120: An integrated motion capture system that obtains the relative position information of large components and the robot by tracking reflective markers attached to key construction points and the robot. S130: Through timestamp alignment and data fusion algorithms, the synchronization of scanning sensor data and motion capture position information is achieved, ensuring that physical environment data is mapped to the virtual environment in real time; S140: Based on the robot operating system, an interactive virtual simulation environment is built, defining and simulating the collaborative roles and interaction logic of the scanning robot and the positioning robot, and realizing the interactive perception between the robot and the digital twin model through the API interface.

3. The method according to claim 2, characterized in that, S120 includes: S121: The two-dimensional image coordinates of the markers attached to the large components and the robot are detected by multiple cameras of the motion capture system, denoted as... ; S122: For each marker, its 3D position is reconstructed using a direct linear transformation algorithm based on its image coordinates in at least two cameras, specifically: camera projection matrix 3D point X satisfies By constructing a system of equations with coefficient matrix A And solve its singular value decomposition to obtain X, where, This is the intrinsic parameter matrix of the camera. , These are the rotation matrix and translation vector of the camera, respectively; S123: Based on the calculated 3D positions of multiple markers, the six-degree-of-freedom attitude of the target is calculated using a rigid body attitude estimation algorithm. Specifically, the rotation matrix is ​​calculated by solving the following optimization problem. Translation vector : ； in, and Let be the rigid body rotation matrix and translation vector to be determined, respectively. Given the current pose, let's consider the 3D coordinates of the i-th marker point as measured by the camera. These are the three-dimensional coordinates of the i-th marker point in the model coordinate system. This represents the number of markers used for attitude estimation.

4. The method according to claim 3, characterized in that, S200 includes: S210: Establish a collaborative division of labor mechanism between the scanning robot and the positioning robot, wherein the scanning robot is responsible for performing high-precision data acquisition of the component surface, and the positioning robot is responsible for providing global coordinate reference and real-time path guidance for the scanning robot; S220: The motion capture system tracks the markings on the scanning robot, the positioning robot, and large components in real time, and uses an optimization algorithm to correct the positional errors between the three. S230: Perform state modeling for each robot, representing its state as a multi-dimensional vector containing position, velocity, attitude, energy level, and motion capture correction error, and construct a joint state of the two robots, integrate multimodal data, and combine a visual language model to generate a linguistic description of the robot's state.

5. The method according to claim 4, characterized in that, The optimization algorithm in S220 is a least-squares optimization algorithm based on singular value decomposition, used to calculate transformation parameters to minimize the error between the position of the motion capture observation marker and the reference point. Specifically: ； in This represents the position of the nth marker point observed by motion capture. This represents the position of the nth reference point on the component or robot, where N is the number of marker points used for coordinate system alignment. , These are the transformation parameters, namely the rotation matrix and translation vector for coordinate system alignment.

6. The method according to claim 4, characterized in that, The dual-machine collaborative mechanism is adaptive, dynamically adjusting the division of labor strategy between the scanning robot and the localization robot through a value function, specifically: ； in, The division of labor decision made at time t. For motion capture error term, It is an assignable role variable that limits the scope of division of labor in the dual-machine system. Indicates a scanning robot. Indicates the location of the robot. This represents the value function, used to evaluate the current state. Choose a character Expected returns These are the learning parameters.

7. The method according to any one of claims 5 or 6, characterized in that, The S300 conflict prediction model uses the Monte Carlo tree search algorithm to calculate the path collision probability through multiple simulations. In each simulation, starting from the current state, possible future trajectories are generated through selection, expansion, simulation, and backtracking steps, and the number of times the distance between the trajectories of the scanning robot and the positioning robot is lower than the safe distance threshold is counted. The specific formula for calculating the collision probability is as follows: ； Where M represents the total number of simulations. For indicator functions, For safe distance threshold, Let be the collision probability. and These are the predicted positions of the scanning robot and the localization robot in the m-th simulation, respectively. To correct errors in motion capture; By using collision probability to guide the embodied twin intelligent interactive learning mechanism to optimize strategies, the robot path is dynamically adjusted to avoid collisions.

8. The method according to claim 7, characterized in that, In S400, the coupling model is a graph network structure, specifically a coupled constraint graph, where nodes represent the robot's state information and edges encode the motion path. With scanning field coverage The coupling constraint relationship between them, where For visually visible functions, Let E be the scanning angle, and E be the set of edges in the coupling constraint graph. For the edge The length of the coupling loss function is determined; and global optimization is performed by minimizing the coupling loss function. Specifically: ； in These are the weight coefficients for the path length term, view coverage term, and penalty collision term, respectively. For path length, For view coverage, To punish collisions, For indicator functions, and Let i and j represent the coordinates of the i-th and j-th robots in three-dimensional space, respectively. This is the safe distance threshold.

9. The method according to claim 8, characterized in that, The S500 specifically refers to: The coupling loss function is embedded into the reward function of reinforcement learning as a prior constraint. The scanning robot and the localization robot act as agents. The virtual agent first simulates planning in the digital twin model to generate candidate paths. The physical agent executes the candidate paths and provides feedback on the errors, thereby updating the policy network. Based on the interaction between reinforcement learning and motion capture feedback, the path and perspective are dynamically adjusted. The learning process uses a variational inference optimization strategy, specifically: ； Among them, the reward function Defined as: ； in, This represents the optimal policy, i.e., in state . Select action The probability distribution, Indicates the policy of following Mid-sampling action The average of the results is calculated. For scan coverage, For scanning accuracy, For collision penalty, the collision term in the coupled model Positive correlation This represents the collision probability in a digital twin simulation. These are weighting coefficients used to balance the relative importance of coverage, accuracy, and collision penalty in the reward function. For KL divergence weights, It measures the distribution of variational trajectories With posterior trajectory distribution The difference between them is expressed by the formula: This ensures that the trajectory distribution generated by variational inference closely approximates the trajectory in the real environment. It is a variational trajectory distribution, representing the state... Below, the robot trajectory distribution is approximately generated through variational inference. Used for optimizing paths and perspectives in digital twin-based environment simulations. It is the posterior trajectory distribution, representing the state. Below, based on the trajectory distribution from real-world feedback, combined with collision probability... and motion capture correction error Ensure the authenticity of the trajectory.

10. A collaborative planning system for inspection robots based on digital twins, characterized in that, include: The digital twin model building module is used to construct a digital twin model of the inspection environment. It uses point cloud modeling technology to geometrically reconstruct large components to form a virtual simulation environment, and integrates a motion capture system to obtain the spatial position information of large components, scanning robots and positioning robots in real time, so as to realize the real-time mapping and interaction between the physical inspection environment and the virtual model. The dual-machine collaboration mechanism definition module is used to define the dual-machine collaboration mechanism, in which the scanning robot is responsible for collecting surface data of the component, the positioning robot provides global coordinate reference and path guidance for the scanning robot, the motion capture system corrects the positional error between the scanning robot, the positioning robot and the large component in real time, and the state of each robot is jointly modeled through a digital twin model; The conflict simulation module is used to simulate the robot's motion trajectory based on the robot's state information in the digital twin model, and to predict the potential collision risks between the scanning robot and the positioning robot, as well as between the robot and large components, based on the conflict prediction model. The collaborative planning algorithm design module is used to design collaborative planning algorithms under coupled constraints, construct a coupled model of multi-machine motion planning and scanning view planning, and take path optimization and view coverage as joint optimization objectives based on the coupling loss function. The dynamic interactive learning module is used to achieve dynamic interactive learning through embodied twin intelligence. In the digital twin virtual and real environment, based on the real-time feedback of the coupled model and motion capture system, it dynamically adjusts the path and scanning perspective of the scanning robot and the positioning robot through reinforcement learning to achieve collaborative planning.

Images