An indoor wall building operation simulation and optimization method based on digital twinning

Abstract

The application relates to an indoor wall building operation simulation and optimization method based on digital twinning, in particular to the field of digital twinning, a high-fidelity digital twin body synchronous with a physical world is constructed, real-time twin state data is generated, an accurate basis is provided for subsequent simulation and decision, future operation risks are forward-lookingly quantitatively evaluated by using a conflict risk quantification atlas, an optimization control instruction capable of actively avoiding conflicts and optimizing operation beats is generated by driving a deep reinforcement learning strategy network, meanwhile, model error calculation is realized by comparing state feedback data and predicted state data, and model parameters in the high-fidelity digital twin body are calibrated online according to the model error calculation, a closed loop of perception, simulation, decision and calibration is formed, and finally, safe, efficient and collaborative autonomous operation of a wall building robot double mechanical arm in a complex indoor environment is realized, and the wall building robot double mechanical arm has the ability of continuously self-optimizing and adapting to environmental changes.

Description

Technical Field

[0001] This invention relates to the field of digital twins, and more specifically, to a method for simulating and optimizing indoor bricklaying operations based on digital twins. Background Technology

[0002] With the deepening of the intelligent transformation of the construction industry, the automation and robotization of indoor bricklaying operations have become an important development direction. An intelligent bricklaying robot integrating a mobile chassis, lifting mechanism, dual robotic arms, visual perception, and brick and mortar supply modules is designed to perform continuous bricklaying tasks in complex indoor environments such as residential living rooms and bedrooms. In such scenarios, the robot needs to operate autonomously in a dynamic environment with limited space, door and window openings, and furniture obstacles. Its dual robotic arms are usually planned to perform a series of continuous processes such as picking up bricks, applying mortar, and placing bricks in order to form an efficient assembly line operation rhythm. However, the robot body, its dual robotic arms, end effector, the ever-increasing brickwork, and surrounding obstacles together constitute a tightly coupled, real-time changing closed work space, which makes the collaborative motion planning face severe challenges.

[0003] Currently, the mainstream methods for achieving collaborative operation of such multi-degree-of-freedom robotic systems still rely on teach-in programming, offline programming, or master-slave control strategies. Teach-in and offline programming depend on pre-set static environment models and fixed trajectories, which cannot adapt to the dynamic characteristics of the continuously changing workspace during the wall-building process. Online obstacle avoidance methods based on rules or artificial potential fields are often computationally cumbersome when dealing with potential interference conflicts between two robotic arms and between the robotic arm and the complex environment. Furthermore, it is difficult to optimize global operational efficiency and motion smoothness while satisfying collision-free constraints. More importantly, simple local obstacle avoidance strategies may cause the robotic arm to get stuck in local optima or produce vibrations, failing to guarantee long-term operational stability. The coordination and efficiency of the work sequence are currently lacking in existing technologies. There is a lack of a way to perform pre-operation simulation, predict conflicts in real time during operation, and dynamically replan the global motion sequence. This results in robots either sacrificing the parallel operation potential and overall efficiency of dual robotic arms due to conservative path planning, or experiencing frequent sudden stops or even mechanical collisions due to unforeseen interference, leading to equipment damage, work interruption, and safety risks. Therefore, there is an urgent need for a solution that can deeply integrate real-time environmental perception, perform high-fidelity dynamic modeling of the robot system, and realize intelligent collaborative decision-making to break through the technical bottleneck of high-efficiency and high-reliability collaborative operation of indoor bricklaying robots. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a simulation and optimization method for indoor bricklaying operations based on digital twins, thereby solving the problems mentioned in the background art.

[0005] The technical solution of this invention to solve the above-mentioned technical problems is as follows: specifically, it includes the following steps: Step S1: In response to the start operation command of the bricklaying robot, the multimodal sensor data of the bricklaying robot and the visual perception data of the working environment are collected synchronously. Data fusion and state estimation are performed on the multimodal sensor data and the visual perception data. Based on the fusion and estimation results, a high-fidelity digital twin containing the virtual model of the bricklaying robot, the virtual model of the dual robotic arms and the virtual model of the working environment is driven and updated. The high-fidelity digital twin contains model parameters used to describe the dynamics of the virtual model and generates real-time twin state data. Step S2: In the high-fidelity digital twin, based on the real-time twin state data output in step S1, advance rolling simulation is performed on the pre-planned action sequence of the dual robotic arms. During the simulation, continuous collision detection is performed on the virtual model of the bricklaying robot, the virtual model of the dual robotic arms, and the virtual model of the working environment, and the geometric interference between them is calculated. Based on the geometric interference, the conflict risk quantification map within the future time window is calculated and output. Step S3: Receive the conflict risk quantification map output in step S2 and the real-time twin state data output in step S1, and input them into a pre-trained deep reinforcement learning policy network. The deep reinforcement learning policy network makes online decisions and generates optimized control commands for adjusting the joint spatial trajectory of the dual robotic arms. Step S4: Send the optimized control command generated in step S3 to the bricklaying robot for execution. At the same time, collect the physical state data measured by the sensors during the actual operation of the bricklaying robot as state feedback data. Compare the state feedback data with the predicted state data generated by the simulation calculation based on the optimized control command in the high-fidelity digital twin. Calculate the difference between the two as the model error. Use the model error to calibrate the model parameters in the high-fidelity digital twin that describe the dynamics of the virtual model online, so as to reduce the difference between the system state represented by the high-fidelity digital twin and the real physical state of the bricklaying robot. In a preferred embodiment, in step S1, the multimodal sensor data specifically includes encoder data of each joint of the dual robotic arms, joint torque sensor data, wheel odometer data and inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism; the visual perception data specifically includes point cloud data and color image data of the working environment collected by the depth camera and the color camera.

[0006] In a preferred embodiment, the specific operation of data fusion and state estimation of multimodal sensor data and visual perception data is as follows: The wheeled odometry data, inertial measurement unit data, and visual odometry data derived from visual perception data of the mobile chassis of the bricklaying robot are fused using a tightly coupled graph optimization model to estimate the pose and velocity of the mobile chassis of the bricklaying robot in the global coordinate system. The tightly coupled graph optimization model achieves state estimation by constructing and minimizing a loss function, which is composed of the inertial measurement unit pre-integration constraint residual, the visual odometry relative pose constraint residual, and the loop closure detection constraint residual. During the optimization process, a time-varying adaptive weighting factor is introduced into the visual odometry relative pose constraint residual. The adaptive weighting factor is dynamically calculated and adjusted according to the magnitude of the covariance of the reprojection error generated by the visual odometry in the most recent time window. Meanwhile, a robust kernel function is applied to the loop closure detection constraint residuals, and finally, the optimized pose and velocity estimates of the mobile chassis of the bricklaying robot are output.

[0007] In a preferred embodiment, the process of driving and updating a high-fidelity digital twin containing a virtual model of a bricklaying robot, a virtual model of dual robotic arms, and a virtual model of the working environment based on the fusion and estimation results specifically involves: Using the pose and velocity estimates of the mobile chassis of the bricklaying robot, the global position and orientation of the mobile chassis component of the bricklaying robot virtual model in the high-fidelity digital twin are updated; using the encoder data of each joint of the dual robotic arms obtained from multimodal sensor data, the position and orientation of each link and end effector of the dual robotic arm virtual model in the high-fidelity digital twin are updated through forward kinematics calculation; using the displacement sensor data of the two-stage lifting mechanism obtained from multimodal sensor data, the extension height of the lifting mechanism component of the bricklaying robot virtual model in the high-fidelity digital twin is updated. Simultaneously, the depth point cloud and color image in the visual perception data are processed in real time. First, the continuously collected depth point cloud data are superimposed and aligned in a unified world coordinate system through a point cloud registration algorithm. Then, the registered depth point cloud data is input into a pre-trained semantic segmentation network for pixel-level classification. Based on the classification results, the occupancy probability of each three-dimensional spatial unit is updated and semantic labels are assigned. A three-dimensional dense semantic voxel map representing the working environment is dynamically constructed and updated. Each voxel unit of the three-dimensional dense semantic voxel map contains its spatial location information, the probability value of being occupied by obstacles, and semantic category labels. This continuously updated three-dimensional dense semantic voxel map is used as a virtual model of the working environment in a high-fidelity digital twin. During this process, the high-fidelity digital twin synchronously maintains and updates the model parameters of the bricklaying robot virtual model, the dual-arm virtual model, and the working environment virtual model in computer memory. The model parameters include kinematic constraint parameters that are consistent with the physical chassis for the mobile chassis model, mass, moment of inertia, joint friction coefficient, and damping coefficient configured for the two-stage lifting mechanism model, geometric shape and mass attributes configured for each link of the dual-arm virtual model, transmission stiffness and damping parameters configured for each joint, and physical friction coefficient configured for the obstacle surface in the working environment virtual model. All model parameters together constitute a dynamically adjustable parameter set. Finally, the chassis and lifting mechanism states of the bricklaying robot virtual model, the states of all joints and links of the dual-arm virtual model, and the three-dimensional dense semantic voxel map of the working environment virtual model are integrated to generate real-time twin state data.

[0008] In a preferred embodiment, in step S2, based on the initial state of the virtual model represented by the real-time twin state data output in step S1, the pre-planned action sequence of the dual robotic arms is used as the control input, and dynamic forward extrapolation is performed in the high-fidelity digital twin at a speed faster than the actual operation to simulate the continuous motion process of the wall-building robot virtual model, the dual robotic arm virtual model, and the work environment virtual model within a fixed period of time in the future. Within each calculation step of this advanced rolling simulation, continuous collision detection is performed between each pair of components in the virtual model of the bricklaying robot: the mobile chassis component, the lifting mechanism component, each link and end effector in the virtual model of the dual robotic arms, and the wall and obstacle surfaces in the virtual model of the working environment. Continuous collision detection calculates the spatial interference between the motion trajectory sweep of all moving parts and other parts or environment models from the current simulation time to the next simulation time, and outputs the minimum approach distance between each pair of potential collision objects, as well as the motion trajectory penetration depth if geometric penetration occurs.

[0009] In a preferred embodiment, the specific operation of calculating and outputting the conflict risk quantification map within the future time window based on geometric interference is as follows: Based on the minimum approach distance between each pair of potential collision objects output by continuous collision detection, the motion trajectory penetration depth information when geometric penetration occurs, and combined with the relative motion velocity between collision objects obtained from real-time twin state data and advanced rolling simulation, and the mass and moment of inertia parameters of collision objects obtained from high-fidelity digital twin model parameters, the instantaneous conflict risk value at each discrete moment on the future simulation time axis is calculated. The calculation of the instantaneous conflict risk value integrates the following three dimensions: The first dimension is a geometric risk factor based on the minimum proximity distance. This geometric risk factor maps the minimum proximity distance to a normalized value that is greater than or equal to zero and less than or equal to one through a negative exponential function with the natural constant e as the base and the product of the minimum proximity distance and the distance sensitivity coefficient as the exponent. The second dimension is a motion risk factor based on the relative velocity of the colliding objects in the direction of the normal to the point of contact. The third dimension is the effective mass energy factor derived from the simplified mass model of the collision object; Multiply the first dimension's geometric risk factor by a preset geometric risk weight coefficient, multiply the second dimension's motion risk factor by a preset motion risk weight coefficient, multiply the third dimension's effective mass-energy factor by a preset energy risk weight coefficient, and sum these three weighted values ​​to obtain the final instantaneous conflict risk value. Within the entire future simulation time window, all calculated instantaneous conflict risk values ​​and their corresponding collision object pairs and location information are organized in chronological order to form a structured conflict risk quantification map.

[0010] In a preferred embodiment, step S3 involves receiving the conflict risk quantification map output from step S2 and the real-time twin state data output from step S1, and inputting them into a pre-trained deep reinforcement learning policy network. First, features are extracted and encoded from the real-time twin state data and the conflict risk quantification map, respectively. The encoding process for real-time twin state data is as follows: extract state information from the chassis and lifting mechanism states of the bricklaying robot virtual model, the joint and link states of the dual robotic arm virtual model, and the three-dimensional dense semantic voxel map of the working environment virtual model contained in the real-time twin state data. This includes the global position and attitude of the mobile chassis component of the bricklaying robot, the position and attitude of each joint and end effector of the dual robotic arm, and the direction and distance features of nearby obstacles calculated from the three-dimensional dense semantic voxel map. Input these features into a multilayer perceptron network composed of multiple fully connected layers and nonlinear activation functions. The state information is then fused and compressed into a fixed-dimensional current state feature vector through calculation. The encoding process for the conflict risk quantification map is as follows: the conflict risk quantification map is regarded as a data sequence arranged in chronological order. Each element of the data sequence contains the instantaneous conflict risk value at a given future moment and its corresponding collision object identification and location information. This data sequence is input into an encoder of a temporal convolutional network or recurrent neural network. The encoder captures the evolution trend of future risks in the time dimension and the distribution pattern in the space through its temporal modeling capability, and outputs a future risk feature vector with a fixed dimension. Subsequently, the current state feature vector obtained by encoding is concatenated with the future risk feature vector to form a comprehensive state representation vector.

[0011] In a preferred embodiment, the specific process of generating optimized control commands for adjusting the spatial trajectory of the dual robotic arms joints is as follows: After receiving the comprehensive state representation vector, the deep reinforcement learning policy network performs calculations and inferences through its internal multi-layer neural network and directly outputs an original action vector. To ensure that the output motion meets the actual range of motion and speed limits of the robotic arm joints, the original motion vector is post-processed. The post-processing process is as follows: First, the hyperbolic tangent function is used to compress each element value of the original motion vector to between negative one and positive one. Then, the compressed original motion vector is multiplied element by element by a preset motion scaling vector. Each component of the motion scaling vector represents the maximum position or speed adjustment allowed for the corresponding joint. After this scaling and limiting process, the final optimized control command is obtained. The pre-training of the deep reinforcement learning policy network is completed in the high-fidelity digital twin virtual environment constructed and maintained in step S1. The training process models the collaborative bricklaying operation of the two robotic arms as a sequential decision-making process, and learns the policy through the interaction between the policy model and the digital twin environment. The reward function used in the training consists of task progress reward, risk avoidance reward, motion smoothness reward, and bi-arm coordination fluency reward; the training objective of the deep reinforcement learning policy network is to maximize the sum of long-term cumulative rewards obtained by performing actions in the digital twin environment by adjusting the network parameters.

[0012] In a preferred embodiment, the specific process of comparing the state feedback data with the predicted state data generated by simulation calculation based on optimized control instructions in a high-fidelity digital twin, and calculating the difference between the two as the model error, is as follows: First, the physical state data collected from the actual operation of the bricklaying robot and measured by sensors are processed. The physical state data includes encoder data of each joint of the dual robotic arms, joint torque sensor data, inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism. By parsing and transforming these multimodal sensor data, state feedback data corresponding to the real-time twin state data structure in step S1 is generated. The state feedback data specifically includes the actual pose and speed of the mobile chassis of the bricklaying robot, and the actual angle and actual end pose of each joint of the dual robotic arms. Meanwhile, in the high-fidelity digital twin, the latest real-time twin state data output in step S1 is used as the initial state, and the optimized control command generated in step S3 is used as the input of the virtual model. A single-step or short-term dynamic forward simulation is performed through the physics engine to calculate the predicted state data that the bricklaying robot virtual model and the dual-arm virtual model should have after the optimized control command is executed. The predicted state data includes the predicted pose and velocity of the moving chassis of the bricklaying robot virtual model, and the predicted angles and predicted end poses of each joint of the dual-arm virtual model. Subsequently, the state feedback data and the corresponding observable state component in the predicted state data are subtracted element by element to obtain the original difference vector. Based on the historical accuracy and reliability of each sensor measurement, a confidence weight is assigned to each component of the original difference vector; Multiply the original differences of all components by the square root of their corresponding confidence weights to form a weighted residual vector.

[0013] In a preferred embodiment, the specific process of online calibration of the model parameters used to describe the dynamics of the virtual model in the high-fidelity digital twin using model errors is as follows: First, calculate the sensitivity matrix of the weighted residual vector relative to the model parameters in the high-fidelity digital twin. Each row of the sensitivity matrix corresponds to a component of the weighted residual vector, and each column corresponds to an adjustable parameter in the set of dynamically adjustable parameters. Next, a parameter update equation is constructed, which is to multiply the transpose of the sensitivity matrix by the sensitivity matrix itself to obtain an information matrix. Based on this, a product of a first regularization coefficient and an identity matrix is ​​introduced and added to the information matrix; at the same time, a structured prior matrix is ​​introduced and added to the information matrix; the information matrix, the product of the first regularization coefficient and the identity matrix, and the structured prior matrix are added together to form a new composite matrix. Then, the inverse of the composite matrix is ​​calculated; subsequently, the transpose of the sensitivity matrix is ​​multiplied by the weighted residual vector to obtain a gradient vector; the calculated inverse matrix is ​​multiplied by the gradient vector to obtain a model parameter update vector. Finally, the model parameter update vector is multiplied by a preset learning rate; the resulting scaled update vector is then added element-wise to the current dynamically adjustable parameter set in the high-fidelity digital twin to complete an online iterative calibration of the dynamically adjustable parameter set. The calibrated dynamically adjustable parameter set will be immediately used to update the dynamic properties of the high-fidelity digital twin.

[0014] The beneficial effects of this invention are as follows: By constructing a high-fidelity digital twin synchronized with the physical world and generating real-time twin state data, a precise foundation is provided for subsequent simulation and decision-making. A conflict risk quantification map is used to conduct a forward-looking quantitative assessment of future operational risks, driving a deep reinforcement learning strategy network to generate optimized control commands that can proactively avoid conflicts and optimize the work rhythm. At the same time, the model error is calculated by comparing the state feedback data with the predicted state data, and the model parameters in the high-fidelity digital twin are calibrated online accordingly, forming a closed loop of perception, simulation, decision-making and calibration. Ultimately, the dual robotic arms of the bricklaying robot can operate safely, efficiently, collaboratively and autonomously in complex indoor environments, and have the ability to continuously self-optimize and adapt to environmental changes. Attached Figure Description

[0015] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0018] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application. Example

[0019] This embodiment provides, for example Figure 1 The method for simulating and optimizing indoor bricklaying operations based on digital twins, as shown, specifically includes the following steps: Step S1: In response to the start operation command of the bricklaying robot, the multimodal sensor data of the bricklaying robot and the visual perception data of the working environment are collected synchronously. Data fusion and state estimation are performed on the multimodal sensor data and the visual perception data. Based on the fusion and estimation results, a high-fidelity digital twin containing the virtual model of the bricklaying robot, the virtual model of the dual robotic arms and the virtual model of the working environment is driven and updated. The high-fidelity digital twin contains model parameters used to describe the dynamics of the virtual model and generates real-time twin state data. Step S2: In the high-fidelity digital twin, based on the real-time twin state data output in step S1, advance rolling simulation is performed on the pre-planned action sequence of the dual robotic arms. During the simulation, continuous collision detection is performed on the virtual model of the bricklaying robot, the virtual model of the dual robotic arms, and the virtual model of the working environment, and the geometric interference between them is calculated. Based on the geometric interference, the conflict risk quantification map within the future time window is calculated and output. Step S3: Receive the conflict risk quantification map output in step S2 and the real-time twin state data output in step S1, and input them into a pre-trained deep reinforcement learning policy network. The deep reinforcement learning policy network makes online decisions and generates optimized control commands for adjusting the joint spatial trajectory of the dual robotic arms. Step S4: Send the optimized control command generated in step S3 to the bricklaying robot for execution. At the same time, collect the physical state data measured by sensors during the actual operation of the bricklaying robot as state feedback data. Compare the state feedback data with the predicted state data generated by simulation calculation based on the optimized control command in the high-fidelity digital twin. Calculate the difference between the two as the model error. Use the model error to calibrate the model parameters in the high-fidelity digital twin that describe the dynamics of the virtual model online, so as to reduce the difference between the system state represented by the high-fidelity digital twin and the real physical state of the bricklaying robot.

[0020] In this embodiment, it is specifically necessary to explain that in step S1, the multimodal sensor data includes encoder data of each joint of the dual robotic arms, joint torque sensor data, wheel odometer data and inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism. Among them, the encoder data is used to analyze the real-time rotation angle of the robotic arm joints, the joint torque sensor data is used to sense the interaction force between the gripper and the brick, the wheel odometer and inertial measurement unit data are used to preliminarily calculate the motion displacement and attitude change of the mobile chassis, and the displacement sensor data is used to determine the precise height of the lifting mechanism. The visual perception data specifically includes point cloud data and color image data of the working environment collected by the depth camera and the color camera. The depth camera can be a sensor based on the time-of-flight principle or the structured light principle. Each point in the output point cloud data contains three-dimensional spatial coordinate information. The RGB images collected synchronously by the color camera provide rich texture and color information. Both have been calibrated and synchronized in time and space to ensure the basic reliability of data fusion. The specific operations for data fusion and state estimation of multimodal sensor data and visual perception data are as follows: The wheeled odometry data, inertial measurement unit data, and visual odometry data derived from visual perception data of the mobile chassis of the bricklaying robot are fused using a tightly coupled graph optimization model to estimate the pose and velocity of the mobile chassis of the bricklaying robot in the global coordinate system. The visual odometry data is obtained by feature point extraction, matching, and motion estimation calculation of continuous color images. The tightly coupled graph optimization model achieves state estimation by constructing and minimizing a loss function, which is composed of the inertial measurement unit pre-integration constraint residual, the visual odometry relative pose constraint residual, and the loop closure detection constraint residual. The inertial measurement unit pre-integration constraint residual reflects the difference between the relative pose change calculated by the inertial measurement unit data through pre-integration technology and the pose change predicted in the state variables between two adjacent image frames. The visual odometry relative pose constraint residual represents the difference between the camera pose transformation between adjacent frames calculated based on visual features and the pose transformation predicted in the state variables. The loop closure detection constraint residual refers to the difference between the spatial constraint relationship between the historical pose and the current pose established when the system detects a match between the current scene and the historical scene and the corresponding pose in the state variables. During the optimization process, a time-varying adaptive weighting factor is introduced into the visual odometry relative pose constraint residual. This adaptive weighting factor is dynamically calculated and adjusted based on the covariance of the reprojection error generated by the visual odometry within the most recent time window. Specifically, the adaptive weighting factor λ_v(t) is calculated as: λ_v(t) = η / (tr(Σ_reproj(t)) + ε), where η is a normalization coefficient used to adjust the weights to an appropriate level; its typical value can be set between 0.5 and 2.0 depending on the sensor accuracy. r(Σ_reproj(t)) represents the trace of the visual odometry reprojection error covariance matrix within the most recent time window, used to measure the overall uncertainty of visual observations; ε is a very small positive number (e.g., 1e-6) to prevent the denominator from being zero, so that when the uncertainty of visual odometry observations increases, the contribution weight of the visual odometry relative pose constraint residual in the tightly coupled graph optimization model is reduced accordingly, thereby enhancing the robustness and accuracy of the estimation of the mobile chassis pose and velocity of the bricklaying robot in indoor working environments where visual features are scarce or dynamic interference exists; Simultaneously, a robust kernel function is applied to the loop closure detection constraint residuals to suppress the interference of erroneous loop closure matching on state estimation. The robust kernel function is preferably the Huber kernel function or the Cauchy kernel function. Its function is to reduce the influence of the constraint in the loss function when the loop closure detection constraint residuals are large, thereby avoiding state estimation divergence caused by accidental erroneous matching. Finally, the optimized pose and velocity estimates of the mobile chassis of the bricklaying robot are output as the basis for subsequent updates of the virtual model state of the bricklaying robot in the high-fidelity digital twin. The process of creating a high-fidelity digital twin based on fusion and estimation results, which includes a virtual model of a bricklaying robot, a virtual model of a dual-arm robot, and a virtual model of the working environment, is as follows: Using the pose and velocity estimates of the mobile chassis of the bricklaying robot, the global position and orientation of the mobile chassis component of the bricklaying robot virtual model in the high-fidelity digital twin are updated. This update is a direct assignment of position and orientation values, ensuring spatial alignment between the virtual model and the physical entity. Using encoder data from the joints of the dual robotic arms obtained from multimodal sensor data, the position and orientation of each link and end effector of the dual robotic arm virtual model in the high-fidelity digital twin are updated through forward kinematics calculations. The forward kinematics calculations derive the poses of the links and end effectors step-by-step from the joint angles based on the DH parameters of the robotic arms or an improved model. Using displacement sensor data from the two-stage lifting mechanism obtained from multimodal sensor data, the extension height of the lifting mechanism component of the bricklaying robot virtual model in the high-fidelity digital twin is updated. This is a synchronous update of a one-dimensional linear parameter. Simultaneously, the depth point cloud and color image in the visual perception data are processed in real time. First, the continuously acquired depth point cloud data are superimposed and aligned in a unified world coordinate system using a point cloud registration algorithm. The point cloud registration algorithm can employ the iterative nearest point algorithm or its variants. By minimizing the distance error between point pairs, the optimal rigid body transformation matrix between adjacent frame point clouds is solved. Subsequently, the registered depth point cloud data is input into a pre-trained semantic segmentation network for pixel-level classification. The semantic segmentation network is a deep learning model that uses an encoder-decoder structure (such as U-Net) or a Transformer-based architecture to classify each 3D point or voxel. Semantic label prediction, such as classifying it as "wall", "unbuilt area", "brick", "door and window" or "obstacle", updates the occupancy probability and assigns semantic labels to each three-dimensional spatial unit based on the classification results, dynamically constructs and updates a three-dimensional dense semantic voxel map representing the working environment. The occupancy probability update can use the Bayesian filtering method to adjust the confidence of each voxel being occupied according to the new observation data. Each voxel unit of the three-dimensional dense semantic voxel map contains its spatial location information, the probability value of being occupied by an obstacle, and a semantic category label. This continuously updated three-dimensional dense semantic voxel map is used as a virtual model of the working environment in a high-fidelity digital twin. During this process, the high-fidelity digital twin synchronously maintains and updates the model parameters of the virtual models of the bricklaying robot, the dual-arm robot, and the working environment in the computer memory. These model parameters are the foundation for driving the physical simulation of the virtual models. These parameters include assigning kinematic constraint parameters consistent with the physical chassis to the mobile chassis model; configuring the mass, moment of inertia, joint friction coefficient, and damping coefficient for the two-stage lifting mechanism model; configuring the geometric shape and mass properties of each link in the dual-arm robot virtual model; configuring the transmission stiffness and damping parameters for each joint; and configuring the physical friction coefficient for the obstacle surfaces in the working environment virtual model. Parameters such as friction coefficient and damping coefficient are usually obtained as initial values ​​based on material properties and experimental measurements. All model parameters together constitute a dynamically adjustable parameter set, which is the direct object of operation for model calibration in subsequent steps. Finally, the chassis and lifting mechanism states of the bricklaying robot virtual model, the states of all joints and links of the dual robotic arm virtual model, and the three-dimensional dense semantic voxel map of the working environment virtual model are integrated to generate real-time twin state data. The real-time twin state data is a structured data set that encapsulates a complete snapshot of all key components in the digital twin world at time t, and is the sole data input source for subsequent simulation and decision-making steps.

[0021] In this embodiment, it is specifically necessary to explain that in step S2, based on the initial state of the virtual model represented by the real-time twin state data output in step S1, the pre-planned action sequence of the dual robotic arms is used as the control input. In the high-fidelity digital twin, dynamic forward extrapolation is performed at a speed faster than the actual operation to simulate the continuous motion process of the wall-building robot virtual model, the dual robotic arm virtual model, and the work environment virtual model within a fixed period of time in the future. The dynamic forward extrapolation is implemented based on the physics engine, and the simulation speed is set to five to twenty times the actual operation speed to complete all prediction calculations before the next control cycle arrives. The simulated fixed period of time in the future is called the prediction time domain, and its length is set according to the wall-building operation cycle and the movement speed of the robotic arms, with a typical value of 0.5 seconds to 2 seconds. Within each computational step of this advanced rolling simulation, continuous collision detection is performed between each pair of components in the virtual model of the bricklaying robot: the mobile chassis component, the lifting mechanism component, each link and end effector in the virtual model of the dual robotic arms, and the wall and obstacle surfaces in the virtual model of the working environment. The continuous collision detection preferably adopts the continuous extended form of the Gilbert-Johnson-Kilty (GJK) algorithm or the scanning volume test method based on the separated axis theorem. These methods can accurately calculate the interference between the sweep volumes formed by the motion trajectories of the moving objects in discrete time steps. Continuous collision detection calculates the spatial interference between the motion trajectories of all moving parts and other parts or the environment model from the current simulation time to the next simulation time. It outputs the minimum approach distance between each pair of potential collision objects, as well as the trajectory penetration depth if geometric penetration occurs. The minimum approach distance is defined as the Euclidean distance between the closest points of the two objects. The trajectory penetration depth is output when geometric penetration occurs, and its value is the maximum depth to which one object is embedded in another object. This information is used to assess the severity of the collision. The specific steps for calculating and outputting a quantitative map of conflict risk within a future time window based on geometric interference are as follows: Based on the minimum approach distance between each pair of potential collision objects output by continuous collision detection, the penetration depth information of the motion trajectory when geometric penetration occurs, and combined with the relative motion velocity between collision objects obtained from real-time twin state data and advanced rolling simulation, and the mass and moment of inertia parameters of collision objects obtained from high-fidelity digital twin model parameters, the instantaneous conflict risk value at each discrete moment on the future simulation time axis is calculated; the calculation is performed pair by pair and time point by time. The calculation of the instantaneous conflict risk value integrates the following three dimensions: The first dimension is a geometric risk factor based on the minimum approach distance. This geometric risk factor maps the minimum approach distance to a normalized value greater than or equal to zero and less than or equal to one through a negative exponential function with the natural constant e as the base and the product of the minimum approach distance and the distance sensitivity coefficient as the exponent. This ensures that the geometric risk factor approaches one sharply when the minimum approach distance approaches zero, thus assigning a highly sensitive nonlinear risk weight to the close-range state. The distance sensitivity coefficient controls the rate at which the geometric risk factor decays with the minimum approach distance, and its typical value ranges from five to twenty per meter. A larger value indicates greater sensitivity to distance. When the minimum approach distance is zero or the penetration depth is greater than zero, the geometric risk factor directly takes the maximum value of one. The second dimension is a motion risk factor based on the relative velocity of the colliding objects in the normal direction of the contact point, which is used to quantify the kinetic energy intensity when a potential collision occurs; the relative velocity is calculated through vector projection. The third dimension is the effective mass energy factor derived from the simplified mass model of the collision object, which is used to assess the magnitude of the impact force that a potential collision may transmit. The effective mass energy factor is calculated based on the mass and moment of inertia in the model parameters of the collision object, taking into account the simplified collision geometry, and its value is positively correlated with non-zero mass. The first dimension's geometric risk factor is multiplied by a preset geometric risk weight coefficient, the second dimension's motion risk factor is multiplied by a preset motion risk weight coefficient, and the third dimension's effective mass-energy factor is multiplied by a preset energy risk weight coefficient. These three weighted values ​​are then summed to obtain the final instantaneous conflict risk value. The geometric risk weight coefficient, motion risk weight coefficient, and energy risk weight coefficient are all preset constants, and their sum is one. Their typical values ​​can be adjusted according to different emphases on safety and efficiency. For example, one configuration is: geometric risk weight coefficient of 0.6, motion risk weight coefficient of 0.3, and energy risk weight coefficient of 0.1. Within the entire future simulation time window, all calculated instantaneous conflict risk values ​​and their corresponding collision object pairs and location information are organized in chronological order to form a structured conflict risk quantification map. The conflict risk quantification map is represented in memory as a data list indexed by timestamps. Each entry contains the time, the identifier of the object involved, the calculated instantaneous conflict risk value, and the spatial coordinates of the potential contact location. This map serves as the core input of step S3, providing a clear and quantifiable optimization objective for the deep reinforcement learning policy network, namely, finding a motion strategy that minimizes the sum of future conflict risks.

[0022] In this embodiment, it is specifically necessary to explain the following steps in step S3: receiving the conflict risk quantification map output from step S2 and the real-time twin state data output from step S1, and inputting them into a pre-trained deep reinforcement learning policy network. First, features are extracted and encoded from the real-time twin state data and the conflict risk quantification map, respectively. The encoding process for real-time twin state data is as follows: extract state information from the chassis and lifting mechanism states of the bricklaying robot virtual model, the joint and link states of the dual-arm virtual model, and the 3D dense semantic voxel map of the working environment virtual model contained in the real-time twin state data. This includes the global position and attitude of the mobile chassis component of the bricklaying robot, the position and attitude of each joint and end effector of the dual-arm, and the direction and distance features of nearby obstacles calculated from the 3D dense semantic voxel map. Input these features into a multilayer perceptron network composed of multiple fully connected layers and nonlinear activation functions. The multilayer perceptron network typically contains three to five hidden layers. The activation function can be a rectified linear unit. Through forward propagation calculation, the high-dimensional, heterogeneous real-time twin state information is mapped into a low-dimensional, dense current state feature vector. The dimension of this vector can be set according to the network capacity, such as 128 dimensions or 256 dimensions. The state information is then fused and compressed into a fixed-dimensional current state feature vector through calculation. The encoding process for the conflict risk quantification map is as follows: The conflict risk quantification map is regarded as a data sequence arranged in chronological order. Each element of the data sequence contains the instantaneous conflict risk value at a given future moment and its corresponding collision object identifier and location information. This data sequence is input into an encoder of a temporal convolutional network or recurrent neural network. If a recurrent neural network encoder is used, a long short-term memory network or gated recurrent unit is preferred to better model the long-term dependence of risk in time. The encoder processes the data sequence step by step in time, and the final hidden state or pooled features are used as the future risk feature vector. The encoder captures the evolution trend of future risks in the time dimension and the distribution pattern in space through its temporal modeling capability, and outputs a future risk feature vector with a fixed dimension. Subsequently, the encoded current state feature vector and future risk feature vector are concatenated to form a comprehensive state representation vector. The concatenation operation links the two feature vectors end to end in terms of dimension, forming a new vector with a higher dimension. For example, if the current state feature vector is 128-dimensional and the future risk feature vector is 64-dimensional, then the comprehensive state representation vector is 192-dimensional. The comprehensive state representation vector carries the complete static and dynamic information of the digital twin environment at the current moment, as well as the predictive assessment information of the risk of motion conflict in the future. The comprehensive state representation vector is used as the input of the pre-trained deep reinforcement learning policy network. The specific process for generating optimized control commands to adjust the spatial trajectory of the dual robotic arms joints is as follows: After receiving the comprehensive state representation vector, the deep reinforcement learning policy network performs calculations and inferences through its internal multi-layer neural network, and directly outputs an original action vector. The dimension of the original action vector is the same as the number of all controllable joints of the dual robotic arm. Each element in the vector corresponds to the expected adjustment amount of a joint in the next control cycle. The policy network itself can be a multilayer perceptron, whose output layer neurons have the same number as the action dimension, and the output layer activation function is usually a linear activation function. To ensure that the output motion meets the actual range of motion and speed limits of the robotic arm joints, the original motion vector is post-processed. The post-processing process is as follows: First, the hyperbolic tangent function is used to compress each element value of the original motion vector to between negative one and positive one. Then, the compressed original motion vector is multiplied element-wise with a preset motion scaling vector. Each component of the motion scaling vector represents the maximum position or speed adjustment allowed for the corresponding joint. The value of each component of the motion scaling vector is determined according to the physical performance manual of the specific robotic arm, such as the maximum angular velocity limit of each joint. After this scaling and limiting process, the final optimized control command is obtained. The pre-training of the deep reinforcement learning policy network is completed in the high-fidelity digital twin virtual environment constructed and maintained in step S1. The training process models the collaborative bricklaying operation of the two robotic arms as a sequential decision-making process, and learns the policy through the interaction between the policy model and the digital twin environment. The training adopts advanced deep reinforcement learning algorithms such as proximal policy optimization or deep deterministic policy gradient, and performs millions to tens of millions of interactive sampling steps in the digital twin environment until the policy performance converges. The reward function used in the training consists of four parts: the first part is the task progress reward, which provides a positive incentive when the robotic arm successfully completes a cycle of picking up a brick, applying mortar, and placing the brick; the task progress reward is a sparse reward, which is given a large positive value, such as positive ten, only when a key milestone is completed. The second part is the risk aversion reward, used to drive the policy network to actively avoid high-risk areas. Its calculation process is as follows: First, based on the conflict risk quantification map output in step S2, all instantaneous conflict risk values ​​within a preset time window are extracted. The length of the preset time window is usually consistent with or slightly shorter than the prediction time domain in step S2, for example, 0.5 seconds to 1.5 seconds. The maximum value is selected from the instantaneous conflict risk values ​​and defined as the highest risk value at that moment. Next, the highest risk values ​​at all moments within the future time window are accumulated to obtain a cumulative risk value. Then, a risk sensitivity coefficient is introduced and multiplied by the cumulative risk value. The risk sensitivity coefficient is used to adjust the risk level. The penalty intensity for risk typically ranges from 0.1 to 1.0, and can be adjusted based on the balance between exploration and safety in the early stages of training. Finally, the product result is substituted into a logarithmic function with the natural constant e as the base, and one is added before the logarithmic operation. The specific calculation process is: base negative natural constant e, one plus the logarithmic value of the product of the risk sensitivity coefficient and the cumulative risk value. Using a logarithmic form instead of a linear penalty can effectively handle situations where the risk value varies greatly. When the cumulative risk value is small, the penalty increases slowly, encouraging exploration; when the cumulative risk value increases sharply, the logarithmic penalty will be amplified dramatically, forcing the strategy to avoid risk. This non-linear characteristic enhances the safety and stability of learning. The third part is the motion smoothness reward, which is used to punish drastic changes in control commands between adjacent time steps in order to encourage the generation of smooth and continuous motion. The motion smoothness reward is specifically calculated as the square of the Euclidean norm of the optimized control command vector between two adjacent time steps, multiplied by a smoothness weight coefficient. The fourth part is the dual-arm coordination smoothness reward. The dual-arm coordination smoothness reward incentivizes the two robotic arms to maintain a coordinated work rhythm by comparing the average time it takes for the two robotic arms to complete a single work cycle, reducing mutual waiting time and improving overall work efficiency. The formula for calculating the dual-arm coordination smoothness reward is: the coordination weight coefficient multiplied by (one minus the ratio of the absolute value of the difference in the two-arm work cycle time to the total time of the two-arm work cycle). This value reaches its maximum when the two arms are fully synchronized. The training objective of the deep reinforcement learning policy network is to maximize the sum of long-term cumulative rewards obtained by performing actions in the digital twin environment by adjusting the network parameters. Guided by these four reward components, the policy network learns autonomously in the digital twin environment through trial and error that it can efficiently complete tasks, actively and smoothly avoid various conflicts predicted in step S2, and promote the matching of the work rhythms of the two robotic arms.

[0023] In this embodiment, it is specifically necessary to explain the process in step S4, which involves comparing the state feedback data with the predicted state data generated by simulation calculation based on optimized control instructions in the high-fidelity digital twin, and calculating the difference between the two as the model error. First, the physical state data collected from the actual operation of the bricklaying robot and measured by sensors is processed. The physical state data includes encoder data of each joint of the dual robotic arms, joint torque sensor data, inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism. By analyzing and transforming these multimodal sensor data, including converting the encoder data from pulse count to angle, denoising and integrating the inertial measurement unit data, and uniformly transforming all data to the body coordinate system centered on the mobile chassis of the bricklaying robot, state feedback data corresponding to the real-time twin state data structure in step S1 is generated. The state feedback data specifically includes the actual pose and velocity of the mobile chassis of the bricklaying robot, and the actual angle and actual end pose of each joint of the dual robotic arms. Meanwhile, in the high-fidelity digital twin, the latest real-time twin state data output in step S1 is used as the initial state, and the optimized control command generated in step S3 is used as the input of the virtual model. A single-step or short-term dynamic forward simulation is performed through the physics engine. The duration of the single-step simulation is usually consistent with the control cycle, such as 0.01 seconds or 0.02 seconds. The physics engine can be a commercial or open-source simulation software based on the Newton-Euler equation or Lagrange mechanics. It calculates the predicted state data that the bricklaying robot virtual model and the dual-arm virtual model should have after executing the optimized control command. The predicted state data includes the predicted pose and velocity of the moving chassis of the bricklaying robot virtual model, and the predicted angles and predicted end poses of each joint of the dual-arm virtual model. Subsequently, the state feedback data and the corresponding observable state component in the predicted state data are subtracted element by element to obtain the original difference vector. To more accurately characterize the degree of model mismatch, a confidence weight is assigned to each component of the original difference vector based on the historical accuracy and reliability of each sensor measurement. The confidence weight can be determined based on the accuracy index calibrated by the sensor at the factory or by calculating the inverse of the variance of the sensor data over a recent period online. The smaller the variance of the sensor data, the higher the confidence weight is assigned. Components with high confidence weights represent more reliable measurements and should have a greater weight in error assessment. Multiplying the original differences of all components by the square root of their corresponding confidence weights creates a weighted residual vector. Multiplying each difference component by the square root of its confidence weight is equivalent to constructing a diagonal matrix with the reciprocals of the confidence weights as its diagonal elements. The square root of this matrix is ​​then used to left-multiply the original difference vector. This is a standard method for whitening observation uncertainty. This weighted residual vector is the quantitative representation of the model error after normalization and reliability weighting. It comprehensively reflects the overall degree of inconsistency between the prediction of the high-fidelity digital twin and the actual response of the physical bricklaying robot across all key observation dimensions, taking into account different observation confidence levels. The specific process of online calibration of model parameters used to describe the dynamics of virtual models in a high-fidelity digital twin using model errors is as follows: First, the sensitivity matrix of the weighted residual vector relative to the model parameters in the high-fidelity digital twin is calculated. Each row of the sensitivity matrix corresponds to a component of the weighted residual vector, and each column corresponds to an adjustable parameter in the dynamically adjustable parameter set. Each element value in the sensitivity matrix represents the amount of change in the corresponding weighted residual component caused by a small change in a certain model parameter. The sensitivity matrix is ​​obtained by automatic differentiation or by calculating the pre-programmed analytical derivative based on the physical model. If automatic differentiation is used, gradient tracking mode is usually enabled in the simulation calculation graph, and all elements of the sensitivity matrix can be calculated at once through a single forward simulation and backpropagation. Next, a parameter update equation is constructed based on the idea of ​​regularized online natural gradient optimization. Specifically, the transpose of the sensitivity matrix is ​​multiplied by the sensitivity matrix itself to obtain an information matrix. The information matrix characterizes the constraint strength of the current observation data on the uncertainty of the model parameters. Building upon this, a first regularization coefficient multiplied by the identity matrix is ​​introduced and added to the information matrix. This first regularization coefficient is a small positive number, typically ranging from 1 to 10^-6 to 1 to 10^-3. It is used to ensure the information matrix remains invertible under ill-conditioned conditions, prevent numerical computational instability when observational information is insufficient, and ensure the mathematical solvability of the parameter update equations. Simultaneously, a structured prior matrix is ​​introduced and added to the information matrix. This structured prior matrix is ​​a diagonal matrix, and each element on the diagonal corresponds to a model parameter. In the physical world, the prior knowledge of the predictable rate of change is inversely proportional to the value of the prior matrix elements. For example, for parameters like the joint friction coefficient, which may increase slowly over time, a larger prior matrix element value is assigned to limit the magnitude of each update. For relatively stable parameters like load mass, a smaller prior matrix element value is assigned to allow for more flexible adjustments based on observations. This constrains the direction and magnitude of parameter updates, ensuring they conform to physical common sense and preventing parameter drift to unreasonable numerical ranges. A new composite matrix is ​​formed by adding the information matrix, the product of the first regularization coefficient and the identity matrix, and the structured prior matrix. Then, the inverse of the composite matrix is ​​calculated. Subsequently, the transpose of the sensitivity matrix is ​​multiplied by the weighted residual vector to obtain a gradient vector, which indicates the direction in which the model parameters should be adjusted to reduce the current weighted residual. The calculated inverse matrix is ​​multiplied by the gradient vector to obtain a model parameter update vector, which indicates the optimal adjustment amount and direction of the dynamically adjustable parameter set in this iteration after considering the current observation constraints, numerical stability, and physical prior knowledge. This update direction is essentially the "natural gradient" of the original gradient direction in the metric space defined by the composite matrix. It takes into account the curvature of the parameter space itself (through the information matrix) and prior constraints, and usually produces more stable and efficient parameter updates than ordinary gradient descent. Finally, the model parameter update vector is multiplied by a preset learning rate, which is a decimal between zero and one. The typical range of the learning rate is 0.001 to 0.1, and the specific value can be adjusted according to the dynamic characteristics of the system and the required convergence speed. This value is used to control the step size of the parameter update and achieve smooth and gradual adjustment. The resulting scaled update vector is then added element-wise to the current dynamically adjustable parameter set in the high-fidelity digital twin, completing an online iterative calibration of the dynamically adjustable parameter set. The calibrated dynamically adjustable parameter set is immediately used to update the dynamic properties of the high-fidelity digital twin, enabling it to more accurately predict the physical response of the bricklaying robot in subsequent actions. This online iterative calibration mechanism constitutes a real-time adaptive loop that can continuously compensate for model mismatch caused by mechanical wear, temperature changes, load variations, etc., thereby maintaining the high fidelity of the digital twin prediction in the long term and providing a fundamental guarantee for the reliability of the entire simulation optimization closed loop.

[0024] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0029] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0030] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A simulation and optimization method for indoor bricklaying operations based on digital twins, characterized in that, Specifically, the steps include the following: Step S1: In response to the start operation command of the bricklaying robot, the multimodal sensor data of the bricklaying robot and the visual perception data of the working environment are collected synchronously. Data fusion and state estimation are performed on the multimodal sensor data and the visual perception data. Based on the fusion and estimation results, a high-fidelity digital twin containing the virtual model of the bricklaying robot, the virtual model of the dual robotic arms and the virtual model of the working environment is driven and updated. The high-fidelity digital twin contains model parameters used to describe the dynamics of the virtual model and generates real-time twin state data. Step S2: In the high-fidelity digital twin, based on the real-time twin state data output in step S1, advance rolling simulation is performed on the pre-planned action sequence of the dual robotic arms. During the simulation, continuous collision detection is performed on the virtual model of the bricklaying robot, the virtual model of the dual robotic arms, and the virtual model of the working environment, and the geometric interference between them is calculated. Based on the geometric interference, the conflict risk quantification map within the future time window is calculated and output. Step S3: Receive the conflict risk quantification map output in step S2 and the real-time twin state data output in step S1, and input them into a pre-trained deep reinforcement learning policy network. The deep reinforcement learning policy network makes online decisions and generates optimized control commands for adjusting the joint spatial trajectory of the dual robotic arms. Step S4: Send the optimized control command generated in step S3 to the bricklaying robot for execution. At the same time, collect the physical state data measured by sensors during the actual operation of the bricklaying robot as state feedback data. Compare the state feedback data with the predicted state data generated by simulation calculation based on the optimized control command in the high-fidelity digital twin. Calculate the difference between the two as the model error. Use the model error to calibrate the model parameters in the high-fidelity digital twin that describe the dynamics of the virtual model online, so as to reduce the difference between the system state represented by the high-fidelity digital twin and the real physical state of the bricklaying robot.

2. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 1, characterized in that: In step S1, the multimodal sensor data specifically includes encoder data of each joint of the dual robotic arms, joint torque sensor data, wheel odometer data and inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism; the visual perception data specifically includes point cloud data and color image data of the working environment collected by depth camera and color camera.

3. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 2, characterized in that: The specific operations for data fusion and state estimation of multimodal sensor data and visual perception data are as follows: The wheeled odometry data, inertial measurement unit data, and visual odometry data derived from visual perception data of the mobile chassis of the bricklaying robot are fused using a tightly coupled graph optimization model to estimate the pose and velocity of the mobile chassis of the bricklaying robot in the global coordinate system. The tightly coupled graph optimization model achieves state estimation by constructing and minimizing a loss function, which is composed of the inertial measurement unit pre-integration constraint residual, the visual odometry relative pose constraint residual, and the loop closure detection constraint residual. During the optimization process, a time-varying adaptive weighting factor is introduced into the visual odometry relative pose constraint residual. The adaptive weighting factor is dynamically calculated and adjusted according to the magnitude of the covariance of the reprojection error generated by the visual odometry in the most recent time window. Meanwhile, a robust kernel function is applied to the loop closure detection constraint residuals, and finally, the optimized pose and velocity estimates of the mobile chassis of the bricklaying robot are output.

4. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 3, characterized in that: The process of driving and updating a high-fidelity digital twin containing a virtual model of a bricklaying robot, a virtual model of dual robotic arms, and a virtual model of the working environment based on the fusion and estimation results is as follows: Using the pose and velocity estimates of the mobile chassis of the bricklaying robot, the global position and orientation of the mobile chassis component of the bricklaying robot virtual model in the high-fidelity digital twin are updated. Using encoder data of each joint of the dual robotic arms obtained from multimodal sensor data, the position and orientation of each link and end effector of the dual robotic arm virtual model in the high-fidelity digital twin are updated through forward kinematics calculation; using displacement sensor data of the two-stage lifting mechanism obtained from multimodal sensor data, the extension height of the lifting mechanism component of the bricklaying robot virtual model in the high-fidelity digital twin is updated. Simultaneously, the depth point cloud and color image in the visual perception data are processed in real time. First, the continuously collected depth point cloud data are superimposed and aligned in a unified world coordinate system through a point cloud registration algorithm. Then, the registered depth point cloud data is input into a pre-trained semantic segmentation network for pixel-level classification. Based on the classification results, the occupancy probability of each three-dimensional spatial unit is updated and semantic labels are assigned. A three-dimensional dense semantic voxel map representing the working environment is dynamically constructed and updated. Each voxel unit of the three-dimensional dense semantic voxel map contains its spatial location information, the probability value of being occupied by obstacles, and semantic category labels. This continuously updated three-dimensional dense semantic voxel map is used as a virtual model of the working environment in a high-fidelity digital twin. During this process, the high-fidelity digital twin synchronously maintains and updates the model parameters of the wall-building robot virtual model, the dual-arm virtual model, and the working environment virtual model in the computer memory. The model parameters include kinematic constraint parameters that are consistent with the physical chassis for the mobile chassis model, mass, moment of inertia, joint friction coefficient, and damping coefficient configured for the two-stage lifting mechanism model, geometric shape and mass attributes configured for each link of the dual-arm virtual model, transmission stiffness and damping parameters configured for each joint, and physical friction coefficient configured for the obstacle surface in the working environment virtual model. All model parameters together constitute a dynamically adjustable parameter set. Finally, by integrating the chassis and lifting mechanism status of the wall-building robot virtual model, the status of all joints and links of the dual robotic arm virtual model, and the three-dimensional dense semantic voxel map of the working environment virtual model, real-time twin status data is generated.

5. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 4, characterized in that: In step S2, based on the initial state of the virtual model represented by the real-time twin state data output in step S1, the pre-planned action sequence of the dual robotic arms is used as the control input. In the high-fidelity digital twin, the dynamics are forward-engineered at a speed faster than the actual operation to simulate the continuous motion process of the wall-building robot virtual model, the dual robotic arm virtual model, and the work environment virtual model within a fixed period of time in the future. Within each calculation step of this advanced rolling simulation, continuous collision detection is performed between each pair of components in the virtual model of the bricklaying robot: the mobile chassis component, the lifting mechanism component, each link and end effector in the virtual model of the dual robotic arms, and the wall and obstacle surfaces in the virtual model of the working environment. Continuous collision detection calculates the spatial interference between the motion trajectory sweep of all moving parts and other parts or environment models from the current simulation time to the next simulation time, and outputs the minimum approach distance between each pair of potential collision objects, as well as the motion trajectory penetration depth if geometric penetration occurs.

6. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 5, characterized in that: The specific operation for calculating and outputting the quantitative map of conflict risk within the future time window based on geometric interference is as follows: Based on the minimum approach distance between each pair of potential collision objects output by continuous collision detection, the motion trajectory penetration depth information when geometric penetration occurs, and combined with the relative motion velocity between collision objects obtained from real-time twin state data and advanced rolling simulation, and the mass and moment of inertia parameters of collision objects obtained from high-fidelity digital twin model parameters, the instantaneous conflict risk value at each discrete moment on the future simulation time axis is calculated. The calculation of the instantaneous conflict risk value integrates the following three dimensions: The first dimension is a geometric risk factor based on the minimum proximity distance. This geometric risk factor maps the minimum proximity distance to a normalized value that is greater than or equal to zero and less than or equal to one through a negative exponential function with the natural constant e as the base and the product of the minimum proximity distance and the distance sensitivity coefficient as the exponent. The second dimension is a motion risk factor based on the relative velocity of the colliding objects in the direction of the normal to the point of contact. The third dimension is the effective mass energy factor derived from the simplified mass model of the collision object; Multiply the first dimension's geometric risk factor by a preset geometric risk weight coefficient, multiply the second dimension's motion risk factor by a preset motion risk weight coefficient, multiply the third dimension's effective mass-energy factor by a preset energy risk weight coefficient, and sum these three weighted values ​​to obtain the final instantaneous conflict risk value. Within the entire future simulation time window, all calculated instantaneous conflict risk values ​​and their corresponding collision object pairs and location information are organized in chronological order to form a structured conflict risk quantification map.

7. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 6, characterized in that: In step S3, the specific operation of receiving the conflict risk quantification map output from step S2 and the real-time twin state data output from step S1, and inputting them into a pre-trained deep reinforcement learning policy network is as follows: First, features are extracted and encoded from the real-time twin state data and the conflict risk quantification map, respectively. The encoding process for real-time twin state data is as follows: extract state information from the chassis and lifting mechanism states of the bricklaying robot virtual model, the joint and link states of the dual robotic arm virtual model, and the three-dimensional dense semantic voxel map of the working environment virtual model contained in the real-time twin state data. This includes the global position and attitude of the mobile chassis component of the bricklaying robot, the position and attitude of each joint and end effector of the dual robotic arm, and the direction and distance features of nearby obstacles calculated from the three-dimensional dense semantic voxel map. Input these features into a multilayer perceptron network composed of multiple fully connected layers and nonlinear activation functions. The state information is then fused and compressed into a fixed-dimensional current state feature vector through calculation. The encoding process for the conflict risk quantification map is as follows: the conflict risk quantification map is regarded as a data sequence arranged in chronological order. Each element of the data sequence contains the instantaneous conflict risk value at a given future moment and its corresponding collision object identification and location information. This data sequence is input into an encoder of a temporal convolutional network or recurrent neural network. The encoder captures the evolution trend of future risks in the time dimension and the distribution pattern in the space through its temporal modeling capability, and outputs a future risk feature vector with a fixed dimension. Subsequently, the current state feature vector obtained by encoding is concatenated with the future risk feature vector to form a comprehensive state representation vector.

8. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 7, characterized in that: The specific process for generating optimized control commands to adjust the spatial trajectory of the dual robotic arms joints is as follows: After receiving the comprehensive state representation vector, the deep reinforcement learning policy network performs calculations and inferences through its internal multi-layer neural network and directly outputs an original action vector. To ensure that the output motion meets the actual range of motion and speed limits of the robotic arm joints, the original motion vector is post-processed. The post-processing process is as follows: First, the hyperbolic tangent function is used to compress each element value of the original motion vector to between negative one and positive one. Then, the compressed original motion vector is multiplied element by element by a preset motion scaling vector. Each component of the motion scaling vector represents the maximum position or speed adjustment allowed for the corresponding joint. After this scaling and limiting process, the final optimized control command is obtained. The pre-training of the deep reinforcement learning policy network is completed in the high-fidelity digital twin virtual environment constructed and maintained in step S1. The training process models the collaborative bricklaying operation of the two robotic arms as a sequential decision-making process, and learns the policy through the interaction between the policy model and the digital twin environment. The reward function used in the training consists of task progress reward, risk avoidance reward, motion smoothness reward, and bi-arm coordination fluency reward; the training objective of the deep reinforcement learning policy network is to maximize the sum of long-term cumulative rewards obtained by performing actions in the digital twin environment by adjusting the network parameters.

9. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 8, characterized in that: In step S4, the specific process of comparing the state feedback data with the predicted state data generated by simulation calculation based on optimized control commands in the high-fidelity digital twin, and calculating the difference between the two as the model error, is as follows: First, the physical state data collected from the actual operation of the bricklaying robot and measured by sensors are processed. The physical state data includes encoder data of each joint of the dual robotic arms, joint torque sensor data, inertial measurement unit data of the mobile chassis of the bricklaying robot, and displacement sensor data of the two-stage lifting mechanism. By parsing and transforming these multimodal sensor data, state feedback data corresponding to the real-time twin state data structure in step S1 is generated. The state feedback data specifically includes the actual pose and speed of the mobile chassis of the bricklaying robot, and the actual angle and actual end pose of each joint of the dual robotic arms. Meanwhile, in the high-fidelity digital twin, the latest real-time twin state data output in step S1 is used as the initial state, and the optimized control command generated in step S3 is used as the input of the virtual model. A single-step or short-term dynamic forward simulation is performed through the physics engine to calculate the predicted state data that the bricklaying robot virtual model and the dual-arm virtual model should have after the optimized control command is executed. The predicted state data includes the predicted pose and velocity of the moving chassis of the bricklaying robot virtual model, and the predicted angles and predicted end poses of each joint of the dual-arm virtual model. Subsequently, the state feedback data and the corresponding observable state component in the predicted state data are subtracted element by element to obtain the original difference vector. Based on the historical accuracy and reliability of each sensor measurement, a confidence weight is assigned to each component of the original difference vector; Multiply the original differences of all components by the square root of their corresponding confidence weights to form a weighted residual vector.

10. The method for simulation and optimization of indoor bricklaying operations based on digital twins according to claim 9, characterized in that: The specific process of online calibration of model parameters used to describe the dynamics of virtual models in a high-fidelity digital twin using model errors is as follows: First, calculate the sensitivity matrix of the weighted residual vector relative to the model parameters in the high-fidelity digital twin. Each row of the sensitivity matrix corresponds to a component of the weighted residual vector, and each column corresponds to an adjustable parameter in the set of dynamically adjustable parameters. Next, a parameter update equation is constructed, which is to multiply the transpose of the sensitivity matrix by the sensitivity matrix itself to obtain an information matrix. Based on this, a product of a first regularization coefficient and an identity matrix is ​​introduced and added to the information matrix; at the same time, a structured prior matrix is ​​introduced and added to the information matrix; the information matrix, the product of the first regularization coefficient and the identity matrix, and the structured prior matrix are added together to form a new composite matrix. Then, the inverse of the composite matrix is ​​calculated; subsequently, the transpose of the sensitivity matrix is ​​multiplied by the weighted residual vector to obtain a gradient vector; the calculated inverse matrix is ​​multiplied by the gradient vector to obtain a model parameter update vector. Finally, the model parameter update vector is multiplied by a preset learning rate; the resulting scaled update vector is then added element-wise to the current dynamically adjustable parameter set in the high-fidelity digital twin to complete an online iterative calibration of the dynamically adjustable parameter set. The calibrated dynamically adjustable parameter set will be immediately used to update the dynamic properties of the high-fidelity digital twin.

Images