Calibration method for robot motion process and electronic device

Abstract

The application provides a robot motion process calibration method and an electronic device. The method comprises: collecting multi-source synchronous data in a robot motion process to obtain a time-aligned first multi-source synchronous data frame; preprocessing the first multi-source synchronous data frame to obtain a second multi-source data frame; updating a first particle map of a current scene to a second particle map based on first image data in the second multi-source data frame, a robot state vector group and a second sliding window data set, and determining a plurality of stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set; performing real-time particle state prediction based on basic parameters of the robot and the stable observation anchor particle set, and determining an error between a predicted pose and an actual pose to obtain a real-time error residual vector; and correcting a pose of an end effector of the robot according to the real-time error residual vector to achieve dynamic calibration control.

Description

Technical Field

[0001] This application belongs to the field of robot control technology, specifically relating to a calibration method and electronic equipment for robot motion processes. Background Technology

[0002] In the field of robotics, pre-task calibration refers to the core technical means of identifying and compensating for geometric errors of the links, joint zero-point offsets, and visual calibration deviations in the multi-degree-of-freedom full-body structure and onboard visual perception system of the robot, in order to achieve precise and controllable accuracy in whole-body motion, end-effector manipulation, and visual positioning. Robots are characterized by multi-joint serial / parallel structures, dynamic full-body walking, frequent posture switching, real-time load changes with posture, significant body vibration and deformation, and onboard cameras that sway with the torso. During production assembly and final installation, there are inherent errors in the installation position of each link, installation rotation errors, joint mechanical zero-point offsets, and deviations in DH kinematic parameters. Furthermore, during long-term walking and operation, dynamic drift errors are continuously generated due to joint temperature drift, torso deformation, dynamic load changes, vibration and impact, and slight shifts in the pose of the onboard camera.

[0003] Currently, most robot calibration algorithms are offline static calibration modes, which can only complete parameter calibration when the robot is stationary. They cannot adapt to parameter drift problems caused by joint temperature drift, load changes, and slight changes in camera pose during the robot's dynamic movement. At the same time, the parameters calibrated offline cannot be corrected online, resulting in a continuous decay of calibration accuracy after the robot has been running for a long time, and it is impossible to maintain accuracy throughout the entire life cycle. Summary of the Invention

[0004] This application provides a calibration method and electronic device for robot motion processes, aiming to achieve dynamic calibration during robot motion.

[0005] Firstly, this application provides a calibration method for robot motion processes, comprising: Collect multi-source synchronization data during the robot's movement and perform time alignment to obtain the first time-aligned multi-source synchronization data frame; The first multi-source synchronous data frame is preprocessed to obtain a second multi-source data frame; the second multi-source data frame includes first image data, robot state vector group and second sliding window dataset; Based on the first image data, the robot state vector group, and the second sliding window dataset, the first particle map of the current scene is updated to the second particle map, and multiple stable observation anchor particles are determined from the second particle map to obtain a stable observation anchor particle set. Based on the robot's basic parameters and the stable observation anchor particle set, real-time particle state prediction is performed, and the error between the predicted pose and the actual pose is determined to obtain the real-time error residual vector. The pose of the robot end effector is corrected based on the real-time error residual vector to achieve dynamic calibration control.

[0006] In conjunction with the first aspect, in one possible embodiment, the robot state vector set includes joint angle vectors, joint temperature vectors, and joint torque vectors; preprocessing the first multi-source synchronous data frame to obtain a second multi-source data frame includes: acquiring a dual-loop calibration hyperparameter set, a camera intrinsic parameter matrix, and camera distortion coefficients; performing camera lens distortion correction on the first RGB original image based on the camera intrinsic parameter matrix and the distortion coefficients, and performing Gaussian filtering and adaptive white balance correction to obtain first image data; performing sliding mean filtering on the joint angle vectors, and low-pass filtering on the joint temperature vectors and joint torque vectors to obtain the robot state vector set; adding the first image data and the robot state vector set to a first sliding window dataset, and updating the first sliding window dataset to obtain a second sliding window dataset.

[0007] In conjunction with the first aspect, in one possible embodiment, updating the first particle map of the current scene to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset, and determining multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set, includes: extracting a set of two-dimensional feature points from the first image data based on the ORB feature extraction algorithm; matching each two-dimensional feature point in the set of two-dimensional feature points with historical feature points of multiple keyframes in the second sliding window dataset to obtain multiple sets of matching point combinations, each set of matching point combinations including the association relationship between the two-dimensional feature points and the historical feature points; performing three-dimensional spatial coordinate calculation based on the multiple sets of matching point combinations corresponding to each two-dimensional feature point and the camera intrinsic parameter matrix to obtain an observation particle set, the observation particle set including the three-dimensional spatial coordinates corresponding to multiple two-dimensional feature points; fusing the observation particle set with the first particle map of the current scene to update the observation area corresponding to the observation particle set in the particle map to obtain an updated second particle map; performing a consistency score on each second particle in the second particle map, and filtering out third particles with a consistency score less than a preset threshold to obtain a stable observation anchor particle set.

[0008] In conjunction with the first aspect, in one possible embodiment, the real-time particle state prediction based on the robot's basic parameters and the stable observation anchor particle set, and the determination of the error between the predicted pose and the actual pose to obtain a real-time error residual vector, includes: obtaining a first global calibration parameter; the first global calibration parameter is generated by the previous calibration cycle; inputting the first global calibration parameter and the joint angle vector into the improved MDH forward kinematics model, calculating the homogeneous transformation matrix of the robot end effector relative to the base coordinate system; the homogeneous transformation matrix is ​​used to indicate the pose of the robot end effector; projecting each third particle in the stable observation anchor particle set onto the plane corresponding to the first image data to obtain a two-dimensional predicted projection point set; extracting a two-dimensional true feature point set from the first image data, calculating the chamfer distance between the two-dimensional predicted projection point set and the two-dimensional true feature point set to obtain a real-time error residual vector; and decomposing the real-time error residual vector into joint kinematic error components and camera extrinsic parameter error components based on the Jacobian matrix.

[0009] In conjunction with the first aspect, in one possible embodiment, the step of correcting the pose of the robot end effector based on the real-time error residual vector to achieve dynamic calibration control includes: constructing a particle-filtered state-space model of the robot end effector pose using the robot motion sub-model; predicting the prior state particle set at the current moment using the robot motion sub-model based on the first pose correction amount of the previous calibration cycle and the robot motion model, wherein the prior state particle set includes the prior states of all third particles in the stable observation anchor particle set; calculating the weight of each third particle using the observation sub-model with the real-time error residual vector as the observation value; selecting third particles with weights greater than a first threshold as high-weight particles through particle filtering resampling, and calculating the current optimal end effector pose correction amount based on the high-weight particles; converting the optimal end effector pose correction amount into a joint control correction amount through the inverse kinematics module, and directly sending it to the servo controller to correct the robot motion trajectory in real time.

[0010] In conjunction with the first aspect, in one possible embodiment, the method further includes: iteratively updating the first global calibration parameter based on the multi-frame cumulative error using an optimization algorithm to complete the outer loop parameter calibration; performing compliance verification and amplitude limitation on the updated first global calibration parameter, and saving the updated first global calibration parameter after confirming that it is correct; storing the running data of this dynamic calibration, monitoring the device status, executing the loop operation logic, jumping to the step of acquiring the first multi-source synchronous data frame, and continuously executing the next dynamic calibration loop.

[0011] In conjunction with the first aspect, in one possible embodiment, the step of iteratively updating the first global calibration parameter based on multi-frame cumulative error using an optimization algorithm to complete the outer loop parameter calibration includes: determining whether the outer loop global parameter update condition is met; if the outer loop global parameter update condition is met, constructing a global optimization objective function for all frames in the second sliding window, the objective function being used to minimize the cumulative chamfer distance of all frames in the second sliding window, the cumulative chamfer distance referring to the sum of the chamfer distances of all frames in the second sliding window; iteratively optimizing and solving the objective function to obtain the globally optimal calibration parameter and the optimized objective function value, the globally optimal calibration parameter referring to the updated first global calibration parameter; if the outer loop global parameter update condition is not met, keeping the current first global calibration parameter unchanged.

[0012] In conjunction with the first aspect, in one possible embodiment, the step of performing compliance verification and amplitude limitation on the updated first global calibration parameters, and saving the updated first global calibration parameters after confirming that they are correct, includes: verifying the validity of the globally optimal calibration parameters; recalculating the average projection error within the second sliding window dataset based on the globally optimal calibration parameters; if the average projection error is greater than a preset accuracy threshold, then determining that the globally optimal calibration parameters are invalid, immediately freezing the global parameter update, outputting a parameter abnormality alarm signal, and keeping the original first global calibration parameters unchanged; if the average projection error is less than or equal to the preset accuracy threshold, then determining that the globally optimal calibration parameters are valid, and determining that the globally optimal calibration parameters are valid. The system checks whether the change in calibration parameters exceeds a preset safety threshold. If the change exceeds the preset safety threshold, the change is limited to within the preset safety threshold to prevent parameter jumps from causing robot motion loss of control. The globally optimal calibration parameters that pass the safety check are decomposed to obtain first update parameters for multiple joints and second update parameters for the camera. Based on the first update parameters and the second update parameters, a first write instruction and a second write instruction are generated, and the first write instruction is sent to the robot servo control cabinet and the second write instruction is sent to the camera parameter storage unit to complete the online, non-stop update of robot kinematic parameters and camera extrinsic parameters. At the same time, the full data of this parameter update is recorded.

[0013] In conjunction with the first aspect, in one possible embodiment, the steps of storing the running data of this dynamic calibration, monitoring the device status, executing the loop operation logic, jumping to the acquisition of the first multi-source synchronous data frame, and continuously executing the next dynamic calibration cycle include: real-time monitoring of the robot calibration status, recording and updating the error residual sequence, parameter change curve, and calibration accuracy data, and generating a visualized calibration status interface; wherein, the error residual sequence is a dataset formed by arranging the real-time error residual vectors of each frame image within the second sliding window in chronological order; the calibration accuracy data includes the average projection error corresponding to the error residual sequence; the parameter change curve includes the error change curve and the calibration parameter change curve, the error change curve being generated based on the error residual sequence, and the calibration parameter change curve being generated by the calibration accuracy data; fully storing the calibration data, running log, and parameter update records of this cycle to construct a robot full lifecycle calibration database; executing the loop operation logic, jumping to the acquisition of the first multi-source synchronous data frame, and continuously executing the entire process of the next calibration cycle, thereby realizing continuous online real-time calibration during the robot's dynamic motion process.

[0014] Secondly, this application provides a calibration device for robot motion processes, comprising: The acquisition unit is used to acquire multi-source synchronization data during the robot's movement and perform time alignment to obtain the first time-aligned multi-source synchronization data frame. The preprocessing unit is used to preprocess the first multi-source synchronous data frame to obtain a second multi-source data frame; the second multi-source data frame includes first image data, robot state vector group and second sliding window dataset. The first calibration processing unit is used to update the first particle map of the current scene to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset, and to determine multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set; to perform real-time particle state prediction based on the robot's basic parameters and the stable observation anchor particle set, and to determine the error between the predicted pose and the actual pose to obtain a real-time error residual vector; and to correct the pose of the robot's end effector according to the real-time error residual vector to achieve dynamic calibration control.

[0015] Thirdly, this application provides an electronic device including a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of any one of the first to third aspects of this application.

[0016] Fourthly, this application provides a computer-readable storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in any one of the first to third aspects of this application.

[0017] Fifthly, this application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps described in any one of the first to third aspects of this application. The computer program product may be a software installation package.

[0018] As can be seen, compared with the current offline static calibration mode, the calibration method of robot motion process in this application simultaneously collects multi-source data during robot motion, preprocesses the multi-source data, builds a particle map based on the preprocessed multi-source data, generates predicted pose and actual pose based on the data in the particle map, calculates the error between the predicted pose and the actual pose, and finally adjusts the pose in real time based on the error. This breaks through the limitations of traditional offline static calibration and realizes online real-time calibration during the robot's dynamic motion process. It can correct parameter drift caused by joint temperature drift, load changes, and slight changes in camera pose in real time, improve the positioning and operation accuracy retention rate of the robot during long-term operation, and solve the problem of continuous accuracy decay caused by the inability to correct offline calibration parameters online. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a system block diagram of a robot provided in an embodiment of this application; Figure 2 This is a structural diagram of a robot provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the updating of a robot's second sliding window dataset according to an embodiment of this application; Figure 4 This is a flowchart illustrating the calibration method for robot motion provided in an embodiment of this application; Figure 5 This is a schematic block diagram of the particle filter state-space model provided in the embodiments of this application; Figure 6This is a schematic diagram illustrating the generation process of the second particle map provided in an embodiment of this application; Figure 7 This is a schematic diagram of the screening process for stable observation anchor particles provided in an embodiment of this application; Figure 8 This is a schematic block diagram of the structure of the first calibration device for robot motion provided in the embodiments of this application; Figure 9 This is a schematic block diagram of the structure of the second type of calibration device for robot motion provided in the embodiments of this application; Figure 10 This is a schematic block diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0022] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, systems, products, or apparatuses.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Currently, most robot calibration algorithms are offline static calibration modes, which can only complete parameter calibration when the robot is stationary. They cannot adapt to parameter drift problems caused by joint temperature drift, load changes, and slight changes in camera pose during the robot's dynamic movement. At the same time, the parameters calibrated offline cannot be corrected online, resulting in a continuous decay of calibration accuracy after the robot has been running for a long time, and it is impossible to maintain accuracy throughout the entire life cycle.

[0025] To address the aforementioned issues, this application provides a calibration method for robot motion processes. This calibration method can be applied to scenarios involving dynamic pose calibration during robot motion. It involves simultaneously acquiring multi-source data during motion, preprocessing the multi-source data, constructing a particle map based on the preprocessed data, generating predicted and actual poses based on the data in the particle map, calculating the error between the predicted and actual poses, and finally adjusting the pose in real time based on the error. This method overcomes the limitations of traditional offline static calibration, achieving online real-time calibration during robot dynamic motion. It can correct parameter drift caused by joint temperature drift, load changes, and slight changes in camera pose in real time, improving the long-term positioning and operational accuracy retention rate of the robot and solving the problem of continuous accuracy decay caused by the inability to correct offline calibration parameters online. This solution is applicable to various scenarios, including but not limited to the applications mentioned above.

[0026] The system architecture involved in the embodiments of this application is described below.

[0027] Please see Figure 1 and Figure 2 This application provides a robot 10, including a camera module 11 and multiple joints 12. The number of cameras and the number of joints can be selected as needed and are not limited to a single type.

[0028] The robot 10 includes a camera module 11, a temperature sensor module, a torque sensor module, an angle sensor module, and an end effector. The camera module includes a first camera and a second camera to acquire visual information. The temperature sensor module, torque sensor module, and angle sensor module are each equipped with multiple sets of corresponding sensors to collect joint temperature, force, and rotation angle data. All sensor and visual signals are used to control the end effector at the bottom to ensure the overall operation accuracy of the robot.

[0029] For example, the temperature sensor module includes multiple temperature sensors, such as Figure 1 The first temperature sensor, the second temperature sensor, and so on up to the m-th temperature sensor are included; the torque sensor module includes multiple torque sensors, such as... Figure 1 The first torque sensor, the second torque sensor, and so on up to the h-th torque sensor are included; the angle sensor module includes multiple joint encoders, such as... Figure 1 The first joint encoder, the second joint encoder, and the b-th joint encoder are included.

[0030] The specific methods will be described in detail below.

[0031] Please see Figure 4 This application also provides a calibration method for robot motion processes, including: Step S401: Collect multi-source synchronization data during the robot's movement and perform time alignment to obtain the first time-aligned multi-source synchronization data frame; In practice, before performing multi-source synchronous data acquisition, the robot needs to be powered on. After the robot is powered on, all electrical components, including the robot body, servo drivers, two RGB cameras, edge computing host, time synchronization controller, and data acquisition card, are powered on. The power supply, network connection, EtherCAT (Ethernet for Control Automation Technology) communication, camera trigger circuit, and sensor data feedback links are checked in sequence to ensure they are normal. After confirming that all hardware is error-free and data can be sent and received normally, the hardware layer initialization is completed.

[0032] Obtain the robot's basic parameters, including the robot's factory-set Denavit parameters (DH), the camera's factory-calibrated intrinsic parameter matrix, the initial RGB image sequence of the calibration scene, and the system accuracy threshold. Among these, the robot's factory-set DH (Denavit) parameters... Hartenberg, Denawi The Hartenberg parameter specifically refers to MDH (Modified) DH, Improved De Nave Hartenberg parameters. All basic parameters are factory presets / algorithm presets; the robot's factory DH parameters and camera intrinsic parameter matrix are factory calibration parameters; the initial RGB image sequence of the calibration scene is the initial data acquired in the current scene; the system accuracy threshold is a fixed parameter preset by the algorithm.

[0033] Specifically, after obtaining the basic parameters, perform the following operations: First, we construct a global parameter mathematical vector for the robot. This global parameter mathematical vector is a vector of dimension 'a', where 'a' can be any value, determined based on the robot's physical structure. For example, assuming a joint includes 5 calibration parameters (2 installation position errors, 2 installation rotation errors, and 1 zero-point error), then based on the number of joints 'b', we can obtain 5b dimensions. The number of calibration parameters for each joint can be freely set as needed, and is not limited to 5 here. Similarly, corresponding calibration parameters can be set for devices such as cameras and radar. Ultimately, all calibration parameters constitute the global parameter mathematical vector. The traditional offline global calibration method is called. With the robot in a static state, the robot's factory DH parameters are used for basic calibration to obtain a set of initial reference parameters. These initial reference parameters are filled into the framework of the global parameter mathematical vector to form the initial global calibration parameter vector (initial zero error compensation state, providing a parameter carrier for subsequent iterations). Based on the initial RGB image sequence, image feature points are extracted, and then processed through multi-view... Figure 3 An angularization algorithm recovers 3D spatial points; these 3D points are treated as static particles and fixed as the scene's baseline map, with the total number of particles... The determination can be made based on the actual situation, and no uniqueness is required here, to form an initial static particle map of the scene (to provide a scene benchmark for subsequent real-time particle matching). Configure and solidify all algorithm parameters to obtain a dual-loop calibration hyperparameter set. The dual-loop calibration hyperparameter set includes parameters such as inner loop frequency, outer loop frequency, sliding window length, error residual threshold, temperature mutation threshold, load mutation threshold, particle filter state dimension, and number of cross-entropy optimization samples.

[0034] Finally, the initial global calibration parameter vector and the initial scene static particle map are obtained. The robot's software parameters are initialized using a dual-loop calibration hyperparameter set and a camera intrinsic parameter matrix. This includes the initial global calibration parameter vector. The initial static particle map of the scene is used as the global calibration parameter vector and particle map when the robot performs dynamic calibration for the first time. In subsequent iterations, the global calibration parameter vector and particle map obtained in the previous round of dynamic calibration are used to participate in the calibration.

[0035] As can be seen, in this embodiment, the entire machine completes power-on self-test, EtherCAT industrial bus link detection, and time synchronization of multiple sensors and vision units. At the same time, the sliding window, communication configuration, and basic operating parameters are initialized. This not only troubleshoots hardware faults and ensures smooth communication links for each unit, but also achieves nanosecond-level timing alignment for all network devices. It completely avoids the original errors caused by timing discrepancies and communication anomalies, and builds a stable and standard basic operating environment for the entire data acquisition, calibration calculation, and motion control process.

[0036] Furthermore, in one possible embodiment, the process of acquiring multi-source synchronous data and performing time alignment during the robot's dynamic motion to obtain a time-aligned first multi-source data frame includes: the sampling frequency of synchronizing robot joint data, camera images, temperature sensor data, and torque sensor data; acquiring joint angle vectors, joint temperature vectors, joint torque vectors, and a first RGB original image at the current moment in real time according to the sampling frequency; and adding a timestamp to each set of acquired data to obtain the first multi-source data frame with the same timestamp.

[0037] In the specific implementation, after initialization, the robot's preset hardware synchronization configuration parameters are first obtained. The sampling frequency of the first multi-source synchronization data is determined based on the dual-loop calibration hyperparameter set and the hardware synchronization configuration parameters. First, the required sampling and update frequencies (inner loop frequency, outer loop frequency, camera frame rate, joint sampling rate, etc.) are read from the dual-loop calibration hyperparameter set to determine the acquisition rate of all devices. The synchronization controller is configured according to the hardware synchronization configuration parameters to ensure that all hardware triggers sampling at the same time. It is understandable that... Specifically, the hardware synchronization configuration parameters specify the data sampling frequency for multiple sources, including but not limited to robot servo motors, dual cameras, temperature sensors, torque sensors, and other structures, components, or devices. To integrate the data collected in each frame, the sampling frequencies of these multiple sources need to be synchronized. Specifically, based on the input hardware synchronization parameters, the time synchronization controller is set to output synchronization pulses. These pulses are simultaneously connected to the robot servo motors, dual cameras, temperature sensors, and torque sensors to ensure that all devices begin sampling at the same time.

[0038] The robot is driven to execute dynamic operation commands and its motion status is fed back in real time to ensure normal robot movement. During robot movement, the joint angle vector, joint temperature vector, joint torque vector, and first RGB raw image (including first RGB raw sub-images) are acquired in real time through synchronous trigger signals. Among them, the joint angle vector is the feedback value of the robot's full-joint encoder, the joint temperature vector is acquired by the temperature sensor, the joint torque vector is acquired by the torque sensor, and the RGB raw image is acquired by the dual cameras.

[0039] Finally, all data collected at the same time are added with the same timestamp to complete the spatiotemporal binding of single-frame multi-source data (i.e., within the same time period), forming a complete frame of multi-source data, and obtaining the first multi-source synchronous data frame.

[0040] As can be seen, in this embodiment, multi-source data such as robot joint posture, torque, temperature and visual feature points are collected in real time, and preprocessing such as filtering, noise reduction and data alignment are completed. At the same time, a continuous and effective sliding window dataset is constructed to effectively filter out invalid data generated by environmental interference and instantaneous jitter, ensuring the integrity and consistency of all input data, and improving the reliability of subsequent error calculation and parameter optimization results from the data source.

[0041] Step S402: Preprocess the first multi-source synchronization data frame to obtain a second multi-source data frame; the second multi-source data frame includes first image data, robot state vector group and second sliding window dataset; In one possible embodiment, please refer to the following: Figure 3 The robot state vector set includes joint angle vectors, joint temperature vectors, and joint torque vectors. The first multi-source synchronous data frame is preprocessed to obtain a second multi-source data frame, including: acquiring a dual-loop calibration hyperparameter set, a camera intrinsic parameter matrix, and camera distortion coefficients; performing camera lens distortion correction on the first RGB original image based on the camera intrinsic parameter matrix and the distortion coefficients, and performing Gaussian filtering and adaptive white balance correction to obtain first image data; performing sliding mean filtering on the joint angle vectors, and low-pass filtering on the joint temperature vectors and joint torque vectors to obtain the robot state vector set; adding the first image data and the robot state vector set to a first sliding window dataset, and updating the first sliding window dataset to obtain a second sliding window dataset. Figure 3 As shown, two sets of historical images after acquisition time are listed, namely the historical image set at time t-1 and the historical image set at time t. The current time is time t. The first image dataset acquired at time t-1 forms the first sliding window dataset. The first sliding window dataset includes X-1 first image data. After the first image data of frame t is acquired at time t, the first image data of frame t is added to the first sliding window dataset, so that the sliding window moves forward one step and forms the second sliding window dataset.

[0042] In specific implementation, the distortion coefficients are optical distortion correction parameters calibrated at the camera's factory, used in conjunction with the camera's intrinsic parameter matrix. They include radial and tangential distortion coefficients, used to eliminate errors such as image edge curvature and positional shift caused by lens optical distortion. These are fixed parameters of the device and do not participate in online iterative updates. The first sliding window dataset is the sliding window dataset obtained in the previous calibration cycle.

[0043] Based on camera intrinsic parameter matrix With distortion coefficients, lens distortion correction is performed on the first RGB original image, image noise is eliminated by using a 5×5 Gaussian filter kernel, and illumination color shift is corrected by adaptive white balance to obtain the preprocessed first image data, which includes first image sub-data and second image sub-data. The joint angle vector in the first multi-source synchronous data frame is subjected to sliding mean filtering with a sliding window size of 3 to remove jitter noise and outliers; and the joint temperature vector and joint torque vector are subjected to first-order low-pass filtering with a cutoff frequency of 10Hz to eliminate high-frequency noise, resulting in a preprocessed robot state vector group (including preprocessed joint angle vector, joint temperature vector and joint torque vector). Based on the sliding window mechanism, the first image data and the robot state vector group are added to the sliding window dataset. If the number of data frames in the window exceeds the preset number of frames, the earliest frame of data in the window is discarded to complete the dynamic update of the sliding window and obtain the updated sliding window (i.e., the second sliding window dataset). The window always contains only the most recent preset number of valid data.

[0044] As can be seen, in this embodiment, the single-frame projection error is calculated in real time based on the kinematic and visual projection model, and high-frequency inner loop pose closed-loop correction is performed. This can quickly compensate for the instantaneous positioning deviation and small dynamic error that occur during the robot's movement. Relying on high responsiveness, the real-time positioning accuracy of the robot during operation is continuously stabilized, forming the inner loop control link responsible for rapid correction in the entire calibration system.

[0045] Step S403: Based on the first image data, the robot state vector group and the second sliding window dataset, update the first particle map of the current scene to the second particle map, and determine multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set. In one possible embodiment, please refer to the following: Figure 6 and Figure 7 The process of updating the first particle map of the current scene to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset, and determining multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set, includes: extracting a set of two-dimensional feature points from the first image data based on the ORB feature extraction algorithm; matching each two-dimensional feature point in the set with historical feature points of multiple keyframes in the second sliding window dataset to obtain multiple sets of matching point combinations, each set including the association between the two-dimensional feature points and the historical feature points; performing three-dimensional spatial coordinate calculation based on the multiple sets of matching point combinations corresponding to each two-dimensional feature point and the camera intrinsic parameter matrix to obtain an observation particle set, the observation particle set including the three-dimensional spatial coordinates corresponding to multiple two-dimensional feature points; fusing the observation particle set with the first particle map of the current scene to update the observation area corresponding to the observation particle set in the particle map to obtain an updated second particle map; performing a consistency score on each second particle in the second particle map, and selecting third particles with a consistency score less than a preset threshold to obtain a stable observation anchor particle set.

[0046] In the specific implementation, the first image data, the updated sliding window, the current scene static particle map, and the dual closed-loop calibration hyperparameter set are obtained; First, keyframes related to and adjacent to the current frame are identified within the updated sliding window. Then, the ORB feature extraction algorithm extracts multiple stable image feature points (i.e., 2D feature points) from the first image data (i.e., the image data of the current frame), resulting in a set of 2D feature points. The ORB feature extraction algorithm is used to quickly and stably obtain image feature points, i.e., 2D feature points, in 3D positioning. Specifically, the ORB feature extraction algorithm traces the 3D particles in the first image data back to the feature points corresponding to the current frame (generating the original 2D features of the particle), generating an ORB descriptor to obtain the 2D feature points. A fixed-length binary number is generated for each 3D particle in the first image data (e.g., corners, edges, texture points); this binary number is the ORB descriptor. ORB is a binary local feature descriptor used to characterize the inherent properties of each 2D feature point, which remain unchanged by rotation, scaling, and brightness changes. It is the most commonly used real-time feature in robot vision, SLAM, and dynamic calibration. It is understandable that other extraction methods capable of achieving the same or corresponding functions can also be used.

[0047] Then, the keyframes adjacent to the current frame are traversed. Each two-dimensional feature point in the set of two-dimensional feature points is matched with the feature points of the adjacent keyframe images in the updated sliding window. This ensures that each two-dimensional feature point matches multiple historical feature points (real feature points), resulting in multiple sets of matching points between the two-dimensional feature point and multiple historical feature points. Specifically, a keyframe in the updated sliding window refers to the image frame retained after removing motion blur, abnormal lighting, and duplicate / redundant frames. Matching involves comparing the ORB descriptor of each two-dimensional feature with the ORB descriptors of feature points in adjacent keyframes. The more similar the binary numbers are, the more likely the two feature points are the same physical point, i.e., they are identified as historical feature points of the two-dimensional feature point. Multiple historical feature points are identified from multiple adjacent keyframes.

[0048] Furthermore, based on multiple sets of matching point combinations and the camera intrinsic parameter matrix, a triangulation algorithm is used to calculate the three-dimensional spatial coordinates corresponding to each two-dimensional feature point, generating the observation particle set for the current frame. The triangulation algorithm involves projecting the same three-dimensional spatial point onto two camera images from different viewpoints to obtain two two-dimensional feature points, and then using geometric relationships to deduce the true coordinates of this point in three-dimensional space. The triangulation algorithm can be used to calculate the true coordinates of this point in three-dimensional space based on the two-dimensional feature points and the camera intrinsic parameter matrix. The three-dimensional spatial coordinates corresponding to the two-dimensional coordinate point are calculated.

[0049] Furthermore, an incremental multi-view particle fusion method is adopted to fuse the observed particle set of the current frame (current calibration period) with the first particle map of the previous calibration period. Only the newly appearing particles in the current frame are added to the first particle map, and the positions of the particles in the historically observed areas are optimized. The updated second particle map can be obtained without global map redrawing.

[0050] Furthermore, for each second particle (which is a three-dimensional particle) in the second particle map, its multi-view observation consistency score is calculated, and the lower the observation consistency score, the higher the particle stability.

[0051] Specifically, for the successfully matched i-th keyframe, based on the camera imaging model and the robot's current calibration parameters, the second particle is projected forward onto the image plane of the i-th keyframe to obtain the i-th predicted projection point. Then, the L2 distance between the i-th predicted projection point and the i-th historical feature point in the i-th keyframe is calculated to obtain the first error distance. After traversing all keyframes and obtaining the first error distances for all keyframes, the first error distances of the i-th keyframes are accumulated to obtain the sum of the first error distances. For example, the first error distance + the second error distance + ... + the i-th error distance = the sum of the first error distances. Finally, the sum of the first error distances is divided by the number of effective matching frames (i.e., i-th frames) to obtain the average error, i.e., the particle observation consistency score.

[0052] Finally, the third particle with a consistency score of less than 0.5 pixels was selected as the stable observation anchor particle set, while the interference particles of dynamic objects in the second particle map were removed to obtain the final incrementally updated static particle map of the scene.

[0053] As can be seen, in this embodiment, dynamically updating the visual particle map and completing the feature point matching operation can effectively resist the impact of changes in lighting and robot posture on visual recognition, enhance the robustness and continuity of visual positioning, ensure stable and reliable visual feature extraction during long-term operation of the device, and provide a continuous and effective visual data source for error calculation and parameter calibration.

[0054] Step S404: Based on the robot's basic parameters and the stable observation anchor particle set, perform real-time particle state prediction, determine the error between the predicted pose and the actual pose, and obtain the real-time error residual vector. In one possible embodiment, the real-time particle state prediction based on the robot's basic parameters and the stable observation anchor particle set, and the determination of the error between the predicted pose and the actual pose to obtain a real-time error residual vector, includes: obtaining a first global calibration parameter; the first global calibration parameter is generated by the previous calibration cycle; inputting the first global calibration parameter and the joint angle vector into the improved MDH forward kinematics model, and calculating a first homogeneous transformation matrix of the robot end effector relative to the base coordinate system; the homogeneous transformation matrix is ​​used to indicate the pose of the robot end effector. Project each third particle in the stable observation anchor particle set onto the plane corresponding to the first image data to obtain a two-dimensional predicted projection point set; Extract a set of two-dimensional real feature points from the first image data, calculate the chamfer distance between the set of two-dimensional predicted projection points and the set of two-dimensional real feature points, and obtain the real-time error residual vector; The real-time error residual vector is decomposed into joint kinematic error components and camera extrinsic error components based on the Jacobian matrix.

[0055] In the specific implementation, the stable observation anchor point particle set, the first global calibration parameters, the robot state vector group and the camera intrinsic parameter matrix are obtained; among them, the first global calibration parameters are generated after dynamic calibration based on the first multi-source synchronous data frame of the previous sampling period.

[0056] For example, first, read the joint angle vectors of each end effector of the robot to obtain a joint angle vector sequence; input this joint angle vector sequence and the first global calibration parameter into the improved MDH forward kinematics model; optionally, the first global calibration parameter contains a calibration vector of a dimensions corresponding to each joint. Extract the calibration vectors of a dimensions corresponding to each of the b joints from the first global calibration parameter to obtain a×b sets of calibration vectors. Input the a×b sets of calibration vectors and the b joint angle vectors into the improved MDH forward kinematics model to calculate the homogeneous transformation matrix of each joint. The improved MDH forward kinematics model calculates the homogeneous transformation matrix of each robot end effector (i.e., each joint) from the base coordinate system to the robot end effector coordinate system, obtaining multiple homogeneous transformation matrices; multiply these multiple homogeneous transformation matrices together to obtain the second homogeneous transformation matrix from the robot base coordinate system (world coordinate system) to the end effector coordinate system.

[0057] Furthermore, based on the robot's current end-effector pose (i.e., the second transformation matrix from the base coordinate system to the robot's end-effector coordinate system), and combined with the fixed installation relationship between the camera and the end-effector (i.e., the transformation matrix from the robot's end-effector coordinate system to the camera coordinate system), the two transformation matrices are multiplied through coordinate transformation cascade calculation to obtain the extrinsic parameter matrix from the robot's base coordinate system to the camera coordinate system. This extrinsic parameter matrix includes multiple joints.

[0058] The extrinsic parameter matrix is ​​multiplied by each third particle in the stable observation anchor particle set to obtain the first spatial coordinate vector of the third particle transformed from the world coordinate system to the camera coordinate system. Since the first spatial coordinate vector may contain non-coordinate parameters, the spatial coordinate vector in the first spatial coordinate vector is extracted to obtain the second spatial coordinate vector containing only spatial coordinates. Finally, the second spatial coordinate vector is multiplied by the camera intrinsic parameter matrix to obtain the two-dimensional predicted projection point of each third particle. Multiple two-dimensional predicted projection points form a two-dimensional predicted projection point set.

[0059] It is understood that the calculation process of the above-mentioned two-dimensional predicted projection point set can be obtained based on the pinhole camera imaging model. The calculation process of the pinhole camera imaging model is the same as the above calculation process. It is used to describe the central perspective projection mapping relationship from three-dimensional spatial points to two-dimensional image planes. In this embodiment, it is used for two-dimensional projection calculation and error solving of three-dimensional particles.

[0060] Extract all two-dimensional real feature points from the current frame image (i.e., the first image data) to form a two-dimensional real feature point set; calculate the Chamfer distance (i.e., chamfer distance, matching error) between the two-dimensional predicted projection point set and the two-dimensional real feature point set to obtain the real-time error residual vector. Finally, the real-time error residual vector is decomposed into joint kinematic error components and camera extrinsic error components using the Jacobian matrix, providing multi-dimensional error feedback for dual closed-loop calibration, separating robot error and camera error for easy separate calibration.

[0061] As can be seen, in this embodiment, the error residuals of all frames within the sliding window are summarized, statistically analyzed, and distinguished between instantaneous disturbance errors and systematic deviations caused by long-term operation of the equipment. The overall deviation status of the whole machine is quantified, providing comprehensive and objective data basis for whether the outer loop global parameters are updated and for parameter iteration optimization, thus connecting the two major links of inner loop real-time correction and outer loop global calibration.

[0062] Step S405: Correct the pose of the robot end effector according to the real-time error residual vector to achieve dynamic calibration control.

[0063] In one possible embodiment, please refer to the following: Figure 5The step of correcting the pose of the robot end effector based on the real-time error residual vector to achieve dynamic calibration control includes: constructing a particle filter state space model 50 for the robot end effector pose; the particle filter state space model 50 includes a robot motion sub-model 51 and an observation sub-model 52; the robot motion sub-model 51 predicts the prior state particle set at the current moment based on the first pose correction amount of the previous calibration cycle and the robot motion model; the observation sub-model 52 calculates the weight of each particle based on the prior state particle set and the real-time error residual vector as the observation value; through particle filter resampling, the third particle with a weight greater than a first threshold is selected as a high-weight particle, and the current optimal end effector pose correction amount is calculated based on the high-weight particles; the optimal end effector pose correction amount is converted into a joint control correction amount through the inverse kinematics module and directly sent to the servo controller to correct the robot motion trajectory in real time.

[0064] In the specific implementation, the real-time error residual vector, the robot's real-time motion control command, the robot's state vector set, and the dual-closed-loop calibration hyperparameter set are obtained; a particle filter state space model 50 of the pose of the 6-DOF robot end effector is constructed, and multiple vectors are defined as pose correction quantities of the robot end effector, including 3 translation correction quantities and 3 rotation correction quantities; among them, the translation correction quantity is used to compensate for the deviation of the robot end effector's linear position in three-dimensional space, describing the position offset compensation of the end effector along the three orthogonal axes of the Cartesian coordinate system; the rotation correction quantity is used to compensate for the deviation of the robot end effector's attitude (orientation, angle), describing the rotation angle compensation of the end effector around the three orthogonal coordinate axes, also called attitude correction quantity; the common expression for industrial robots is Euler angle (roll angle, pitch angle, yaw angle).

[0065] Furthermore, the first pose correction of all particles in the previous calibration cycle and the robot motion control commands in the current calibration cycle are read. The robot motion sub-model 51 calls the robot motion state transition function and, combined with the first pose correction and the robot real-time motion control commands, calculates the basic prediction value of each third particle. The robot motion sub-model 51 is a mathematical model describing the robot end effector pose correction (i.e., the first pose correction) in the previous calibration cycle, the current control commands (robot real-time motion control commands), and deriving the pose correction at the current moment (current calibration cycle). This model simulates the state evolution of the robot end effector in continuous motion and introduces motion noise to simulate uncertain disturbances such as mechanical jitter and transmission backlash. The basic prediction value of each third particle is superimposed with process noise following a zero-mean Gaussian distribution to obtain the prior state of each third particle at the current moment. This completes the prediction of all third particles, resulting in the prior state particle set.

[0066] Furthermore, the real-time error residual vector of the current frame is read as the observation value; the observation likelihood function is constructed based on a Gaussian distribution by the observation sub-model 52, and the prior state of each third particle is substituted into it to calculate the observation likelihood value corresponding to each third particle. This value is used as the weight of the third particle; the smaller the error residual, the higher the particle weight. A resampling operation is performed on the weight of each third particle, removing low-weight particles, copying and retaining high-weight particles, and updating the particle set; for example, weights less than the first threshold are removed, and weights greater than or equal to the first threshold are retained. A weighted sum is performed on all resampled particles to calculate the optimal end effector pose correction amount of the robot at the current time.

[0067] The optimal end-effector pose correction is substituted into the robot's inverse kinematics model to obtain the control correction for each joint. These joint control corrections are then superimposed on the original motion control commands to generate corrected real-time joint control commands. These new commands are then sent to the robot servo system at a preset operating frequency (e.g., 100Hz, or other frequencies) to complete the inner-loop high-frequency pose correction. The inverse kinematics model and the forward kinematics model are inverse operations of each other. The forward kinematics model calculates the actual pose of the robot's end-effector given the joint angles, while the inverse kinematics model calculates the required rotation angles of each joint given the target end-effector pose.

[0068] As can be seen, in this embodiment, by combining the robot's operating conditions such as temperature and load with error thresholds, it is possible to intelligently determine whether to initiate the outer-loop global parameter update process and trigger parameter optimization actions as needed. This avoids the waste of system resources caused by frequent parameter updates and can respond in a timely manner to systemic accuracy degradation caused by mechanical temperature drift and load changes, thereby achieving reasonable control over calibration timing and balancing system operating efficiency with the timeliness of accuracy compensation.

[0069] In one possible embodiment, the method further includes: Based on the accumulated error of multiple frames, an optimization algorithm is used to iteratively update the first global calibration parameters to complete the outer loop parameter calibration; The updated first global calibration parameters are subjected to compliance verification and amplitude limitation. After confirming that there are no errors, the updated first global calibration parameters are safely saved. Store the running data of this dynamic calibration, monitor the equipment status, execute the loop operation logic, jump to the step of collecting the first multi-source synchronous data frame, and continue to execute the next dynamic calibration loop.

[0070] In one possible embodiment, based on the cumulative error of multiple frames, the first global calibration parameters are iteratively updated using an optimization algorithm to complete the outer loop parameter calibration. This includes: iteratively updating the first global calibration parameters based on the cumulative error of multiple frames using an optimization algorithm to complete the outer loop parameter calibration; performing compliance verification and amplitude limitation on the updated parameters, and safely saving the first global calibration parameters after confirming that they are correct; storing the running data of this dynamic calibration, monitoring the device status, executing the loop operation logic, jumping to the step of acquiring the first multi-source synchronous data frame, and continuously executing the next dynamic calibration loop.

[0071] Specifically, the step of iteratively updating the first global calibration parameters based on multi-frame cumulative error using an optimization algorithm to complete the outer loop parameter calibration includes: determining whether the outer loop global parameter update condition is met; if the outer loop global parameter update condition is met, constructing a global optimization objective function for all frames in the second sliding window, wherein the global optimization objective function is used to minimize the cumulative chamfer distance of all frames in the second sliding window, wherein the cumulative chamfer distance refers to the sum of the chamfer distances of all frames in the second sliding window; iteratively optimizing and solving the objective function to obtain the updated first global calibration parameters and the optimized objective function value; if the outer loop global parameter update condition is not met, keeping the current first global calibration parameters unchanged.

[0072] In the specific implementation, the following are obtained: joint kinematic error components, camera extrinsic error components, robot real-time state vector set, first global calibration parameters, second sliding window dataset, and dual closed-loop calibration hyperparameter set; Obtain the real-time error residual vector of N consecutive frames (including the current frame) within the second sliding window dataset, and calculate the average error residual of the N consecutive frames; calculate the joint temperature change of the current frame (specifically obtained by subtracting the joint temperature vector of the current frame from the joint temperature vector of the previous week), and calculate the joint load change of the current frame (specifically obtained by subtracting the joint torque vector of the current frame from the joint torque vector of the previous week).

[0073] Determine the conditions for updating the outer loop global parameters. If any one of the outer loop global parameter update conditions is met, the outer loop global parameter update is initiated; otherwise, the global parameters remain unchanged. The outer loop global parameter update conditions include: Condition 1: The average residual error of N consecutive frames is greater than 0.8 pixels; Condition 2: Joint temperature change > 5℃; Condition 3: Joint load change > 20%; Condition 4: Reaching the fixed update cycle.

[0074] If an outer-loop global parameter update is triggered, the objective function for global optimization is constructed with the goal of minimizing the cumulative Chamfer distance across all frames within the sliding window. Specifically, all valid frames within the sliding window dataset are traversed, and the ground truth feature point set corresponding to each frame is extracted. The first global calibration parameter is substituted into the objective function, and the predicted projection point set is calculated frame by frame, combining forward kinematics and the camera projection model. The Chamfer distance operator is called to calculate the matching error (i.e., Chamfer distance) between the predicted point set and the ground truth feature point set for each frame. The Chamfer distances of all frames within the sliding window are accumulated to obtain the objective function value corresponding to the current first global calibration parameter. A smaller objective function value indicates higher accuracy of the current calibration parameter.

[0075] Based on the global calibration parameters of the previous frame, a Gaussian sampling distribution is constructed. ;in The mean of the distribution is... Given the covariance matrix. Determine the hyperparameters. Group; by hyperparameter Randomly sample from this Gaussian distribution to generate A sample of candidate calibration parameters.

[0076] For each group of candidate calibration parameter samples, calculate the objective function value for each group of candidate samples to obtain the objective function value corresponding to each group of candidate calibration parameter samples; sort all samples according to the objective function value from smallest to largest, and select the candidate calibration parameter samples with the smallest objective function value as elite samples; based on the selected elite samples, recalculate the mean and covariance matrix of the elite samples to obtain new mean and new covariance matrix; replace the original distribution parameters with the new mean and new covariance matrix to obtain the iterated Gaussian distribution.

[0077] The process involves repeated sampling, calculation of the objective function, selection of elite samples, and updating of the Gaussian distribution. The optimization is considered complete and the iteration is terminated when the change in the objective function value between two adjacent iterations is less than the preset convergence threshold. Then, the updated parameters are output as the mean of the final Gaussian distribution, which is used as the globally optimal calibration parameters after this round of updates.

[0078] As can be seen, in this embodiment, after the update is triggered, the optimization algorithm iteratively optimizes the 153-dimensional global calibration parameters, including joints and cameras, to accurately solve the optimal parameter combination. It specifically corrects the inherent systematic deviations caused by mechanical clearance, component aging, and camera pose offset, thereby improving the overall positioning accuracy of the robot in long-term operation from the root. This serves as the core link of the outer loop calibration responsible for eradicating deviations in the entire system.

[0079] In one possible embodiment, the step of performing compliance verification and amplitude limitation on the updated first global calibration parameters, and securely saving the updated first global calibration parameters after confirming that they are correct, includes: verifying the validity of the globally optimal calibration parameters; recalculating the average projection error within the second sliding window dataset based on the globally optimal calibration parameters; if the average projection error is greater than a preset accuracy threshold, then determining that the globally optimal calibration parameters are invalid, immediately freezing the global parameter update, outputting a parameter abnormality alarm signal, and keeping the original first global calibration parameters unchanged; if the average projection error is less than or equal to the preset accuracy threshold, then determining that the globally optimal calibration parameters are valid, and determining the calibration of the globally optimal calibration parameters. The system checks whether the change in calibration parameters exceeds a preset safety threshold. If the change exceeds the preset safety threshold, the change is limited to within the preset safety threshold to prevent parameter jumps from causing robot motion loss of control. The globally optimal calibration parameters that pass the safety verification are decomposed to obtain first update parameters for multiple joints and second update parameters for the camera. Based on the first update parameters and the second update parameters, a first write instruction and a second write instruction are generated, and the first write instruction is sent to the robot servo control cabinet and the second write instruction is sent to the camera parameter storage unit to complete the online, non-stop update of robot kinematic parameters and camera extrinsic parameters. At the same time, all data of this parameter update is recorded.

[0080] In the specific implementation, the system acquires globally optimal calibration parameters, real-time error residual vectors, preset safety thresholds, and robot operating status. The robot operating status is a real-time operational data set for the entire robot, including information such as joint angles, angular velocities, torques, temperatures, servo system operating conditions, safety loop status, and communication status. During parameter verification and distribution, this data is used for safety interlocking, operational condition adaptation, and anomaly interception. When an abnormal operating condition such as emergency stop, servo failure, or communication interruption is detected, parameter distribution is immediately terminated and safety protection is triggered to ensure the safe operation of the robot during online calibration.

[0081] Using the globally optimal calibration parameters as the unified calibration parameters, the entire process of forward kinematics, particle projection, and Chamfer distance calculation is re-executed for all frames in the second sliding window dataset to obtain the real-time error residual vector for each frame. The average value of all real-time error residual vectors is calculated to obtain the average projection error within the second sliding window.

[0082] Determine if the average projection error exceeds a preset accuracy threshold. If the average projection error exceeds the preset accuracy threshold, it is determined to be invalid, and a safety control signal and abnormal alarm information are generated and output. Finally, the data from the entire process of parameter verification, correction, and distribution are summarized and written into the parameter update record. If the average projection error is less than or equal to the preset accuracy threshold, proceed to the next step of parameter safety boundary verification. It is understood that the specific value of the preset accuracy threshold can be selected according to the actual situation and is not limited to a single value here.

[0083] Furthermore, the parameter safety boundary verification process is as follows: Obtain the preset safety threshold for each parameter component in the global optimal calibration parameters; calculate the change in calibration parameters of each parameter component relative to the first global calibration parameters. Perform the following processing on each parameter component: If the change in the calibration parameter of a parameter component exceeds the corresponding preset safety threshold, the parameter component is truncated and corrected to the threshold boundary value to prevent parameter jumps from causing the robot to lose control. If the change in the calibration parameter of a parameter component is within the preset safety threshold range, the original calculation result is retained and no modification is made. After verifying and correcting all parameter components of the globally optimal calibration parameters, the final calibration parameters that have passed the verification are obtained.

[0084] Furthermore, the final calibration parameters are decomposed to obtain the first updated parameters for each joint and the second updated parameters for the camera. The first updated parameters include the installation position error, installation rotation error, and zero-point error calibration value, while the second updated parameters include the installation position error, installation rotation error, and trigger zero-point error calibration value.

[0085] Then, based on the decomposed first and second update parameters, standardized parameter writing instructions conforming to the EtherCAT (Ethernet for Control Automation Technology) protocol are generated; according to the parameter application and the hardware unit to which it belongs, the first update parameters for joints are sent to the robot servo control cabinet, and the second update parameters for cameras are sent to the camera parameter storage unit; finally, a non-stop online update is performed, and the complete data of this parameter update is recorded synchronously.

[0086] After confirming the successful execution of the first and second parameter update actions, a status flag is generated and output to mark the normal completion of this round of parameter update process. This flag may include messages such as "Parameters successfully issued, process complete," or other icons, text, or language; uniqueness is not required. Finally, the data from the entire parameter verification, correction, and issuance process is summarized and written into the parameter update record.

[0087] As can be seen, in this embodiment, two levels of verification are completed successively through parameter validity judgment threshold and preset safety threshold. The parameters exceeding the limit are truncated to suppress the risk of parameter jump. Then, the qualified parameters are split and distributed online without downtime by distinguishing objects through EtherCAT bus. Relying on multiple safety protection mechanisms to eliminate invalid parameters and avoid the risk of motion loss of control, the global calibration parameters are updated and iterated smoothly under the premise of ensuring the safe operation of the equipment.

[0088] In one possible embodiment, the process includes storing the running data of this dynamic calibration, monitoring the device status, executing loop operation logic, jumping to the step of acquiring the first multi-source synchronous data frame, and continuously executing the next dynamic calibration cycle. This includes: real-time monitoring of the robot calibration status, recording and updating the error residual sequence, parameter change curves, and calibration accuracy data, and generating a visual calibration status interface. The error residual sequence is a dataset formed by arranging the real-time error residual vectors of each frame image within the second sliding window in chronological order. The calibration accuracy data includes the average projection error corresponding to the error residual sequence. The parameter change curve includes an error change curve and a calibration parameter change curve, where the error change curve is generated based on the error residual sequence, and the calibration parameter change curve is generated from the calibration accuracy data. The entire calibration data, running logs, and parameter update records of this cycle are fully stored, and the robot's full lifecycle calibration database is updated. The loop operation logic is executed, jumping to the step of acquiring the first multi-source synchronous data frame, and continuously executing the next calibration cycle, achieving continuous online real-time calibration during the robot's dynamic movement.

[0089] In practice, the system acquires globally optimal calibration parameters, error residual sequences, parameter update records, and robot operating status data. It also acquires in real-time data such as the error residual sequence generated during the current calibration cycle, changes in calibration parameters, and average projection error. Based on the error residual sequence, it dynamically plots an error variation curve, where the horizontal axis represents time / calibration cycle, and the vertical axis represents the error residual value in pixels. The curve includes a parameter preset accuracy threshold reference line to display the fluctuation of the overall positioning accuracy of the system in real time.

[0090] The calibration parameter change curve is plotted based on the change of calibration parameters. The horizontal axis is synchronized with the error curve (i.e., time / calibration period), and the vertical axis represents the physical value of the corresponding parameter. The curves are divided into joint parameter curves and camera parameter curves according to the parameter affixation, and the preset safety threshold boundaries of each dimension are marked to monitor the temporal changes, drift and truncation status of the calibration parameters.

[0091] Based on the aforementioned error residual sequence, calibration parameter changes, and average projection error, a visual calibration status interface is generated. The corresponding error change curve and calibration parameter change curve are displayed on the visual calibration status interface, intuitively presenting the equipment operating status and online calibration effect.

[0092] The calibration data, equipment operation logs, parameter update records, and error residual data generated during this calibration cycle are classified and organized; all data is persistently stored, and a full life cycle calibration database for the robot is continuously built through iteration; an online calibration report for this cycle is generated according to a standard format, and complete calibration process data is retained to support subsequent historical data backtracking, querying, and analysis.

[0093] Confirm that all calibration, verification, distribution, and log recording work for this calibration cycle is completed; issue the cycle operation instruction, and the system automatically jumps to step S401; restart the entire process of multi-source synchronous data acquisition, data preprocessing, particle map update, and dual closed-loop online calibration to achieve uninterrupted cycle operation of the entire solution.

[0094] As can be seen, in this embodiment, time-series curves are plotted based on overall calibration accuracy and calibration parameters to achieve a visual display of calibration status. At the same time, the entire process data is classified and archived into the lifecycle database and a calibration report is generated, and the operation and calibration records are completely preserved for traceability analysis. In addition, the entire process is driven to run continuously in a loop, enabling the system to achieve unattended online calibration around the clock, maintain the overall accuracy of the robot in a stable manner over a long period of time, and provide complete data support for equipment operation and maintenance and performance evaluation.

[0095] In summary, compared with the current offline static calibration mode, the robot motion calibration method in this application simultaneously collects multi-source data during robot motion, preprocesses the multi-source data, builds a particle map based on the preprocessed multi-source data, generates predicted pose and actual pose based on the data in the particle map, calculates the error between the predicted pose and the actual pose, and finally adjusts the pose in real time based on the error. This method breaks through the limitations of traditional offline static calibration, realizes online real-time calibration during robot dynamic motion, and can correct parameter drift caused by joint temperature drift, load changes, and slight changes in camera pose in real time. It improves the long-term positioning and operation accuracy retention rate of the robot and solves the problem of continuous accuracy decay caused by the inability to correct offline calibration parameters online.

[0096] The above primarily describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, mobile electronic devices include corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments provided herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0097] This application embodiment can divide the electronic device into functional units according to the above method example. For example, each function can be divided into a separate functional unit, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in hardware or as a software functional unit. It should be noted that the unit division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0098] Please see Figure 8 This application also provides a calibration device 70 for robot motion processes, comprising: The acquisition unit 71 is used to acquire multi-source synchronization data during the robot's movement and perform time alignment to obtain the first time-aligned multi-source synchronization data frame. Preprocessing unit 72 is used to preprocess the first multi-source synchronous data frame to obtain a second multi-source data frame; the second multi-source data frame includes first image data, robot state vector group and second sliding window dataset; The first calibration processing unit 73 is used to update the first particle map of the current scene to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset, and to determine multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set; to perform real-time particle state prediction based on the robot's basic parameters and the stable observation anchor particle set, and to determine the error between the predicted pose and the actual pose to obtain a real-time error residual vector; and to correct the pose of the robot end effector according to the real-time error residual vector to achieve dynamic calibration control.

[0099] In one possible embodiment, the robot state vector set includes joint angle vectors, joint temperature vectors, and joint torque vectors; preprocessing the first multi-source synchronous data frame to obtain a second multi-source data frame includes: acquiring a dual-loop calibration hyperparameter set, a camera intrinsic parameter matrix, and camera distortion coefficients; performing camera lens distortion correction on the first RGB original image based on the camera intrinsic parameter matrix and the distortion coefficients, and performing Gaussian filtering and adaptive white balance correction to obtain first image data; performing sliding mean filtering on the joint angle vectors, and low-pass filtering on the joint temperature vectors and joint torque vectors to obtain the robot state vector set; adding the first image data and the robot state vector set to a first sliding window dataset, and updating the first sliding window dataset to obtain a second sliding window dataset.

[0100] In one possible embodiment, updating the first particle map of the current scene to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset, and determining multiple stable observation anchor particles from the second particle map to obtain a stable observation anchor particle set, includes: extracting a set of two-dimensional feature points from the first image data based on the ORB feature extraction algorithm; matching each two-dimensional feature point in the set with historical feature points of multiple keyframes in the second sliding window dataset to obtain multiple sets of matching point combinations, each set including the association between the two-dimensional feature points and the historical feature points; performing three-dimensional spatial coordinate calculation based on the multiple sets of matching point combinations corresponding to each two-dimensional feature point and the camera intrinsic parameter matrix to obtain an observation particle set, the observation particle set including the three-dimensional spatial coordinates corresponding to multiple two-dimensional feature points; fusing the observation particle set with the first particle map of the current scene to update the observation area corresponding to the observation particle set in the particle map to obtain an updated second particle map; performing a consistency score on each second particle in the second particle map, and selecting third particles with a consistency score less than a preset threshold to obtain a stable observation anchor particle set.

[0101] In one possible embodiment, the real-time particle state prediction based on the robot's basic parameters and the stable observation anchor particle set, and the determination of the error between the predicted pose and the actual pose to obtain a real-time error residual vector, includes: obtaining a first global calibration parameter; the first global calibration parameter is generated by the previous calibration cycle; inputting the first global calibration parameter and the joint angle vector into the improved MDH forward kinematics model, and calculating the homogeneous transformation matrix of the robot end effector relative to the base coordinate system; the homogeneous transformation matrix is ​​used to indicate the pose of the robot end effector; projecting each third particle in the stable observation anchor particle set onto the plane corresponding to the first image data to obtain a two-dimensional predicted projection point set; extracting a two-dimensional true feature point set from the first image data, calculating the chamfer distance between the two-dimensional predicted projection point set and the two-dimensional true feature point set to obtain a real-time error residual vector; and decomposing the real-time error residual vector into joint kinematic error components and camera extrinsic parameter error components based on the Jacobian matrix.

[0102] In one possible embodiment, the step of correcting the pose of the robot end effector based on the real-time error residual vector to achieve dynamic calibration control includes: constructing a particle-filtered state-space model of the robot end effector pose using the robot motion sub-model; predicting the prior state particle set at the current moment using the robot motion sub-model based on the first pose correction amount of the previous calibration cycle and the robot motion model, wherein the prior state particle set includes the prior states of all third particles in the stable observation anchor particle set; calculating the weight of each third particle using the observation sub-model with the real-time error residual vector as the observation value; selecting third particles with weights greater than a first threshold as high-weight particles through particle filtering resampling, and calculating the current optimal end effector pose correction amount based on the high-weight particles; converting the optimal end effector pose correction amount into a joint control correction amount through the inverse kinematics module, and directly sending it to the servo controller to correct the robot's motion trajectory in real time.

[0103] In one possible embodiment, please refer to Figure 9 The calibration device 70 for the robot motion process further includes: a second calibration processing unit 74, used to iteratively update the first global calibration parameters based on the cumulative error of multiple frames using an optimization algorithm to complete the outer loop parameter calibration; and to perform compliance verification and amplitude limitation on the updated first global calibration parameters, and save the updated first global calibration parameters after confirming that they are correct; and to store the running data of this dynamic calibration, monitor the device status, execute the loop operation logic, jump to the step of collecting the first multi-source synchronous data frame, and continue to execute the next dynamic calibration loop.

[0104] In one possible embodiment, the aspect of iteratively updating the first global calibration parameter based on multi-frame cumulative error using an optimization algorithm to complete the outer loop parameter calibration, the second calibration processing unit 74 is specifically used for: determining whether the outer loop global parameter update condition is met; if the outer loop global parameter update condition is met, constructing a global optimization objective function for all frames in the second sliding window, the objective function being used to minimize the cumulative chamfer distance of all frames in the second sliding window, the cumulative chamfer distance being the sum of the chamfer distances of all frames in the second sliding window; iteratively optimizing and solving the objective function to obtain the globally optimal calibration parameter and the optimized objective function value, the globally optimal calibration parameter being the updated first global calibration parameter; if the outer loop global parameter update condition is not met, keeping the current first global calibration parameter unchanged.

[0105] In one possible embodiment, the aspect of performing compliance verification and amplitude limitation on the updated first global calibration parameters, and saving the updated first global calibration parameters after confirmation, is specifically configured by the second calibration processing unit 74 as follows: verifying the validity of the globally optimal calibration parameters; recalculating the average projection error within the second sliding window dataset based on the globally optimal calibration parameters; if the average projection error is greater than a preset accuracy threshold, determining the globally optimal calibration parameters as invalid, immediately freezing the global parameter update, outputting a parameter abnormality alarm signal, and keeping the original first global calibration parameters unchanged; if the average projection error is less than or equal to the preset accuracy threshold, determining the globally optimal calibration parameters as valid, and determining the global... The system checks whether the change in the optimal calibration parameters exceeds a preset safety threshold. If the change exceeds the preset safety threshold, the change is limited to within the threshold to prevent parameter jumps from causing robot motion loss of control. The globally optimal calibration parameters that pass safety verification are decomposed to obtain first update parameters for multiple joints and second update parameters for the camera. Based on the first and second update parameters, a first write instruction and a second write instruction are generated, and the first write instruction is sent to the robot servo control cabinet and the second write instruction is sent to the camera parameter storage unit to complete the online, non-stop update of robot kinematic parameters and camera extrinsic parameters. At the same time, all data of this parameter update is recorded.

[0106] In one possible embodiment, the steps of storing the running data of this dynamic calibration, monitoring the device status, executing the loop operation logic, jumping to the acquisition of the first multi-source synchronous data frame, and continuously executing the next dynamic calibration cycle are specifically used by the second calibration processing unit 74 to: monitor the robot calibration status in real time, record and update the error residual sequence, parameter change curve, and calibration accuracy data, and generate a visual calibration status interface; wherein, the error residual sequence is a dataset formed by arranging the real-time error residual vectors of each frame image in the second sliding window in chronological order; the calibration accuracy data includes the average projection error corresponding to the error residual sequence; the parameter change curve includes the error change curve and the calibration parameter change curve, the error change curve is generated based on the error residual sequence, and the calibration parameter change curve is generated by the calibration accuracy data; fully store the calibration data, running log, and parameter update records of this cycle to construct a robot full life cycle calibration database; execute the loop operation logic, jump to the acquisition of the first multi-source synchronous data frame, and continuously execute the entire process of the next calibration cycle to realize continuous online real-time calibration during the robot's dynamic movement.

[0107] As can be seen, compared with the current offline static calibration mode, the calibration method of robot motion process in this application simultaneously collects multi-source data during robot motion, preprocesses the multi-source data, builds a particle map based on the preprocessed multi-source data, generates predicted pose and actual pose based on the data in the particle map, calculates the error between the predicted pose and the actual pose, and finally adjusts the pose in real time based on the error. This breaks through the limitations of traditional offline static calibration and realizes online real-time calibration during the robot's dynamic motion process. It can correct parameter drift caused by joint temperature drift, load changes, and slight changes in camera pose in real time, improve the positioning and operation accuracy retention rate of the robot during long-term operation, and solve the problem of continuous accuracy decay caused by the inability to correct offline calibration parameters online.

[0108] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0109] This application also provides an electronic device 80, such as... Figure 10 As shown, it includes at least one processor 81; a display screen 82; and a memory 83, and may also include a communications interface 85 and a bus 84. The processor 81, display screen 82, memory 83, and communications interface 85 can communicate with each other via the bus 84. The display screen 82 is configured to display a preset user guide interface in the initial setup mode. The communications interface 85 can transmit information. The processor 81 can call logical instructions in the memory 83 to execute the methods described in the above embodiments.

[0110] Optionally, the electronic device 80 may be a mobile electronic device, an electronic device, or other devices, and is not limited to any particular type.

[0111] Furthermore, the logic instructions in the aforementioned memory 83 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0112] The memory 83, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 81 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 83, thereby implementing the methods in the above embodiments.

[0113] The memory 83 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the electronic device 80. Furthermore, the memory 83 may include high-speed random access memory (RAM) and may also include non-volatile memory. For example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, may be used, or they may be transient storage media.

[0114] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0115] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments. The computer program product may be a software installation package, and the computer may include an electronic device.

[0116] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0117] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatuses, and systems can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0118] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0119] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0120] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a USB flash drive, a portable hard disk, a magnetic disk, an optical disk, volatile memory, or non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), etc., which are various media capable of storing program code.

[0121] While this application discloses the above information, it is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions without departing from the spirit and scope of this application, and can make various alterations and modifications, including combinations of the different functions and implementation steps described above, as well as software and hardware implementation methods, all of which are within the protection scope of this application.

Claims

1. A calibration method for robot motion processes, characterized in that, include: Collect multi-source synchronization data during the robot's movement and perform time alignment to obtain the first time-aligned multi-source synchronization data frame; The first multi-source synchronous data frame is preprocessed to obtain a second multi-source data frame; the second multi-source data frame includes first image data, robot state vector group and second sliding window dataset; Based on the first image data, the robot state vector group, and the second sliding window dataset, the first particle map of the current scene is updated to the second particle map, and multiple stable observation anchor particles are determined from the second particle map to obtain a stable observation anchor particle set. Based on the robot's basic parameters and the stable observation anchor particle set, real-time particle state prediction is performed, and the error between the predicted pose and the actual pose is determined to obtain the real-time error residual vector. The pose of the robot end effector is corrected based on the real-time error residual vector to achieve dynamic calibration control.

2. The method according to claim 1, characterized in that, The robot state vector set includes joint angle vector, joint temperature vector, and joint torque vector; The first multi-source synchronous data frame is preprocessed to obtain the second multi-source data frame, which includes: Obtain the dual-loop calibration hyperparameter set, camera intrinsic parameter matrix, and camera distortion coefficients; Based on the camera intrinsic parameter matrix and the distortion coefficients, the first RGB original image is subjected to camera lens distortion correction, Gaussian filtering and adaptive white balance correction to obtain the first image data; The joint angle vector is subjected to sliding mean filtering, and the joint temperature vector and joint torque vector are subjected to low-pass filtering to obtain the robot state vector group; The first image data and the robot state vector group are added to the first sliding window dataset, and the first sliding window dataset is updated to obtain the second sliding window dataset.

3. The method according to claim 2, characterized in that, The first particle map of the current scene is updated to a second particle map based on the first image data, the robot state vector group, and the second sliding window dataset. Multiple stable observation anchor particles are determined from the second particle map to obtain a set of stable observation anchor particles, including: A set of two-dimensional feature points is extracted from the first image data based on the ORB feature extraction algorithm; Each two-dimensional feature point in the set of two-dimensional feature points is matched with historical feature points of multiple keyframes in the second sliding window dataset to obtain multiple sets of matching point combinations. Each set of matching point combinations includes the association relationship between the two-dimensional feature points and the historical feature points. Based on the combination of multiple matching points corresponding to each two-dimensional feature point and the camera intrinsic parameter matrix, the three-dimensional spatial coordinates are calculated to obtain the observation particle set, which includes the three-dimensional spatial coordinates corresponding to multiple two-dimensional feature points. The observed particle set is fused with the first particle map of the current scene to update the observation area corresponding to the observed particle set in the particle map, resulting in an updated second particle map. A consistency score is calculated for each second particle in the second particle map, and third particles with consistency scores less than a preset threshold are selected to obtain a stable observation anchor particle set.

4. The method according to claim 3, characterized in that, The real-time particle state prediction is performed based on the robot's basic parameters and the stable observation anchor particle set, and the error between the predicted pose and the actual pose is determined to obtain the real-time error residual vector, including: Obtain the first global calibration parameter; the first global calibration parameter is generated from the previous calibration cycle; The first global calibration parameters and the joint angle vector are input into the improved MDH forward kinematics model to calculate the homogeneous transformation matrix of the robot end effector relative to the base coordinate system; the homogeneous transformation matrix is ​​used to indicate the pose of the robot end effector. Project each third particle in the stable observation anchor particle set onto the plane corresponding to the first image data to obtain a two-dimensional predicted projection point set; Extract a set of two-dimensional real feature points from the first image data, calculate the chamfer distance between the set of two-dimensional predicted projection points and the set of two-dimensional real feature points, and obtain the real-time error residual vector; The real-time error residual vector is decomposed into joint kinematic error components and camera extrinsic error components based on the Jacobian matrix.

5. The method according to claim 4, characterized in that, The step of correcting the pose of the robot end effector based on the real-time error residual vector to achieve dynamic calibration control includes: Construct a particle-filtered state-space model of the robot's end-effector pose; the particle-filtered state-space model includes a robot motion sub-model and an observation sub-model; Based on the first pose correction value of the previous calibration cycle and the robot motion model, the robot motion sub-model predicts the prior state particle set at the current moment. The prior state particle set includes the prior states of all third particles in the stable observation anchor particle set. The weight of each third particle is calculated based on the observation sub-model using the real-time error residual vector as the observation value; By resampling with particle filtering, the third particle with a weight greater than the first threshold is selected as a high-weight particle, and the current optimal end pose correction amount is calculated based on the high-weight particle. The optimal end-effector pose correction is converted into joint control correction by the inverse kinematics module and directly sent to the servo controller to correct the robot's motion trajectory in real time.

6. The method according to claim 4 or 5, characterized in that, The method further includes: Based on the accumulated error of multiple frames, the first global calibration parameter is iteratively updated using an optimization algorithm to complete the outer loop parameter calibration; The updated first global calibration parameters are subjected to compliance verification and amplitude limitation. After confirming that there are no errors, the updated first global calibration parameters are saved. Store the running data of this dynamic calibration, monitor the equipment status, execute the loop operation logic, jump to the step of collecting the first multi-source synchronous data frame, and continue to execute the next dynamic calibration loop.

7. The method according to claim 6, characterized in that, The process of iteratively updating the first global calibration parameters based on multi-frame cumulative error using an optimization algorithm to complete the outer loop parameter calibration includes: Determine whether the conditions for updating the global parameters of the outer ring are met; If the outer ring global parameter update condition is met, then a global optimization objective function for all frames in the second sliding window is constructed. The objective function is used to minimize the cumulative chamfer distance of all frames in the second sliding window. The cumulative chamfer distance refers to the sum of the chamfer distances of all frames in the second sliding window. The objective function is iteratively optimized to obtain the globally optimal calibration parameters and the optimized objective function value. The globally optimal calibration parameters refer to the updated first global calibration parameters. If the conditions for updating the outer loop global parameters are not met, the current first global calibration parameter remains unchanged.

8. The method according to claim 7, characterized in that, The process of performing compliance verification and amplitude limiting on the updated first global calibration parameters, and saving the updated first global calibration parameters after confirming that they are correct, includes: The validity of the global optimal calibration parameters is verified, and the average projection error in the second sliding window dataset is recalculated based on the global optimal calibration parameters. If the average projection error is greater than the preset accuracy threshold, the global optimal calibration parameter is determined to be an invalid parameter, the global parameter update is immediately frozen, a parameter abnormality alarm signal is output, and the original first global calibration parameter remains unchanged. If the average projection error is less than or equal to the preset accuracy threshold, the global optimal calibration parameter is determined to be valid, and it is determined whether the change in the calibration parameter of the global optimal calibration parameter is greater than the preset safety threshold; if the change in the calibration parameter is greater than the preset safety threshold, the change in the calibration parameter is limited to within the preset safety threshold to avoid parameter jumps causing the robot to lose control of its motion. The globally optimal calibration parameters that have passed the safety verification are decomposed to obtain the first update parameters of multiple joints and the second update parameters of the camera. Based on the first update parameters and the second update parameters, a first write instruction and a second write instruction are generated respectively. The first write instruction is sent to the robot servo control cabinet and the second write instruction is sent to the camera parameter storage unit to complete the online, non-stop update of the robot kinematic parameters and camera extrinsic parameters. At the same time, the full data of this parameter update is recorded.

9. The method according to claim 8, characterized in that, The steps of storing the running data of this dynamic calibration, monitoring the device status, executing the loop operation logic, jumping to the acquisition of the first multi-source synchronous data frame, and continuously executing the next dynamic calibration loop include: The robot's calibration status is monitored in real time, and the error residual sequence, parameter change curves, and calibration accuracy data are recorded and updated to generate a visual calibration status interface. The error residual sequence is a dataset formed by arranging the real-time error residual vectors of each frame within the second sliding window in chronological order. The calibration accuracy data includes the average projection error corresponding to the error residual sequence. The parameter change curves include error change curves and calibration parameter change curves; the error change curves are generated based on the error residual sequence, and the calibration parameter change curves are generated from the calibration accuracy data. Store all calibration data, operation logs, and parameter update records for this cycle, and update the robot's full lifecycle calibration database; The loop execution logic jumps to the step of acquiring the first multi-source synchronous data frame, and continues to execute the entire process of the next calibration cycle, realizing continuous online real-time calibration during the robot's dynamic motion.

10. An electronic device, characterized in that, The method includes a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to be executed by the processor, said programs including instructions for performing the steps of the method as described in any one of claims 1-9.

Images