Joint calibration method and system for coupling errors of robot joints and cameras

Abstract

The application discloses a joint calibration method and system for coupling errors of a robot joint and a camera. The method comprises: acquiring synchronous observation data corresponding to a plurality of observation angles and a parameter set to be calibrated; generating a foreground mask of a scene object according to a camera image, and constructing a unified particle state representation according to the parameter set to be calibrated and the foreground mask; forming a projection particle point set and a mask sampling point set according to robot joint state data of each observation angle; constructing a neighborhood distance objective function according to the projection particle point set and the mask sampling point set of the plurality of observation angles, and performing joint iterative updating on the parameter set to be calibrated according to the neighborhood distance objective function to obtain a target calibration parameter set; and performing projection error checking on the plurality of observation angles according to the target calibration parameter set, and outputting the target calibration parameter set that passes the checking and a parameter update instruction. The application is beneficial to improving the overall calibration consistency of a robot motion vision system.

Description

Technical Field

[0001] This application relates to the fields of robot calibration and machine vision technology, and in particular to a joint calibration method and system for coupling error between robot joints and cameras. Background Technology

[0002] Robots typically form a kinematic chain through multiple connected joints and acquire environmental images through cameras mounted on the robot body, head gimbal, or end effector. The joint mounting position, rotation, and zero-point state affect the actual motion pose of the camera, and the camera mounting state in turn affects the imaging position of scene targets in the image. Therefore, there is a correlation between robot motion errors and visual perception errors.

[0003] Current robot calibration typically performs joint kinematic parameter calibration and camera mounting parameter calibration separately. That is, robot-side parameters are determined first, and then the camera-side parameters are determined using the aforementioned calibration results. If residual errors still exist in the previous calibration stage, these residual errors will continue to participate in the subsequent visual calibration stage, causing errors from different sources to propagate and accumulate.

[0004] Therefore, in multi-joint, multi-view robot motion vision systems, how to establish the correlation between robot motion error and camera perception error using the same set of observation information, and reduce the error accumulation caused by step-by-step processing, is a technical problem that needs to be solved. Summary of the Invention

[0005] This application proposes a joint calibration method and system for the coupling error of robot joints and cameras, aiming to correlate robot joint errors and camera errors in a unified data processing process, and to use multi-view observation information to jointly calibrate the two types of parameters, thereby reducing error propagation and accumulation in step-by-step calibration.

[0006] In a first aspect, embodiments of this application provide a joint calibration method for coupling errors between robot joints and cameras, executed by a calibration processing device, comprising: acquiring synchronous observation data corresponding to multiple observation perspectives and a set of parameters to be calibrated, wherein the synchronous observation data includes robot joint state data and camera images, and the set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters; generating a foreground mask of scene objects based on the camera images, and constructing a unified particle state representation based on the set of parameters to be calibrated and the foreground mask, wherein the unified particle state representation includes a joint particle set, a camera particle set, and a scene object particle set; mapping the scene object particle set to the corresponding camera image based on the robot joint state data of each observation perspective, forming a projected particle point set and a mask sampling point set; constructing a neighborhood distance objective function based on the projected particle point set and the mask sampling point set of multiple observation perspectives, and performing joint iterative updates on the set of parameters to be calibrated based on the neighborhood matching error represented by the neighborhood distance objective function, to obtain a target calibration parameter set; performing projection error verification on multiple observation perspectives based on the target calibration parameter set, and outputting a verified target calibration parameter set and parameter update instructions.

[0007] As can be seen, this embodiment incorporates parameter changes in the robot's kinematic space and projection errors in the camera's visual space into a unified particle state representation. It also utilizes neighborhood matching errors formed from multiple observation perspectives to simultaneously constrain joint calibration parameters and camera calibration parameters. This eliminates the dependence of the two types of parameters on their independent sequential calibration processes, which helps reduce error propagation and improve the overall calibration consistency of the robot's motion vision system.

[0008] In one possible example, the step of performing joint iterative updates on the parameter set to be calibrated based on the neighborhood matching error characterized by the neighborhood distance objective function to obtain the target calibration parameter set includes: inputting the current parameter set to be calibrated into a differentiable computational chain, calculating the objective function value of the neighborhood distance objective function and obtaining the parameter gradient; applying gradient magnitude constraints to the parameter gradient to obtain a constrained parameter gradient; updating the current parameter set to be calibrated based on the constrained parameter gradient to obtain an updated parameter set; determining whether the updated parameter set satisfies a preset convergence condition; if it does, then determining the updated parameter set as the target calibration parameter set; if it does not, then using the updated parameter set as the new current parameter set to be calibrated and continuing to perform joint iterative updates.

[0009] As can be seen, this example propagates the neighborhood matching error to the parameters to be calibrated through a differentiable computation chain, and limits the magnitude of abnormal gradients before parameter updates, so that robot joint parameters and camera parameters can be stably updated in the same iterative chain. At the same time, it uses convergence conditions to control the termination time of joint optimization.

[0010] In one possible example, the step of inputting the current set of parameters to be calibrated into a differentiable computation chain, calculating the objective function value of the neighborhood distance objective function, and obtaining the parameter gradient includes: mapping the current set of parameters to be calibrated to the unified particle state representation to obtain an updated unified particle state representation; generating a corresponding projected particle point set based on the updated unified particle state representation and robot joint state data from each observation perspective; calculating the objective function value based on the projected particle point set and the corresponding masked sampling point set; and performing inverse automatic differentiation on the objective function value along the differentiable computation chain to obtain the parameter gradient.

[0011] As can be seen, this example applies parameter changes sequentially to the particle space state, the image projection state, and the neighborhood matching error, and then transmits the final objective function value back to the parameter side, thereby forming a closed-loop differentiable computational relationship from the parameter to be calibrated to the observation error, and then from the observation error back to the parameter to be calibrated.

[0012] In one possible example, mapping the current set of parameters to be calibrated to the unified particle state representation to obtain the updated unified particle state representation includes: extracting the joint calibration parameters and the camera calibration parameters from the current set of parameters to be calibrated; writing the joint calibration parameters into the robot's cascaded kinematics chain to update the spatial state of the joint particle set; writing the camera calibration parameters into the camera mounting pose to update the spatial state of the camera particle set; and combining the updated joint particle set and camera particle set with the scene object particle set to obtain the updated unified particle state representation.

[0013] As can be seen, this example transforms the parameters that originally existed in the robot kinematics model and the camera mounting model into spatial states in a unified particle state representation, enabling different types of parameters to participate in subsequent projection and error calculation through the same data structure, thus providing a unified data foundation for joint optimization.

[0014] In one possible example, writing the joint calibration parameters into the robot's serial kinematics chain and updating the spatial state of the joint particle set includes: writing the installation position error, installation rotation error, and zero-point error corresponding to each joint into the kinematic transformation relationship of the corresponding joint to obtain the correction transformation of each joint; combining the correction transformation of each joint along the robot's serial kinematics chain according to the joint connection order to obtain the cumulative pose of each joint; and updating the spatial position of the corresponding joint particles according to the cumulative pose of each joint to obtain the updated spatial state of the joint particle set.

[0015] As can be seen, this example does not treat multiple joint errors as independent spatial deviations, but propagates joint correction transformations step by step according to the actual connection order of the robot's serial kinematic chain, so that the chain effect of the previous joint error on the pose of the subsequent joint can be preserved in the unified particle state representation.

[0016] In one possible example, the step of combining the corrected transformations of each joint along the robot's serial kinematic chain according to the joint connection sequence to obtain the cumulative pose of each joint includes: starting from the first-level joint corresponding to the robot's base coordinate system, combining the corrected transformation of the current joint with the cumulative pose of the previous level to obtain the cumulative pose of the current joint; using the cumulative pose of the current joint as the previous cumulative pose of the next level joint, and performing the combination step by step according to the joint connection sequence; after completing the step-by-step combination of the robot's serial kinematic chain, the cumulative pose corresponding to each joint is obtained.

[0017] As can be seen, this example uses the cumulative pose of the previous stage as the input for the joint pose calculation of the next stage, thereby preserving the cascade relationship in the serial kinematic chain and reducing the loss of pose correlation caused by calculating multi-joint errors separately.

[0018] In one possible example, acquiring synchronous observation data corresponding to multiple observation perspectives includes: selecting static objects in the scene as observation objects and planning multi-view observation trajectories of the robot relative to the static objects; driving the robot to sequentially reach the observation poses corresponding to the multi-view observation trajectories, and synchronously triggering robot joint state acquisition and camera image acquisition at each observation pose; establishing the correspondence between the robot joint state data and the camera images according to the synchronous triggering time, and performing validity screening on the acquired data to obtain the synchronous observation data.

[0019] As can be seen, this example uses static objects in the scene to form a visual observation benchmark, and establishes a temporal correspondence between the robot joint state and the camera image through synchronous triggering, so that the data participating in the joint calibration from the same observation perspective have consistent acquisition time and spatial state, while reducing the dependence on special calibration objects.

[0020] In one possible example, after obtaining the synchronous observation data, generating a foreground mask for scene objects based on the camera image includes: performing distortion correction on the camera image based on camera intrinsic parameters to obtain a corrected image; performing noise suppression and color cast correction on the corrected image to obtain a preprocessed image; performing foreground and background segmentation on the preprocessed image to obtain an initial foreground mask; and performing morphological processing on the initial foreground mask to remove isolated noise regions and internal holes to obtain the foreground mask.

[0021] As can be seen, this example reduces lens distortion, image noise, color cast, and local mask defects before using scene objects to form mask sampling points, so that the neighborhood matching error between the subsequent projected particle points and the mask sampling points can better reflect the projection differences caused by the robot's motion state and the camera's installation state.

[0022] In one possible example, after the joint iterative update satisfies the preset convergence condition and before the target calibration parameter set is obtained, the method further includes: regenerating multiple projection particle point sets for observation perspectives based on the parameter set to be calibrated that satisfies the preset convergence condition; identifying outliers based on the neighborhood distance of each projection particle and removing the outliers from the unified particle state representation to obtain a filtered particle state representation; performing the joint iterative update again based on the filtered particle state representation, and determining the parameter set obtained from the second joint iterative update as the target calibration parameter set.

[0023] As can be seen, this example uses the spatial neighborhood relationship of projected particles to identify anomalous particles again after the convergence condition is met for the first time, and re-jointly optimizes after removing the influence of anomalous observations, so that the target calibration parameter set is less affected by outlier projection points and local noise in the mask.

[0024] Secondly, embodiments of this application provide a joint calibration system for coupling errors between robot joints and cameras, comprising: a data acquisition module, used to acquire synchronous observation data corresponding to multiple observation perspectives and a set of parameters to be calibrated, wherein the synchronous observation data includes robot joint state data and camera images, and the set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters; a unified characterization module, used to generate a foreground mask of scene objects based on the camera images, and construct a unified particle state characterization including a set of joint particles, a set of camera particles, and a set of scene object particles based on the set of parameters to be calibrated and the foreground mask; and a coupling observation module, The system is used to map the set of scene objects to the corresponding camera images based on the robot joint state data from each observation perspective, forming a projected particle point set and a mask sampling point set; the joint optimization module is used to construct a neighborhood distance objective function based on the projected particle point set and the mask sampling point set from multiple observation perspectives, and to perform joint iterative updates on the parameter set to be calibrated based on the neighborhood matching error represented by the neighborhood distance objective function, to obtain the target calibration parameter set; the result output module is used to perform projection error verification on multiple observation perspectives based on the target calibration parameter set, and to output the verified target calibration parameter set and parameter update instructions.

[0025] The system provided in this application embodiment can perform data acquisition, unified characterization, coupled observation, joint optimization and result output processing in the above method embodiments through different functional modules, and achieve the technical effects corresponding to the above methods.

[0026] Thirdly, embodiments of this application provide a calibration processing device, including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps in the first aspect of embodiments of this application.

[0027] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this embodiment.

[0028] As can be seen, in this embodiment, compared with the prior art of performing robot joint calibration and camera calibration separately, this application first obtains synchronous observation data and parameter sets to be calibrated corresponding to multiple observation perspectives by a calibration processing device. Then, it forms a foreground mask based on the camera image and constructs a unified particle state representation including joint particle sets, camera particle sets, and scene object particle sets. Then, it forms a projection particle point set and a mask sampling point set based on the robot joint state data of each observation perspective. Then, it constructs a neighborhood distance objective function based on the two types of point sets of multiple observation perspectives, and jointly iterates and updates the parameter set to be calibrated based on the neighborhood matching error represented by the neighborhood distance objective function to obtain the target calibration parameter set. Finally, it performs projection error verification on the target calibration parameter set and outputs parameter update instructions. Since the parameter changes in the robot kinematic space and the projection errors in the camera visual space are incorporated into the same particle representation and the same optimization process, the joint errors and camera errors can be associated and constrained by common multi-view observation errors, thereby reducing the error transmission and accumulation caused by step-by-step calibration, which is beneficial to improving the overall calibration consistency of the robot motion vision system. Attached Figure Description

[0029] Figure 1 A schematic diagram of the hardware architecture of the joint calibration system for coupling error between robot joints and cameras provided in this application embodiment; Figure 2 A schematic diagram of the software architecture of the joint calibration system for coupling error between robot joints and cameras provided in the embodiments of this application; Figure 3 This is a schematic diagram of the robot multi-view open scene joint calibration service provided in the embodiments of this application; Figure 4 A schematic flowchart illustrating the joint calibration method for coupling error between robot joints and cameras provided in this application embodiment; Figure 5 A schematic diagram of unified particle state characterization and multi-view projection matching provided for embodiments of this application; Figure 6This is a schematic diagram of the joint iterative update process for the parameters to be calibrated provided in the embodiments of this application; Figure 7 A flowchart illustrating the differentiable computational chain and parameter gradient generation process provided in the embodiments of this application; Figure 8 A schematic flowchart illustrating the mapping process from the parameters to be calibrated to the unified particle state characterization provided in the embodiments of this application; Figure 9 A flowchart illustrating the process of writing joint calibration parameters into the robot's serial kinematics chain, as provided in the embodiments of this application; Figure 10 A schematic diagram of the step-by-step combination process of joint cumulative pose provided in this application embodiment; Figure 11 A flowchart illustrating the multi-view synchronous observation data acquisition process provided in this application embodiment; Figure 12 This is a schematic diagram of the scene object foreground mask generation process provided in the embodiments of this application; Figure 13 A flowchart illustrating the outlier particle filtering and subsequent joint iterative update process provided in this application embodiment; Figure 14 This is a schematic diagram of the functional module architecture of the joint calibration system for coupling error between robot joints and cameras provided in the embodiments of this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are used to illustrate the technical solutions of this application, and are not intended to limit the scope of protection of this application. In the absence of technical conflicts, the different embodiments and technical features described below can be combined with each other.

[0031] The "calibration processing equipment" referred to in this application can be an industrial computer, edge computing host, or computing device with corresponding data processing capabilities that can receive robot joint state data and camera images and perform joint calibration calculations. The "joint calibration parameters" are used to characterize the kinematic correction amounts of one or more joints in the robot's cascaded kinematic chain; the "camera calibration parameters" are used to characterize the camera's installation state relative to the robot's kinematic chain or the state to be corrected related to image acquisition. Together, they constitute the set of parameters to be calibrated.

[0032] The term "unified particle state representation" used in this application does not require that real physical objects must be composed of particles, but rather refers to a data organization format for calibration processing equipment to perform unified numerical processing. Robot joints, cameras, and scene objects are represented by data units with spatial position and associated attributes, thereby enabling state changes in the robot's kinematic space and projection changes in the visual space to be correlated within the same data processing chain.

[0033] The "neighborhood distance objective function" referred to in this application is used to summarize the degree of neighborhood matching between projected particle points and mask sampling points from multiple observation perspectives. A neighborhood can be a number of sampling points that are relatively close to a projection point, or a number of projection points that are relatively close to a sampling point. The specific mathematical form of the objective function is not the only form that must be adopted in the various embodiments of this application, as long as it can utilize the differences in point set matching in the local neighborhood to form an error metric that can be used to jointly update the set of parameters to be calibrated.

[0034] See Figure 1 The hardware architecture of the joint calibration system 100 for coupling error between robot joints and cameras includes a robot body 110, a robot control cabinet 120, an industrial camera 130, a supplementary lighting component 140, a calibration processing device 150, a synchronization trigger 160, and a communication network 170.

[0035] The robot body 110 can be composed of multiple joints connected in series to form a kinematic chain. Each joint can be equipped with an encoder, a driver, and sensing components related to joint motion state detection. The encoder is used to generate robot joint state data such as joint angles. The robot control cabinet 120 is used to receive motion control commands and control the robot body 110 to move according to a predetermined observation trajectory.

[0036] The industrial camera 130 can be fixed to the head gimbal of the robot body 110, the end effector of the robotic arm, or other positions that can change the observation pose with the movement of the robot's joints. The industrial camera 130 is used to acquire images of scene objects from different observation angles. The supplementary lighting assembly 140 can be placed near the industrial camera 130 and improve the imaging conditions of scene objects when needed.

[0037] The calibration processing device 150 communicates with the robot control cabinet 120, the industrial camera 130, and the synchronization trigger 160. The calibration processing device 150 may include a processor, a memory, a graphics computing unit, and a communication interface. The processor is used to perform process control, parameter management, and data preprocessing; the graphics computing unit can be used for particle state updates, projection calculations, neighborhood matching, and parameter gradient calculations; the memory is used to store robot kinematic data, camera configuration data, synchronous observation data, the set of parameters to be calibrated, and calibration processing data.

[0038] Synchronization trigger 160 provides a common synchronization trigger reference to robot control cabinet 120 and industrial camera 130, enabling a time correspondence between robot joint state data and camera images from the same observation perspective. Communication network 170 is used to transmit control commands, robot joint state data, camera images, and parameter update commands between the aforementioned hardware devices.

[0039] In one specific implementation, the robot body 110 can be a full-body robot with multiple degrees of freedom, and the industrial camera 130 is mounted in an "eye-on-hand" manner. The set of parameters to be calibrated can cover the mounting position errors, mounting rotation errors, and zero-point errors of multiple joints, as well as the camera's mounting position errors, mounting rotation errors, and errors related to acquisition triggering. The specific number of parameters can be adjusted according to the robot's degrees of freedom and the object to be calibrated.

[0040] Thus, the joint calibration system 100 for coupling errors between robot joints and cameras can use the synchronous trigger 160 to correspond the robot motion state with the visual observation state, and the same calibration processing device 150 performs joint processing on the robot-side parameters and camera-side parameters, providing a hardware foundation for subsequent reduction of cross-device error propagation.

[0041] See Figure 2 The software architecture of the joint calibration system 100 for coupling error between robot joints and cameras may include a device driving and data synchronization layer 210, an image processing layer 220, a unified representation layer 230, a coupled observation layer 240, a joint optimization layer 250, and a result verification and output layer 260.

[0042] The device drive and data synchronization layer 210 is used to interact with the robot control cabinet, industrial camera and synchronization trigger, control the execution of multi-view observation trajectory, and establish the time correspondence between robot joint state data and camera images.

[0043] The image processing layer 220 is used to perform distortion correction, noise suppression, color shift correction, foreground segmentation, and mask optimization on the synchronously acquired camera images to obtain a foreground mask that can characterize the two-dimensional observation range of static objects in the scene.

[0044] The unified representation layer 230 is used to manage the set of parameters to be calibrated and to convert joint calibration parameters, camera calibration parameters and spatial observation information formed by scene objects into a unified particle state representation.

[0045] The coupled observation layer 240 is used to combine robot joint state data and updated unified particle state representation to calculate visual projection state under different observation perspectives, and form projection particle point set and mask sampling point set.

[0046] The joint optimization layer 250 is used to construct an objective function using the neighborhood matching error between two types of point sets, propagate the error corresponding to the objective function along the differentiable computation chain to the side of the parameter to be calibrated, and obtain the target calibration parameter set through multiple parameter updates.

[0047] The result verification and output layer 260 is used to recalculate the projection errors of multiple observation perspectives and generate parameter update instructions when the target calibration parameter set meets the accuracy requirements. These parameter update instructions can be used to update robot control parameters and camera installation parameters or corresponding software configuration parameters.

[0048] The aforementioned software layers are interconnected according to their data processing relationships. Data output from the previous stage serves as input for the next stage, preventing robot state, camera state, and visual observation errors from entering separate processing chains.

[0049] See Figure 3 In the schematic diagram of the robot multi-view open scene joint calibration service, the static scene object 310 can be an ordinary object whose position remains unchanged in the calibration environment and has a distinguishable outer contour or visual area, without the need to specially set up a calibration plate with a predetermined pattern. In an exemplary calibration task, the calibration processing equipment controls the robot 320 to change its observation pose around the static scene object 310, and multiple observation perspectives 340 can cover frontal observation, side observation, top observation, and observation states at different tilt angles.

[0050] Each time robot 320 reaches a new observation pose, camera 330 acquires the corresponding scene image and simultaneously obtains the state data of each joint at that moment. By observing from multiple different directions, the same static scene object 310 can be projected into different states in the camera images.

[0051] For example, when a robot joint has a zero-point deviation, this deviation will change the actual pose of the camera 330 along the robot's kinematic chain; when the camera 330 itself has a mounting rotation deviation, it will also change the projected position of the static scene object 310 in the image. If only a single viewpoint is observed, the two types of errors may exhibit similar image displacement. Multiple observation viewpoints 340 can form common observation constraints under different motion states, so that the influence of the two types of parameter changes on image projection is more fully constrained in the joint optimization process.

[0052] Therefore, this business instance can establish multi-view visual observations using ordinary static objects in open scenarios, without requiring the calibration object to always be used with dedicated calibration facilities.

[0053] In one possible example, see Figure 4 The joint calibration method for the coupling error between the robot joint and the camera can be achieved through the following steps S101 to S105.

[0054] S101, acquire synchronous observation data and a set of parameters to be calibrated corresponding to multiple observation perspectives; the synchronous observation data includes robot joint state data and camera images, and the set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters.

[0055] Synchronous observation data refers to robot state data and visual data that can establish a temporal correspondence between acquisitions from the same observation perspective. Robot joint state data may include encoder feedback values ​​for each joint, and camera images may be RGB images output from industrial cameras. The parameter set to be calibrated is used to centrally store the robot-side parameters and camera-side parameters that are allowed to vary during this joint calibration.

[0056] In one example, for a robot with multiple tandem joints, the installation position error, installation rotation error, and joint zero-point error of each joint can be used as joint calibration parameters, and the camera installation position and installation rotation related parameters can be used as camera calibration parameters. If the acquisition trigger state also needs to be jointly corrected with the robot's motion process, the calibration parameters can also set the calibrated quantities related to the trigger timing.

[0057] S102, generate a foreground mask for scene objects based on camera images, and construct a unified particle state representation based on the parameter set to be calibrated and the foreground mask. The unified particle state representation includes joint particle set, camera particle set and scene object particle set.

[0058] Among them, the joint particle set is used to convert the joint states and associated calibration parameters in the robot's kinematics chain into updatable spatial data units; the camera particle set is used to characterize the camera mounting space state; and the scene object particle set is used to characterize the scene spatial information that serves as a visual observation benchmark from multiple observation perspectives.

[0059] Scene object particles are not limited to being directly equivalent to image pixels. The calibration processing device can select foreground sampling locations from multiple foreground masks and establish corresponding spatial observation points using multi-view geometry, thereby forming a scene object particle set. The scene object particle set can remain fixed during a round of joint parameter optimization, while the joint particle set and camera particle set change as the parameter set to be calibrated is updated.

[0060] In one extended implementation, duplicate point merging and local neighborhood consistency checks can be performed on the spatial observation points used to form scene object particles from different perspectives before adding the processed spatial observation points to the scene object particle set. This extended scheme mainly addresses the problem of uneven local point density when multiple perspectives sample and form duplicate particles with similar spatial positions, which helps to reduce the bias of local duplicate observations on subsequent neighborhood distance statistics.

[0061] S103: Based on the robot joint state data from each observation perspective, the scene object particle set is mapped to the corresponding camera image to form a projected particle point set and a mask sampling point set.

[0062] The mapping process involves sequentially using the robot's kinematics chain to determine the camera's pose relative to the robot's reference coordinates from the current observation viewpoint, and then using the camera's imaging relationship to transform the scene's object particles into a two-dimensional image coordinate space. The resulting two-dimensional spatial points constitute the projected particle point set.

[0063] The mask sampling point set is obtained from the foreground mask corresponding to the same observation viewpoint. Mask sampling can employ uniform sampling, partitioned sampling, or other sampling methods that can cover the foreground region. To maintain an appropriate data size during neighborhood comparisons between the two types of point sets, the number of mask sampling points can be controlled according to the number of scene object particles.

[0064] In a specific example, if the mounting rotation state of a certain front-stage joint of the robot changes, this change will alter the overall pose of the camera along the kinematic chain, thereby causing a change in the two-dimensional projection position of the same set of scene object particles; a change in the camera mounting rotation state will also affect the same two-dimensional projection position. Therefore, the projected particle point set serves as a common intermediate data connecting changes in robot joint parameters, changes in camera parameters, and visual observation errors.

[0065] S104. A neighborhood distance objective function is constructed based on the projection particle point set and mask sampling point set from multiple observation perspectives. The neighborhood matching error characterized by the neighborhood distance objective function is used to perform a joint iterative update on the parameter set to be calibrated, thereby obtaining the target calibration parameter set.

[0066] The neighborhood distance objective function can find several nearby mask sampling points around the projected particle points, and then find several nearby projected particle points around the mask sampling points. The neighborhood matching error is then formed by combining the local distances from multiple observation perspectives. One specific implementation of this application's scheme can use the K-nearest neighbor form of the Chamfer distance, where K takes multiple nearest neighbor points instead of just a single nearest point to form the point-to-point error.

[0067] Using multiple neighboring points allows a local projection point to be constrained by multiple visual observation points within a certain range. When the foreground mask contour has local spikes, holes, or random sampling biases, the parameter update direction is not entirely determined by a single nearest point, thereby improving the adaptability of the high-dimensional parameter joint optimization process to local observation disturbances.

[0068] The term "joint iterative update" refers to using the neighborhood matching error calculated in the same round of objective function calculations simultaneously to update both joint calibration parameters and camera calibration parameters, rather than first fixing the camera parameters and then fixing the joint parameters and then fixing the joint parameters and then fixing the camera parameters. After multiple joint updates, the parameter set that meets the termination condition is determined as the target calibration parameter set.

[0069] S105 performs projection error verification on multiple observation angles based on the target calibration parameter set, and outputs the verified target calibration parameter set and parameter update instructions.

[0070] The projection error verification process involves rewriting the target calibration parameter set into the robot's kinematic relationships and camera installation status, and regenerating projected particle points from multiple observation perspectives. The calibration processing equipment can compare the average projection difference between the regenerated projected particle points and the corresponding foreground observations, and then compare this difference with preset accuracy conditions.

[0071] When the target calibration parameter set passes verification, the joint calibration parameters and camera calibration parameters can be output separately. Parameter update commands can instruct the robot control cabinet to update the corresponding kinematic parameters and instruct the camera parameter storage location or related software configuration location to update the camera calibration parameters. If the verification conditions are not met, data acquisition or parameter optimization can be re-executed.

[0072] Through steps S101 to S105, the robot state, camera state, and scene visual observation no longer form independent calibration links. Instead, they form a closed-loop correlation through unified particle state representation, common projection process, and common neighborhood matching error. This allows multi-joint kinematic errors and camera errors to be corrected together under the same multi-view observation conditions, thereby reducing error propagation in staged calibration.

[0073] In one possible example, see Figure 5 In the unified particle state representation and multi-view projection matching diagram, the particle positions in the joint particle set 510 are jointly determined by the robot's kinematic relationships and the current joint calibration parameters. The camera particle set 520 is used to reflect the current camera mounting pose. The scene object particle set 530 is used to provide spatial observation points as visual spatial references during the joint optimization process.

[0074] For any given observation viewpoint, the robot joint state data 540, together with the current joint calibration parameters, determines the spatial state of the robot's kinematics chain. The camera calibration parameters further contribute to determining the camera's observation pose. Based on this, the projection transformation 550 converts the scene object particle set 530 into a projected particle point set 560 on the image plane.

[0075] The foreground mask 570 corresponding to the same observation viewpoint is sampled to form a mask sampling point set 580. The projected particle point set 560 and the mask sampling point set 580 are in the same two-dimensional image space, so the local neighborhood distance relationship can be directly established.

[0076] For example, in a curved contour region of the foreground mask 570, when the predicted set of projected particle points 560 shifts as a whole towards one side of the contour, multiple local projected points and mask sampling points all experience a consistent change in neighborhood distance. This change can be inversely correlated to joint calibration parameters and camera calibration parameters via projection transformation 550, so that both types of parameters are subject to common constraints under the same visual bias.

[0077] This implementation establishes a data bridge between the kinematic space and the visual observation space by unifying particle state representation and two-dimensional projection matching relationship, providing a continuous data processing chain for subsequent differentiable joint optimization.

[0078] In one possible example, see Figure 6 The joint iterative update process in step S104 can be implemented through the following steps S201 to S204.

[0079] S201: Input the current set of parameters to be calibrated into the differentiable computation chain, calculate the objective function value of the neighborhood distance objective function and obtain the parameter gradient.

[0080] The current set of parameters to be calibrated represents the actual state of the parameters involved in the calculation during the current iteration. When step S201 is executed for the first time, the current set of parameters to be calibrated can be the values ​​to be calibrated corresponding to the robot's factory kinematic parameters and the camera's initial installation parameters; in subsequent iterations, the current set of parameters to be calibrated is formed from the parameter set updated in the previous round.

[0081] Differentiable computational chains are used to maintain differentiable correlations between parameters to be calibrated, kinematic states, camera pose, particle projections, and even neighborhood matching errors. Through this correlation, the calibration processing device can determine the direction and extent to which the objective function value varies with different parameters to be calibrated.

[0082] In one extended implementation, the gradient statistics of the joint calibration parameters and the camera calibration parameters can be recorded separately, and the parameter update magnitude can be adjusted using their respective historical gradient information in the same iteration. This extended implementation can solve the problem that a uniform step size is difficult to take into account various parameters when the parameter scales corresponding to different physical quantities are different, enabling different types of parameters to maintain a relatively stable update process.

[0083] S202 applies a gradient magnitude constraint to the parameter gradient to obtain a constrained parameter gradient.

[0084] The gradient magnitude limit is used to restrict abnormally large parameter gradients during a single backpropagation. It can be limited based on the overall gradient norm, or allowable ranges can be set for different parameter groups. One implementation of this application's scheme can employ gradient pruning, which proportionally reduces the gradient when it exceeds a preset range while maintaining its primary update direction.

[0085] In one extended implementation, gradient magnitude statistics can be performed separately for the joint parameter set and the camera parameter set. When only a few parameters produce abnormally large gradients, restrictions can be applied only to the abnormal parameter set without altering the effective gradients of other parameter sets. This approach reduces the risk that local anomaly observations will simultaneously pull the entire set of parameters to be calibrated in the anomaly direction.

[0086] S203, update the current set of parameters to be calibrated based on the gradient of the constrained parameters to obtain the updated set of parameters.

[0087] One approach to generating the updated parameter set is gradient optimization with an adaptive learning rate. One specific implementation of this application uses the Adam optimizer to determine the actual update amount of different parameters based on the current gradient and historical gradient statistics.

[0088] It should be noted that the specific optimizer used does not change the processing relationship of "jointly updating joint calibration parameters and camera calibration parameters based on the matching error of the same neighborhood". Other optimization methods that can use parameter gradients to update the current set of parameters to be calibrated can also be used in this embodiment.

[0089] S204, determine whether the updated parameter set meets the preset convergence condition; if it does, then the updated parameter set is determined as the target calibration parameter set; if it does not, then the updated parameter set is used as the new current parameter set to be calibrated and the joint iterative update continues.

[0090] The preset convergence conditions can consider at least one or more of the following: the magnitude of change in the objective function, the amount of parameter updates, and the maximum allowed number of iterations. For example, when the amount of parameter updates in continuous iterations is lower than a preset threshold, it can be considered that the state change brought about by continued updates is already small; when the maximum allowed number of iterations is reached, the current optimization process can also be terminated, and the subsequent accuracy verification step can be used to confirm whether the result is usable.

[0091] Through steps S201 to S204, the common visual error corresponding to the objective function can be continuously applied to the robot joint parameters and camera parameters, and the target calibration parameter set is gradually obtained under gradient constraints and convergence control, thus taking into account both multi-dimensional joint update capability and iterative numerical stability.

[0092] In one possible example, see Figure 7 Step S201 can be achieved through the following steps S301 to S304.

[0093] S301, map the current set of parameters to be calibrated to the unified particle state representation to obtain the updated unified particle state representation.

[0094] In this process, mapping is not simply about appending parameter values ​​to a data structure. Instead, the parameter values ​​actually change the spatial state of the corresponding joint particles and camera particles, so that the subsequent projection results can change with the current set of parameters to be calibrated.

[0095] S302, generate the corresponding projected particle point set based on the updated unified particle state representation and the robot joint state data from each observation perspective.

[0096] Each observation viewpoint has corresponding robot joint state data. The calibration processing equipment incorporates this joint state along with the updated joint calibration parameters into the robot's kinematics calculations, and then combines this with the updated camera state to determine the projection relationship of the current viewpoint.

[0097] In one extended implementation, for multi-view data, projected particle point sets can be generated in parallel by grouping according to the observation viewpoint, and then the outputs of each viewpoint can be aggregated before the objective function calculation. This extended implementation can solve the problem of increased computational overhead of serial projection as the number of observation views increases, while not changing the coupling relationship established between the views through a common set of parameters to be calibrated.

[0098] S303, calculate the objective function value based on the projected particle point set and the corresponding mask sampling point set.

[0099] In this process, the projected particle point set in different observation viewpoints is only compared with the mask sampling point set corresponding to that observation viewpoint, and then the local errors formed by different observation viewpoints are summarized, thereby avoiding meaningless cross-view matching of two-dimensional sampling points between different images.

[0100] In one extended implementation, a bidirectional neighborhood relationship can be established between the projected particle points and the mask sampling points within each observation viewpoint, and then the bidirectional neighborhood distances can be aggregated. A further subdivision is that unidirectional matching may result in multiple projected points converging on a small number of mask sampling points; through bidirectional neighborhood relationships, the coverage of the projected points on the observation foreground and the coverage of the observation foreground on the projected points can be constrained simultaneously, improving the objective function's ability to express the differences in local point set distribution.

[0101] S304 performs inverse automatic differentiation on the objective function value along the differentiable computation chain to obtain the parameter gradient.

[0102] In this process, the backward automatic differentiation, following the parameter dependencies preserved in the forward calculation, returns to the parameter side to be calibrated from the objective function value through neighborhood distance calculation, two-dimensional particle projection, camera state, and robot kinematic state. Therefore, the parameter gradient can represent the direction of influence of different joint calibration parameters and camera calibration parameters on the current neighborhood matching error.

[0103] Through steps S301 to S304, the changes in the parameters to be calibrated can form observable image projection changes in the forward phase and be converted into gradient information for the parameters to be calibrated in the reverse phase, thereby establishing a complete end-to-end parameter solution chain.

[0104] In one possible example, see Figure 8 Step S301 can be achieved through the following steps S401 to S404.

[0105] S401, extract joint calibration parameters and camera calibration parameters from the current set of parameters to be calibrated.

[0106] The two types of parameters can be distinguished by parameter index, parameter type flag, or parameter device flag. Parameter classification is used to determine the objects to which different parameters act in the subsequent spatial state update process.

[0107] S402, writes the joint calibration parameters into the robot's cascade kinematics chain and updates the spatial state of the joint particle set.

[0108] The joint calibration parameters can be applied to the robot joint's mounting position, mounting rotation, and zero-point state. After the parameters are changed, the spatial state of the relevant joint and its subsequent joints needs to be recalculated, rather than simply modifying the joint's local representation.

[0109] In one extended implementation, the range of affected subsequent joints can be saved for each joint. When the change of a certain parameter in an iteration is less than the preset calculation accuracy, the intermediate state of the corresponding unchanged kinematic chain segment can be reused. This extended scheme mainly solves the problem that recalculating the entire joint chain in every round of a high-degree-of-freedom robot may result in a large amount of redundant calculation, which helps to reduce redundant kinematic calculations in iteration.

[0110] S403 writes the camera calibration parameters into the camera mounting pose and updates the spatial state of the camera particle set.

[0111] The camera mounting pose describes the spatial relationship between the camera and its mounting carrier. The mounting carrier can be an end effector, a stage of a robotic arm, or a head-mounted gimbal. When the camera's mounting position and rotation change, the spatial position and orientation of the camera particles change accordingly.

[0112] S404 combines the updated joint particle set and camera particle set with the scene object particle set to obtain the updated unified particle state representation.

[0113] The scene object particle set represents a relatively fixed spatial observation anchor point in the calibration environment, which does not change with the parameters to be calibrated during a round of joint parameter solving. When combined with the dynamically changing joint particle set and camera particle set, the same unified particle state representation simultaneously contains both parameter-driven state and fixed observation state.

[0114] Through steps S401 to S404, the parameters to be calibrated from different sources and with different physical meanings can be converted into a unified data state, so that subsequent projection calculations do not require the establishment of separate robot calibration chains and camera calibration chains.

[0115] In one possible example, see Figure 9 Step S402 can be achieved through the following steps S501 to S503.

[0116] S501, write the installation position error, installation rotation error and zero point error corresponding to each joint into the kinematic transformation relationship of the corresponding joint to obtain the correction transformation of each joint.

[0117] Specifically, the installation position error is used to correct the positional relationship between adjacent joints, the installation rotation error is used to correct the attitude relationship between adjacent joints, and the zero-point error is used to correct the deviation between the joint encoder feedback state and the actual mechanical zero position. Each error can be merged into the local kinematic transformation of the joint according to its corresponding physical position.

[0118] In one extended implementation, the correction parameters for each joint can be divided into a fixed-installation correction component and a zero-point correction component that works in conjunction with the joint feedback state, and these components are combined according to the actual transformation order of the kinematic chain when forming the correction transformation. This extended implementation can solve the problem that the direct superposition of different types of joint errors in coordinate transformation, which may lead to confusion in physical meaning, can resolve the issue that different types of errors affect the final pose through their corresponding kinematic links.

[0119] S502, according to the joint connection sequence, combines the correction transformations of each joint along the robot's serial kinematic chain to obtain the cumulative pose of each joint.

[0120] In this context, the tandem kinematic chain means that the subsequent joints are built upon the spatial state of the preceding joints. Therefore, the correction transformation of the preceding joints not only affects themselves, but also continues to affect the joints located after them and the camera mounted on the subsequent structure.

[0121] S503 updates the spatial position of the corresponding joint particles based on the cumulative pose of each joint, thus obtaining the spatial state of the updated joint particle set.

[0122] One or more joint particles can be set for each joint's cumulative pose. For cases where only the joint center state needs to be represented, a single representative spatial particle can be used; for cases where the direction of the joint connection segment needs to be expressed, multiple particles distributed along the local structure can be set. Regardless of the number of particles used, the spatial state is uniformly determined by the corresponding cumulative pose.

[0123] Through steps S501 to S503, multiple joint errors participate in the spatial state calculation according to the robot's actual serial relationship, so that the chain effect of upstream joint errors on downstream camera pose can be preserved and can be subject to unified constraints in subsequent visual projection errors.

[0124] In one possible example, see Figure 10 Step S502 can be achieved through the following steps S601 to S603.

[0125] S601: Starting from the first joint corresponding to the robot's base coordinate system, the correction transformation of the current joint is combined with the cumulative pose of the previous joint to obtain the cumulative pose of the current joint.

[0126] For the first-level joint, its cumulative pose from the previous levels can be defined by the robot's base coordinate system. The cumulative pose of the current joint represents the spatial state after reaching the current joint from the robot's base coordinate system through all processed previous motion relationships.

[0127] S602, the cumulative pose of the current joint is used as the previous cumulative pose of the next joint, and the combination is executed level by level according to the joint connection sequence.

[0128] This processing method allows the correction state of one joint to be naturally incorporated into the calculation path of all subsequent joints. For example, a small rotational deviation in the lumbar joint may further alter the pose of the shoulder, the end effector of the robotic arm, and the camera mounted on the end effector; this effect can be propagated through step-by-step accumulation.

[0129] In one extended implementation, when there are multiple branches in the cascaded kinematic chain, the cumulative pose of a common preceding joint can be used as the starting point for each branch, and the cumulative pose can continue along the corresponding branch. This approach is applicable to robots with dual arms or head and upper limb branch structures, while ensuring that the upstream joint errors shared by each branch are propagated consistently.

[0130] S603 obtains the cumulative pose of each joint after completing the step-by-step combination of the robot's serial kinematic chain.

[0131] Among them, each accumulated pose can not only be used to update joint particles, but also to determine the current reference pose of the structure where the camera is installed, so that the changes in robot-side parameters and the installation state of the camera-side meet in the same spatial relationship.

[0132] Through steps S601 to S603, robot joint errors can propagate along their true topological order, avoiding the problem that the overall chain motion state cannot be recovered after calculating the local errors of individual joints separately.

[0133] In one possible example, see Figure 11 The synchronous observation data acquisition process in step S101 can be achieved through the following steps S701 to S703.

[0134] S701 selects static objects in the scene as observation objects and plans the robot's multi-view observation trajectory relative to the static objects.

[0135] Static objects refer to objects that remain basically fixed relative to the environment during this multi-view data acquisition process. They are not required to have a predefined checkerboard pattern, dot array, or other special calibration pattern; they only need to form an extractable foreground area in the camera image.

[0136] Multi-view observation trajectories are used to enable robots to observe the same static object from different joint configurations. The trajectory can cover the frontal, lateral, top, and tilt directions, thus allowing the robot's kinematic parameters and camera mounting parameters to have different combinations of influence on the projection results under different spatial states.

[0137] S702 drives the robot to sequentially reach the observation poses corresponding to the multi-view observation trajectory, and simultaneously triggers robot joint state acquisition and camera image acquisition at each observation pose.

[0138] Synchronization triggering can employ a unified hardware trigger signal or, provided time alignment accuracy is met, a unified clock synchronization method. Each trigger generates a set of robot joint state data and camera images corresponding to the same robot state.

[0139] In one extended implementation, after the robot reaches the observation pose, it can first determine whether the joint velocity is lower than a preset stability condition before triggering the corresponding observation data acquisition. This extended implementation can solve the problem of inconsistency between image acquisition and joint feedback caused by rapid changes in a short period of time when the robot still has significant movement, which is beneficial to improving the stability of data from a single observation perspective.

[0140] S703 establishes the correspondence between robot joint state data and camera images according to the synchronization trigger time, and performs validity screening on the collected data to obtain synchronous observation data.

[0141] The validity screening feature can identify observation data that is overexposed, underexposed, has missing foreground areas, or where the robot's joint states exceed the allowable range. When data from a particular observation view fails the screening, that view's data can be deleted or the data can be re-acquired at that observation pose.

[0142] Through steps S701 to S703, multiple time-aligned robot-image data pairs can be formed using static objects in an open environment, providing a data basis for subsequent joint correction of joint parameters and camera parameters from common observation errors.

[0143] In one possible example, see Figure 12 After obtaining the synchronous observation data, the foreground mask generation process in step S102 can be implemented through the following steps S801 to S804.

[0144] S801 performs distortion correction on the camera image based on the camera's intrinsic parameters to obtain a corrected image.

[0145] The camera intrinsic parameters can be predetermined during the camera manufacturing or system deployment phase and remain fixed during this joint calibration. Distortion correction is used to reduce the impact of lens radial and tangential distortion on two-dimensional spatial position.

[0146] S802 performs noise suppression and color cast correction on the corrected image to obtain a preprocessed image.

[0147] Noise suppression can be achieved using image smoothing methods such as Gaussian filtering, while color cast correction can be achieved using processing methods such as adaptive white balance. This step is mainly used to reduce imaging noise and the impact of ambient lighting changes on foreground segmentation.

[0148] S803 performs foreground and background segmentation on the preprocessed image to obtain the initial foreground mask.

[0149] This can be achieved using an image segmentation model capable of outputting foreground probabilities or binary foreground regions. One specific implementation of this application's scheme can employ a pre-trained U2Net saliency detection network to identify static object regions in a scene. The foreground segmentation model is not the only model limited by the joint calibration main link in this application; other visual segmentation methods capable of obtaining foreground regions of scene objects can also be used.

[0150] In one extended implementation, the foreground area changes from multiple adjacent viewing angles can be used as a mask quality check condition. When the foreground area of ​​a certain viewpoint changes abnormally compared to adjacent viewpoints, that viewpoint is marked as a viewpoint to be reviewed, and foreground segmentation is re-executed or the image is re-acquired. This extended implementation can solve the problem of single-frame segmentation errors continuously affecting parameter updates after entering subsequent joint optimization.

[0151] S804 performs morphological processing on the initial foreground mask to remove isolated noise regions and internal holes, thus obtaining the foreground mask.

[0152] In this process, opening operations can be used to eliminate isolated small regions, and closing operations can be used to fill small holes inside the foreground, so that the final foreground mask has a relatively continuous spatial range.

[0153] Through steps S801 to S804, the subsequent mask sampling point set is established on the visual data after distortion, noise, color shift and local mask defects have been processed, which helps to reduce the interference of non-calibration error factors on the neighborhood matching error.

[0154] In one possible example, see Figure 13 After the joint iterative update meets the preset convergence condition and before the target calibration parameter set is obtained, particle filtering and joint iterative update can be performed through the following steps S901 to S903.

[0155] S901, regenerates multiple sets of projected particle points from different observation perspectives based on the set of parameters to be calibrated that meet the preset convergence conditions.

[0156] The fact that the preset convergence condition is met indicates that the current parameter state has entered a relatively stable state after the aforementioned joint iteration, but it does not necessarily mean that every projected particle is a reliable observation. The projected particle point set is regenerated to re-examine the local spatial consistency of the particles under stable parameter states.

[0157] S902 identifies outlier particles based on the neighborhood distance of each projected particle and removes outlier particles from the unified particle state representation to obtain the filtered particle state representation.

[0158] This involves calculating the average distance between a projected particle and its surrounding neighborhood points. When a particle consistently exhibits a significantly larger neighborhood distance than its normal local range across multiple viewing angles, the corresponding scene object particle can be identified as an outlier particle.

[0159] In one extended implementation, instead of directly deleting particles based on a single observation viewpoint, the abnormal states of particles in the same scene are statistically analyzed across multiple observation views. Deletion is only performed when an abnormal condition is met in multiple views. This approach primarily addresses the problem of erroneous deletion of normal spatial particles caused by single-view occlusion, localized reflections, or changes in mask edges, thus improving the stability of outlier particle detection.

[0160] S903, perform joint iterative update again based on the state characterization of filtered particles, and determine the parameter set obtained from the second joint iterative update as the target calibration parameter set.

[0161] The subsequent joint iterative update still utilizes joint calibration parameters and camera calibration parameters to jointly form the projection result, and uses the neighborhood matching error formed by the filtered scene object particles to update both types of parameters together. This subsequent update can use fewer iterations, or it can continue to terminate according to the aforementioned convergence criteria.

[0162] Through steps S901 to S903, after the existing parameters have basically converged, anomalous spatial particles are further identified, and the parameters to be calibrated are corrected again using the cleaned observation structure. This can reduce the impact of local flypoints, erroneously segmented regions, or anomalous spatial reconstruction points on the final target calibration parameter set.

[0163] In one possible example, see Figure 14 The functional module architecture of the joint calibration system 100 for coupling error between robot joints and cameras includes a data acquisition module 1410, a unified characterization module 1420, a coupled observation module 1430, a joint optimization module 1440, and a result output module 1450.

[0164] The data acquisition module 1410 is used to acquire synchronous observation data and a set of parameters to be calibrated corresponding to multiple observation perspectives. The synchronous observation data includes robot joint state data and camera images, and the set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters.

[0165] The data acquisition module 1410 can call the robot control interface, camera acquisition interface and synchronous trigger interface to obtain robot joint state data and camera images from the same observation perspective, and read the set of parameters to be calibrated for this calibration task from the parameter storage area.

[0166] The unified representation module 1420 is used to generate a foreground mask of scene objects based on camera images, and to construct a unified particle state representation including joint particle set, camera particle set and scene object particle set based on the parameter set to be calibrated and the foreground mask.

[0167] The unified representation module 1420 can perform image preprocessing, foreground segmentation, scene space particle initialization, and robot and camera particle state initialization, thereby enabling data from different sources to enter a common data organization structure.

[0168] The coupled observation module 1430 is used to map the set of scene objects to the corresponding camera image based on the robot joint state data from each observation perspective, forming a set of projected particle points and a set of mask sampling points.

[0169] Among them, the coupled observation module 1430 establishes a forward propagation relationship between parameter states and two-dimensional observation through robot kinematic calculation relationship and camera imaging relationship, so that the common influence of robot parameters and camera parameters on image projection can form a comparable two-dimensional point set.

[0170] The joint optimization module 1440 is used to construct a neighborhood distance objective function based on the projection particle point set and mask sampling point set from multiple observation perspectives, and to perform joint iterative updates on the parameter set to be calibrated based on the neighborhood matching error characterized by the neighborhood distance objective function, so as to obtain the target calibration parameter set.

[0171] The joint optimization module 1440 can maintain an end-to-end differentiable computational chain and perform objective function calculation, parameter gradient generation, gradient magnitude constraint, parameter update, and convergence determination. The joint optimization module 1440 can also perform outlier identification and further joint optimization after the first parameter convergence.

[0172] The result output module 1450 is used to perform projection error verification on multiple observation angles according to the target calibration parameter set, and output the verified target calibration parameter set and parameter update instructions.

[0173] The result output module 1450 can generate robot joint parameter update data and camera parameter update data respectively, and write them to the corresponding devices or parameter storage areas. The result output module 1450 can also save the input data, iteration status, verification results and final parameters of this calibration task for use in equipment maintenance or subsequent calibration tasks.

[0174] As can be seen, in this example, the system forms a continuous data processing chain through the data acquisition module 1410, the unified representation module 1420, the coupled observation module 1430, the joint optimization module 1440, and the result output module 1450, so that the joint errors in the robot's kinematic space and the camera errors in the visual space can be jointly processed under the unified observation error constraint, thereby achieving the technical effect corresponding to the aforementioned method embodiment.

[0175] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.

[0176] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0177] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0178] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0179] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0180] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0181] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0182] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0183] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A joint calibration method for coupling error between robot joints and a camera, characterized in that, Performed by the calibration processing equipment, including: Acquire synchronous observation data and a set of parameters to be calibrated corresponding to multiple observation perspectives. The synchronous observation data includes robot joint state data and camera images. The set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters. A foreground mask for scene objects is generated based on the camera image, and a unified particle state representation is constructed based on the set of parameters to be calibrated and the foreground mask. The unified particle state representation includes a joint particle set, a camera particle set, and a scene object particle set. Based on the robot joint state data from each observation perspective, the scene object particle set is mapped to the corresponding camera image to form a projection particle point set and a mask sampling point set. The mask sampling point set is obtained from the foreground mask corresponding to the same observation perspective. A neighborhood distance objective function is constructed based on the projected particle point set and the mask sampling point set from multiple observation perspectives. A joint iterative update is then performed on the parameter set to be calibrated based on the neighborhood matching error represented by the neighborhood distance objective function to obtain the target calibration parameter set. This includes: inputting the current parameter set to be calibrated into a differentiable computation chain, calculating the objective function value of the neighborhood distance objective function and obtaining the parameter gradient; mapping the current parameter set to be calibrated to the unified particle state representation to obtain an updated unified particle state representation; and generating a pair of... The system calculates the target function value based on the projected particle point set and the corresponding mask sampling point set; performs inverse automatic differentiation on the target function value along the differentiable computation chain to obtain the parameter gradient; applies gradient magnitude constraint to the parameter gradient to obtain the constrained parameter gradient; updates the current parameter set to be calibrated based on the constrained parameter gradient to obtain the updated parameter set; determines whether the updated parameter set meets the preset convergence condition; if it does, the updated parameter set is determined as the target calibration parameter set; if it does not, the updated parameter set is used as the new current parameter set to be calibrated and joint iterative updates are continued. Based on the target calibration parameter set, projection error verification is performed on multiple observation perspectives, and the verified target calibration parameter set and parameter update instructions are output.

2. The method according to claim 1, characterized in that, The step of mapping the current set of parameters to be calibrated to the unified particle state representation to obtain the updated unified particle state representation includes: Extract the joint calibration parameters and the camera calibration parameters from the current set of parameters to be calibrated; The joint calibration parameters are written into the robot's tandem kinematics chain to update the spatial state of the joint particle set; Write the camera calibration parameters into the camera mounting pose and update the spatial state of the camera particle set; The updated joint particle set and the camera particle set are combined with the scene object particle set to obtain the updated unified particle state representation.

3. The method according to claim 2, characterized in that, The step of writing the joint calibration parameters into the robot's cascaded kinematics chain and updating the spatial state of the joint particle set includes: The installation position error, installation rotation error, and zero point error corresponding to each joint are written into the kinematic transformation relationship of the corresponding joint to obtain the correction transformation of each joint; The cumulative pose of each joint is obtained by combining the correction transformations of each joint along the robot's serial kinematic chain according to the joint connection sequence. The spatial position of the corresponding joint particle is updated based on the cumulative pose of each joint to obtain the updated spatial state of the joint particle set.

4. The method according to claim 3, characterized in that, The step of combining the correction transformations of each joint along the robot's serial kinematic chain according to the joint connection sequence to obtain the cumulative pose of each joint includes: Starting from the first joint corresponding to the robot's base coordinate system, the correction transformation of the current joint is combined with the cumulative pose of the previous joint to obtain the cumulative pose of the current joint. The cumulative pose of the current joint is used as the previous cumulative pose of the next joint, and the combination is executed step by step according to the joint connection order. After completing the stepwise combination of the robot's serial kinematics chain, the cumulative pose corresponding to each joint is obtained.

5. The method according to any one of claims 1-4, characterized in that, The acquisition of synchronous observation data corresponding to multiple observation perspectives includes: Select static objects in the scene as observation objects, and plan the robot's multi-view observation trajectory relative to the static objects; The robot is driven to sequentially reach the observation poses corresponding to the multi-view observation trajectory, and the robot joint state acquisition and camera image acquisition are triggered synchronously at each observation pose. The correspondence between the robot joint state data and the camera images is established according to the synchronization trigger time, and the validity screening of the collected data is performed to obtain the synchronization observation data.

6. The method according to claim 5, characterized in that, After obtaining the synchronous observation data, generating a foreground mask for scene objects based on the camera image includes: Distortion correction is performed on the camera image based on the camera intrinsic parameters to obtain a corrected image; The corrected image is subjected to noise suppression and color cast correction to obtain a preprocessed image; Perform foreground and background segmentation on the preprocessed image to obtain an initial foreground mask; Morphological processing is performed on the initial foreground mask to remove isolated noise regions and internal holes, thus obtaining the foreground mask.

7. The method according to any one of claims 1-4, characterized in that, After the joint iterative update satisfies the preset convergence condition and before the target calibration parameter set is obtained, the method further includes: Based on the set of parameters to be calibrated that meet the preset convergence conditions, multiple sets of projected particle points from different observation perspectives are regenerated. Outlier particles are identified based on the neighborhood distance of each projected particle, and the outlier particles are removed from the unified particle state representation to obtain the filtered particle state representation. The joint iterative update is performed again based on the filtered particle state characterization, and the parameter set obtained from the second joint iterative update is determined as the target calibration parameter set.

8. A joint calibration system for coupling error between robot joints and a camera, characterized in that, include: The data acquisition module is used to acquire synchronous observation data and a set of parameters to be calibrated corresponding to multiple observation perspectives. The synchronous observation data includes robot joint state data and camera images, and the set of parameters to be calibrated includes joint calibration parameters and camera calibration parameters. The unified representation module is used to generate a foreground mask of scene objects based on the camera image, and to construct a unified particle state representation including a joint particle set, a camera particle set, and a scene object particle set based on the parameter set to be calibrated and the foreground mask. The coupled observation module is used to map the scene object particle set to the corresponding camera image based on the robot joint state data from each observation viewpoint, forming a projected particle point set and a mask sampling point set. The mask sampling point set is obtained from the foreground mask corresponding to the same observation viewpoint. The joint optimization module is used to construct a neighborhood distance objective function based on the projected particle point set and the mask sampling point set from multiple observation perspectives, and to perform joint iterative updates on the parameter set to be calibrated based on the neighborhood matching error represented by the neighborhood distance objective function to obtain the target calibration parameter set. This includes: inputting the current parameter set to be calibrated into a differentiable computation chain, calculating the objective function value of the neighborhood distance objective function and obtaining the parameter gradient; mapping the current parameter set to be calibrated to the unified particle state representation to obtain the updated unified particle state representation; and using the updated unified particle state representation and the number of robot joint states from each observation perspective. Generate a corresponding set of projected particle points; calculate the objective function value based on the set of projected particle points and the corresponding set of masked sampling points; perform inverse automatic differentiation on the objective function value along the differentiable computation chain to obtain the parameter gradient; perform gradient magnitude constraint on the parameter gradient to obtain the constrained parameter gradient; update the current parameter set to be calibrated based on the constrained parameter gradient to obtain the updated parameter set; determine whether the updated parameter set meets the preset convergence condition; if it does, determine the updated parameter set as the target calibration parameter set; if it does not, use the updated parameter set as the new current parameter set to be calibrated and continue to perform joint iterative updates. The result output module is used to perform projection error verification on multiple observation angles according to the target calibration parameter set, and output the target calibration parameter set that has passed the verification and the parameter update instruction.

Images