A deformation online measurement and management method for multi-robot collaborative assembly

Abstract

The application relates to a deformation online measurement and management method for multi-robot collaborative assembly, which comprises the following steps: collecting multi-view surface images of a workpiece in a real-time manner during assembly through a distributed robot and a camera arranged at the tail end of the robot; inputting each view surface image into a deep learning model based on a digital image correlation technology, outputting optimal pixel displacements corresponding to each view surface image, and converting the optimal pixel displacements corresponding to each view surface image into spatial displacement labels; the deep learning model takes an encoder-decoder as a backbone network; fusing the spatial displacement labels corresponding to each view surface image to obtain a fused displacement field; determining the tail end pose of the distributed robot based on the fused displacement field and a nominal path planning point; calculating an assembly error based on a reference target positioning and the tail end pose, and adjusting the joint space of the distributed robot based on a PID controller and the assembly error. The method significantly reduces the calculation overhead and improves the measurement accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-robot collaborative assembly, in particular to a deformation online measurement and control method for multi-robot collaborative assembly. BACKGROUND

[0002] In the assembly process of complex products (such as aviation thin-walled structures, automobile body-in-white, and precision instrument assembly), parts are easily affected by multiple factors such as clamping, positioning, and fastening force, resulting in elastic or plastic deformation. Especially in the complex scenario of multi-robot collaborative assembly, parts are prone to non-uniform micron-level deformation due to distributed load, assembly dynamics, and environmental disturbance, leading to assembly error accumulation and geometric consistency decline, thereby affecting the overall precision and functional performance of the product.

[0003] Traditional digital image correlation (DIC) deformation measurement methods have large computational load and poor real-time performance, making it difficult to achieve efficient global data fusion and precision protection in multi-view robot systems. Moreover, there is a lack of deep linkage with the online control link, which cannot quickly compensate for assembly errors or respond to on-site abnormalities. SUMMARY

[0004] Therefore, it is necessary to provide a deformation online measurement and control method for multi-robot collaborative assembly, comprising:

[0005] S1: Real-time acquisition of multi-view surface images of workpieces in the assembly process by distributed robots and cameras arranged at the ends thereof; and input of each view surface image into a deep learning model based on digital image correlation technology, output of the optimal pixel displacement corresponding to each view surface image, and conversion of the optimal pixel displacement corresponding to each view surface image into a spatial displacement label; the deep learning model takes an encoder-decoder as the backbone network;

[0006] S2: Fusion of the spatial displacement labels corresponding to each view surface image to obtain a fused displacement field;

[0007] S3: Determination of the end pose of the distributed robot based on the fused displacement field and the nominal path planning point; calculation of the assembly error based on the reference target positioning and the end pose; and adjustment of the joint space of the distributed robot by the PID controller based on the assembly error.

[0008] Preferably, the training process of the deep learning model comprises:

[0009] Step 1: Obtain the surface speckle images and each deformation frame image of the workpiece in the historical assembly process by the cameras arranged at the ends of the distributed robots;

[0010] Step 2: Perform pixel de-distortion processing on each deformation frame image based on the distortion parameters and the intrinsic parameters of the camera;

[0011] Step 3: Based on the surface speckle image and the distortion-processed deformed frame images, the optimal pixel displacement corresponding to each deformed frame image and the pixel-level strain field component are calculated by using the digital image correlation technology.

[0012] Step 4: Taking each deformed frame image and the camera parameter as the input of the deep learning model, and taking the segmentation prediction probability, the optimal pixel displacement corresponding to each deformed frame image and the pixel-level strain field component as the output, the deep learning model is trained.

[0013] Preferably, the calculation of the optimal pixel displacement corresponding to each deformed frame image comprises:

[0014] The zero-mean normalized cross-correlation search coefficient is calculated based on the surface speckle image and the deformed frame image in the pixel window, and the calculation formula is as follows:

[0015] ;

[0016] Wherein, represents the zero-mean normalized cross-correlation search coefficient when the pixel point is taken as the center and the displacement is assumed, represents the pixel window, represents the coordinates of any one pixel point in the pixel window, represents the pixel gray value of the surface speckle image at , represents the gray mean value of the surface speckle image in the pixel window, represents the pixel gray value of the deformed frame image at , represents the pixel gray value of the deformed frame image at , represents the gray mean value of the deformed frame image in the pixel window.

[0017] The optimal pixel displacement on each deformed frame image is obtained by maximizing the zero-mean normalized cross-correlation search coefficient corresponding to each deformed frame image.

[0018] Preferably, the calculation of the pixel-level strain field component corresponding to each deformed frame image comprises:

[0019] The central difference is made to the optimal pixel displacement to obtain the strain field component of each pixel point in the deformed frame image, and the calculation formula comprises:

[0020] ;

[0021] ;

[0022] ;

[0023] Wherein, represents a pixel point a principal strain component in the x direction, represents a pixel point a principal strain component in the y direction, represents a pixel point a shear strain component, represents a component of the optimal pixel displacement in the x direction, represents a component of the optimal pixel displacement in the y direction, represents a partial derivative.

[0024] Preferably, step 4 comprises:

[0025] based on the segmentation prediction probability, the optimal pixel displacement corresponding to each deformed frame image, the pixel-level strain field component, and the corresponding true value, a joint loss is calculated, and the joint loss is minimized to train the deep learning model;

[0026] The joint loss is represented as:

[0027] ;

[0028] wherein, represents the joint loss, represents the first loss weight, represents the second loss weight, represents the third loss weight, represents the fourth loss weight, represents the fifth loss weight, represents the segmentation prediction loss, represents the pixel displacement prediction loss, represents the true value of the optimal pixel displacement, represents the predicted optimal pixel displacement, represents the strain field prediction loss, represents the true value of the pixel-level strain field component, represents the predicted pixel-level strain field component, represents a smoothing regular, represents a multi-view consistency term.

[0029] Preferably, the optimal pixel displacement is converted into a spatial displacement label, represented as:

[0030] ;

[0031] ;

[0032] wherein, represents the spatial displacement label of the pixel point in the th view, Ri, i = 1, 2,..., N rotation matrix from the camera coordinate system corresponding to the i-th view to the world coordinate system, tangential displacement vector of pixel point predicted optimal pixel displacement, partial derivative, x direction, y direction.

[0033] Preferably, the calculation formula of the fusion displacement field is:

[0034] ;

[0035] wherein, fusion displacement field of pixel point spatial displacement label of pixel point total number of views, fusion confidence weight of pixel point

[0036] Preferably, the calculation formula of the assembly error is:

[0037] ;

[0038] ;

[0039] wherein, assembly error at time reference target positioning at time end pose of the distributed robot at time nominal path planning point at time fusion displacement field of pixel point

[0040] Preferably, the PID controller adjusts the joint space of the distributed robot based on the assembly error, comprising:

[0041] The PID controller calculates the compensation amount in the Cartesian space based on the assembly error, and the calculation formula is:

[0042] ;

[0043] wherein, assembly error at time​​​​​​​​​​​​ the compensation amount at the time point, the proportional of the PID controller, the integral of the PID controller, the differential gain of the PID controller, the compensation amount at the time point, the assembly error at the time point, the compensation amount at the time point, the assembly error at the time point, the integral of the assembly error;

[0044] convert the compensation amount into the increment of the joint space of the distributed robot, and the conversion formula is:

[0045]

[0046] wherein, the compensation amount at the time point, the increment of the joint space of the distributed robot at the time point, the compensation amount at the time point, the pseudo-inverse matrix of the Jacobian at the joint configuration at the time point;

[0047] adjust the joint space of the distributed robot according to the increment of the joint space of the distributed robot.

[0048] Preferably, S4 is further included:

[0049] S4.1: calculate the fusion strain field of the pixel point based on the fusion displacement field of the pixel point, and the calculation formula is:

[0050]

[0051]

[0052]

[0053] wherein, the principal strain component of the fusion pixel point in the x direction, the principal strain component of the fusion pixel point in the y direction, the shear strain component of the fusion pixel point , wherein, the component of the fusion displacement field at the time point in the x direction, the component of the fusion displacement field at the time point in the x direction, the component of the fusion displacement field at the time point in the x direction, the component of the fusion displacement field at the time point ​​​​a component of the fusion displacement field in the x direction, represents a pixel physical size difference in the x direction, represents a pixel physical size difference in the y direction;

[0054] S4.2: Calculate the maximum principal strain value of the pixel point based on the fusion strain field of the pixel point, and the calculation formula is:

[0055] ;

[0056] wherein, represents the maximum principal strain value of the pixel point ;

[0057] S4.3: Compare the maximum principal strain value of the pixel point with the principal strain safety threshold value, when the maximum principal strain value of the pixel point is greater than the principal strain safety threshold value, it is determined that the pixel point strain is abnormal, triggering an alarm and re-planning the assembly path and adjusting the process parameters; otherwise, it is determined that the pixel point strain is normal, and the next deformation measurement and control cycle is continued.

[0058] Beneficial effects: the method effectively solves the deficiencies of the existing method in real-time, fusion efficiency and control linkage, significantly reduces the calculation overhead, and improves the measurement accuracy and robustness. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0060] Figure 1 The flowchart of the deformation online measurement and control method for multi-robot collaborative assembly in the embodiments of the present application. DETAILED DESCRIPTION

[0061] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the specific embodiments of the present application will be described in detail below in conjunction with the drawings. In the following description, a lot of specific details are set forth in order to fully understand the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0062] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and are not to be construed as indicating or implying relative importance or an ordered ranking of the indicated technical features. Thus, features defined with "first", "second" or "third" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly and specifically limited.

[0063] Embodiment 1

[0064] As shown in the embodiment, a deformation online measurement and control method for multi-robot collaborative assembly is provided, comprising: Figure 1

[0065] S1: Real-time acquisition of multi-view surface images of workpieces in the assembly process by distributed robots and cameras arranged at the ends thereof; and input of each view surface image into a deep learning model trained based on a digital image correlation technology, output of optimal pixel displacement corresponding to each view surface image, and conversion of the optimal pixel displacement corresponding to each view surface image into a spatial displacement label; the deep learning model takes an encoder-decoder as a backbone network.

[0066] Specifically, the training process of the deep learning model comprises:

[0067] Step 1: Obtaining surface speckle images and each deformation frame image of workpieces in the historical assembly process by cameras arranged at the ends of distributed robots;

[0068] Step 2: Performing pixel de-distortion processing on each deformation frame image based on distortion parameters and intrinsic parameters of the camera;

[0069] Step 3: Calculating the optimal pixel displacement, pixel-level strain field component corresponding to each deformation frame image based on the surface speckle image and the de-distortion processed deformation frame image by using the digital image correlation technology;

[0070] Step 4: Training the deep learning model with each deformation frame image and camera parameters as input, and segmentation prediction probability, the optimal pixel displacement corresponding to each deformation frame image, and the pixel-level strain field component as output, the deep learning model is represented as: , represents the deep learning model, represents the predicted optimal pixel displacement, represents the predicted pixel-level strain field component , represents the segmentation prediction probability.

[0071] Further, the calculation of the optimal pixel displacement corresponding to each deformation frame image comprises:

[0072] ​The zero-mean normalized cross-correlation search coefficient is calculated based on the surface speckle image and the deformed frame image on a pixel window, and the calculation formula is:

[0073]

[0074] wherein, represents the zero-mean normalized cross-correlation search coefficient when the pixel point is taken as the center and the displacement is assumed, represents the pixel window (including the window pixel set taken the pixel point as the center), represents the coordinates of any one pixel point in the pixel window, represents the pixel gray value of the surface speckle image at , represents the gray mean value of the surface speckle image in the pixel window, represents the pixel gray value of the deformed frame image at , represents the pixel gray value of the deformed frame image at , represents the gray mean value of the deformed frame image in the pixel window.

[0075] The optimal pixel displacement on each deformed frame image is obtained by maximizing the zero-mean normalized cross-correlation search coefficient corresponding to each deformed frame image.

[0076] Further, the pixel-level strain field components corresponding to each frame of deformed image are calculated, including:

[0077] The central difference is made to the optimal pixel displacement to obtain the strain field components of each pixel point in the deformed frame image, and the calculation formula includes:

[0078]

[0079]

[0080]

[0081] wherein, represents the principal strain component of the pixel point in the x direction, represents the principal strain component of the pixel point in the y direction, represents the shear strain component of the pixel point , represents the component of the optimal pixel displacement in the x direction, represents the component of the optimal pixel displacement in the y direction, represents the partial derivative. ​​​​

[0082] In the present embodiment, step 4 comprises:

[0083] The segmentation prediction loss is calculated based on the segmentation prediction probability and the segmentation ground truth label, and the calculation formula is:

[0084] ;

[0085] wherein, represents the segmentation prediction loss, represents the set of pixel points, represents the segmentation ground truth label of the i-th pixel point, represents the segmentation prediction probability of the i-th pixel point, represents a smoothing parameter to prevent the denominator from being zero. The pixel displacement prediction loss is calculated based on the optimal pixel displacement corresponding to each deformed frame image and the true displacement, and the calculation formula is:

[0086]

[0087] ; ;

[0088] wherein, represents the pixel displacement prediction loss, represents the ground truth of the optimal pixel displacement, represents the predicted optimal pixel displacement, represents the confidence weight of the i-th pixel point, represents the predicted optimal pixel displacement of the i-th pixel point, represents the true displacement of the i-th pixel point, represents the Charbonnier loss function, represents the residual (the difference between the predicted optimal pixel displacement component and the true displacement component), represents a constant to prevent the denominator from being zero. The strain field prediction loss is calculated based on the pixel-level strain field component and the ground truth of the strain component, and the calculation formula is:

[0089]

[0090] ;

[0091] wherein, represents the strain field prediction loss, represents the ground truth of the pixel-level strain field component, represents the predicted pixel-level strain field component, represents the predicted pixel point main strain component in the x direction,​​​​​ represents a predicted pixel point a principal strain component in the y direction, represents a predicted pixel point a shear strain component of, represents a real pixel point a principal strain component in the x direction, represents a real pixel point a principal strain component in the y direction, represents a real pixel point a shear strain component of, represents a shear strain loss weight.

[0092] calculating a joint loss based on the segmentation prediction loss, the pixel displacement prediction loss, and the strain field prediction loss, and minimizing the joint loss to train the deep learning model;

[0093] The joint loss is represented as:

[0094] ;

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] wherein, represents a joint loss, represents a first loss weight, represents a second loss weight, represents a third loss weight, represents a fourth loss weight, represents a fifth loss weight, represents a segmentation prediction loss, represents a pixel displacement prediction loss, represents a ground truth of an optimal pixel displacement, represents a predicted optimal pixel displacement, represents a strain field prediction loss, represents a ground truth of a pixel-level strain field component, represents a predicted pixel-level strain field component, represents a smoothing regular, represents a multi-view consistency term, represents a multi-view geometric consistency loss, represents a weight parameter for measuring the confidence and the consistency loss, This represents the difference in loss between network confidence and DIC physical confidence across multiple perspectives. Indicates the first A collection of pixels from multiple camera views , They represent the first , From the perspective of the first Confidence weight of each pixel This represents the Charbonnier loss function. , They represent the first , Camera pixels from various perspectives Predicted global spatial displacement Represents the Euclidean distance norm. Indicates the first Pixels from various perspectives The network prediction confidence, Indicates the first Pixels from various perspectives The physical confidence level of DIC.

[0100] Furthermore, the optimal pixel displacement is converted into a spatial displacement label, represented as:

[0101] ;

[0102] ;

[0103] in, Indicates the first Pixels from various perspectives Spatial displacement label, Indicates from the first The rotation matrix from the camera coordinate system to the world coordinate system corresponding to each viewpoint. Represents pixels The tangential displacement vector, This represents the predicted optimal pixel displacement. This indicates finding the partial derivative. Indicates the x-direction. Indicates the y-direction.

[0104] S2: By fusing the spatial displacement labels corresponding to the surface images from various viewpoints, a fused displacement field is obtained. The formula for calculating the fused displacement field is:

[0105] ;

[0106] in, Represents pixels The fusion displacement field, Indicates the first pixel point under the i-th view angle , a spatial displacement tag of the pixel point under the i-th view angle , a total number of views , a fusion confidence weight of the pixel point under the i-th view angle

[0107] Further, the fusion confidence weight calculation formula is:

[0108] ;

[0109] wherein, a fusion confidence weight of the pixel point under the i-th view angle , an adjustment parameter , a zero-mean normalized cross-correlation search coefficient when the optimal pixel displacement is centered on the pixel point , a partial derivative , a normalization of all pixel points in the surface image of any one view angle , a zero-mean normalized cross-correlation search coefficient when the optimal pixel displacement is centered on the pixel point , and a minimum absolute value of x-direction and y-direction gradients.

[0110] S3: determining the end pose of the distributed robot based on the fusion displacement field and the nominal path planning point; calculating the assembly error based on the reference target positioning and the end pose, and adjusting the joint space of the distributed robot based on the assembly error by the PID controller.

[0111] Specifically, the calculation formula of the assembly error is:

[0112] ;

[0113] ;

[0114] wherein, an assembly error at the moment , a reference target positioning at the moment , an end pose of the distributed robot at the moment , a nominal path planning point at the moment , and a fusion displacement field at the moment ​​​​​​​​​​​pixel point at time t fusion displacement field.

[0115] Further, the PID controller adjusts the joint space of the distributed robot based on the assembly error, comprising:

[0116] The compensation amount of the PID controller in the Cartesian space is calculated based on the assembly error, and the calculation formula is:

[0117] ;

[0118] wherein, represents the compensation amount at time t, represents the proportional of the PID controller, represents the integral of the PID controller, represents the differential gain of the PID controller, represents the assembly error at time t, represents the assembly error at time t, represents the integral of the assembly error; ;

[0119] The compensation amount is converted into the increment of the joint space of the distributed robot, and the conversion formula is:

[0120] ;

[0121] wherein, represents the increment of the joint space of the distributed robot at time t, represents the pseudo-inverse matrix of the Jacobian at time t under the joint configuration; The joint space of the distributed robot is adjusted according to the increment of the joint space of the distributed robot.

[0122] In this embodiment, S4 is also included:

[0123] S4.1: Calculate the fusion strain field of the pixel point based on the fusion displacement field of the pixel point, and the calculation formula is:

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] wherein, represents the principal strain component of the fused pixel point in the x direction, represents the principal strain component of the fused pixel point​​​ a principal strain component in the y direction, a shear strain component of the fusion pixel point at the fusion displacement field a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a component of the fusion displacement field in the x direction at a pixel physical size difference in the x direction; a pixel physical size difference in the y direction;

[0129] S4.2: Calculate the maximum principal strain value of the pixel point based on the fusion strain field of the pixel point, and the calculation formula is:

[0130]

[0131] wherein, the maximum principal strain value of the pixel point ;

[0132] S4.3: Compare the maximum principal strain value of the pixel point with the principal strain safety threshold value, when the maximum principal strain value of the pixel point is greater than the principal strain safety threshold value, it is determined that the pixel point strain is abnormal, triggering an alarm and re-planning the assembly path and adjusting the process parameters; otherwise, it is determined that the pixel point strain is normal, and returning to step S1 to continue the next deformation measurement and control cycle.

[0133] The deformation online measurement and control method for multi-robot collaborative assembly provided by the embodiment has the following beneficial effects:

[0134] The method can realize distributed, high-precision, multi-source spatio-temporal data synchronous collection in the assembly process, and obtain accurate and real-time global deformation field under a wide range and multiple angles; at the same time, it supports real-time compensation of assembly deviation caused by deformation, and integrates automatic abnormal detection and re-planning mechanism, effectively deals with sudden working conditions, and comprehensively improves the assembly quality and safety of the system.

[0135] Embodiment 2

[0136] ​​To verify the application effect of the deformation online measurement and control method for multi-robot collaborative assembly, the thin-walled aviation skin and frame assembly is selected as the object, and the assembly experiment platform composed of two six-axis industrial robots (model: ABB IRB 4600, degree of freedom: 6) and multi-view high-resolution industrial cameras (model: Basler acA4112-30um, resolution: 4112×3008 pixels, pixel size: 3.45μm) is built. The platform integrates digital image correlation (DIC) and deep learning end-to-end prediction to form an integrated closed-loop process of measurement, feature fusion, assembly control and abnormal handling. The specific implementation process is as follows, described according to the method module, and the actual parameters and effects of each module are emphasized.

[0137] 1. Distributed sensing and data acquisition: In the experimental platform, first, the robot collaborative assembly unit is constructed, and the industrial camera is installed at the end of each robot. The spatial coordinate calibration of base, end and camera is completed through chain homogeneous transformation (calibration accuracy: ±0.05 mm, ArUco marker plate is used for hand-eye calibration). The camera intrinsic matrix K is obtained through chessboard calibration (focal length f x = f y =5000 pixels, center offset c x = c y =2056 pixels), and the distortion parameters are calibrated by OpenCV (radial distortion k 1=-0.1, k 2=0.05). During the assembly process, the system synchronously collects reference frame and deformation frame images at a frequency of 40 Hz (image resolution: 2048×1536 pixels, depth d is obtained by laser range finder, accuracy ±0.1 mm), combined with the real-time recorded robot pose (obtained through ROS system, sampling rate 50 Hz) and process parameters, covering the dynamic deformation process of clamping, positioning and fastening.

[0138] 2. DIC (digital image correlation) deformation measurement and multi-view fusion: based on the collected image data, high-contrast random speckle (speckle size: 0.5-1 mm, evenly distributed using a spray machine, contrast >80%) is sprayed on the workpiece surface, and zero-mean normalized cross-correlation technology (window size S=31×31 pixels, sub-pixel interpolation accuracy 0.01 pixels) is used, combined with correlation coefficient threshold (C>0.8) and edge intensity (minimum gradient >0.1) to obtain high-precision pixel displacement and strain. According to the physical scale of pixels δ x = δ y=3.45 μm) and camera-to-world coordinate transformation (rotation matrix R extracted from calibration), map each view data to a unified spatial displacement label.

[0139] 3. Deep learning end-to-end prediction: take the encoder-decoder as the backbone network (based on U-Net architecture, layer number: 5 layers of encoding + 5 layers of decoding, training data set: 5000 pairs of images, batch size 32), input the de-distorted speckle pattern (resolution 2048x1536) and camera parameters (internal parameter K and transformation T), output pixel-level horizontal and vertical displacement and strain field, and simultaneously predict the probability of semantic segmentation to filter the non-interest area (segmentation threshold 0.5). Take the DIC calculation result as a high-precision supervision signal, and use loss combination (loss weight λ 1=1、 λ 2=0.5、 λ 3=0.5、 λ 4=0.2、 λ 5=0.3) and confidence weighted training method (optimizer: Adam, learning rate 0.001, iteration 200 rounds).

[0140] Take the displacement and label calculated by DIC as the "physical teacher" supervision signal, train the pixel-level deep learning model, replace the computationally intensive full DIC method, and realize millisecond-level online prediction while maintaining physical credibility. In actual implementation, the innovation optimizes displacement and strain output through joint loss function, and filters non-interest areas combined with semantic segmentation prediction probability, thereby improving the efficiency and accuracy of industrial vision applications.

[0141] 4. Multi-view fusion and confidence weighting: through coordinate index alignment (spatial resolution 0.1 mm), unify the multi-view spatial displacement prediction to the world coordinate system (view M=2), based on network confidence (threshold >0.7) and quality weight adaptive weighted fusion (outlier threshold: displacement deviation >20 μm removed), obtain the global displacement field, and further calculate the global strain tensor and principal strain distribution (center difference step size 2 pixels, pixel physical scale δ x = δ y ==3.45μm)。

[0142] Based on the unified space-time alignment of the full-link homogeneous transformation, combined with the fusion confidence weight for adaptive weighted fusion, the global displacement / strain field is robustly constructed. In the experiment, the innovation eliminates the multi-view blind area, ensures data consistency through consistency constraint, is suitable for distributed robot systems, and greatly improves the precision and robustness of large-scale assembly monitoring.

[0143] 5. Online error compensation and feedback control: The fused displacement is used to calculate the real-time end-assembly error (the nominal path is generated by the CAD model, the target accuracy is ±0.1 mm), and drive the Cartesian space PID controller (proportional K p = 0.8, integral K i = 0.2, differential gain K d = 0.1) to generate compensation instructions in real time, which are converted into robot joint space adjustments (joint speed limit 50° / s) through Jacobian pseudo-inverse mapping (joint configuration q is obtained through the robot API, and pseudo-inverse calculation uses the SVD method). For large deformation areas, additional adjustments are made to the clamping force (from 50 N to 30 N) and the tightening sequence (preferably in the low strain area).

[0144] 6. Strain judgment and assembly re-planning: Compare the maximum principal strain value with the principal strain safety threshold (principal strain safety threshold threshold ε_thresh = 0.05%) and make a judgment. If the result is abnormal, trigger an automatic alarm (through the PLC system, response time <0.5s), and optimize the assembly path (re-plan using the A* algorithm, path length reduced by 10%), adjust the force control parameters (clamping force reduced by 20%) or change the tightening strategy (pause the current process, jump to the backup sequence), to ensure the stability of the assembly process. After processing the abnormal data, the system automatically returns to the closed-loop iteration process.

[0145] Map the fused deformation field to the assembly error, link the PID controller for path and force control compensation, and trigger abnormal detection and multi-dimensional re-planning with the principal strain threshold and composite score. In actual testing, this innovation effectively prevents damage risks without interrupting production, and improves multi-cycle assembly consistency through closed-loop iteration.

[0146] This method realizes online deformation measurement and control in multi-robot collaborative assembly by integrating DIC high-precision measurement and deep learning end-to-end prediction, uses distributed perception, multi-view fusion and closed-loop compensation mechanism to form a real-time, accurate and safe closed-loop process. This method uses DIC physical label to supervise the training of efficient model, combines confidence weighted fusion and abnormal re-planning, significantly reduces the computational overhead, improves the measurement accuracy and system robustness, is suitable for complex assembly scenes such as aviation thin-walled structures, and effectively solves the problems of poor real-time performance and insufficient control linkage of traditional methods.

[0147] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but as long as the combinations of the technical features do not contradict, they should be considered within the scope of the present disclosure.

[0148] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A deformation online measurement and management method for multi-robot collaborative assembly, characterized in that, The method comprises the following steps: S1: acquiring multi-view surface images of a workpiece in a real-time manner through a distributed robot and a camera arranged at the end of the robot during assembly; and inputting each view surface image into a deep learning model trained based on a digital image correlation technology to output an optimal pixel displacement corresponding to each view surface image, and converting the optimal pixel displacement corresponding to each view surface image into a spatial displacement label; the deep learning model takes an encoder-decoder as a backbone network; the training process of the deep learning model comprises the following steps: Step 1: acquiring surface speckle images and each deformation frame image of the workpiece in a historical assembly process through a camera arranged at the end of the distributed robot; Step 2: performing pixel distortion correction on each deformation frame image based on distortion parameters and internal parameters of the camera; Step 3: calculating the optimal pixel displacement corresponding to each deformation frame image, and a pixel-level strain field component based on the surface speckle image and the distortion-corrected deformation frame image by using a digital image correlation technology; Step 4: training the deep learning model by taking each deformation frame image and camera parameters as inputs and taking segmentation prediction probability, the optimal pixel displacement corresponding to each deformation frame image and the pixel-level strain field component as outputs, comprising: calculating a joint loss based on the segmentation prediction probability, the optimal pixel displacement corresponding to each deformation frame image and the pixel-level strain field component and a corresponding true value, and minimizing the joint loss to train the deep learning model; the joint loss is expressed as: ； wherein, denotes a joint loss, denotes a first loss weight, denotes a second loss weight, denotes a third loss weight, denotes a fourth loss weight, denotes a fifth loss weight, denotes a segmentation prediction loss, denotes a pixel displacement prediction loss, denotes a ground truth of an optimal pixel displacement, denotes a predicted optimal pixel displacement, denotes a strain field prediction loss, denotes a ground truth of a pixel-level strain field component, denotes a predicted pixel-level strain field component, denotes a smoothing regularizer, denotes a multi-view consistency term; S2: fusing the spatial displacement labels corresponding to each view surface image to obtain a fused displacement field; S3: determining an end pose of the distributed robot based on the fused displacement field and a nominal path planning point, and calculating an assembly error based on the reference target positioning and the end pose, wherein the calculation formula of the assembly error is: ； ； wherein, represents assembly error at a time instant, represents reference target positioning at a time instant, represents end pose of a distributed robot at a time instant, represents nominal path planning point at a time instant, represents fusion displacement field of pixel points at a time instant fusion displacement field of pixel points at a time instant adjusting a joint space of the distributed robot based on the assembly error by using a PID controller.

2. The morphing on-line measurement and management method for multi-robot collaborative assembly according to claim 1, characterized in that, calculating the optimal pixel displacement corresponding to each deformation frame image comprises: calculating a zero-mean normalized cross-correlation search coefficient on a pixel window based on the surface speckle image and the deformation frame image, and the calculation formula is: ； wherein, represents a zero-mean normalized cross-correlation search coefficient of a pixel point as the center and assuming displacement , represents a pixel window, represents the coordinates of any one pixel point in the pixel window, represents the pixel gray value of the surface speckle image at , represents the average gray value of the surface speckle image in the pixel window, represents the pixel gray value of the deformed frame image at , represents the pixel gray value of the deformed frame image at , represents the average gray value of the deformed frame image in the pixel window; maximizing the zero-mean normalized cross-correlation search coefficient of each deformation frame image to obtain the optimal pixel displacement on the deformation frame image.

3. The morphing on-line measurement and control method for multi-robot collaborative assembly according to claim 1, wherein, calculating the pixel-level strain field component corresponding to each deformation frame image comprises: performing central difference on the optimal pixel displacement to obtain a strain field component of each pixel point in the deformation frame image, and the calculation formula comprises: ； ； ； wherein, denotes a pixel point a principal strain component in the x direction, denotes a pixel point a principal strain component in the y direction, denotes a pixel point a shear strain component, denotes a component of the optimal pixel displacement in the x direction, denotes a component of the optimal pixel displacement in the y direction, denotes a partial derivative.

4. The morphing on-line measurement and control method for multi-robot collaborative assembly according to claim 1, wherein, the optimal pixel displacement is converted into a spatial displacement label, and the conversion is expressed as: ； ； in, Indicates the first Pixels from various perspectives Spatial displacement label, Indicates from the first The rotation matrix from the camera coordinate system to the world coordinate system corresponding to each viewpoint. Represents pixels The tangential displacement vector, This represents the predicted optimal pixel displacement. This indicates finding the partial derivative. Indicates the x-direction. Indicates the y-direction.

5. The morphing on-line measurement and control method for multi-robot collaborative assembly according to claim 1, wherein, the calculation formula of the fused displacement field is: ； wherein, denotes the fused displacement field of the pixel point , denotes the spatial displacement label of the pixel point at the th view, denotes the total number of views, denotes the fused confidence weight of the pixel point at the th view.

6. The morphing on-line measurement and control method for multi-robot collaborative assembly according to claim 1, wherein, adjusting the joint space of the distributed robot based on the assembly error by using the PID controller comprises: calculating a compensation amount of the PID controller in a Cartesian space based on the assembly error, and the calculation formula is: ； wherein represents compensation amount at the time point, represents a proportion of the PID controller, represents an integral of the PID controller, represents a differential gain of the PID controller, represents assembly error at the time point, represents assembly error at the time point, represents an integral of the assembly error; converting the compensation amount into an increment of the joint space of the distributed robot, and the conversion formula is: ； wherein, denotes the incremental distributed robot joint space at the time instant, denotes the pseudo-inverse of the Jacobian matrix at the joint configuration at the time instant adjusting the joint space of the distributed robot based on the increment of the joint space of the distributed robot.

7. The morphing on-line measurement and control method for multi-robot collaborative assembly according to claim 5, wherein, Further comprising S4: S4.1: calculating a fused strain field of a pixel point based on the fused displacement field of the pixel point, and the calculation formula is: ； ； ； wherein, denotes the pixel point after fusion the principal strain component in the x direction, denotes the pixel point after fusion the principal strain component in the y direction, denotes the pixel point after fusion the shear strain component of, denotes the component of the fusion displacement field at in the x direction, denotes the component of the fusion displacement field at in the x direction, denotes the component of the fusion displacement field at in the x direction, denotes the component of the fusion displacement field at in the x direction, denotes the difference in the physical size of the pixels in the x direction, denotes the difference in the physical size of the pixels in the y direction; S4.2: calculating a maximum principal strain value of a pixel point based on the fused strain field of the pixel point, and the calculation formula is: ； wherein represents the maximum principal strain value of the pixel point ; S4.3: comparing the maximum principal strain value of the pixel point with the principal strain safety threshold value, when the maximum principal strain value of the pixel point is greater than the principal strain safety threshold value, determining that the pixel point strain is abnormal, triggering an alarm and re-planning the assembly path and adjusting the process parameters; otherwise, determining that the pixel point strain is normal, continuing to the next deformation measurement and control cycle.

Images