Robot vision external parameter calibration method based on blind deblurring and natural vibration measurement

Abstract

The invention discloses a robot vision external parameter calibration method based on blind deblurring and natural vibration measurement, and the method comprises the steps: collecting a continuous image sequence in a vibration environment through a high-frame-rate camera, extracting pixel-level motion information in combination with an advanced image deblurring deep network and an optical flow feature tracking technology, and carrying out the self-vibration measurement. And further restoring a three-dimensional displacement signal caused by natural vibration of the camera based on a pinhole imaging model, and obtaining ideal position information through frequency domain analysis to realize high-precision visual system external parameter calibration under a vibration condition.

Description

Technical Field

[0001] The present invention relates to the technical field of extrinsic parameter calibration, and in particular to a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement. Background Art

[0002] Robot vision systems are key perception units in modern industrial automation and intelligent manufacturing, widely used in tasks such as robot navigation, target recognition, 3D measurement, and path planning. As a crucial sensing tool in fields such as intelligent manufacturing, precision assembly, and high-end testing, their performance relies heavily on accurate extrinsic calibration (i.e., determining the position and attitude relationship between the camera and the robot's end effector or world coordinate system). To ensure the accuracy of the spatial coordinate relationship between visual information and the robot's motion control system, the vision system requires extrinsic calibration. This involves precisely determining the position and attitude parameters of the camera coordinate system relative to the robot or world coordinate system. This accuracy directly impacts the spatial precision and robustness of subsequent visual measurement, vision-guided control, and other tasks.

[0003] Traditional external parameter calibration methods mainly rely on the assistance of calibration plates. They collect images in specific postures and solve them with mathematical models, or use external high-precision sensors (such as laser trackers, accelerometers, IMU inertial measurement units, etc.) for posture detection. Although these methods can achieve high accuracy in static or experimental environments, in actual applications, in the presence of mechanical vibration or high-speed dynamic conditions of robots, they are often affected by dynamic interference, image blur, and accumulated external sensor errors. These factors not only increase system complexity and cost, but also lead to unstable calibration results and reduced accuracy, seriously affecting the actual deployment and engineering application of the system. Summary of the Invention

[0004] The present invention provides a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement to solve the problems of unstable calibration results and decreased accuracy in traditional extrinsic parameter calibration methods under mechanical vibration or high-speed robot operation dynamic conditions.

[0005] According to the first aspect, an embodiment provides a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement, the method comprising: The high-frame-rate HD camera of the robot vision system is used to continuously shoot the target area in a vibrating environment to obtain an image sequence and synchronize the time stamp; performing a blind deblurring operation on blurred images in the image sequence by an image restoration network comprising a blur kernel estimation network and a multi-scale deblurring network; Detect image feature points in the deblurred image sequence and perform inter-frame pixel motion tracking to obtain the pixel displacement signal of the feature points and the three-dimensional displacement time history signal of the camera's self-oscillation; Perform frequency domain analysis on the three-dimensional displacement time-history signal, obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix.

[0006] Furthermore, the high-frame-rate HD camera of the robot vision system is used to continuously shoot the target area in a vibrating environment to obtain an image sequence and synchronize the time stamp, specifically including: The camera's frame rate must be sufficient to clearly capture the camera's own tiny vibrations. A high-frame-rate camera captures image sequences with minimal inter-frame displacement, facilitating accurate tracking of its own vibrations. The camera's resolution must be sufficient to clearly capture the details of the target object, facilitating subsequent blind deblurring and feature point detection. The camera should be rigidly fixed to the end of the robot or on a stable bracket in a vibration-controlled environment. The installation position should ensure that the camera can observe the target area globally. The camera should not be actively moved during the extrinsic parameter calibration process to avoid introducing complex dynamic errors caused by non-self-vibration. The target area captured by the camera should include multiple stationary objects or high-texture areas to facilitate the subsequent extraction of trackable feature points in the image sequence and reduce tracking failures or errors caused by texture repetition and weak texture; the scene in the target area should remain stationary; the acquisition time should cover multiple vibration cycles to facilitate subsequent frequency domain analysis to extract the main frequency and directional components of the vibration.

[0007] Furthermore, a blind deblurring operation is performed on the blurred images in the image sequence by using an image restoration network including a blur kernel estimation network and a multi-scale deblurring network, specifically comprising: Perform fuzzy judgment on the collected image sequence data frame by frame; No processing is performed on image frames that are judged to be non-blurred; For image frames that are judged to be blurred, an image restoration network is used to perform deblurring processing, including: Use the blur kernel estimation network to dynamically estimate the shape and parameters of the blur kernel; The estimated blur kernel output by the blur kernel estimation network is upsampled to the size of the original blurred image, and then input together with the original blurred image into the Wiener Filtering (WF) for preliminary image motion blur restoration operation; The original blurred image and the preliminary restored image output by the Wiener filter are feature-concatenated and input into a multi-scale deblurring network for deblurring, and the final restored image is output.

[0008] Furthermore, the fuzzy judgment is performed on the collected image sequence data frame by frame, specifically including: Image blur judgment includes the Laplacian variance method. The Laplacian operator is used for edge detection in images. The size of its response value reflects the degree of change in pixel intensity at the edge. The edge richness in the image is quantified by calculating the pixel value variance of the Laplacian response image. If the variance is lower than the set threshold, the image is judged to be blurred.

[0009] Furthermore, in the image deblurring step, the image restoration network includes: 1. Use a blur kernel estimation network based on an encoder-decoder structure to dynamically estimate the shape and parameters of the blur kernel. Then perform Wiener filtering on the estimated blur kernel output by the blur kernel estimation network and the original blurred image, transforming the blind image processing problem into a non-blind processing problem, guiding the deblurring operation of the subsequent image restoration network. 2. A multi-scale deblurring network based on the Transformer structure and U-shaped framework is used to deblur the original blurred image. The low-level feature embedding is converted into deep features through a U-shaped network composed of a three-layer symmetric encoder-decoder. The original blurred image is downsampled three times in sequence and the results are input into different encoders respectively. An attention fusion module is added at the encoder input to extract features from input images of multiple scales to handle vibration blur of different scales and types. The encoder and decoder exchange feature information through jump connections, and each level of the encoder and decoder is connected by a multi-scale high-frequency feature extraction module to realize the screening of high-frequency feature information. The deep features are refined at high spatial resolution to retain the texture information of the original blurred image, making the restored image closer to the original blurred image.

[0010] In this embodiment, the multi-scale deblurring network includes a multi-scale feature fusion enhancement block (MSFEBlock) consisting of a multi-scale high-frequency feature extraction block (Multi-Scale High-Frequency Feature Extraction Block, HFFEB), a low-level convolutional block (Low-Level Convolutional Block, LLCB), an attention fusion block (Attention Fusion Block, AFB), a transformer block (Transformer Block, TB) and a lightweight residual block (Lightweight Residual Block, LRB).

[0011] Furthermore, the blur kernel estimation network specifically includes: The Blur Kernel Estimation Block (BKEB) adopts an encoder-decoder network structure. The encoder consists of a 3×3 convolution and four downsampling layers, and the decoder consists of four upsampling layers and a softmax layer. Each downsampling layer consists of two 3×3 depth-wise separable convolutions, and the upsampling layers 1, 2, and 3 are composed of one deconvolution and two 3×3 depth-wise separable convolutions. Finally, after the upsampling layer 4 and the Softmax layer consisting of one 2×2 convolution and two 3×3 convolutions, the output is an estimated blur kernel of size 16×16.

[0012] Furthermore, the multi-scale deblurring network specifically includes: The multi-scale deblurring network includes a three-level downsampling module, a three-layer shallow convolution module, a U-shaped network and a refinement layer; the U-shaped network adopts a three-layer symmetric encoder-decoder structure, including a three-layer encoder, a three-layer decoder and an intermediate layer, the three-layer encoder includes an encoder 1, an encoder 2 and an encoder 3 connected in sequence, the three-layer decoder includes a decoder 1, a decoder 2 and a decoder 3 connected in sequence, the encoder 3 and the decoder 3 are connected through the intermediate layer, the encoder 1 and the encoder 2, the encoder 2 and the encoder 3, and the encoder 3 and the intermediate layer are all connected through the attention fusion module, the encoder 1 and the decoder 1, the encoder 2 and the decoder 2, and the encoder 3 and the decoder 3 are all jump-connected through the multi-scale high-frequency feature extraction module; the three-level downsampling module is connected to the three-layer shallow convolution module, and the three-layer shallow convolution modules are respectively connected to the three-layer attention fusion module; the encoder 1 is also connected to a 3×3 convolution layer 1, the decoder 1 is connected to the refinement layer, and the refinement layer is also connected to a 3×3 convolution layer 2; The spliced features obtained by splicing the original blurred image and the preliminary restored image output by the Wiener filter are input into the convolution layer 1. The low-level feature embedding of the spliced features is obtained by using the convolution layer 1 and input into the encoder 1. The input original blurred image is downsampled three times in sequence through the three-stage downsampling module to obtain a three-sampled image; The three sampled images are input into three shallow convolution modules respectively to obtain shallow blur features at corresponding scales; The multi-scale shallow fuzzy features output by the three-layer shallow convolution module are input into the three-layer attention fusion module respectively. The input splicing features and the multi-scale shallow fuzzy features are fused step by step through the three-layer attention fusion module and the three-layer encoder. Each level of codecs is connected by a multi-scale high-frequency feature extraction module. The multi-scale high-frequency feature extraction module extracts high-frequency features from the shallow feature information generated by the encoder of the layer, and concatenates them with the low-resolution potential features output by the decoder of the next layer to form a composite feature, which is then input to the next decoder. The deep features output by decoder 1 are input to the refinement layer for processing at high spatial resolution. The convolution layer 2 is applied to the refined features to generate a residual image. Finally, the residual image is added to the original blurred image to obtain the final restored image.

[0013] Furthermore, the three encoders, three decoders, intermediate layers, and refinement layers all use a multi-scale feature fusion enhancement module, which includes a Transformer module and a lightweight residual module. The data processing process includes: For the input feature map, the corresponding global feature information and local feature information are first extracted through the Transformer module and the lightweight residual module respectively, and then added element by element; The added result is concatenated with the local feature information extracted from the input feature map by the lightweight residual module to increase the network receptive field and learn multi-scale information; Finally, 1×1 convolution is used to reduce the number of channels to keep it consistent with the initial number of channels; The residual connection is introduced to add the output feature map to the input feature map element by element, which makes the model converge faster and retains low-dimensional features.

[0014] Furthermore, the multi-scale high-frequency feature extraction module is mainly composed of a fully connected layer and an average pooling layer. The process of extracting high-frequency feature information includes: For the input feature map, it first passes through the fully connected layer to perform pixel-by-pixel full connection operations to generate a feature map after feature recoding; The re-encoded feature map is subjected to average pooling operations with kernel sizes of 2×2 and 4×4 respectively to obtain low-frequency feature maps of two scales, and then upsampled to the size of the original feature map respectively; Calculate the absolute value of the element-by-element subtraction between the two output results and the re-encoded feature map to obtain the high-frequency feature maps at two scales; The high-frequency feature maps at two scales are concatenated in the channel dimension, and finally the channels are fused and compressed through 1×1 convolution to obtain multi-scale high-frequency fusion features.

[0015] Furthermore, the shallow convolution module is mainly composed of 3×3 convolution and 1×1 convolution. Its feature extraction process for the downsampled blurred image includes: The input features undergo 4 convolution operations to extract local texture and edge information of the image; The output features after convolution are concatenated with the input features in the channel dimension; Finally, a 1×1 convolution is used to refine the features, compress the channel dimension and refine the feature expression.

[0016] Furthermore, the attention fusion module is mainly composed of a channel attention mechanism, which fuses the feature information extracted by the shallow convolution module with the features output by the previous scale encoder. The process includes: The feature information extracted by the shallow convolution module is multiplied element-by-element with the features output by the previous scale encoder, and the multiplication results are weighted by channel attention; Channel attention first performs global average pooling on the multiplied features and compresses them into a global description vector for each channel along the spatial dimension; The pooled channel description vector is input into a 1×1 convolutional fully connected network with a ReLU nonlinear activation function to model the dependencies between channels. Then, through a Sigmoid activation function, the network output is mapped to between 0 and 1 to generate the attention weight of each channel; Multiply these channel attention weights with each channel of the input multiplication feature channel by channel and output them to achieve channel-level weighted enhancement; The output of the channel attention is added element by element to the features output by the previous scale encoder to refine the features.

[0017] Furthermore, in the image deblurring step, the multi-scale loss calculation process includes: Comparing the true clear image with the restored image to obtain a first gap; comparing the true clear image downsampled once with the restored image downsampled once to obtain a second gap; comparing the true clear image downsampled twice with the restored image downsampled twice to obtain a third gap; comparing the true clear image downsampled three times with the restored image downsampled three times to obtain a fourth gap; and calculating the loss based on the first gap, the second gap, the third gap, and the fourth gap; For each scale gap, the first loss function, namely the content loss function, is calculated based on the L1 distance between the true clear image and the restored image; For each scale gap, the second loss function, namely the frequency loss function, is calculated based on the L1 distance between the true clear image and the restored image in the frequency domain; For each scale of the gap, the third loss function, namely the edge loss function, is calculated based on the L2 distance of the Laplacian operator between the true clear image and the restored image.

[0018] Furthermore, the image feature points of the deblurred image sequence are detected and pixel motion tracking is performed between frames to obtain pixel displacement signals of the feature points and three-dimensional displacement time history signals of the camera's self-oscillation, specifically including: Preprocess the deblurred image sequence so that each pixel accurately corresponds to the actual spatial position; Selecting a region of interest from the preprocessed image sequence and cropping the selected region to obtain a cropped image sequence; The ORB feature point detection algorithm is used to detect the corner features in the cropped image of the first frame; The Lucas-Kanade (LK) optical flow method is used to obtain the relative displacement of feature points between adjacent frame images, track the motion trajectory of feature points between adjacent frame images, and obtain the pixel displacement time-course signal of each feature point in the region of interest; Based on the pinhole camera model principle, the pixel displacement signal is reconstructed to obtain the discrete time-course signal of the camera's actual displacement in three-dimensional space. The three-dimensional displacement discrete time history signal is processed by numerical differentiation method to obtain the three-dimensional velocity time history signal of each characteristic point; The displacement signal is interpolated based on the velocity correction through the velocity signal and the displacement signal to obtain the reconstructed continuous three-dimensional displacement time history signal of the camera self-oscillation.

[0019] Furthermore, the deblurred image sequence is preprocessed, including: Use a checkerboard calibration plate to calibrate the camera and calculate the camera's intrinsic parameter matrix and distortion coefficients; Use the correction algorithm to correct the image distortion and remove the image distortion caused by the lens.

[0020] Furthermore, the process of the ORB feature point detection algorithm includes: Use FAST operator to quickly detect corner features; The candidate points are sorted using the Harris corner response function scoring mechanism, and the N points with the highest Harris response values are selected as key feature points. Using the grayscale value method, the main direction of each key feature point is calculated to achieve rotation invariance; A rotated BRIEF descriptor is constructed around each key feature point to achieve feature description.

[0021] Furthermore, the process of motion tracking using the Lucas-Kanade optical flow method includes: Define the neighborhood window and establish the optical flow constraint equation based on the constant brightness assumption; Combine the information of other pixels in the neighborhood window to establish a set of equations, and use the least squares method to solve the optical flow constraint equation; Update the feature point position based on the displacement obtained in the last calculation, recalculate the relevant parameters and solve the displacement, and iterate until the displacement change is less than a certain threshold; Multiply the calculated velocity by the time interval to obtain the displacement vector of the feature point between the two frames.

[0022] Furthermore, the displacement signal interpolation process includes: Extract two adjacent moments from the actual displacement discrete time-history signal and the actual velocity time-history signal and The displacement value 、 And the corresponding speed value 、 ; Calculate the time steps for these two moments ; Will and Add and divide by 2 to get the linear interpolation estimate at the midpoint; Calculating speed difference , and multiply by , get the velocity term used to correct the midpoint displacement; Add the linear interpolation estimate to the velocity correction term to get the midpoint time The final interpolated displacement

[0023] Furthermore, the three-dimensional displacement time-history signal is subjected to frequency domain analysis to obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and the extrinsic parameter matrix is calculated, specifically including: Perform frequency domain analysis on the component signals of the camera's self-vibration three-dimensional displacement time history signal in the three coordinate axis directions in the world coordinate system; The three DC components obtained after Fourier transform are used as the ideal world coordinates of the camera coordinate origin corresponding to the input original image sequence, and the ideal translation matrix extrinsic parameters of the camera at the corresponding position are obtained.

[0024] According to the second aspect, an embodiment provides a robot vision extrinsic parameter calibration system based on blind deblurring and self-vibration measurement, the system comprising: An image acquisition module is used to continuously capture the target area in a vibrating environment using a high-definition camera with a high frame rate of the robot vision system to acquire an image sequence and synchronize the time stamp; an image deblurring module, configured to perform a blind deblurring operation on blurred images in the image sequence by using an image restoration network comprising a blur kernel estimation network and a multi-scale deblurring network; The vibration signal extraction module is used to detect image feature points in the deblurred image sequence and track pixel motion between frames to obtain the pixel displacement signal of the feature points and the three-dimensional displacement time history signal of the camera's self-vibration; The ideal position calculation module is used to perform frequency domain analysis on the three-dimensional displacement time-history signal, obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix.

[0025] According to a third aspect, an embodiment provides an electronic device, the device comprising: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement as described in any one of the above items.

[0026] According to the fourth aspect, an embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement as described in any one of the above items are implemented.

[0027] The present invention provides a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement, which has the following beneficial effects: (1) The present invention performs blind deblurring on image sequences collected in a vibration environment, effectively eliminating image blurring interference caused by vibration, significantly improving image quality and thus improving calibration accuracy; (2) Traditional external parameter calibration methods mostly rely on static or low-interference environments. The present invention can effectively calibrate the visual system in the micro-vibration environment of actual operation, has good environmental adaptability, and is particularly suitable for robotic systems in high-precision visual control tasks; (3) The present invention relies only on the image sequence and timestamp information obtained by the robot vision system itself, combined with the motion tracking algorithm and the pinhole camera model. It can realize the construction of the camera's self-vibration signal and the external parameter calibration process without the help of additional external sensors such as accelerometers and IMU inertial measurement units, thereby simplifying system integration and reducing hardware costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 2 A flowchart of image blur restoration in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 3 A schematic diagram of an image deblurring network framework in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 4 A schematic diagram of a blur kernel estimation network structure in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 5 A schematic diagram of the structure of a shallow convolution module in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 6 A schematic diagram of the structure of an attention fusion module in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 7 A schematic diagram of the structure of a multi-scale feature fusion enhancement module in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 8 A schematic diagram of the Transformer module structure in a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement provided by one embodiment of the present invention; Figure 9 A schematic diagram of the structure of a lightweight residual module in a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 10 A schematic diagram of the structure of a multi-scale high-frequency feature extraction module in a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement provided by one embodiment of the present invention; Figure 11 A flowchart of a multi-component loss function supervision method for robot vision extrinsic parameter calibration based on blind deblurring and self-vibration measurement provided by one embodiment of the present invention; Figure 12 A flow chart of displacement vibration signal extraction in a robot vision extrinsic parameter calibration method based on blind defuzzification and self-vibration measurement is provided in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below by means of specific embodiments in conjunction with the accompanying drawings. Similar elements in different embodiments are numbered with associated similar elements. In the following embodiments, many detailed descriptions are provided to enable the present invention to be better understood. However, those skilled in the art will readily appreciate that some of the features may be omitted under different circumstances, or may be replaced by other elements, materials, or methods. In some cases, some operations related to the present invention are not shown or described in the specification. This is to avoid the core of the present invention being overwhelmed by excessive descriptions, and for those skilled in the art, it is not necessary to describe these related operations in detail. They can fully understand the related operations based on the description in the specification and the general technical knowledge in the art.

[0030] In addition, the features, operations, or characteristics described in the specification may be combined in any appropriate manner to form various embodiments. Furthermore, the steps or actions in the method description may be reordered or adjusted in a manner readily apparent to those skilled in the art. Therefore, the various sequences in the specification and drawings are provided solely for the purpose of clearly describing a particular embodiment and are not intended to be mandatory, unless otherwise specified.

[0031] The first embodiment of the present invention provides a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement, which realizes the calibration of the ideal extrinsic parameter matrix of the camera in a vibration environment by blind image processing and real displacement signal construction. Figure 1 Provide detailed explanation.

[0032] like Figure 1 As shown, in step S100, a high-definition camera with a high frame rate of the robot vision system is used to continuously shoot the target area in a vibrating environment to obtain an image sequence, and synchronize the time stamp.

[0033] Specifically, this embodiment uses a method to construct a camera self-oscillation signal and extrinsic parameter calibration process using image sequences and timestamp information acquired by the vision system itself. This process places the following requirements on camera selection, mounting, and shooting: The camera's frame rate must be sufficient to clearly capture the camera's own tiny vibrations. A high-frame-rate camera captures image sequences with minimal inter-frame displacement, facilitating accurate tracking of its own vibrations. The camera's resolution must be sufficient to clearly capture the details of the target object, facilitating subsequent blind deblurring and feature point detection. The camera is rigidly fixed to the end of the robot or on a stable bracket, but is placed in a vibration-controlled environment and installed in a position that ensures that the camera can observe the target area globally. The camera should not actively move during the extrinsic parameter calibration process to avoid introducing complex dynamic errors caused by non-self-vibration. The target area captured by the camera should contain multiple stationary objects or high-texture areas to facilitate the subsequent extraction of trackable feature points in the image sequence and reduce tracking failures or errors caused by texture repetition and weak texture. The target area scene should remain stationary, otherwise the image will have both pixel displacement caused by self-vibration and interference caused by the movement of external objects, affecting the subsequent execution of the method. The acquisition time covers multiple vibration cycles to facilitate subsequent frequency domain analysis to extract the main frequency and directional components of the vibration.

[0034] like Figure 1 As shown, in step S200, a blind deblurring operation is performed on the blurred images in the image sequence through an image restoration network including a blur kernel estimation network and a multi-scale deblurring network.

[0035] Specifically, the image deblurring process in this embodiment is as follows: Figure 2 As shown, the acquired image sequence data is subjected to blur judgment frame by frame. The image frames judged as not blurry are not processed. The image frames judged as blurry are deblurred using an image restoration network.

[0036] In this embodiment, the Laplacian variance method is used to determine image blur. The Laplacian operator is used for edge detection in images. The magnitude of its response value reflects the degree of change in pixel intensity at the edge. The edge richness in the image is quantified by calculating the pixel value variance of the Laplacian response image. If the variance is lower than a set threshold, the image is judged to be blurred. The calculation process is as follows: Laplacian is a second-order derivative operator, and the commonly used discrete kernel is:

[0037] Then the Laplacian response image is:

[0038] Assume that the image has N pixels in total, and define the mean of the Laplacian response value as:

[0039] Compute the Laplacian variance:

[0040] Then, the image frames that are judged to be non-blurred are not processed, and the image frames that are judged to be blurred are deblurred using the image restoration network.

[0041] In order to solve the problem that current image deblurring algorithms lack an explanation for the intermediate degradation process, this embodiment first proposes a blur kernel estimation network (BKEB), and performs Wiener filtering (WF) on the output of the BKEB and the original image to provide prior information for subsequent image deblurring; secondly, an attention fusion module (AFB) is designed to effectively aggregate feature information extracted from the downsampled blurred image; then, considering that images with severe blur usually require a larger receptive field to capture more nonlinear information, the present invention designs a multi-scale feature fusion enhancement module (MSFEBlock), which uses the Transformer module (TB) to capture long-range dependencies between different positions of the image and effectively learns global feature information, while using an improved lightweight residual module (LRB) to extract image features at different scales; finally, the present invention proposes a novel skip connection, designs and introduces a multi-scale high-frequency feature extraction module (HFFEB), and realizes multi-scale extraction of high-frequency feature information.

[0042] The overall framework of the image restoration network in this embodiment is as follows Figure 3 As shown in Figure 2, the original blurred image is input into the image restoration network, and the process of obtaining the output restored image is as follows: (1) A BKEB module based on a codec structure is used to dynamically estimate the shape and parameters of the blur kernel, and the estimated blur kernel output by the BKEB module and the original blurred image are subjected to WF processing, which converts the blind processing problem of the image into a non-blind processing problem and guides the deblurring operation of the subsequent image restoration network.

[0043] The BKEB module adopts a codec network structure, such as Figure 4 As shown in the figure, the encoder consists of a 3×3 convolution and 4 downsampling layers, and the decoder consists of 4 upsampling layers and a Softmax layer. Each downsampling layer consists of two 3×3 depth-separable convolutions, and upsampling layers 1, 2, and 3 consist of one deconvolution and two 3×3 depth-separable convolutions. Finally, after an upsampling layer 4 consisting of one 2×2 convolution and two 3×3 convolutions and a softmax layer, the output is an estimated blur kernel of size 16×16. The estimated blur kernel output by the BKEB module is upsampled to the original image size and then input into the WF together with the original blurred image to perform preliminary image motion blur restoration operations; When the blur kernel is determined, WF is expressed as:

[0044] in, , represents the complex conjugate, 、 Fourier transforms representing degradation functions and degradation images; It represents the ratio of the noise power spectrum to the image power spectrum. An empirical constant is usually used as a substitute. Here we take =0.01.

[0045] (2) The original blurred image and the preliminary restored image output by WF are feature-concatenated and input into the deblurring network. A 3×3 convolutional layer is used to obtain the low-level feature embedding of the concatenated features as the input of the first encoder. (3) Downsample the blurred image once, downsample the blurred image twice, and downsample the blurred image three times, input them into the deblurring network, and process them through the LLCB module respectively to obtain the shallow blur features of the corresponding scale; The first codec corresponds to the original scale of the original blurred image, the second codec corresponds to the scale of the original blurred image after downsampling once, the third codec corresponds to the scale of the original blurred image after downsampling twice, and the middle layer corresponds to the scale of the blurred image after downsampling three times; The structure of LLCB module is as follows Figure 5 As shown in Figure 1, the LLCB module is mainly composed of 3×3 convolution and 1×1 convolution. Its feature extraction process for the downsampled blurred image includes: The input features undergo 4 convolution operations to extract local texture and edge information of the image; The output features after convolution are concatenated with the input features in the channel dimension; Finally, a 1×1 convolution is used to refine the features, compress the channel dimension and refine the feature expression.

[0046] The LLCB module uses two pairs of 3×3 and 1×1 convolutional layers, and the features of the last 1×1 layer are respectively The feature concatenation is performed on the channel dimension at the same scale, and an additional 1×1 convolution layer is used to further refine the connected feature output. , to extract local texture and edge information of the image.

[0047] (4) The AFB module is used to gradually fuse the splicing features of the original blurred image and the preliminary restored image output by WF with the multi-scale shallow features generated by downsampling the blurred image once, downsampling the blurred image twice, and downsampling the blurred image three times.

[0048] The structure of the AFB module is as follows Figure 6 As shown, the feature information extracted by the LLCB module Multiply the feature output by the encoder of the previous scale element by element, and weight the multiplication result through channel attention; Channel attention first performs global average pooling on the multiplied features and compresses them along the spatial dimension into a global description vector for each channel. The pooled channel description vector is then input into a 1×1 convolutional fully connected network with a ReLU nonlinear activation function to model the dependencies between channels. A sigmoid activation function is then used to map the network output to a range of 0 to 1 to generate the attention weight for each channel. Multiply these channel attention weights with each channel of the multiplied features channel by channel and output them to achieve channel-level weighted enhancement; Finally, the output of the channel attention is added element by element to the features output by the previous scale encoder to refine the features; The specific process can be expressed as follows:

[0049] in, Represents the channel attention mechanism; represents element-wise multiplication; Indicates the encoder output; Indicates the feature information extracted by LLCB .

[0050] (5) Each level of codecs is connected by a jump connection through the HFFEB module to extract high-frequency features from the shallow feature information generated by the encoder, and concatenate them with the low-resolution potential features output by the next scale decoder to form a composite feature as the input of the next decoder.

[0051] The structure of the HFFEB module is as follows Figure 10 As shown in the figure, the HFFEB module mainly consists of a fully connected layer and an average pooling layer. The process of extracting high-frequency feature information includes: For the input feature map First, a pixel-by-pixel full connection operation is performed through the fully connected layer to generate a feature map after feature recoding ; The re-encoded feature map is subjected to average pooling operations with kernel sizes of 2×2 and 4×4 respectively to obtain low-frequency feature maps of two scales. , and upsample to the original feature map size respectively ; Calculate the absolute value of the element-by-element subtraction between the two output results and the re-encoded feature map to obtain the high-frequency feature maps at two scales; The high-frequency feature maps at two scales are concatenated in the channel dimension, and finally channel fusion and compression are performed through 1×1 convolution to obtain multi-scale high-frequency fusion features; The specific process can be expressed as follows:

[0052] in, Represent upsampling operation and fully connected layer operation respectively, represents the average pooling operation with a kernel size of k×k, represents feature map concatenation, represents a 1×1 convolution with channel compression.

[0053] (6) The deep features output by the first decoder are refined at a high spatial resolution, and a convolution layer is applied to the refined features to generate a residual image. Finally, the residual image is added to the original blurred image to be restored to obtain the final restored image.

[0054] Each level, intermediate layer, and refine layer of the first codec, the second codec, and the third codec are composed of an MSFEBlock module; The structure of the MSFEBlock module is as follows Figure 7 As shown in Figure 1, the MSFEBlock module mainly consists of the Transformer Block (TB) and the Lightweight Residual Block (LRB). The data processing process is as follows: For the input feature map F, the corresponding global feature information is first extracted through the TB module and the LRB module respectively. and local feature information , and then perform element-by-element addition; The result after adding and local feature information Perform feature splicing to increase the network receptive field and learn multi-scale information; Finally, we use 1×1 convolution to The number of channels is reduced to keep it consistent with the initial number of channels, and a residual connection is introduced to add the output feature map to the input feature map element by element, so that the model converges faster and retains low-dimensional features.

[0055] The specific process can be expressed as:

[0056]

[0057] in, represents feature map concatenation, represents a 1×1 convolution with channel compression.

[0058] The structure of TB module is as follows Figure 8As shown in the figure, it consists of a multi-head self-attention mechanism (MHSA) and a feed-forward fully connected network (FFN).

[0059] The structure of the LRB module is as follows Figure 9 As shown in the figure, compared with the traditional residual network, the batch normalization processing layer of the jump connection block is deleted, making the entire network structure simpler and reducing the computational complexity. The simplified network structure reduces the amount of computation and improves the training efficiency of the model. At the same time, the use of 3×3 convolutional layers can better capture the local features of the image, effectively reducing the gradient attenuation problem, making the model more stable and easier to converge during training.

[0060] (7) Calculate the multi-scale loss function and supervise the training process of the model; The process of multi-component and multi-scale loss supervision in this embodiment is as follows Figure 11 As shown, the true clear image is compared with the restored image, and the first gap is obtained as the loss of the first scale; the true clear image downsampled once is compared with the restored image downsampled once, and the second gap is obtained as the loss of the second scale; the true clear image downsampled twice is compared with the restored image downsampled twice, and the third gap is obtained as the loss of the third scale; the true clear image downsampled three times is compared with the restored image downsampled three times, and the fourth gap is obtained as the loss of the fourth scale; Based on the L1 distance between the real clear image and the restored image, the content loss function is calculated and expressed as follows:

[0061] Among them, where: Indicates scale, represents the clear image output by the network, Indicates a true and clear image. Represents the L1 norm and divides the calculated loss by the total number of elements Perform normalization processing; Based on the L1 distance between the true clear image and the restored image in the frequency domain, the frequency loss function is calculated and expressed as follows:

[0062] in, Represents the fast Fourier transform, while dividing the calculated loss by the total number of elements Perform normalization processing; Based on the L2 distance of the Laplacian operator between the real clear image and the restored image, the edge loss function is calculated and expressed as follows:

[0063] in, represents the Laplace operator, represents the L2 norm, is a very small number used to stabilize square root calculations; The final loss function of the training network is determined as follows:

[0064] In this embodiment, and Set to 0.01 and 0.05 respectively.

[0065] like Figure 1 As shown, in step S300, the image feature points of the deblurred image sequence are detected and inter-frame pixel motion tracking is performed to obtain pixel displacement signals of the feature points and three-dimensional displacement time history signals of the camera's self-oscillation.

[0066] Specifically, the process of extracting vibration signals in this embodiment is as follows: Figure 12 As shown in Figure 2, the process of obtaining the three-dimensional displacement time history signal of the camera's self-oscillation using the deblurred image sequence is as follows: (1) Perform image distortion correction preprocessing on the deblurred image sequence so that each pixel accurately corresponds to the actual spatial position, and select the region of interest on the preprocessed image sequence, crop the selected region, and obtain the cropped image sequence.

[0067] The preprocessing operation mainly uses camera calibration technology to remove lens distortion. The process steps are as follows: a. Prepare the calibration plate: This example uses a checkerboard calibration plate with a regular black and white checkered pattern. The size of the calibration plate and the grid size should be selected based on the actual situation. Generally, the grid size ranges from a few millimeters to tens of millimeters. In this example, a calibration plate with a grid side length of 25 mm is selected.

[0068] b. Capture calibration images: Place the calibration plate at different positions and angles, and use the camera to capture multiple images containing the calibration plate. Ensure that the calibration plate is completely within the image range and that the images are clear and unblurred. Typically, capture 10-20 images; in this example, 15 images are captured.

[0069] c. Detect corner points: Use image processing algorithms to detect the corner points of the chessboard in each calibration image. In this embodiment, the findChessboardCorners function in OpenCV is used for corner point detection; the corner points are the vertices of the chessboard squares, and the detected corner point coordinates will be used for subsequent calibration calculations.

[0070] d. Calibration calculation: Using the detected corner coordinates and the known calibration grid size, calculate the camera's intrinsic parameter matrix (including focal length, principal point coordinates, etc.) and distortion coefficients (including radial distortion coefficients and tangential distortion coefficients) through a calibration algorithm (such as Zhang Zhengyou calibration method). In this embodiment, the calibrateCamera function of OpenCV is used for calibration.

[0071] e. Image correction: Based on the calculated intrinsic parameter matrix and distortion coefficient, a correction algorithm is used to correct the large image sequence after deblurring to remove lens distortion. In this embodiment, the undistort function in OpenCV is used for image correction.

[0072] (2) Use the ORB feature point detection algorithm to detect the corner features in the cropped image of the first frame.

[0073] First, an image pyramid is constructed to perform multi-level scaling on the original image to adapt to feature detection at different scales. The image pyramid is expressed as follows:

[0074] in, is the original image, For the image level, Indicates the image scaling operation; scale represents the scaling factor between adjacent layers in the image pyramid, which is a number less than 1. In this embodiment, scale=0.8; In each layer of the image, the FAST algorithm is used to detect corner points and determine whether the grayscale difference of pixels in the ring around the pixel point is significant enough. The FAST detection conditions are:

[0075] in, is the center pixel intensity, is the ring pixel intensity, is the threshold, It is a set of 16 pixels on the ring; Then, the Harris response function is used to score the corner points obtained by FAST, and the N points with the highest Harris response values are selected as key feature points. The points with low response values are removed. The Harris response function is expressed as follows:

[0076] in, is the gradient of the image in the x and y directions, ; The main direction of each key feature point is calculated using the grayscale value method to ensure the rotation invariance of the descriptor. The main direction calculation formula is as follows:

[0077] in, is the moment of image grayscale, θ is the main direction angle; After the main direction is aligned, a rotated BRIEF descriptor is constructed around each key feature point for subsequent feature matching. The descriptor construction is expressed as follows:

[0078] in, is a pair of sampling points, rotated relative to the main direction of the key point.

[0079] (3) The LK optical flow method is used to obtain the relative displacement of feature points between adjacent frame images, track the motion trajectory of feature points between adjacent frame images, and obtain the pixel displacement time-course signal of each feature point in the region of interest.

[0080] The LK optical flow method is based on three major assumptions: Brightness constancy assumption: the pixel brightness of the same object does not change in two adjacent frames; Small motion assumption: the pixel positions between two adjacent frames cannot change significantly; Local consistency assumption: the motion state between adjacent pixels is consistent.

[0081] In this embodiment, the process of obtaining the pixel displacement time course signal based on the image sequence using the LK optical flow method includes: I(x,y,t) is the grayscale value of the pixel at coordinate (x,y) in the video frame at time t in the vibration video. Assuming that the pixel moves to the coordinate position (x+Δx,y+Δy) after time Δt, the following formula is obtained based on the optical flow brightness consistency constraint:

[0082] Wherein, Δx represents the distance that the pixel point moves along the x-axis within the time Δt, and Δy represents the distance that the pixel point moves along the y-axis within the time Δt; Expand the above equation using Taylor's formula, and ignore the second-order terms based on the assumption of small displacement motion to obtain the following equation:

[0083] Eliminate the same terms I(x, y, t) on both sides of the above equation and simplify it to:

[0084] in, 、 are the displacements in the x-axis and y-axis directions, is the time difference between the two frames. 、 and Represents the first-order partial derivatives of the gray value I with respect to x, y, and t, respectively, and can be calculated using the gradient operator Barron; Solving based on the least squares method and , the sub-pixel displacement of the target area in the horizontal and vertical directions can be obtained.

[0085] (4) Reconstruct the pixel displacement signal based on the pinhole camera model principle to obtain the discrete time-course signal of the camera’s actual displacement in three-dimensional space; The pinhole camera model is expressed as follows:

[0086] Where s represents the scaling factor, K is the camera intrinsic matrix, (u, v) is the image coordinate point, (X, Y, Z) is the world coordinate point, R is the rotation extrinsic matrix, and t is the translation extrinsic matrix. K and t are obtained through the degradation calibration process in the image sequence preprocessing operation. By bringing the pixel coordinates in the image into the pinhole camera model, the coordinates of the pixel point in the world coordinate system can be obtained. Therefore, by bringing the pixel displacement time-series signal obtained by the LK optical flow method into the pinhole camera model, the actual discrete time-series signal of the camera's displacement in the three-dimensional space can be obtained.

[0087] (5) The three-dimensional displacement discrete time history signal is processed by numerical differentiation method to obtain the three-dimensional velocity time history signal of each characteristic point; The three-dimensional velocity signal can be calculated from the three-dimensional displacement signal using the first-order difference approximation (i.e., forward difference or central difference) method. In this embodiment, the central difference method is used. Assuming that x(t), y(t), and z(t) are the displacement signals in the three directions at each moment, the velocity signals in the three directions obtained by the central difference method are expressed as:

[0088] (6) Perform displacement signal interpolation based on velocity correction through the velocity signal and displacement signal to obtain the reconstructed continuous camera self-oscillation three-dimensional displacement time history signal; Furthermore, the process of displacement signal interpolation includes: Extract two adjacent moments from the actual displacement discrete time-history signal and the actual velocity time-history signal and The displacement value 、 And the corresponding speed value 、 ; Calculate the time steps for these two moments ; Will and Add and divide by 2 to get the linear interpolation estimate at the midpoint; Calculating speed difference , and multiply by , get the velocity term used to correct the midpoint displacement; Add the linear interpolation estimate to the velocity correction term to get the midpoint time The final interpolated displacement

[0089] The specific process can be expressed as follows:

[0090] Among them, Y express The displacement signal at the moment, Y express The displacement signal at time express The speed signal at the moment, express Time-sensitive response, , f represents the sampling rate of the camera, take , the above four formulas are obtained and The reconstructed camera self-vibration three-dimensional displacement continuous time history signal z(t) is obtained by combining them in time sequence.

[0091] like Figure 1 As shown, in step S400, the three-dimensional displacement time history signal is subjected to frequency domain analysis, the DC components of the signal in three directions are obtained as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and the extrinsic parameter matrix is calculated.

[0092] The above steps specifically include: Perform frequency domain analysis on the component signals of the camera's self-vibration three-dimensional displacement time history signal in the three coordinate axis directions in the world coordinate system; The three DC components obtained after Fourier transform are used as the ideal world coordinates of the camera coordinate origin corresponding to the input original image sequence, and the ideal translation matrix extrinsic parameters of the camera at the position are obtained; Assuming that x(t), y(t), and z(t) are the displacement signals in the three directions at each moment, the Fourier transform of the camera's self-oscillation displacement signal x(t) in the x direction of the world coordinate system is expressed as:

[0093] When f=0, the DC component can be obtained:

[0094] Then the average DC component of the displacement signal is expressed as:

[0095] Where T is the period of the finite time signal; The three DC components obtained after Fourier transform are used as the ideal world coordinates of the camera coordinate origin corresponding to the input original image sequence, and the ideal translation matrix extrinsic parameters of the camera at the position are obtained.

[0096] The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement proposed in an embodiment of the present invention has the following advantages: (1) The image deblurring network based on the encoding-decoding structure and Transformer architecture provided by the present invention has excellent image restoration capabilities, can significantly improve the clarity of vibration-blurred images, and ensure the accuracy of subsequent feature extraction and calibration.

[0097] (2) The present invention constructs a mapping relationship between the camera pixel displacement signal and the real space displacement through a self-vibration measurement algorithm, accurately restores the three-dimensional motion law of the camera in a vibrating environment, and provides accurate dynamic posture information for external parameter calibration.

[0098] (3) The present invention utilizes image sequence deblurring and optical flow tracking to extract camera pixel displacement signals, thus avoiding dependence on external high-precision sensors and reducing system cost and complexity.

[0099] Corresponding to the above-disclosed robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement, an embodiment of the present invention further discloses a robot vision extrinsic parameter calibration system based on blind deblurring and self-vibration measurement, which specifically includes: An image acquisition module is used to continuously capture the target area in a vibrating environment using a high-definition camera with a high frame rate of the robot vision system to acquire an image sequence and synchronize the time stamp; An image deblurring module is used to blindly deblur blurred images in an image sequence through an image restoration network including a blur kernel estimation network and a multi-scale deblurring network; The vibration signal extraction module is used to detect image feature points in the deblurred image sequence and track pixel motion between frames to obtain the pixel displacement signal of the feature points and the three-dimensional displacement time history signal of the camera's self-vibration; The ideal position calculation module is used to perform frequency domain analysis on the three-dimensional displacement time-history signal, obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix.

[0100] It should be noted that, for a detailed description of a robot vision extrinsic parameter calibration system based on blind deblurring and self-vibration measurement provided in an embodiment of the present invention, reference can be made to the relevant description of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided in an embodiment of the present invention, which will not be repeated here.

[0101] In addition, an embodiment of the present invention also provides an electronic device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement as any of the above items.

[0102] It should be noted that, for the detailed description of an electronic device provided in an embodiment of the present invention, reference can be made to the relevant description of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided in an embodiment of the present application, which will not be repeated here.

[0103] In addition, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement as described in any of the above items are implemented.

[0104] It should be noted that, for a detailed description of a computer-readable storage medium provided in an embodiment of the present invention, reference can be made to the relevant description of a robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement provided in an embodiment of the present application, which will not be repeated here.

[0105] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer program. When all or part of the functions in the above embodiments are implemented by computer program, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer program, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and saved in the memory of the local device by downloading or copying, or the system of the local device is updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented. The above examples are used to illustrate the present invention, which are only used to help understand the present invention and are not intended to limit the present invention. Those skilled in the art can make several simple deductions, modifications or substitutions based on the concept of the present invention.

Claims

1. A robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement, characterized in that: The method comprises: The high-frame-rate HD camera of the robot vision system is used to continuously shoot the target area in a vibrating environment to obtain an image sequence and synchronize the time stamp; performing a blind deblurring operation on blurred images in the image sequence by an image restoration network comprising a blur kernel estimation network and a multi-scale deblurring network; Detect image feature points in the deblurred image sequence and perform inter-frame pixel motion tracking to obtain the pixel displacement signal of the feature points and the three-dimensional displacement time history signal of the camera's self-oscillation; Perform frequency domain analysis on the three-dimensional displacement time-history signal, obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix.

2. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 1, characterized in that: The robot vision system uses a high-frame-rate, high-definition camera to continuously capture the target area in a vibrating environment to acquire an image sequence and synchronize the time stamps. Specifically, the following steps are performed: The camera's frame rate must be sufficient to clearly capture the camera's own tiny vibrations. A high-frame-rate camera captures image sequences with minimal inter-frame displacement, facilitating accurate tracking of its own vibrations. The camera's resolution must be sufficient to clearly capture the details of the target object, facilitating subsequent blind deblurring and feature point detection. The camera should be rigidly fixed to the end of the robot or on a stable bracket in a vibration-controlled environment. The installation position should ensure that the camera can observe the target area globally. The camera should not be actively moved during the extrinsic parameter calibration process to avoid introducing complex dynamic errors caused by non-self-vibration. The target area captured by the camera should include multiple stationary objects or high-texture areas to facilitate the subsequent extraction of trackable feature points in the image sequence and reduce tracking failures or errors caused by texture repetition and weak texture; the scene in the target area should remain stationary; the acquisition time should cover multiple vibration cycles to facilitate subsequent frequency domain analysis to extract the main frequency and directional components of the vibration.

3. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 1, characterized in that: The blind deblurring operation on the blurred images in the image sequence is performed by an image restoration network including a blur kernel estimation network and a multi-scale deblurring network, specifically comprising: Perform fuzzy judgment on the collected image sequence data frame by frame; No processing is performed on image frames that are judged to be non-blurred; For image frames that are judged to be blurred, an image restoration network is used to perform deblurring processing, including: Use the blur kernel estimation network to dynamically estimate the shape and parameters of the blur kernel; The estimated blur kernel output by the blur kernel estimation network is upsampled to the size of the original blurred image, and then input together with the original blurred image into the Wiener filter for preliminary image motion blur restoration operation; The original blurred image and the preliminary restored image output by the Wiener filter are feature-concatenated and input into a multi-scale deblurring network for deblurring, and the final restored image is output.

4. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 3, characterized in that: Perform fuzzy judgment on the collected image sequence data frame by frame, specifically including: Image blur judgment includes the Laplacian variance method. The Laplacian operator is used for edge detection in images. The size of its response value reflects the degree of change in pixel intensity at the edge. The edge richness in the image is quantified by calculating the pixel value variance of the Laplacian response image. If the variance is lower than the set threshold, the image is judged to be blurred.

5. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 3, characterized in that: The fuzzy kernel estimation network specifically includes: The blur kernel estimation network adopts an encoder-decoder network structure. The encoder is composed of a 3×3 convolution and four downsampling layers, and the decoder is composed of four upsampling layers and a Softmax layer. Each downsampling layer consists of two 3×3 depth-wise separable convolutions, and the upsampling layers 1, 2, and 3 are composed of one deconvolution and two 3×3 depth-wise separable convolutions. Finally, after the upsampling layer 4 and the Softmax layer consisting of one 2×2 convolution and two 3×3 convolutions, the output is an estimated blur kernel of size 16×16.

6. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 3, characterized in that: The multi-scale deblurring network specifically includes: The multi-scale deblurring network includes a three-level downsampling module, a three-layer shallow convolution module, a U-shaped network and a refinement layer; the U-shaped network adopts a three-layer symmetric encoder-decoder structure, including a three-layer encoder, a three-layer decoder and an intermediate layer, the three-layer encoder includes an encoder 1, an encoder 2 and an encoder 3 connected in sequence, the three-layer decoder includes a decoder 1, a decoder 2 and a decoder 3 connected in sequence, the encoder 3 and the decoder 3 are connected through the intermediate layer, the encoder 1 and the encoder 2, the encoder 2 and the encoder 3, and the encoder 3 and the intermediate layer are all connected through the attention fusion module, the encoder 1 and the decoder 1, the encoder 2 and the decoder 2, and the encoder 3 and the decoder 3 are all jump-connected through the multi-scale high-frequency feature extraction module; the three-level downsampling module is connected to the three-layer shallow convolution module, and the three-layer shallow convolution modules are respectively connected to the three-layer attention fusion module; the encoder 1 is also connected to a 3×3 convolution layer 1, the decoder 1 is connected to the refinement layer, and the refinement layer is also connected to a 3×3 convolution layer 2; The spliced features obtained by splicing the original blurred image and the preliminary restored image output by the Wiener filter are input into the convolution layer 1. The low-level feature embedding of the spliced features is obtained by using the convolution layer 1 and input into the encoder 1. The input original blurred image is downsampled three times in sequence through the three-stage downsampling module to obtain a three-sampled image; The three sampled images are input into three shallow convolution modules respectively to obtain shallow blur features at corresponding scales; The multi-scale shallow fuzzy features output by the three-layer shallow convolution module are input into the three-layer attention fusion module respectively. The input splicing features and the multi-scale shallow fuzzy features are fused step by step through the three-layer attention fusion module and the three-layer encoder. Each level of codecs is connected by a multi-scale high-frequency feature extraction module. The multi-scale high-frequency feature extraction module extracts high-frequency features from the shallow feature information generated by the encoder of the layer, and concatenates them with the low-resolution potential features output by the decoder of the next layer to form a composite feature, which is then input to the next decoder. The deep features output by decoder 1 are input to the refinement layer for processing at high spatial resolution. The convolution layer 2 is applied to the refined features to generate a residual image. Finally, the residual image is added to the original blurred image to obtain the final restored image.

7. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 6, characterized in that: The three encoders, three decoders, intermediate layers, and refinement layers all use a multi-scale feature fusion enhancement module, which includes a Transformer module and a lightweight residual module. The data processing process includes: For the input feature map, the corresponding global feature information and local feature information are first extracted through the Transformer module and the lightweight residual module respectively, and then added element by element; The added result is concatenated with the local feature information extracted from the input feature map by the lightweight residual module to increase the network receptive field and learn multi-scale information; Finally, 1×1 convolution is used to reduce the number of channels to keep it consistent with the initial number of channels; The residual connection is introduced to add the output feature map to the input feature map element by element, which makes the model converge faster and retains low-dimensional features.

8. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 1, characterized in that: Detect image feature points in the deblurred image sequence and perform inter-frame pixel motion tracking to obtain the pixel displacement signals of the feature points and the three-dimensional displacement time-course signals of the camera's self-oscillation. Specifically, the following steps are performed: Preprocess the deblurred image sequence so that each pixel accurately corresponds to the actual spatial position; Selecting a region of interest from the preprocessed image sequence and cropping the selected region to obtain a cropped image sequence; The ORB feature point detection algorithm is used to detect the corner features in the cropped image of the first frame; The Lucas-Kanade optical flow method is used to obtain the relative displacement of feature points between adjacent frame images, track the motion trajectory of feature points between adjacent frame images, and obtain the pixel displacement time history signal of each feature point in the region of interest; Based on the pinhole camera model principle, the pixel displacement signal is reconstructed to obtain the discrete time-course signal of the camera's actual displacement in three-dimensional space. The three-dimensional displacement discrete time history signal is processed by numerical differentiation method to obtain the three-dimensional velocity time history signal of each characteristic point; The displacement signal is interpolated based on the velocity correction through the velocity signal and the displacement signal to obtain the reconstructed continuous three-dimensional displacement time history signal of the camera self-oscillation.

9. The robot vision extrinsic parameter calibration method based on blind deblurring and self-vibration measurement according to claim 1, characterized in that: Perform frequency domain analysis on the three-dimensional displacement time-history signal to obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix, including: Perform frequency domain analysis on the component signals of the camera's self-vibration three-dimensional displacement time history signal in the three coordinate axis directions in the world coordinate system; The three DC components obtained after Fourier transform are used as the ideal world coordinates of the camera coordinate origin corresponding to the input original image sequence, and the ideal translation matrix extrinsic parameters of the camera at the corresponding position are obtained.

10. A robot vision extrinsic parameter calibration system based on blind deblurring and self-vibration measurement, characterized in that: The system comprises: An image acquisition module is used to continuously capture the target area in a vibrating environment using a high-definition camera with a high frame rate of the robot vision system to acquire an image sequence and synchronize the time stamp; an image deblurring module, configured to perform a blind deblurring operation on blurred images in the image sequence by using an image restoration network comprising a blur kernel estimation network and a multi-scale deblurring network; The vibration signal extraction module is used to detect image feature points in the deblurred image sequence and track pixel motion between frames to obtain the pixel displacement signal of the feature points and the three-dimensional displacement time history signal of the camera's self-vibration; The ideal position calculation module is used to perform frequency domain analysis on the three-dimensional displacement time-history signal, obtain the DC components of the signal in three directions as the ideal three-dimensional world coordinates corresponding to the camera in the image sequence, and calculate the extrinsic parameter matrix.

Images