A flexible collaboration method for unmanned systems in intelligent packaging and testing scenarios

Through real-time image preprocessing and jitter correction technology, combined with the data fusion of the binocular ranging system, the problem of inaccurate lens jitter and positioning distance measurement in the encapsulated and tested scene is solved, and high-precision, flexible coordinated control of the unmanned system is realized, improving the improvement of the Work efficiency and accuracy.

CN118797214BInactive Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411260730.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-10
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In traditional packaging and testing scenarios, there are problems of lens jitter and inaccurate positioning and distance measurement, which affects the flexible coordination ability and operation efficiency of the unmanned system.

Method used

Using real-time image preprocessing, jitter discrimination and correction, jitter and attitude weight calculation, and binocular ranging system data fusion technology, through the coordinated work of three cameras, ensure the consistency and accuracy of image data, reduce external interference, and improve measurement Accuracy.

Benefits of technology

It improves the stability and measurement accuracy of image acquisition, enhances flexible collaborative control capabilities, and meets the precise operation needs of encapsulated and tested scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118797214B_ABST
    Figure CN118797214B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer vision and swarm intelligence, and specifically provides an unmanned system flexible collaboration method for intelligent packaging and testing scenarios, so as to solve the problems of lens jitter, inaccurate positioning and ranging, etc. existing in traditional packaging and testing scenarios; first, the captured images of each camera are acquired in real time, and the images are preprocessed to obtain a feature matrix; then, for each camera, jitter is identified according to the feature matrix, and jitter correction is performed on the jittered captured images, and then the jitter weight and posture weight of the camera at the current moment are calculated; finally, the measurement results of the three binocular ranging systems are data fused according to the jitter weight and posture weight of the main camera in the binocular ranging system, and the coordinates and size of the target object at the current moment are obtained to complete flexible collaborative control; in summary, the present invention can improve the flexible collaboration capability of the unmanned system, thereby meeting the demand for precise operation in the packaging and testing scenarios.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and swarm intelligence, and involves key technologies such as lens jitter elimination, binocular positioning and ranging, and precise flexible operation of robotic arms. It specifically provides an unmanned system flexible collaboration method for intelligent packaging and testing scenarios. Background Art

[0002] With the continuous improvement of industrial automation and the rapid development of artificial intelligence technology, unmanned systems are increasingly used in various fields; especially in packaging and testing scenarios, the demand for precise positioning, ranging and flexible operations is growing; traditional packaging and testing scenarios have problems such as lens jitter and inaccurate positioning and ranging, which affect the flexible coordination capability and operating efficiency of unmanned systems. Summary of the invention

[0003] The purpose of the present invention is to provide a flexible collaboration method for unmanned systems for intelligent packaging and testing scenarios, so as to solve the problems of lens jitter, inaccurate positioning and ranging in traditional packaging and testing scenarios, aiming to improve the flexible collaboration capabilities of unmanned systems, thereby meeting the requirements of packaging and testing scenarios for precise operations.

[0004] To achieve the above object, the technical solution adopted by the present invention is:

[0005] A flexible collaboration method for unmanned systems in intelligent packaging and testing scenarios includes the following steps:

[0006] Step 1. Calibrate the coordinates of the three cameras in the intelligent packaging and testing scene, obtain the captured images of each camera in real time, and pre-process the images to obtain the feature matrix of the captured images;

[0007] Step 2. For each camera, jitter is identified based on the feature matrix of the captured image, and jitter correction is performed on the jittered captured image;

[0008] Step 3. For each camera, calculate the camera's jitter weight at the current moment based on the jitter identification result;

[0009] Step 4. For each camera, the camera's posture weight at the current moment is calculated based on the acquired image after the shake correction is completed;

[0010] Step 5. The three cameras are placed in pairs in order to form a binocular ranging system. The coordinates and size of the target object are measured by each binocular ranging system. Then, the measurement results of the three binocular ranging systems are fused according to the jitter weight and posture weight of the main camera in the binocular ranging system to obtain the coordinates and size of the target object at the current moment, which is used to complete flexible collaborative control.

[0011] Furthermore, in step 1, the preprocessing process is:

[0012] For each frame image, its gray matrix is ​​expressed as :

[0013] ,

[0014] in, Represents the grayscale value of the mth row and nth column in the grayscale matrix, m=1,2,...,M, n=1,2,...,N, M represents the number of rows of the grayscale matrix, and N represents the number of columns of the grayscale matrix;

[0015] According to the gray matrix Calculate the lateral difference matrix of the image , specifically:

[0016] ,

[0017] Among them, the element in the mth row and nth column It is expressed as:

[0018] ,

[0019] According to the gray matrix Calculate the vertical difference matrix of the image , specifically:

[0020] ,

[0021] Among them, the element in the mth row and nth column It is expressed as:

[0022] ,

[0023] Combine the horizontal difference matrix and the vertical difference matrix to obtain the feature matrix corresponding to the image , specifically:

[0024] ,

[0025] in, The dimension is An all-zero matrix.

[0026] Furthermore, in step 2, the superscript t is used to mark the time, and for the current time t, the jitter discrimination feature is calculated , specifically:

[0027] ,

[0028] in, represents the jitter discrimination feature of the image captured by the camera at the current time t, and They represent the feature matrices of the images captured by the camera at time t and time t+1 respectively, Represents the XOR operation of the matrix;

[0029] Set the jitter detection time window K, and detect the camera jitter according to the jitter detection characteristics. The jitter detection conditions are:

[0030] ,

[0031] in, represents the jitter discrimination threshold, Represents modulo operation;

[0032] If the jitter determination condition is met, it is determined that the camera jitter occurs at time t, jitter correction is performed on the image, and the jitter occurrence time is recorded in sequence in the jitter time set T; otherwise, it is determined that no jitter occurs.

[0033] Furthermore, in step 2, the jitter correction is expressed as:

[0034] ,

[0035] ,

[0036] in, Represents the grayscale matrix of the image captured by the camera at time t, Represents the grayscale matrix The jitter correction result is represents the inverse filtering operation; Represents the grayscale matrix The result of inverse filtering.

[0037] Furthermore, in step 3, the camera jitter weight at the current time t is calculated :

[0038] ,

[0039] Where T represents the jitter time set, Indicates that the current time t does not belong to the jitter set T, Indicates that the current time t belongs to the jitter set T, It represents the time when the next jitter occurs after time t in the jitter set T, and K represents the jitter judgment time window.

[0040] Furthermore, in step 4, for the collected image at the current time t, the boundary of the target object is extracted, and the boundary pixel set is obtained in a clockwise direction. At the same time, a square frame is used to frame the target object, starting from the upper boundary, and the frame pixel set is obtained in a clockwise direction. ; Calculate the overlap between the boundary pixel set and the border pixel set to obtain the target object's posture estimation vector :

[0041] ,

[0042] in, represents the estimated pose vector of the target object at time t, Represents the boundary pixel set Pixel set with border The overlap of A rotation matrix representing a two-dimensional rotation, represents the parameter vector of the straight line fitting arc, represents the modulo operation, Represents the XOR operation of vectors;

[0043] Estimate the vector from the pose Calculate the current camera's posture weight , specifically expressed as:

[0044] .

[0045] Furthermore, in step 5, the three cameras are numbered in a clockwise direction and two of them form a binocular ranging system. In the first binocular ranging system, the first camera is used as the main camera and the second camera is used as the auxiliary camera. In the second binocular ranging system, the second camera is used as the main camera and the third camera is used as the auxiliary camera. In the third binocular ranging system, the third camera is used as the main camera and the first camera is used as the auxiliary camera. For the current time t, the three binocular ranging systems respectively measure the coordinates of the target object. With size , ;

[0046] The measurement results of the three binocular ranging systems are fused according to the posture weight and jitter weight of each camera to obtain the coordinates of the target object at the current time t With size , specifically expressed as:

[0047] ,

[0048] ,

[0049] ,

[0050] ,

[0051] in, represents the distance between the main camera and the target object measured by the jth binocular ranging system, Indicates scientific notation. represents the posture weight of the j-th camera at time t, represents the jitter weight of the jth camera at time t.

[0052] Based on the above technical solution, the beneficial effects of the present invention are:

[0053] The present invention provides a flexible collaborative method for unmanned systems in intelligent packaging and testing scenarios, which uses real-time image preprocessing, jitter discrimination and correction, jitter and posture weight calculation, and binocular ranging system data fusion technology to ensure the data consistency and accuracy of the camera when collecting images, and external interference can be discovered and corrected in time, and the measurement data is more accurate. Finally, the present invention has the advantages of improving image acquisition stability, enhancing measurement accuracy, and realizing flexible collaborative control, and can meet the demand for precise operations in packaging and testing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of the unmanned system flexible collaboration method for intelligent packaging and testing scenarios in the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0056] This embodiment provides an unmanned system flexible collaboration method for intelligent packaging and testing scenarios, and its process is as follows: Figure 1 As shown, the specific steps include:

[0057] Step 1. The intelligent packaging and testing scenario includes three cameras. The coordinates of the three cameras are calibrated, the captured images of each camera are acquired in real time, and the images are preprocessed to obtain the feature matrix of the captured images;

[0058] For each frame image, its gray matrix is ​​expressed as :

[0059] ,

[0060] in, Represents the gray value of the mth row and nth column in the gray matrix, m=1,2,...,M, n=1,2,...,N, M represents the number of rows of the gray matrix, and N represents the number of columns of the gray matrix;

[0061] According to the gray matrix Calculate the lateral difference matrix of the image , specifically:

[0062] ,

[0063] Among them, the element in the mth row and nth column It is expressed as:

[0064] ,

[0065] According to the gray matrix Calculate the vertical difference matrix of the image , specifically:

[0066] ,

[0067] Among them, the element in the mth row and nth column It is expressed as:

[0068] ,

[0069] Combine the horizontal difference matrix and the vertical difference matrix to obtain the feature matrix corresponding to the image , specifically:

[0070] ,

[0071] in, The dimension is The all-zero matrix of

[0072] Step 2. For each camera, jitter is identified based on the feature matrix of the captured image, and jitter correction is performed on the jittered captured image;

[0073] For each camera, use the superscript t to mark the time, and for the current time t, calculate the jitter discrimination feature , specifically:

[0074] ,

[0075] in, represents the jitter discrimination feature of the image captured by the camera at the current time t, and They represent the feature matrices of the images captured by the camera at time t and time t+1 respectively, Represents the XOR operation of the matrix;

[0076] Set the jitter detection time window K, and detect the camera jitter according to the jitter detection characteristics. The jitter detection conditions are:

[0077] ,

[0078] in, represents the jitter discrimination threshold, which is an empirical constant pre-set according to the application scenario; It represents the modulo operation, and the jitter discrimination time window K is set to 4;

[0079] If the jitter determination condition is met, it is determined that the camera jitters at time t, and the image is jitter-corrected and the jitter occurrence time is recorded in sequence in the jitter set T; otherwise, it is determined that no jitter occurs; The jitter correction is expressed as:

[0080] ,

[0081] ,

[0082] in, Represents the grayscale matrix of the image captured by the camera at time t, Represents the grayscale matrix The jitter correction result is Represents inverse filtering (deconvolution) operation; Represents the grayscale matrix The result after inverse filtering, specifically, the grayscale matrix of the deghosted image obtained by inverse filtering;

[0083] Step 3. For each camera, calculate the camera's jitter weight at the current moment based on the jitter identification result;

[0084] For each camera, calculate the camera's jitter weight at the current time t :

[0085] ,

[0086] in, Indicates that the current time t does not belong to the jitter set T, that is, the camera does not jitter at the current time; Indicates that the current moment t belongs to the jitter set T, that is, the camera is jittering at the current moment; Indicates the time when the next jitter occurs after time t in the jitter set T;

[0087] Step 4. For each camera, the camera's posture weight at the current moment is calculated based on the acquired image after the shake correction is completed;

[0088] For each camera, the superscript t is used to mark the time. For the collected image (grayscale matrix) at the current time t, the boundary of the target object is extracted. The boundary is taken as the starting point, and the boundary pixel set is obtained in a clockwise direction. It should be noted that the boundary extraction process is a well-known technology in the art and will not be described here. At the same time, a square frame is used to frame the target object, and the above boundary is taken as the starting point, and the frame pixel set is obtained in a clockwise direction. ; Obtain the target object’s pose estimation vector by calculating the overlap between the boundary pixel set and the border pixel set :

[0089] ,

[0090] in, represents the estimated pose vector of the target object at time t, Represents the boundary pixel set Pixel set with border The overlap of A rotation matrix representing a two-dimensional rotation, represents the parameter vector of the straight line fitting arc, represents the modulo operation, Represents the XOR operation of vectors;

[0091] Estimate the vector from the pose Calculate the current camera's posture weight , specifically expressed as:

[0092] ;

[0093] Step 5. The three cameras are placed in pairs in order to form a binocular ranging system. The coordinates and size of the target object are measured by each binocular ranging system. Then, the measurement results of the three binocular ranging systems are fused according to the jitter weight and attitude weight of the main camera in the binocular ranging system to obtain the coordinates and size of the target object at the current moment, so as to complete flexible collaborative control.

[0094] The three cameras are numbered in clockwise direction and form a binocular ranging system in pairs. In the first binocular ranging system, the first camera is used as the main camera and the second camera is used as the auxiliary camera. In the second binocular ranging system, the second camera is used as the main camera and the third camera is used as the auxiliary camera. In the third binocular ranging system, the third camera is used as the main camera and the first camera is used as the auxiliary camera. For the current time t, the three binocular ranging systems measure the coordinates of the target object respectively. With size , It should be noted that the measurement process of the binocular ranging system is a well-known technology in the art and will not be described in detail here;

[0095] The measurement results of the three binocular ranging systems are fused according to the posture weight and jitter weight of each camera to obtain the coordinates of the target object at the current time t With size , specifically expressed as:

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] in, represents the distance between the main camera and the target object measured by the jth binocular ranging system, Indicates scientific notation. represents the posture weight of the j-th camera at time t, represents the jitter weight of the jth camera at time t;

[0101] Therefore, according to the coordinates of the target object at the current time t With size Flexible collaborative control of the unmanned system can be completed.

[0102] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A flexible collaboration method for unmanned systems in intelligent packaging and testing scenarios, characterized in that: The following steps are involved: Step 1. Calibrate the coordinates of the three cameras in the intelligent packaging and testing scene, obtain the captured images of each camera in real time, and pre-process the images to obtain the feature matrix of the captured images; Step 2. For each camera, jitter determination is performed based on the feature matrix of the captured image, and jitter correction is performed on the captured image that has jitter; The specific jitter discrimination is as follows: the superscript t is used to mark the time, and for the current time t, the jitter discrimination feature Sa is calculated t , specifically: Among them, Sa t represents the jitter discrimination feature of the image captured by the camera at the current time t, D t With D t+1 They represent the feature matrices of the images captured by the camera at time t and time t+1 respectively, Represents the XOR operation of the matrix; Set the jitter detection time window K, and detect the camera jitter according to the jitter detection characteristics. The jitter detection conditions are: Among them, S th represents the jitter discrimination threshold, ‖·‖ represents the modulo operation; If the jitter determination condition is met, it is determined that the camera jitters at time t, and the image is jitter-corrected and the jitter occurrence time is recorded in sequence in the jitter time set T; otherwise, it is determined that no jitter occurs; The jitter correction is expressed as: Among them, I t Represents the grayscale matrix of the image captured by the camera at time t, Represents the gray matrix I t The jitter correction result of f(·) represents the inverse filtering operation; Represents the gray matrix I t The result after inverse filtering; Step 3. For each camera, calculate the camera's jitter weight at the current moment based on the jitter identification result; calculate the camera's jitter weight s at the current moment t t : Where T represents the jitter time set, Indicates that the current time t does not belong to the jitter time set T, t∈T indicates that the current time t belongs to the jitter time set T, t next represents the time when the next jitter occurs after time t in the jitter time set T, and K represents the jitter judgment time window; Step 4. For each camera, the posture weight of the camera at the current moment is calculated based on the acquired image after the shake correction is completed; For the captured image at the current time t, the boundary of the target object is extracted, starting from the boundary, and the boundary pixel set is obtained in a clockwise direction. At the same time, a square frame is used to frame the target object. The upper boundary is taken as the starting point, and the frame pixel set is obtained in a clockwise direction. Calculate the overlap between the boundary pixel set and the border pixel set to obtain the target object’s posture estimation vector G t : Among them, G t represents the estimated pose vector of the target object at time t, Represents the boundary pixel set Pixel set with border The coincidence degree, R represents the rotation matrix of the two-dimensional rotation form, P represents the parameter vector of the straight line fitting arc, ‖·‖ represents the modulus operation, and ⊙ represents the vector exclusive OR operation; According to the pose estimation vector G t Calculate the current camera's posture weight Specifically expressed as: Step 5. The three cameras are placed in pairs in order to form a binocular ranging system. The coordinates and size of the target object are measured by each binocular ranging system. Then, the measurement results of the three binocular ranging systems are fused according to the jitter weight and attitude weight of the main camera in the binocular ranging system to obtain the coordinates and size of the target object at the current moment, so as to complete the flexible collaborative control. The three cameras are numbered in clockwise direction and form a binocular ranging system in pairs. In the first binocular ranging system, the first camera is used as the main camera and the second camera is used as the auxiliary camera. In the second binocular ranging system, the second camera is used as the main camera and the third camera is used as the auxiliary camera. In the third binocular ranging system, the third camera is used as the main camera and the first camera is used as the auxiliary camera. For the current time t, the three binocular ranging systems measure the coordinates of the target object (x j ,y j ,z j ) and size o j , j = 1, 2, 3; The measurement results of the three binocular ranging systems are fused according to the posture weight and jitter weight of each camera to obtain the coordinates (x t ,y t ,z t ) and size o t , specifically expressed as: Among them, d j represents the distance between the main camera and the target object measured by the jth binocular ranging system, e represents the scientific notation symbol, represents the posture weight of the j-th camera at time t, represents the jitter weight of the jth camera at time t.

2. The unmanned system flexible collaboration method for intelligent packaging and testing scenarios according to claim 1 is characterized in that: In step 1, the preprocessing process is: For each frame of image, its grayscale matrix is ​​represented as I: Among them, a m,n Represents the gray value of the mth row and nth column in the gray matrix, m=1,2,...,M, n=1,2,...,N, M represents the number of rows of the gray matrix, and N represents the number of columns of the gray matrix; The lateral difference matrix I1 of the image is calculated according to the gray matrix I, specifically: Among them, the element b in the mth row and nth column m,n It is expressed as: The vertical difference matrix I2 of the image is calculated according to the gray matrix I, specifically: Among them, the element c in the mth row and nth column m,n It is expressed as: Combine the horizontal difference matrix and the vertical difference matrix to obtain the feature matrix D corresponding to the image, which is: D=[I1|0 M×N ]+[0 M×N |I2], Among them, 0 M×N represents an all-zero matrix of dimension M×N.

Citation Information

Patent Citations

  • Mobile-terminal-oriented method for restoring blurred image caused by jitter

    CN105069753A

  • Scene reconstruction method and apparatus based on mobile device monocular camera

    CN106447766A

  • Robustness control method and system for multi-source fusion positioning

    CN114648584A

  • Method for removing imaging jitter of linear array camera on multi-axis motion platform

    CN117479013A

  • Image processing device

    JP2020102786A