Target cooperative control method and system based on deep learning

Abstract

The invention provides a target cooperative control method and system based on deep learning, and the method comprises the steps: obtaining the real-time visual data of a target region through a multi-view image collection device, carrying out the recognition and classification of object features through a deep learning model, and generating feature distribution data containing a class identifier and boundary coordinates, a space calibration algorithm is adopted to map the two-dimensional features into three-dimensional coordinate system data, a space position and posture angle parameters are output in combination with posture calculation, a control instruction including path planning, tail end executor posture adjustment and self-adaptive clamping force is generated according to physical attribute parameters of an object, an operation device is driven to complete the precise grabbing action, and the grabbing precision is improved. Therefore, through dynamic multi-view perception, three-dimensional space modeling and physical attribute driven closed-loop control, the problems of feature deficiency, positioning errors, operation rigidity and the like of a traditional visual system in a complex scene are effectively solved, and the precision and adaptability of target operation in an unstructured environment are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular, to an object collaborative control method and system based on deep learning. Background Art

[0002] In the field of industrial automation, in view of the need for precise operation of objects in complex scenarios, traditional technical solutions have significant limitations. Most existing vision-guided operating systems use monocular cameras or fixed-view sensor arrays, resulting in feature loss or misrecognition problems in object occlusion, lighting changes, or unstructured environments. Typical deep learning-based object detection methods can extract two-dimensional image features, but lack a spatial dimension correlation mechanism and are difficult to directly map to a three-dimensional operation space. For grasping tasks that require precise pose control, traditional solutions usually rely on pre-modeled object template libraries or offline calibration parameters and cannot adapt to dynamically changing working scenarios. In addition, the end effector control strategy generally adopts a fixed parameter drive mode and does not establish a correlation model between the physical properties of the object and the operating force, resulting in clamping failure or surface damage when dealing with objects of different materials, shapes, or weights. Summary of the Invention

[0003] In view of the above-mentioned problems, in combination with the first aspect of the present invention, embodiments of the present invention provide an object collaborative control method based on deep learning, and the method includes: Obtain real-time multi-view image data of a target area through an image acquisition device, where the real-time multi-view image data includes visual feature information of an object to be operated; Call a real-time recognition model to perform object feature recognition on the real-time multi-view image data, and obtain feature distribution data of the object to be operated, where the feature distribution data includes category identification information and boundary region coordinate information of the object to be operated; Perform calibration processing on the feature distribution data to generate coordinate data of the object to be operated in a three-dimensional space coordinate system; Perform three-dimensional pose calculation processing based on the coordinate data to obtain a set of pose parameters of the object to be operated, where the set of pose parameters includes spatial position parameters and attitude angle parameters of the object to be operated; Generate a control instruction according to the set of pose parameters and the object physical property parameters corresponding to the category identification information, and drive an operation execution device to complete the object grasping action of the object to be operated. The control instruction includes a path control signal, an end effector attitude control signal, and a clamping force control signal based on the category identification information.

[0004] In another aspect, an embodiment of the present invention further provides an object collaborative control system based on deep learning, including a processor and a machine-readable storage medium. The machine-readable storage medium is connected to the processor. The machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0005] Based on the above aspects, the embodiment of the present invention realizes the end-to-end optimization from multi-source heterogeneous visual data to precise mechanical operations. The dynamic multi-view image fusion mechanism effectively overcomes the occlusion defect of traditional monocular vision. The cross-view feature correlation extracted by the deep learning model significantly improves the object recognition integrity in complex scenarios. By combining the camera array calibration parameters with the deep learning error compensation network, the two-dimensional detection results are accurately mapped to the three-dimensional coordinate system, reducing the spatial positioning error to the sub-centimeter level. The closed-loop control system based on physical attribute parameters establishes a mapping model between the object category and the operation force, enabling the end effector to adaptively adjust the clamping strategy, avoiding physical damage to vulnerable objects while ensuring operation stability. Through the real-time data interaction between the pose calculation module and the motion planner, the dynamic optimization of the operation path and the synchronous response of the execution mechanism are realized, significantly improving the operation success rate and operation efficiency in complex working conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a schematic execution flow diagram of the object collaborative control method based on deep learning provided by the embodiment of the present invention.

[0007] Figure 2 is a schematic diagram of exemplary hardware and software components of the object collaborative control system based on deep learning provided by the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0008] The present invention will be specifically described below with reference to the accompanying drawings of the specification. Figure 1 is a schematic flow diagram of the object collaborative control method based on deep learning provided by an embodiment of the present invention. The object collaborative control method based on deep learning will be introduced in detail below.

[0009] Step S110: Obtain real-time multi-view image data of the target area through an image acquisition device. The real-time multi-view image data includes visual feature information of the object to be operated.

[0010] In the production scenario of printed circuit board assemblies (PCBA), to complete the task of removing the auxiliary components for chip mounting of some components, it is necessary to accurately obtain the information of the components to be operated in the target area. For example, high-precision industrial cameras can be used as the image acquisition device. The selection of these industrial cameras is based on parameters such as their resolution, frame rate, and sensitivity to ensure that they can capture images clearly and quickly.

[0011] Exemplarily, in terms of the installation layout, according to the actual situation of the PCBA production line, cameras are arranged at different positions and angles. For example, one camera is installed directly above the production line to obtain the top visual features of the components; multiple cameras are installed at different side angles to capture features such as the side profiles of the components. In this way, images are collected from multiple perspectives, enabling a more comprehensive acquisition of the visual feature information of the components to be operated on.

[0012] The frequency of image acquisition by the camera needs to be dynamically adjusted according to the running speed of the production line and the movement of the components. If the production line speed is fast, the camera acquisition frequency is correspondingly increased to ensure that the captured images can reflect the state of the components in real time; conversely, if the speed is slow, the acquisition frequency can be appropriately reduced to reduce the data volume and processing burden. The real-time multi-perspective image data collected contains visual feature information such as the color, shape, and texture of the components.

[0013] Step S120: Invoke the real-time recognition model to perform object feature recognition on the real-time multi-perspective image data, obtaining the feature distribution data of the object to be operated on, where the feature distribution data includes the category identification information and boundary region coordinate information of the object to be operated on.

[0014] After obtaining the real-time multi-perspective image data, it is necessary to invoke the real-time recognition model to process the data in order to identify the feature distribution of the object to be operated on. In this embodiment, the improved YOLOv8 algorithm is used as the real-time recognition model.

[0015] Step S121: Perform parallel multi-scale feature extraction on the real-time multi-perspective image data through the multi-level dilated convolutional network in the real-time recognition model, generating a multi-level feature set containing feature maps of different resolutions, where the dilation rate of the dilated convolutional kernel at each level is dynamically adjusted according to the reciprocal of the corresponding input image resolution to match the object size change.

[0016] The multi-level dilated convolutional network in the real-time recognition model consists of multiple levels of dilated convolutional layers, and each level of dilated convolutional layer has a different dilation rate. The setting of the dilation rate is dynamically adjusted according to the reciprocal of the corresponding input image resolution.

[0017] Assume that the input real-time multi-perspective image data has different resolutions. For an image with a higher resolution, its reciprocal is smaller, and the corresponding dilation rate of the dilated convolutional kernel is also smaller. This is because a high-resolution image contains more detailed information, and a smaller dilation rate can extract these detailed features more precisely. For example, for a high-resolution image block A with a resolution of R_A, the corresponding dilation rate E_A is calculated according to the formula, and the dilated convolutional kernel with a dilation rate of E_A is used to perform a convolutional operation on the image block A to extract its detailed features.

[0018] For an image with a lower resolution, its reciprocal is larger, and the dilation rate of the dilated convolutional kernel increases accordingly. This is to expand the receptive field of the convolutional kernel and capture more macroscopic features. For example, for a low-resolution image patch B with a resolution of R_B, the calculated dilation rate is E_B (E_B > E_A). The dilated convolutional kernel with a dilation rate of E_B is used to perform a convolutional operation on the image patch B to obtain more macroscopic feature information.

[0019] During the parallel processing, dilated convolutional layers at different levels simultaneously perform convolutional operations on the input real-time multi-view image data. The convolutional operations at each level are independent, which can improve the processing efficiency. After these convolutional operations, a multi-level feature set containing feature maps with different resolutions is generated, which can reflect the feature information of the component to be operated from different scales and resolutions.

[0020] Step S122: Input the multi-level feature set into the cross-level feature pyramid in the real-time recognition model. Use bilinear interpolation to increase the resolution of the low-level large receptive field feature map to align its size with the high-level small receptive field feature map, and perform channel-wise weighted fusion on the aligned feature maps to generate an enhanced feature map that fuses global semantics and local details.

[0021] After the multi-level feature set is generated, it is input into the cross-level feature pyramid in the real-time recognition model. The role of the cross-level feature pyramid is to fuse the information of feature maps at different levels to obtain a more comprehensive and accurate feature representation.

[0022] First, process the low-level large receptive field feature map. Since the resolution of the low-level feature map is relatively low, in order to effectively fuse it with the high-level small receptive field feature map, the bilinear interpolation method is used to increase its resolution. Bilinear interpolation is a method of calculating new pixel values based on the weighted average of surrounding pixels.

[0023] Suppose the low-level large receptive field feature map is F_L and the high-level small receptive field feature map is F_H. Through bilinear interpolation, the resolution of F_L is increased to the same size as F_H to obtain a new feature map F_L'. During the interpolation process, the value of each pixel in F_L' is calculated according to the gray values and positional relationships of the surrounding pixels in F_L.

[0024] Then, perform channel-wise weighted fusion on the aligned feature maps F_L' and F_H. For each channel, a weight is assigned. The weights are determined based on the importance of the channel features. These weights can be determined by training a small neural network or using statistical methods. Suppose the weight of channel i is w_i, and the value of the fused feature map at channel i is F_fused_i = w_i * F_L'_i + (1 - w_i) * F_H_i, where F_L'_i and F_H_i are the values of F_L' and F_H at channel i respectively. Through this channel-wise weighted fusion method, an enhanced feature map that combines global semantics and local details is generated. The enhanced feature map synthesizes the information of different-level feature maps, which is more conducive to subsequent object recognition and localization.

[0025] Step S123: Apply channel dimension attention weighting and spatial dimension attention weighting to the enhanced feature map to obtain a weighted feature map, where the channel attention weights are generated by a two-layer fully connected network after global average pooling, and the spatial attention weights are generated by the spatial compression features of a single-layer convolutional kernel.

[0026] To further highlight the important feature information in the enhanced feature map, it is necessary to apply channel dimension attention weighting and spatial dimension attention weighting to the enhanced feature map.

[0027] In terms of channel dimension attention weighting, first perform global average pooling on the enhanced feature map. Global average pooling averages the feature map in the spatial dimension to obtain the average value of each channel. Suppose the enhanced feature map is F_enhanced, with a size of C×H×W (C is the number of channels, H is the height, and W is the width). After global average pooling, a vector V of length C is obtained, where each element V_c of V is the average value of all pixels of F_enhanced at channel c.

[0028] Then, input the vector V into a two-layer fully connected network. The two-layer fully connected network consists of an input layer, a hidden layer, and an output layer. The input layer receives the vector V, and after the non-linear transformation of the hidden layer, the output layer outputs the channel attention weight vector W_c. Each element W_c_i of the channel attention weight vector W_c represents the importance degree of channel i.

[0029] In terms of spatial dimension attention weighting, a single-layer convolutional kernel is used to perform a spatial compression feature generation operation on the enhanced feature map. Parameters such as the size and stride of the single-layer convolutional kernel are set according to the actual situation. Assuming the single-layer convolutional kernel is K, the enhanced feature map F_enhanced is convolved to obtain the spatial compression feature map F_spatial. Then, the spatial attention weight matrix W_s is calculated based on the values of F_spatial. Each element W_s_ij of the spatial attention weight matrix W_s represents the degree of importance of the position (i, j) in the feature map.

[0030] Finally, the channel attention weight vector W_c and the spatial attention weight matrix W_s are respectively applied to the enhanced feature map F_enhanced. For each element F_enhanced_ijk in the enhanced feature map F_enhanced (where i represents the row of the spatial position, j represents the column of the spatial position, and k represents the channel), the weighted feature map element F_weighted_ijk = W_c_k * W_s_ij * F_enhanced_ijk. In this way, the weighted feature map is obtained, highlighting important feature information and suppressing unimportant feature information.

[0031] Step S124: Divide the weighted feature map along the channel dimension into a global classification branch and a local segmentation branch. Among them, the global classification branch is input into a multi-layer perceptron classifier after dimensionality reduction through global max pooling, and outputs the class identification information and corresponding confidence of the object to be operated, and filters out the class identification information with a confidence lower than the dynamic threshold.

[0032] The weighted feature map is divided along the channel dimension into a global classification branch and a local segmentation branch. The purpose of the global classification branch is to classify and identify the object to be operated.

[0033] First, a global max pooling dimensionality reduction operation is performed on the feature map of the global classification branch. Global max pooling takes the maximum value of the feature map in the spatial dimension, thereby reducing the dimension of the feature map. Assuming the feature map of the global classification branch is F_global, with a size of C_g × H_g × W_g, after global max pooling, a vector V_global with a length of C_g is obtained.

[0034] Then, input the vector V_global into a multi-layer perceptron classifier. The multi-layer perceptron classifier consists of multiple neuron layers, including an input layer, a hidden layer, and an output layer. The input layer receives the vector V_global. After undergoing non-linear transformation in the hidden layer, the output layer outputs the class identification information of the object to be operated on and the corresponding confidence level. The class identification information can be represented by a class vector. Each element in the vector corresponds to a class. A value of 1 for an element indicates belonging to that class, and a value of 0 indicates not belonging to that class. The confidence level is a probability value corresponding to the class identification information, representing the reliability of the classification result.

[0035] To improve the classification accuracy, a dynamic threshold is set. The dynamic threshold will be adjusted according to the actual situation, such as determining an appropriate threshold based on different production environments, component types, and other factors. For the output class identification information and confidence level, filter out the class identification information with a confidence level lower than the dynamic threshold. This can remove unreliable classification results and improve the classification precision.

[0036] Step S125: After the local segmentation branch is restored to the original input image resolution through a transposed convolutional layer, it is input into a bounding box coordinate regressor to generate the normalized coordinate parameters of the initial bounding box, and a non-maximum suppression algorithm based on the intersection over union threshold is used to eliminate redundant bounding boxes with spatial overlap.

[0037] The main task of the local segmentation branch is to determine the boundary region of the object to be operated on. First, the feature map of the local segmentation branch is processed through a transposed convolutional layer. The transposed convolutional layer is a convolutional layer that can perform upsampling, and it can restore the resolution of the feature map to the resolution of the original input image.

[0038] Assume the feature map of the local segmentation branch is F_local, and its size is smaller than the resolution of the original input image. Through the transposed convolutional layer, the resolution of F_local is increased to the same size as the original input image, obtaining the feature map F_local'. The parameters of the transposed convolutional layer, such as the convolutional kernel size, stride, padding, etc., are set as needed to ensure that the resolution can be accurately restored.

[0039] Then, input the feature map F_local' with restored resolution into a bounding box coordinate regressor. The bounding box coordinate regressor is a model for predicting the coordinates of an object's bounding box. It generates the normalized coordinate parameters of the initial bounding box through the analysis and calculation of the feature map F_local'. The normalized coordinate parameters normalize the coordinate values of the bounding box to make their range within a set interval, facilitating subsequent processing and comparison.

[0040] To eliminate redundant bounding boxes with spatial overlap, a non-maximum suppression algorithm based on the intersection over union (IoU) threshold is adopted. The IoU is the ratio of the intersection area to the union area of two bounding boxes. First, all the generated initial bounding boxes are sorted according to their confidence levels. Then, the bounding box with the highest confidence level is selected as the reference bounding box, and the IoU of other bounding boxes with the reference bounding box is calculated. If the IoU of a certain bounding box with the reference bounding box is greater than the set IoU threshold, then this bounding box is regarded as a redundant bounding box and removed. Repeat this process until all bounding boxes are processed, and finally only the bounding boxes with the highest scores and no overlap are retained.

[0041] Step S126: Match and associate the filtered class identification information with the suppressed normalized bounding box coordinates according to the spatial grid regions, and output the feature distribution data.

[0042] After the previous processing, the filtered class identification information and the suppressed normalized bounding box coordinates are obtained. To effectively integrate these two parts of information, they are matched and associated according to the spatial grid regions.

[0043] First, the target area is divided into multiple spatial grid regions. The division method can be selected according to the actual situation. For example, it can be divided at equal intervals or adaptively according to the feature distribution of the image. Each spatial grid region has a certain range and coordinates.

[0044] Then, determine the spatial grid region to which it belongs according to the normalized bounding box coordinates. For each normalized bounding box, calculate the coordinates of its center point, and judge the spatial grid region to which it belongs according to the coordinates of the center point. Associate the corresponding class identification information with this grid region. For example, if the center point of a normalized bounding box is located within a certain spatial grid region, and the class identification information corresponding to this bounding box is a certain component category, then associate this class identification information with this spatial grid region.

[0045] Through this way of matching and association, the class identification information and the normalized bounding box coordinates are integrated together, and the feature distribution data containing the class identification information and the boundary region coordinate information of the object to be operated is output. These feature distribution data will be used for subsequent calibration and positioning processing.

[0046] Step S130: Perform calibration processing on the feature distribution data to generate the coordinate data of the object to be operated in the three-dimensional space coordinate system.

[0047] After obtaining the feature distribution data, it is necessary to perform calibration processing on it to generate the accurate coordinate data of the object to be operated in the three-dimensional space coordinate system.

[0048] Step S131: According to the preset reference feature, compensate the position offset in the feature distribution data to obtain the compensated feature position data. The compensation process includes pixel-level error correction based on the calibration template and reconstruction of the coordinate system mapping relationship.

[0049] First of all, it is necessary to obtain the preset reference feature. The reference feature can be some specific marks or known feature points preset on the production line. These reference features have accurate position information for subsequent calibration processing.

[0050] After obtaining the feature distribution data, it is found that there is a position offset in it. The position offset may be caused by factors such as the installation error of the camera and lens distortion. In order to eliminate these position offsets, a compensation process is carried out.

[0051] The first step of the compensation process is pixel-level error correction based on the calibration template. The calibration template is a template with known features, such as a checkerboard template. By comparing the calibration template image captured by the camera with the known calibration template information, the distortion parameters of the camera, such as the radial distortion coefficient and tangential distortion coefficient, can be calculated. According to these distortion parameters, the coordinate information of the boundary region in the feature distribution data is subjected to distortion correction processing to eliminate the pixel-level error caused by lens distortion.

[0052] Next, the coordinate system mapping relationship is reconstructed. Obtain the internal parameter matrix of the calibration template and the external parameter matrix of the calibration template of the image acquisition device. The internal parameter matrix of the calibration template contains focal length and principal point coordinate parameters, and the external parameter matrix of the calibration template contains camera pose parameters. According to these two matrices, a coordinate system conversion relationship can be constructed to map the image pixel coordinates to three-dimensional physical coordinates.

[0053] Assume that the internal parameter matrix of the calibration template is K, the external parameter matrix of the calibration template is [R|t] (R is the rotation matrix, t is the translation vector), and the image pixel coordinate is p. Through the formula P = K^(-1) * [R|t]^(-1) * p, the image pixel coordinate p can be converted into the three-dimensional physical coordinate P. According to this coordinate system conversion relationship, the coordinate information of the boundary region in the feature distribution data is converted to obtain the corrected boundary coordinate data.

[0054] Based on the difference value between the reference feature and the corrected boundary coordinate data, calculate the compensation coefficient of the position offset. The difference value can be determined by Euclidean distance measurement and least squares optimization. Assume that the coordinate of the reference feature is P_ref, and the corrected boundary coordinate data is P_corrected. Through the least squares optimization method, find a compensation coefficient α such that ∑(P_corrected + α - P_ref)^2 is the smallest.

[0055] Finally, update the coordinate point data in the feature distribution data according to the compensation coefficient. The update process includes coordinate translation transformation and scaling adjustment. For each coordinate point p in the feature distribution data, the updated coordinate point p' = p + α, completing the compensation process to obtain the compensated feature position data.

[0056] Step S132: Map the compensated feature position data to the target grid area in the image coordinate system, and extract the set of feature points within the target grid area. The target grid area is dynamically divided according to the image resolution and the object distribution density. The set of feature points includes the edge feature points and internal key points of the object to be operated on.

[0057] To further accurately extract the feature information of the object to be operated on, map the compensated feature position data to the target grid area in the image coordinate system.

[0058] First, according to the resolution parameters of the real-time multi-view image data and the preset object distribution density threshold, the image coordinate system is dynamically divided into multiple grid cells. The object distribution density threshold is determined by statistically averaging the pixel spacing of object distributions in historical data. If the image resolution is high and the object distribution density is large, the size of the divided grid cells is small to capture object features more finely; conversely, if the image resolution is low and the object distribution density is small, the size of the divided grid cells is large.

[0059] Then, according to the coordinate point distribution density in the compensated feature position data, determine the coverage range of the target grid area. Density clustering algorithms and region growing methods can be used to determine the coverage range. Density clustering algorithms, such as the DBSCAN algorithm, divide the coordinate points into different clustering clusters according to their density. The region growing method starts from a seed point and continuously expands the region according to the set growth rules until the stop condition is met.

[0060] Cluster the coordinate point data in the compensated feature position data according to the grid cells. Algorithms such as Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and mean shift algorithm can be used. Through these clustering algorithms, the coordinate point data is divided into different clusters to generate a set of coordinate point clusters within the target grid area.

[0061] Extract the target coordinate point cluster with the highest density from the set of coordinate point clusters. The target coordinate point cluster can be determined by counting the number of points within the cluster and evaluating the compactness of the spatial distribution. For example, select the cluster with the largest number of points within the cluster and relatively small distances between the points as the target coordinate point cluster.

[0062] Calculate the center point of the target coordinate point cluster as the mapping reference point. The center point can be determined by geometric median calculation or centroid coordinates. Assume that the coordinate points in the target coordinate point cluster are p_1, p_2,..., p_n, and the centroid coordinate is C = (∑p_i) / n.

[0063] Map the compensated feature position data to the target grid area in the image coordinate system based on the mapping reference point. Translate and scale each coordinate point relative to the mapping reference point so that it falls within the target grid area. Then, extract the set of feature points within the target grid area. These feature points include the edge feature points and internal key points of the object to be operated on. The edge feature points can be extracted by an edge detection algorithm, such as the Canny edge detection algorithm; the internal key points can be extracted by a corner detection algorithm, such as the Harris corner detection algorithm.

[0064] Step S133: Perform geometric transformation processing on the set of feature points to generate the center point coordinate data and boundary contour coordinate data of the object to be operated on. The geometric transformation processing includes affine transformation and perspective correction.

[0065] After obtaining the set of feature points, in order to more accurately determine the position and pose of the object to be operated on in three-dimensional space, geometric transformation processing needs to be performed on the set of feature points. Here, the geometric transformation mainly includes affine transformation and perspective correction.

[0066] First, perform an affine transformation. An affine transformation is a linear transformation between two-dimensional coordinates, which can maintain the parallelism of lines and can perform operations such as translation, rotation, scaling, and shearing. In this process, the affine transformation matrix needs to be determined. At least three non-collinear feature points in the set of feature points can be selected as reference points. Let the coordinates of these three reference points in the original coordinate system be A, B, C, and the expected corresponding coordinates in the target coordinate system be A', B', C'. The affine transformation matrix M can be obtained by solving a set of linear equations. This set of linear equations is constructed based on the properties of the affine transformation, that is, for each reference point, its coordinates in the original coordinate system can obtain the corresponding coordinates in the target coordinate system after the action of the affine transformation matrix M. By solving this set of linear equations, each element in the affine transformation matrix M can be determined.

[0067] After obtaining the affine transformation matrix M, perform an affine transformation on each feature point in the set of feature points. Let the coordinate of a feature point in the set of feature points be P, and the coordinate after affine transformation be P'. Then P' is obtained by multiplying P by the affine transformation matrix M. Perform such operations on all feature points in the set of feature points to obtain the set of feature points after affine transformation.

[0068] Next, we will perform perspective correction. The purpose of perspective correction is to eliminate the perspective distortion caused by the camera's viewing angle so that the objects in the image present a normal proportion and shape. To perform perspective correction, it is also necessary to select four non-collinear feature points in the feature point set as reference points. Suppose the coordinates of these four reference points in the original image are Q1, Q2, Q3, and Q4, and the corresponding coordinates in the desired corrected image are Q1', Q2', Q3', and Q4'. The perspective transformation matrix H can be calculated through the coordinates of these four sets of corresponding points. The calculation process of the perspective transformation matrix H is also based on a set of linear equations, which describe the relationship between the points in the original image and the corresponding points in the corrected image after the perspective transformation.

[0069] After obtaining the perspective transformation matrix H, the feature point set after affine transformation is transformed again. Let the coordinates of a feature point after affine transformation be R, and the coordinates after perspective transformation be R', which is obtained by performing corresponding operations on R and the perspective transformation matrix H. All feature points in the feature point set after affine transformation are transformed in perspective, so as to obtain a feature point set after complete geometric transformation.

[0070] After completing the geometric transformation processing, the center point coordinate data and boundary contour coordinate data of the object to be operated are generated according to the processed feature point set. For the calculation of the center point coordinate data, the minimum circumscribed geometric figure is constructed according to the distribution of coordinate points in the feature point set. The minimum circumscribed geometric figure can be a rectangle, a polygon, or an ellipse, etc. The specific selection of the figure depends on the distribution of the feature points. If the distribution of the feature points is approximately a rectangle, the minimum circumscribed rectangle is constructed; if the distribution is more complex, the minimum circumscribed polygon or ellipse is constructed. Taking the construction of the minimum circumscribed rectangle as an example, the minimum circumscribed rectangle is generated by the rotating caliper algorithm or the convex hull algorithm. The rotating caliper algorithm finds the circumscribed rectangle of the minimum area by continuously rotating a convex hull surrounding the feature point set; the convex hull algorithm first finds the convex hull of the feature point set, and then determines the minimum circumscribed rectangle based on the convex hull.

[0071] Calculate the geometric center point coordinates of the minimum circumscribed geometric figure as the center point coordinate data of the object to be operated. If it is a rectangle, the geometric center point can be determined by weighted average of vertex coordinates or area centroid calculation. Suppose the four vertex coordinates of the rectangle are V1, V2, V3, and V4 respectively. The weighted average of vertex coordinates is calculated by adding the x coordinates of the four vertex coordinates and taking the average value, and the y coordinates are added and taken the average value. The obtained coordinates are the coordinates of the geometric center point. The calculation of the area centroid takes into account the area distribution of the rectangle and determines the center point coordinates through more complex calculations.

[0072] For the generation of the boundary contour coordinate data, the vertex coordinate data of the minimum circumscribed geometric figure is extracted. The vertex coordinate data can be obtained through edge detection and corner recognition. The edge detection algorithm can detect the edges of the figure formed by the set of feature points, and the corner recognition algorithm can accurately locate the intersection points of the edges, that is, the vertices. For example, the Canny edge detection algorithm is used to detect edges, and the Harris corner detection algorithm is used to recognize corners. The boundary contour coordinate data is generated according to the connection relationship between the vertex coordinates. The connection relationship is realized through polygon fitting and spline curve interpolation. Polygon fitting is to connect the vertices in a set order to form a polygon; spline curve interpolation is to generate a smoother curve on the basis of the polygon through an interpolation algorithm to more accurately describe the boundary contour of the object to be operated on.

[0073] Step S134: Construct the spatial coordinate mapping relationship of the object to be operated on according to the center point coordinate data and the boundary contour coordinate data, and convert the coordinate data in the image coordinate system into the coordinate data in the three-dimensional space coordinate system through the spatial coordinate mapping relationship.

[0074] After obtaining the center point coordinate data and the boundary contour coordinate data of the object to be operated on, it is necessary to construct the spatial coordinate mapping relationship of the object to be operated on in order to convert the coordinate data in the image coordinate system into the coordinate data in the three-dimensional space coordinate system.

[0075] First, analyze the relationship between the image coordinate system and the three-dimensional space coordinate system. The image coordinate system is two-dimensional, which is the coordinate system used by the camera to capture images, while the three-dimensional space coordinate system is the coordinate system that describes the position and posture of an object in the real space. To establish the mapping relationship between these two coordinate systems, the internal parameters and external parameters of the camera need to be considered. The internal parameters of the camera include parameters such as focal length and principal point coordinates, which describe the internal optical characteristics of the camera; the external parameters of the camera include the position and posture of the camera, that is, the coordinates and orientation of the camera in the three-dimensional space.

[0076] According to the center point coordinate data and the boundary contour coordinate data, combined with the internal parameters and external parameters of the camera, construct the spatial coordinate mapping relationship. This mapping can be achieved through a series of transformations. Assume that the coordinate of a point in the image coordinate system is I, and the coordinate of the corresponding point in the three-dimensional space coordinate system is S. First, convert the point I in the image coordinate system to the camera coordinate system. This requires using the internal parameter matrix K of the camera. By multiplying I by the inverse matrix of K, the point I_c in the camera coordinate system is obtained.

[0077] Then, convert the point I_c in the camera coordinate system to the three-dimensional space coordinate system. This requires using the external parameter matrix [R|t] of the camera, where R is the rotation matrix and t is the translation vector. By multiplying I_c by the rotation matrix R and then adding the translation vector t, the point S in the three-dimensional space coordinate system is obtained.

[0078] For each point in the center point coordinate data and the boundary contour coordinate data, the conversion is performed according to the above steps. In this way, the center point coordinate data and the boundary contour coordinate data in the image coordinate system can be converted into coordinate data in the three-dimensional space coordinate system. By constructing and converting this spatial coordinate mapping relationship, the position of the object to be operated in the three-dimensional space can be accurately determined.

[0079] Step S140: Perform three-dimensional pose solution processing based on the coordinate data to obtain a set of pose parameters of the object to be operated, where the set of pose parameters includes the spatial position parameters and the attitude angle parameters of the object to be operated.

[0080] After obtaining the coordinate data of the object to be operated in the three-dimensional space coordinate system, it is necessary to perform three-dimensional pose solution processing based on these coordinate data to obtain a set of pose parameters of the object to be operated, which includes spatial position parameters and attitude angle parameters.

[0081] Step S141: Perform spatial projection processing on the coordinate data according to the real-time multi-view image data to generate multi-view projection feature data of the object to be operated. The multi-view image data includes synchronous observation information of the object to be operated from different perspectives. The spatial projection processing includes multi-view projection matrix construction and depth information fusion. The multi-view projection feature data includes the spatial distribution information of the object surface features from different perspectives.

[0082] Perform spatial projection processing on the coordinate data using the real-time multi-view image data. The real-time multi-view image data includes synchronous observation information of the object to be operated from different perspectives and can provide more comprehensive object features.

[0083] First, construct the multi-view projection matrix. For each camera in each perspective, there is its corresponding projection matrix. The projection matrix describes how points in three-dimensional space are projected onto a two-dimensional image plane. To construct the multi-view projection matrix, the internal and external parameters of each camera need to be known. The internal parameters include the focal length, principal point coordinates, etc., and the external parameters include the position and attitude of the camera. Based on these parameters, the projection matrix of each camera can be calculated. Let the internal parameter matrix of the i-th camera be K_i and the external parameter matrix be [R_i|t_i], then the projection matrix P_i of the i-th camera can be obtained by multiplying K_i with [R_i|t_i].

[0084] Then, project the coordinate data in the three-dimensional space coordinate system through the projection matrix of each camera to obtain the two-dimensional projection coordinates in each perspective. Let the coordinates of a point in three-dimensional space be X, and the two-dimensional projection coordinates obtained by projecting through the projection matrix P_i of the i-th camera be x_i.

[0085] While performing projection, it is also necessary to perform depth information fusion. Depth information can be obtained through methods such as binocular vision and structured light. Depth information from different perspectives may vary and needs to be fused to obtain more accurate depth information. The weighted average method can be used for depth information fusion. Let the depth information at the i-th perspective be d_i, and the corresponding weight be w_i. The fused depth information D can be obtained by weighted summing the depth information from all perspectives, that is, D = ∑(w_i * d_i).

[0086] Through the construction of multi-perspective projection matrices and depth information fusion, multi-perspective projection feature data of the object to be operated is generated. These multi-perspective projection feature data contain the spatial distribution information of the object surface features from different perspectives.

[0087] Step S142: Based on the multi-perspective projection feature data, construct a pose solution matrix, and update the rotation parameters and translation parameters in the pose solution matrix through an iterative optimization algorithm until the reprojection error converges to a preset threshold, obtaining the optimized rotation parameters and translation parameters. The pose solution matrix contains the initial estimated values of the rotation parameters and translation parameters, and the iterative optimization algorithm includes error minimization based on gradient descent and parameter convergence verification.

[0088] Construct a pose solution matrix based on the multi-perspective projection feature data. The pose solution matrix contains the initial estimated values of the rotation parameters and translation parameters. The rotation parameters describe the rotation state of the object in three-dimensional space, and the translation parameters describe the translation state of the object in three-dimensional space.

[0089] First, obtain the observation position data of the operation execution device from different perspectives. The observation position data of the operation execution device can be obtained through encoder feedback and forward kinematic solution calculation. The encoder can provide real-time feedback on the joint angle information of the operation execution device. Through the forward kinematic algorithm, the position and orientation of the operation execution device in three-dimensional space can be calculated based on the joint angle information. Let the observation position data of the operation execution device at the j-th perspective be O_j.

[0090] Construct an observation matrix based on the observation position data. The observation matrix contains camera perspective transformation parameters and object projection relationships. Let the observation matrix at the j-th perspective be M_j, which is related to the internal and external parameters of the camera and the observation position data of the operation execution device.

[0091] Based on the correspondence between the multi-view projection feature data and the observation matrix, the initial parameters of the pose calculation matrix are constructed. The correspondence can be determined through feature point matching and projection error calculation. Feature point matching refers to finding the corresponding feature points in the multi-view projection feature data and the observation matrix, and the projection error refers to the distance error between these corresponding feature points. By minimizing the projection error, the initial parameters of the pose calculation matrix can be determined.

[0092] After obtaining the initial parameters of the pose calculation matrix, an iterative optimization algorithm is used to update the rotation parameters and translation parameters. The iterative optimization algorithm adopts error minimization based on gradient descent and parameter convergence verification. The gradient descent algorithm is a commonly used optimization algorithm, which updates the parameters by continuously moving along the negative gradient direction of the error function to reduce the error. Let the rotation parameter in the pose calculation matrix be R, the translation parameter be t, and the error function be E(R, t). Then in each iteration, according to the gradient of the error function Update the rotation parameters and translation parameters.

[0093] After each iteration, parameter convergence verification is required. Determine whether the reprojection error converges to a preset threshold. The reprojection error refers to the error between the projection result obtained by applying the updated pose parameters to the multi-view projection feature data and the observation matrix. If the reprojection error is less than the preset threshold, it is considered that the parameters have converged and the iteration ends; otherwise, continue the iterative update.

[0094] After multiple iterations, until the reprojection error converges to the preset threshold, the optimized rotation parameters and translation parameters are obtained.

[0095] Step S143: Generate the pose parameter set according to the optimized rotation parameters and translation parameters. The pose parameter set includes attitude angle parameters and spatial position parameters. The attitude angle parameters are used to describe the rotation state of the object in three-dimensional space, and the spatial position parameters are used to describe the translation state of the object in three-dimensional space.

[0096] After obtaining the optimized rotation parameters and translation parameters, a pose parameter set is generated according to these parameters.

[0097] The attitude angle parameters can be derived from the optimized rotation parameters. The rotation parameters are usually represented by a rotation matrix or a quaternion. If the rotation matrix R is used to represent the rotation parameters, the rotation matrix can be converted into attitude angle parameters through a set conversion method, such as Euler angles (including the rotation angle around the X-axis, the rotation angle around the Y-axis, and the rotation angle around the Z-axis). The conversion process involves the mathematical relationship between the elements of the rotation matrix and the attitude angles, and the attitude angle parameters can be obtained by solving a system of equations.

[0098] The spatial position parameters can be directly obtained from the optimized translation parameters. The translation parameters describe the translation state of an object in three-dimensional space, and the three components respectively correspond to the translation distances of the object in the X, Y, and Z axis directions.

[0099] Combine the attitude angle parameters and the spatial position parameters to form a pose parameter set. This pose parameter set accurately describes the position and attitude of the object to be operated in three-dimensional space.

[0100] Step S150: Generate a control command according to the pose parameter set and the object physical property parameters corresponding to the category identification information, and drive the operation execution device to complete the object grasping action of the object to be operated. The control command includes a path control signal, an end effector attitude control signal, and a clamping force control signal based on the category identification information.

[0101] After obtaining the pose parameter set and the category identification information, generate a control command according to the object physical property parameters corresponding to the category identification information to drive the operation execution device to complete the grasping action of the object to be operated.

[0102] Step S151: Call the preset grasping strategy library according to the category identification information, and select the clamping force parameters and motion trajectory constraint conditions that match the category identification information.

[0103] The preset grasping strategy library stores the grasping strategies corresponding to different category identification information, including clamping force parameters and motion trajectory constraint conditions. According to the category identification information of the object to be operated, search for the matching information in the grasping strategy library.

[0104] For objects to be operated of different categories, their physical properties are different, and the required clamping forces are also different. For example, for objects with a harder texture and a larger weight, a larger clamping force is required; for objects with a softer texture and being fragile, a smaller clamping force is required. According to the category identification information, the corresponding clamping force parameters can be determined.

[0105] The motion trajectory constraint condition refers to the trajectory rule that the operation execution device needs to follow during the process of grasping an object. Objects of different categories may have different motion trajectory requirements. For example, for some objects with special shapes, the operation execution device may need to approach the object along a specific path to avoid collision or damage to the object. According to the category identification information, select the corresponding motion trajectory constraint conditions from the grasping strategy library.

[0106] Step S152: Input the set of pose parameters into the motion planning model. Based on inverse kinematics and path optimization algorithms, the motion planning model generates the motion trajectory data and end - effector pose data of the operation execution device. The motion trajectory data includes a joint motion sequence and speed planning parameters and satisfies the motion trajectory constraint conditions.

[0107] For example, step S1521: Based on the spatial position parameters and attitude angle parameters in the set of pose parameters, construct a target homogeneous transformation matrix for the end - effector of the robotic arm. The target homogeneous transformation matrix includes a translation component and a rotation component.

[0108] The spatial position parameters in the set of pose parameters describe the position of the end - effector of the robotic arm in three - dimensional space, and the attitude angle parameters describe the attitude of the end - effector of the robotic arm. The spatial position parameters are represented as a three - dimensional vector, and the attitude angle parameters are represented by a rotation matrix. The target homogeneous transformation matrix is a 4×4 matrix. The upper - left 3×3 sub - matrix is the rotation matrix, representing the rotation information corresponding to the attitude angle parameters; the first three elements of the fourth column are the three components of the spatial position parameters, representing the translation information; the elements of the last row are [0, 0, 0, 1]. In this way, the spatial position parameters and attitude angle parameters are integrated into the target homogeneous transformation matrix for subsequent processing.

[0109] Step S1522: Perform a feasible - space verification on the target homogeneous transformation matrix according to the motion trajectory constraint conditions. The feasible - space verification includes joint motion range limit detection and avoidance of the interference area of the end - effector.

[0110] The motion trajectory constraint conditions are rules set according to the category of the object to be operated and the actual working environment. The joint motion range limit detection is to check whether the robotic - arm joint angles corresponding to the target homogeneous transformation matrix are within the allowable motion ranges of each joint. By presetting the minimum and maximum motion angles of each joint, the joint angles calculated from the target homogeneous transformation matrix through inverse kinematics are compared with these ranges. If the angle of a certain joint exceeds the allowable range, the target homogeneous transformation matrix is not feasible.

[0111] Avoidance of the interference area of the end - effector means checking whether the end - effector will interfere with surrounding objects in the pose corresponding to the target homogeneous transformation matrix. By establishing a three - dimensional model of the working environment and comparing the model of the end - effector with this three - dimensional model, it can be determined whether there is an interference situation. If there is interference, the target homogeneous transformation matrix needs to be adjusted or re - planned.

[0112] Step S1523: Based on inverse kinematics, perform multi-solution screening on the verified target homogeneous transformation matrix to generate a set of candidate joint angles that satisfy the motion trajectory constraint conditions. The screening criterion preferentially selects the candidate solution with the shortest joint motion path and the lowest energy consumption. The multi-solution screening includes Jacobian matrix singularity analysis and joint angular velocity threshold traversal.

[0113] Inverse kinematics is the process of solving the joint angles based on the position and orientation of the end effector of the robotic arm. For a given target homogeneous transformation matrix, there may be multiple solutions for inverse kinematics. To screen out the appropriate solutions, first, Jacobian matrix singularity analysis is performed. The Jacobian matrix describes the relationship between the joint velocities of the robotic arm and the velocity of the end effector. If the Jacobian matrix is singular, it means that the robotic arm is in a singular configuration. At this time, a small change in the joint may cause a large movement of the end effector, which is not conducive to control. Therefore, singular solutions need to be excluded.

[0114] Then, joint angular velocity threshold traversal is performed, that is, check whether the joint angular velocity corresponding to each candidate solution is within the allowable range. If the joint angular velocity of a certain candidate solution is too large, it will cause the robotic arm to move unstably or exceed the capacity of the motor. Therefore, it also needs to be excluded. Finally, according to the screening criterion, the candidate solution with the shortest joint motion path and the lowest energy consumption is preferentially selected to generate a set of candidate joint angles that satisfy the motion trajectory constraint conditions.

[0115] Step S1524: Use a path optimization algorithm to perform smooth trajectory planning on the set of candidate joint angles to generate an optimized joint motion sequence and velocity planning parameters. The path optimization algorithm includes dynamic time warping processing and joint acceleration continuity constraint.

[0116] The purpose of the path optimization algorithm is to make the motion of the robotic arm smoother and more stable, reducing vibration and impact. Dynamic time warping processing can adjust the time sequence of joint motion to make the motion of each joint more coordinated. By performing dynamic time warping on each joint angle sequence in the set of candidate joint angles, the rhythm of joint motion can be made more reasonable.

[0117] The joint acceleration continuity constraint means ensuring that the joint acceleration changes continuously during the motion and avoiding sudden changes. By interpolating and fitting the joint angle sequence, the joint acceleration can change smoothly between adjacent time points. Through these path optimization algorithms, the set of candidate joint angles is processed to generate an optimized joint motion sequence and velocity planning parameters. The joint motion sequence describes the angles of each joint of the robotic arm at different time points, and the velocity planning parameters specify the velocity of each joint during the motion.

[0118] Step S1525: Perform attitude synchronous interpolation on the optimized joint motion sequence according to the end effector attitude adjustment parameters to generate an end effector attitude angle change sequence corresponding to each joint motion node, where the attitude synchronous interpolation includes rotation matrix linear interpolation and attitude angular rate consistency matching.

[0119] The end effector attitude adjustment parameters are determined according to the grasping requirements and actual situation of the object to be operated, and they specify the attitude that the end effector needs to reach during the grasping process. The purpose of attitude synchronous interpolation is to synchronize the attitude change of the end effector with the motion of the robotic arm joints.

[0120] Rotation matrix linear interpolation is to linearly interpolate the rotation matrix of the end effector between different joint motion nodes to ensure smooth transition of the attitude. Attitude angular rate consistency matching means ensuring that the attitude angular rate of the end effector matches the speed of the joint motion. By combining the end effector attitude adjustment parameters with the optimized joint motion sequence, attitude synchronous interpolation is performed to generate an end effector attitude angle change sequence corresponding to each joint motion node. This end effector attitude angle change sequence describes the attitude angle of the end effector at each moment during the motion of the robotic arm.

[0121] Step S1526: Associate the joint motion sequence, speed planning parameters, and attitude angle change sequence as motion trajectory data and end effector attitude data that meet the motion trajectory constraint conditions.

[0122] Associate the optimized joint motion sequence, speed planning parameters, and the end effector attitude angle change sequence obtained through attitude synchronous interpolation to form complete motion trajectory data and end effector attitude data. The motion trajectory data details information such as the motion sequence, speed, and time of each joint of the operation execution device when performing the grasping task, and the end effector attitude data specifies the attitude angle of the end effector at different joint motion nodes. In this way, through the motion planning model, considering inverse kinematics and path optimization algorithms comprehensively, and combining the motion trajectory constraint conditions, motion trajectory data and end effector attitude data suitable for the operation execution device to complete the grasping task are generated.

[0123] Step S153: Generate a path control signal according to the motion trajectory data, and the path control signal drives the operation execution device to move to the target grasping position through pulse width modulation and a servo drive interface.

[0124] After obtaining the motion trajectory data, it is necessary to convert it into a path control signal that can drive the operation execution device. The motion trajectory data contains the joint motion sequence and speed planning parameters, which determine the position and motion speed of each joint of the operation execution device at different times.

[0125] The generation of the path control signal is based on the joint motion sequence and velocity planning parameters in the motion trajectory data. First, each joint position and velocity information in the joint motion sequence is encoded and converted into digital signals. These digital signals represent the positions and velocities that each joint of the operation execution device should reach at different times.

[0126] Then, pulse width modulation (PWM) technology is used to process these digital signals. Pulse width modulation is a technology that controls the speed and torque of a motor by changing the duty cycle of a pulse signal. For each joint motor of the operation execution device, the duty cycle of the pulse signal is adjusted according to its corresponding velocity planning parameter. If the joint needs to move quickly, the duty cycle of the pulse signal will increase; if the joint needs to move slowly, the duty cycle will decrease.

[0127] The signals processed by pulse width modulation are transmitted to each joint motor of the operation execution device through a servo drive interface. The servo drive interface is a bridge connecting the control system and the joint motor, and it can convert the received control signal into an electrical signal that the motor can recognize and execute. Through the servo drive interface, the path control signal is accurately transmitted to each joint motor, driving the operation execution device to move to the target grasping position along the path specified by the motion trajectory data.

[0128] Step S154: Generate an end effector attitude control signal and a clamping force control signal based on the end effector attitude data and the clamping force parameter, and respectively execute the rotation axis control of the end effector and the clamping force feedback adjustment based on the end effector attitude control signal and the clamping force control signal, triggering a clamping action to complete the grasping of the object to be operated.

[0129] Based on the end effector attitude data and the clamping force parameter, an end effector attitude control signal and a clamping force control signal are generated respectively. The end effector attitude data describes the attitude angles that the end effector should reach when grasping an object, and corresponding attitude control signals are generated according to this attitude angle information. The attitude control signal is used to control the rotation axis of the end effector so that it can be accurately adjusted to the appropriate attitude.

[0130] The clamping force parameter is selected from the grasping strategy library according to the category identification information of the object to be operated, and it determines the force that the end effector needs to apply when grasping an object. According to the clamping force parameter, a clamping force control signal is generated. The clamping force control signal is used to control the clamping device of the end effector so that it can apply an appropriate force to grasp the object.

[0131] During the execution process, the rotation axis of the end effector is controlled based on the end effector attitude control signal. The attitude control signal is transmitted to the rotation axis motor of the end effector through the corresponding drive circuit, and the motor rotates according to the instructions of the signal, thereby adjusting the attitude of the end effector. At the same time, in order to ensure that the end effector can accurately reach the required attitude, attitude feedback adjustment is also required. An attitude sensor can be installed on the end effector to real-time monitor the actual attitude angle of the end effector and feedback the monitored information to the control system. The control system compares the actual attitude angle with the target attitude angle. If there is a deviation, it will adjust the attitude control signal to make the attitude of the end effector gradually approach the target attitude.

[0132] For the clamping force control signal, feedback adjustment is also required. A force sensor is installed on the clamping device of the end effector to real-time monitor the actual force exerted by the clamping device. The actual force information is feedback to the control system, and the control system compares the actual force with the clamping force parameter. If the actual force is less than the clamping force parameter, it means that the clamping force is insufficient, and the control system will increase the clamping force control signal to make the clamping device exert a greater force; if the actual force is greater than the clamping force parameter, it means that the clamping force is too large, and the control system will decrease the clamping force control signal to avoid damaging the object to be operated.

[0133] When the attitude of the end effector is adjusted to the appropriate position and the clamping force reaches the set parameter, the clamping action is triggered. The clamping device tightly clamps the object to be operated according to the instructions of the clamping force control signal, and completes the grasping task of the object to be operated.

[0134] In the entire object collaborative control method based on deep learning, starting from obtaining real-time multi-view image data by the image acquisition device, through object feature recognition by the real-time recognition model, calibration processing of feature distribution data, 3D pose calculation processing, to finally generating control instructions according to the pose parameter set and object physical attribute parameters to drive the operation execution device to complete the object grasping action, each step is closely connected, forming a complete control process.

[0135] In the data acquisition stage, the high-precision industrial camera is used as the image acquisition device, and the selection of its performance parameters and the layout method directly affect the quality of the real-time multi-view image data collected. Cameras at different positions and angles can capture visual feature information of the object to be operated from multiple aspects.

[0136] The real-time recognition model adopts an improved YOLOv8 algorithm. Through a series of operations such as a multi-level dilated convolutional network, a cross-level feature pyramid, channel-dimension attention weighting, and spatial-dimension attention weighting, it accurately identifies the category identification information and boundary region coordinate information of the object to be operated. The multi-level dilated convolutional network can dynamically adjust the dilation rate according to the input image resolution and extract features at different scales; the cross-level feature pyramid integrates feature maps at different levels through bilinear interpolation and per-channel weighted fusion; channel-dimension attention weighting and spatial-dimension attention weighting further highlight important feature information and improve the accuracy of recognition.

[0137] The calibration process of the feature distribution data is a key step to ensure the accuracy of subsequent operations. Based on pixel-level error correction of the calibration template and reconstruction of the coordinate system mapping relationship, the position offset caused by factors such as camera installation error and lens distortion is eliminated. The compensated feature position data is mapped to the target grid area in the image coordinate system, and a set of feature points is extracted. Then, through geometric transformation processing, the center point coordinate data and boundary contour coordinate data of the object to be operated are generated. Finally, a spatial coordinate mapping relationship is constructed to convert the coordinate data in the image coordinate system into coordinate data in the three-dimensional space coordinate system.

[0138] The three-dimensional pose calculation process generates multi-view projection feature data through spatial projection processing. Based on these data, a pose calculation matrix is constructed, and the rotation parameters and translation parameters are updated through an iterative optimization algorithm to obtain an optimized set of pose parameters. These pose parameter sets accurately describe the position and pose of the object to be operated in three-dimensional space.

[0139] Finally, control instructions are generated based on the set of pose parameters and the physical property parameters of the object to drive the operation execution device to complete the object grasping action. By calling the preset grasping strategy library, appropriate clamping force parameters and motion trajectory constraint conditions are selected; the motion planning model generates motion trajectory data and end effector pose data based on inverse kinematics and path optimization algorithms; the path control signal drives the operation execution device to move to the target grasping position through pulse width modulation and servo drive interfaces; the end effector pose control signal and the clamping force control signal respectively control the pose and clamping force of the end effector, and ensure the accuracy of the operation through feedback adjustment, and finally complete the grasping task of the object to be operated.

[0140] In the above implementation process, it is also necessary to pay attention to data privacy protection and anti-disclosure issues. For the collected real-time multi-view image data, these real-time multi-view image data may contain sensitive information of the object to be operated, such as the design details of the product, etc. To protect these privacy-sensitive data, encryption technology is used to encrypt the data. During data transmission, a secure communication protocol, such as the SSL / TLS protocol, is used to ensure that the data is not stolen or tampered with during transmission. In terms of data storage, the data is stored on a secure server, and strict access permissions are set, and only authorized personnel can access this data.

[0141] In addition, for the training of the real-time recognition model, a large amount of labeled data is required. These labeled data should cover various types of objects to be operated, as well as their images under different perspectives and different lighting conditions. During the training process, optimization algorithms such as stochastic gradient descent are used to continuously adjust the parameters of the model, so as to continuously improve the recognition accuracy of the model. Appropriate parameters such as the learning rate and batch size also need to be set during the training process to ensure the convergence speed and stability of the model.

[0142] For the motion planning model, its performance also directly affects the motion effect of the operation execution device. When performing inverse kinematics calculation and path optimization, it is necessary to consider the dynamic characteristics of the robotic arm, such as the torque limit of the joints, the acceleration and deceleration of the motion, etc. At the same time, the selection of the path optimization algorithm is also very important. Different algorithms may obtain different motion trajectories, and it is necessary to select the most suitable algorithm according to the actual situation to ensure that the operation execution device can efficiently and stably complete the grasping task.

[0143] In practical applications, it is also necessary to monitor and debug the entire system in real time. By monitoring parameters such as the motion state of the operation execution device, the attitude and clamping force of the end effector, problems existing in the system can be discovered in a timely manner and corresponding adjustments can be made. At the same time, continuously collect data in practical applications to optimize and improve the real-time recognition model and the motion planning model, and improve the performance and adaptability of the system.

[0144] In terms of the hardware of the system, the performance of the high-precision industrial camera, the accuracy and stability of the operation execution device, etc. will all affect the effect of the entire system. Therefore, when selecting hardware devices, it is necessary to comprehensively consider factors such as their performance, price, and reliability. For example, parameters such as the resolution, frame rate, and sensitivity of the high-precision industrial camera will affect the quality of the collected images; parameters such as the repeat positioning accuracy and load capacity of the operation execution device will affect the accuracy and stability of its grasping.

[0145] For the software aspect of the system, the interfaces and communication protocols between various modules also need to be carefully designed. Data transmission and interaction are required among the real-time recognition model, calibration processing module, 3D pose calculation module, and control instruction generation module. Therefore, a unified interface and communication protocol need to be defined to ensure the accurate transmission and processing of data. At the same time, the stability and reliability of the software are also very important, and sufficient testing and optimization are required to avoid system crashes or errors.

[0146] In terms of system maintenance, the hardware devices need to be inspected and maintained regularly to ensure their normal operation. For the software system, it needs to be updated and upgraded in a timely manner to fix the discovered vulnerabilities and improve the system performance. At the same time, a perfect fault diagnosis and handling mechanism should be established so that when the system fails, the problem can be quickly located and fixed, reducing the system downtime.

[0147] Figure 2 FIG. shows a schematic diagram of exemplary hardware and software components of a deep learning-based target collaborative control system 100 that can implement the ideas of the present application provided by some embodiments of the present application. For example, the processor 120 can be used on the deep learning-based target collaborative control system 100 and is used to execute the functions in the present application.

[0148] The deep learning-based target collaborative control system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the deep learning-based target collaborative control method of the present application. Although only one server is shown in the present application, for convenience, the functions described in the present application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0149] For example, the deep learning-based target collaborative control system 100 can include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and different forms of storage media 140, such as disks, ROM, or RAM, or any combination thereof. Exemplarily, the deep learning-based target collaborative control system 100 can also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The methods of the present application can be implemented according to these program instructions. The deep learning-based target collaborative control system 100 also includes an I / O interface 150 between the computer and other input / output devices.

[0150] For ease of explanation, only one processor is described in the object collaborative control system 100 based on deep learning. However, it should be noted that the object collaborative control system 100 based on deep learning in this application may also include multiple processors. Therefore, the steps performed by one processor described in this application may also be jointly performed or separately performed by multiple processors. For example, if the processor of the object collaborative control system 100 based on deep learning performs step A and step B, it should be understood that step A and step B may also be jointly performed by two different processors or separately performed in one processor. For example, the first processor performs step A, the second processor performs step B, or the first processor and the second processor jointly perform steps A and B.

[0151] In addition, an embodiment of the present invention further provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the object collaborative control method based on deep learning as described above is implemented.

[0152] It should be noted that, in order to simplify the description of the present invention disclosure and thus help the understanding of one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, sometimes multiple features are merged into one embodiment, drawing, or description thereof.

Claims

1. A target collaborative control method based on deep learning, characterized in that The method includes: Obtaining real-time multi-view image data of a target area through an image acquisition device, where the real-time multi-view image data includes visual feature information of an object to be operated; Invoking a real-time recognition model to perform object feature recognition on the real-time multi-view image data, obtaining feature distribution data of the object to be operated, where the feature distribution data includes class identification information and boundary region coordinate information of the object to be operated; Performing calibration processing on the feature distribution data to generate coordinate data of the object to be operated in a three-dimensional space coordinate system; Performing three-dimensional pose solution processing based on the coordinate data to obtain a pose parameter set of the object to be operated, where the pose parameter set includes spatial position parameters and attitude angle parameters of the object to be operated; Generating a control instruction according to the pose parameter set and object physical attribute parameters corresponding to the class identification information, driving an operation execution device to complete an object grasping action of the object to be operated, where the control instruction includes a path control signal, an end effector attitude control signal, and a clamping force control signal based on the class identification information.

2. The object collaborative control method based on deep learning according to claim 1, wherein The invoking the real-time recognition model to perform object feature recognition on the real-time multi-view image data and obtaining the feature distribution data of the object to be operated includes: Performing parallel multi-scale feature extraction on the real-time multi-view image data through a multi-level dilated convolutional network in the real-time recognition model to generate a multi-level feature set including feature maps with different resolutions, where the dilation rate of the dilated convolutional kernel at each level is dynamically adjusted according to the reciprocal of the corresponding input image resolution to match object size changes; Inputting the multi-level feature set into a cross-level feature pyramid in the real-time recognition model, performing resolution enhancement on a low-level large receptive field feature map through bilinear interpolation to align its size with that of a high-level small receptive field feature map, and performing channel-wise weighted fusion on the aligned feature maps to generate an enhanced feature map integrating global semantics and local details; Applying channel dimension attention weighting and spatial dimension attention weighting to the enhanced feature map to obtain a weighted feature map, where the channel attention weight is generated by a two-layer fully connected network after global average pooling, and the spatial attention weight is generated by spatial compression features of a single-layer convolutional kernel; Dividing the weighted feature map along the channel dimension into a global classification branch and a local segmentation branch, where the global classification branch is input into a multi-layer perceptron classifier after dimensionality reduction through global max pooling, outputting the class identification information and corresponding confidence of the object to be operated, and filtering out class identification information with a confidence lower than a dynamic threshold; The local segmentation branch is input into a boundary coordinate regressor after being restored to the original input image resolution through a transposed convolutional layer, generating normalized coordinate parameters of an initial bounding box, and using a non-maximum suppression algorithm based on an intersection over union threshold to eliminate redundant bounding boxes with spatial overlap; Matching and associating the filtered class identification information with the suppressed normalized bounding box coordinates according to a spatial grid region, and outputting the feature distribution data.

3. The object collaborative control method based on deep learning according to claim 1, characterized in that Calibrating the feature distribution data to generate coordinate data of the object to be operated in a three-dimensional space coordinate system includes: Compensating for the position offset in the feature distribution data according to a preset reference feature to obtain compensated feature position data, where the compensation process includes pixel-level error correction based on a calibration template and reconstruction of the coordinate system mapping relationship; Mapping the compensated feature position data to a target grid area in the image coordinate system, and extracting a set of feature points within the target grid area. The target grid area is dynamically divided according to the image resolution and the object distribution density, and the set of feature points includes edge feature points and internal key points of the object to be operated; Performing geometric transformation on the set of feature points to generate center point coordinate data and boundary contour coordinate data of the object to be operated, where the geometric transformation includes affine transformation and perspective correction; Constructing a spatial coordinate mapping relationship of the object to be operated according to the center point coordinate data and the boundary contour coordinate data, and converting the coordinate data in the image coordinate system into coordinate data in a three-dimensional space coordinate system through the spatial coordinate mapping relationship.

4. The object collaborative control method based on deep learning according to claim 3, characterized in that Compensating for the position offset in the feature distribution data according to a preset reference feature to obtain compensated feature position data includes: Obtaining the internal parameter matrix and external parameter matrix of the calibration template of the image acquisition device. The internal parameter matrix of the calibration template includes focal length and principal point coordinate parameters, and the external parameter matrix of the calibration template includes camera pose parameters; Constructing a coordinate system conversion relationship according to the internal parameter matrix and external parameter matrix of the calibration template, where the coordinate system conversion relationship is used to map image pixel coordinates to three-dimensional physical coordinates; Performing distortion correction on the boundary region coordinate information according to the coordinate system conversion relationship to obtain corrected boundary coordinate data, where the distortion correction process includes radial distortion coefficient compensation and tangential distortion correction; Calculating a compensation coefficient for the position offset based on the difference value between the reference feature and the corrected boundary coordinate data, where the difference value is determined by Euclidean distance measurement and least squares optimization; Updating the coordinate point data in the boundary region coordinate information according to the compensation coefficient to obtain compensated feature position data, where the update process includes coordinate translation transformation and scale adjustment.

5. The object collaborative control method based on deep learning according to claim 3, characterized in that Mapping the compensated feature position data to a target grid area in the image coordinate system and extracting a set of feature points within the target grid area includes: Dynamically dividing the image coordinate system into multiple grid cells according to the resolution parameter of the real-time multi-view image data and a preset object distribution density threshold, where the density threshold is determined by statistically averaging the pixel spacing of object distributions in historical data; Determining the coverage range of the target grid area according to the coordinate point distribution density in the compensated feature position data, where the coverage range is determined by a density clustering algorithm and a region growing method; Cluster the coordinate point data in the compensated feature position data according to grid cells to generate a set of coordinate point clusters within the target grid region. The clustering process includes density-based spatial clustering of applications with noise and the mean shift algorithm; Extract the target coordinate point cluster with the maximum density in the set of coordinate point clusters. The target coordinate point cluster is determined by counting the number of points within the cluster and evaluating the compactness of the spatial distribution; Calculate the center point of the target coordinate point cluster as the mapping reference point. The center point is determined by geometric median calculation or centroid coordinates; Based on the mapping reference point, map the compensated feature position data to the target grid region in the image coordinate system, and extract the set of feature points within the target grid region.

6. The object collaborative control method based on deep learning according to claim 3, wherein The geometric transformation processing of the set of feature points to generate the center point coordinate data and the boundary contour coordinate data of the object to be operated includes: According to the distribution of coordinate points in the set of feature points, construct the minimum circumscribed geometric figure. The minimum circumscribed geometric figure includes a rectangle, a polygon or an ellipse, and is generated by the rotating calipers algorithm or the convex hull algorithm; Calculate the geometric center point coordinate of the minimum circumscribed geometric figure as the center point coordinate data of the object to be operated. The geometric center point is determined by weighted average of vertex coordinates or area centroid calculation; Extract the vertex coordinate data of the minimum circumscribed geometric figure. The vertex coordinate data is obtained through edge detection and corner recognition; Generate the boundary contour coordinate data according to the connection relationship between vertex coordinates. The connection relationship is realized through polygon fitting and spline curve interpolation processing.

7. The method for target collaborative control based on deep learning according to claim 1, wherein The three-dimensional pose solution processing based on the coordinate data to obtain the set of pose parameters of the object to be operated includes: Perform spatial projection processing on the coordinate data according to the real-time multi-view image data to generate multi-view projection feature data of the object to be operated. The multi-view image data contains synchronous observation information of the object to be operated from different perspectives. The spatial projection processing includes construction of a multi-view projection matrix and depth information fusion. The multi-view projection feature data contains spatial distribution information of object surface features from different perspectives; Construct a pose solution matrix based on the multi-view projection feature data, and update the rotation parameters and translation parameters in the pose solution matrix through an iterative optimization algorithm until the reprojection error converges to a preset threshold to obtain the optimized rotation parameters and translation parameters. The pose solution matrix contains initial estimated values of the rotation parameters and translation parameters. The iterative optimization algorithm includes error minimization based on gradient descent and parameter convergence verification; Generate the set of pose parameters according to the optimized rotation parameters and translation parameters. The set of pose parameters includes attitude angle parameters and spatial position parameters. The attitude angle parameters are used to describe the rotation state of the object in three-dimensional space, and the spatial position parameters are used to describe the translation state of the object in three-dimensional space.

8. The object collaborative control method based on deep learning according to claim 7, wherein The construction of the pose solution matrix based on the multi-view projection feature data includes: Obtain the observation position data of the operation execution device at different perspectives, where the observation position data is obtained through encoder feedback and kinematic forward solution calculation; Construct an observation matrix according to the observation position data, where the observation matrix includes camera perspective transformation parameters and object projection relationships; Construct the initial parameters of the pose solution matrix according to the correspondence between the multi-perspective projection feature data and the observation matrix, where the correspondence is determined through feature point matching and projection error calculation; Adjust the initial parameters in the pose solution matrix through a non-linear optimization algorithm to minimize the error value between the multi-perspective projection feature data and the observation matrix, and obtain the final pose solution matrix, where the error value is optimized by reprojection error measurement and covariance matrix weighting.

9. The object collaborative control method based on deep learning according to claim 1, characterized in that Generating a control command according to the pose parameter set and the object physical attribute parameters corresponding to the category identification information to drive the operation execution device to complete the object grasping action of the object to be operated, including: Call a preset grasping strategy library according to the category identification information, and select the clamping force parameter and motion trajectory constraint condition matching the category identification information; Input the pose parameter set into a motion planning model, and generate the motion trajectory data and end effector pose data of the operation execution device through the motion planning model based on inverse kinematics and path optimization algorithms, where the motion trajectory data includes joint motion sequences and speed planning parameters and satisfies the motion trajectory constraint conditions; Generate a path control signal according to the motion trajectory data, and drive the operation execution device to move to the target grasping position through pulse width modulation and a servo drive interface; Generate an end effector pose control signal and a clamping force control signal based on the end effector pose data and the clamping force parameter, and respectively execute the rotation axis control and clamping force feedback adjustment of the end effector based on the end effector pose control signal and the clamping force control signal, and trigger the clamping action to complete the grasping of the object to be operated.

10. A target collaborative control system based on deep learning, characterized in that, Including a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the deep learning-based target collaborative control method according to any one of claims 1-9 above.

Images