Dual-robot collaborative three-dimensional object recognition method and system based on mutual information estimation feature unwrapping

Abstract

The application discloses a kind of based on mutual information estimation feature unwinding dual-robot collaborative three-dimensional target identification method and system, comprising: the three-dimensional point cloud data of aircraft manufacturing scene is collected, and real-time pose information is obtained, and data set is obtained by preprocessing;Point cloud collaborative identification model is constructed, and the features of dual-robot point cloud are extracted by feature extraction module respectively, and input feature unwinding and fusion module are mapped to shared feature space and unique feature space in parallel, to obtain shared feature and unique feature;Based on mutual information estimation, the mutual information lower bound between shared features and the mutual information upper bound between unique features are constrained respectively, and the features after unwinding are fused into agent fusion features;Finally, the three-dimensional target identification result is output through classification and regression module, and after training, verification and test, it is deployed to robot edge computing device.The high-precision, robust three-dimensional target identification in complex manufacturing environment is realized, and the perception ability and overall efficiency of dual-robot collaborative operation are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent manufacturing and multi-robot collaborative perception, in particular to a dual-robot collaborative three-dimensional target recognition method and system based on mutual information estimation feature unwrapping, which can be widely applied to aviation equipment manufacturing, complex structure assembly, and multi-robot collaborative detection and intelligent inspection in industrial scenes. BACKGROUND

[0002] In the assembly operation of typical large-size thin-walled structures such as aircraft skin, wings and fuselages, systematic inspection of key parts is crucial. The inspection content usually covers the initial state of large components such as skin, including stress distribution, structural deformation, attitude angle and ground support condition, etc., which can help to find potential structural hidden dangers or safety risks as soon as possible, and provide reliable data basis and safety guarantee for subsequent high-precision assembly operation.

[0003] At present, the assembly inspection of large aircraft complex parts still mainly relies on manual operation, relying on manual identification, state evaluation and abnormality checking of key structural features. This method not only has low detection efficiency and limited precision, but also is easily disturbed by human subjective factors, resulting in insufficient consistency and reliability, and heavy burden on operators. With the rapid development of robot technology and artificial intelligence, it has become an important trend of intelligent manufacturing to make the assembly inspection process unmanned and intelligent.

[0004] When applying an inspection robot in the aircraft manufacturing process, the core task is to realize its high-precision three-dimensional target recognition in large-scale manufacturing environment. Three-dimensional target recognition can provide spatial position information and geometric dimensions of multi-class objects in the manufacturing scene, including large aircraft components, on-site operators and collaborative robots. With this capability, the robot can perform intelligent and high-precision safety inspection. However, although existing three-dimensional target detection methods based on deep learning have gradually attracted attention in recent years, they still have deficiencies in actual aviation manufacturing scenarios and cannot fully meet the requirements of high precision, stability and real-time engineering applications. The following difficulties are faced in the specific implementation process:

[0005] 1. The aircraft manufacturing scene often has the characteristics of wide spatial range and complex structure. A single robot is limited by its own perception ability and coverage radius, and it is difficult to complete the complete perception of the whole scene. Therefore, it is necessary to introduce the dual-robot collaborative operation mode, to realize wider monitoring and wider visual coverage by sharing and complementary perception information of cluster robots, so as to improve the comprehensiveness and reliability of environmental target detection.

[0006] 2. In dual-robot collaborative perception, due to the different spatial positions of the robots, the point clouds collected by the laser radars often have differences in coverage, density distribution and geometric shape. The same target may exhibit position offset or structural feature difference in different sampling results, which brings challenges to the fusion of point cloud data and the modeling of the environment. To achieve efficient collaboration of the dual-robot perception system, the spatial alignment and feature integration problems between multi-source point clouds need to be solved, so as to improve the accuracy and robustness of the overall perception. SUMMARY

[0007] To solve the above technical problems, the application provides a dual-robot collaborative three-dimensional target recognition method and system based on mutual information estimation feature unwinding, which aims to optimize the technical problems such as limited perception coverage, spatial differences in point cloud collection results and difficulties in multi-source data fusion in the process of three-dimensional target recognition of robots in large-scale aircraft manufacturing scenes.

[0008] The technical solution adopted by the application to solve its technical problems is:

[0009] The dual-robot collaborative three-dimensional target recognition method based on mutual information estimation feature unwinding comprises the following steps:

[0010] S100: Construct a dual-robot collaborative three-dimensional perception platform, use the laser radars mounted on the dual robots to scan and collect three-dimensional point cloud data of the large-scale aircraft manufacturing scene according to the task, simultaneously obtain the real-time pose information of the dual robots, unify the point cloud data collected by the dual robots to the same world coordinate system based on the real-time pose information, label the point cloud data after unification of the coordinate system, and divide it into a training set, a validation set and a test set;

[0011] S200: Construct a point cloud collaborative three-dimensional target recognition model, which comprises a multi-source point cloud feature extraction module, a multi-agent feature unwinding and fusion module based on mutual information estimation, and a collaborative three-dimensional target classification and regression module;

[0012] S300: Input the training set into the multi-source point cloud feature extraction module in the model to extract the agent features of the point cloud data of the dual robots respectively;

[0013] S400: Input the agent features of the dual robots into the multi-agent feature unwinding and fusion module based on mutual information estimation in the model, and map them to a shared feature space and two unique feature spaces in parallel to obtain shared features and unique features; based on mutual information estimation, calculate the lower bound of mutual information between the shared features and the upper bound of mutual information between the unique features; fuse the unwound shared features and unique features to generate agent fusion features for the recognition task;

[0014] S500: parallel input the agent fusion features into a collaborative three-dimensional target classification and regression module in the model, output the three-dimensional target classification and regression results; update the model parameters through the total loss function until the model converges;

[0015] S600: verify and test the trained model using the verification set and the test set, and deploy the final model to the edge computing device of the robot to realize collaborative three-dimensional target identification of targets in a large range of scenes.

[0016] Preferably, S100 comprises:

[0017] S110: install the laser radar at the reserved position of the dual robot and connect it with the edge computing device inside the robot;

[0018] S120: use the laser radar carried by the dual robot to collect systematic point cloud data in the processing, assembly or transfer docking work area in the aircraft manufacturing scene, and obtain the point cloud data of the dual robot;

[0019] S130: acquire the pose information of the dual robot in real time through the motion capture system, map the point cloud data of the dual robot to the same world coordinate system based on the real-time pose information, and generate global collaborative point cloud data;

[0020] S140: label the global collaborative point cloud data with three-dimensional detection boxes and categories to form a point cloud labeled data set ; divide the point cloud labeled data set , randomly take out 70% of the data as the training set , randomly take out an amount of data equivalent to 15% of the remaining data as the verification set , and the remaining part as the test set .

[0021] Preferably, S300 comprises:

[0022] S310: input the point cloud with the laser radar as the coordinate axis center, the forward direction of the robot as the x-axis direction, the left direction of the robot as the y-axis direction, and the upward direction of the robot as the z-axis direction, mathematically represent the point cloud, and represent the point cloud data collected by each robot as an N x 4 size vector, the content of which is , wherein N is the number of point clouds, and the vector content includes the three-dimensional coordinates and the reflection intensity of the point cloud;

[0023] S320: use different PointPillars point cloud feature extraction networks to extract the point cloud data , feature extraction is performed to obtain corresponding agent features 、 .

[0024] Preferably, S400 comprises:

[0025] S410: mapping the agent features 、 to a shared feature space through a shared space mapping layer to obtain shared features 、 ;

[0026] S420: calculating a lower bound estimate of mutual information between the shared features 、 through a lower bound estimate of mutual information method;

[0027] S430: mapping the agent features 、 to different unique feature spaces through a unique space mapping layer to obtain unique features 、 ;

[0028] S440: calculating an upper bound estimate of mutual information between the unique features 、 through an upper bound estimate of mutual information method;

[0029] S450: constructing an untangling loss function based on the lower and upper bound estimates of mutual information to realize the untangling of the shared features and the unique features;

[0030] S460: inputting the shared features 、 and the unique features 、 of the dual robots into a feature fusion module based on an attention mechanism, performing feature interaction and selection through cross-attention and multi-head self-attention, and then performing splicing, pooling operation and three-dimensional convolution fusion to generate agent fusion features for collaborative recognition .

[0031] Preferably, S410 specifically comprises:

[0032] ;

[0033] ;

[0034] wherein is a multi-agent shared space mapping layer, is a fully connected layer, is an activation function, is layer normalization, , is a feature of the agent, , is a shared feature;

[0035] S420 is specifically:

[0036] ;

[0037] wherein is mutual information between is a lower bound estimate of mutual information, is a lower bound estimate of mutual information between is the number of sample pairs involved in one forward pass for estimation, is a trainable discriminative function, is a random rearrangement sample of ;

[0038] S430 is specifically:

[0039] ;

[0040] ;

[0041] wherein , is a multi-agent unique feature space mapping layer, , is a unique feature.

[0042] Preferably, S440 is specifically:

[0043] ;

[0044] ;

[0045] ;

[0046] ;

[0047] wherein is mutual information between is an upper bound estimate of mutual information, is an upper bound estimate of mutual information between is the number of sample pairs involved in one forward pass for estimation, , are negative square error terms of positive sample pairs and negative sample pairs, respectively, ​​​​is the dimension of the feature vector, is the mean vector of the conditional distribution, is the variance vector of the conditional distribution, is the mean vector of the conditional distribution, is the variance vector of the conditional distribution, is the mean vector of the conditional distribution, is the variance vector of the conditional distribution, is the target feature vector, is the target feature vector, is the k-th component of the mean vector of the conditional distribution in feature dimension d, is the k-th component of the target feature vector in feature dimension d;

[0048] S450 is specifically:

[0049] ;

[0050] wherein is the disentanglement loss function, is the upper bound of mutual information between , is the upper bound of mutual information between , is the upper bound of mutual information between , is the lower bound of mutual information between , and distinguishes it from the shared feature , and distinguishes it from the shared feature , and distinguishes it from the shared feature ;

[0051] S460 is specifically:

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] wherein, is the Cross-attention, which refers to the cross-attention mechanism, using to interact the shared feature of the dual robot at the feature level to obtain the unique shared feature , is the multi-head self-attention mechanism, which uses the unique shared feature with unique features intermediate features are obtained by feature selection , is a concatenation operation, with are max-pooling and average-pooling operations respectively, is the concatenated feature, is a feature fusion layer, is a three-dimensional convolution layer, is an activation function, is the agent fusion feature.

[0057] Preferably, S500 comprises:

[0058] S510: input the agent fusion feature into a classification convolution layer to predict target classification scores, and input the agent fusion feature into a regression convolution layer to predict three-dimensional bounding box regression parameters, specifically:

[0059] ;

[0060] ;

[0061] wherein, is used to predict classification scores and consists of a 1x1 two-dimensional convolution layer, is used to regress three-dimensional bounding boxes and consists of a 1x1 two-dimensional convolution layer;

[0062] S520: update the model parameters by minimizing the total loss function containing classification loss, regression loss and feature unwinding loss, specifically:

[0063] ;

[0064] ;

[0065] ;

[0066] wherein, , , is the bounding box classification loss, is the three-dimensional bounding box regression loss, is the unwinding loss function, is the total loss, are the predicted and ground truth values of classification respectively, are the predicted and ground truth values of regression respectively;

[0067] ​​S530: repeat S520 to update the point cloud three-dimensional detection model parameters through the loss function until the model converges.

[0068] Preferably, S600 comprises:

[0069] S610: after the completion of the training batch number, the trained three-dimensional detection model is evaluated, and the validation set is used for model validation, and the point cloud collaborative three-dimensional target recognition model with the best effect is selected The test set is input into the optimal model for testing to obtain the test result to evaluate the generalization performance;

[0070] S620: the optimal model is deployed to the NVIDIA Jetson Orin edge computing platform of the robot to realize collaborative three-dimensional target recognition of multiple types of targets in a large range of scenes.

[0071] The mutual information-based feature unwinding dual-robot collaborative three-dimensional target recognition system comprises:

[0072] A data preprocessing module is configured to construct a dual-robot collaborative three-dimensional perception platform, scan and collect three-dimensional point cloud data of an aircraft manufacturing large range scene by means of a laser radar carried on the dual robots, simultaneously acquire real-time pose information of the dual robots, unify the point cloud data collected by the dual robots to the same world coordinate system based on the real-time pose information, label the point cloud data after unification of the coordinate system, and divide the point cloud data into a training set, a validation set and a test set.

[0073] A point cloud collaborative three-dimensional target recognition model construction module is configured to construct a point cloud collaborative three-dimensional target recognition model, the model comprising a multi-source point cloud feature extraction module, a multi-agent feature unwinding and fusion module based on mutual information estimation, and a collaborative three-dimensional target classification and regression module.

[0074] The multi-source point cloud feature extraction module is configured to receive the training set and extract agent features of the point cloud data of the dual robots respectively.

[0075] The multi-agent feature unwinding and fusion module based on mutual information estimation is configured to receive the agent features of the dual robots, and map the agent features to a shared feature space and two unique feature spaces in parallel to obtain shared features and unique features; based on mutual information estimation, the lower bound of mutual information between the shared features and the upper bound of mutual information between the unique features are calculated respectively; the unwound shared features and unique features are fused to generate agent fusion features for the recognition task.

[0076] A cooperative three-dimensional target classification and regression module is configured to receive the agent fusion features in parallel and output the three-dimensional target classification and regression results; and the model parameters are updated through a total loss function until the model converges.

[0077] A three-dimensional target recognition module is configured to verify and test the trained model using a verification set and a test set, and deploy the final model to the edge computing device of the robot to realize cooperative three-dimensional target recognition of targets in a large-scale scene.

[0078] A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the method for cooperative three-dimensional target recognition of dual robots based on mutual information estimation of feature unwrapping when executing the computer program.

[0079] The present application collects three-dimensional point cloud data of the aircraft manufacturing site through the edge computing platform and the laser radar, and combines the real-time pose information of the robot to unify the coordinate system, constructs a global scene representation, and proposes a point cloud three-dimensional detection model based on mutual information estimation and feature unwrapping. The model acquires multi-level geometric and semantic features through a point cloud feature extraction module to ensure information integrity; through a feature unwrapping and fusion module, shared features are extracted and unique features are retained, mutual information constraints are used to reduce feature redundancy and enhance unwrapping effect, and then fusion operation is performed to form a high-dimensional representation with stronger discrimination; finally, a three-dimensional target detection module outputs target categories and detection boxes to realize accurate identification and positioning of large components, personnel and equipment. It can maintain high precision and robustness in complex manufacturing environments, effectively improve the perception ability, safety and overall efficiency of dual-robot cooperative operation. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 The flowchart of the method for cooperative three-dimensional target recognition of dual robots based on mutual information estimation of feature unwrapping in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to enable personnel in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the accompanying drawings.

[0082] In one embodiment, as shown in Figure 1 The method for cooperative three-dimensional target recognition of dual robots based on mutual information estimation of feature unwrapping, the method comprising the following steps:

[0083] S100: A dual-robot collaborative three-dimensional perception platform is constructed. Laser radars mounted on the dual robots are used to scan and collect three-dimensional point cloud data of a large-scale aircraft manufacturing scene according to the task. Real-time pose information of the dual robots is obtained. Based on the real-time pose information, the point cloud data collected by the dual robots is unified to the same world coordinate system. The unified coordinate system is labeled and divided into a training set, a validation set, and a test set.

[0084] S200: A point cloud collaborative three-dimensional target recognition model is constructed. The model includes a multi-source point cloud feature extraction module, a multi-agent feature unwinding and fusion module based on mutual information estimation, and a collaborative three-dimensional target classification and regression module.

[0085] S300: The training set is input into the multi-source point cloud feature extraction module in the model to extract the agent features of the dual-robot point cloud data.

[0086] S400: The agent features of the dual robots are input into the multi-agent feature unwinding and fusion module based on mutual information estimation in the model. They are mapped to a shared feature space and two unique feature spaces in parallel to obtain shared features and unique features. Based on mutual information estimation, the mutual information lower bound between shared features and the mutual information upper bound between unique features are calculated. The unwound shared features and unique features are fused to generate agent fusion features for the recognition task.

[0087] S500: The agent fusion features are input into the collaborative three-dimensional target classification and regression module in the model in parallel to output three-dimensional target classification and regression results. The model parameters are updated through the total loss function until the model converges.

[0088] S600: The trained model is verified and tested using the validation set and the test set. The final model is deployed to the edge computing device of the robot to realize collaborative three-dimensional target recognition of targets in a large-scale scene.

[0089] Specifically, the robot terminal, based on an edge computing platform, uses LiDAR to collect 3D point cloud data of a large-scale working environment at the aircraft manufacturing site. Combined with real-time robot pose information acquired by a motion capture system, the collected point cloud is transformed and uniformly mapped to form a global scene representation that can be used for subsequent processing. Based on this, a 3D point cloud detection model based on mutual information estimation and feature untangling is constructed. This model mainly includes a point cloud feature extraction module, a feature untangling and fusion module, and a 3D target detection module. The point cloud feature extraction module extracts multi-level geometric and semantic features from the original point cloud, ensuring the integrity and expressive power of the input information. The feature untangling and fusion module extracts shared features through a shared mapping structure, retains unique features by combining a difference mapping structure, and introduces a mutual information estimation method to constrain the correlation between shared and unique features, ensuring the effectiveness of information untangling. This module further utilizes feature concatenation, pooling, and convolution fusion operations to integrate features from different sources into a high-dimensional fusion representation, thereby improving the discriminativeness and robustness of the features. The 3D target detection module, based on the fused features, outputs the target category and the corresponding 3D detection box, thus achieving high-precision identification and positioning of large components, personnel, and equipment in aircraft manufacturing sites. Through the above design, this invention can maintain stable and reliable detection performance in complex manufacturing environments and improve the safety and efficiency of robot operations in practical applications.

[0090] In one embodiment, S100 includes:

[0091] S110: Install the lidar in the reserved position of the dual robots and connect it to the edge computing device inside the robot;

[0092] S120: Utilizes the lidar carried by dual robots to systematically collect point cloud data of the processing, assembly, or transfer docking operation areas in the aircraft manufacturing scene, and obtains the point cloud data of the dual robots.

[0093] S130: The pose information of the two robots is acquired in real time through the motion capture system. Based on the real-time pose information, the point cloud data of the two robots are uniformly mapped to the same world coordinate system to generate global collaborative point cloud data.

[0094] S140: Perform 3D bounding box annotation and category annotation on the global collaborative point cloud data to form a point cloud annotation dataset. ; for point cloud labeled datasets The dataset is divided into two parts, and 70% of the data is randomly selected as the training set. Randomly select from the remaining data equivalent to 15% of the data was used as the validation set. The remaining part is used as the test set. .

[0095] Specifically, first, the mechanical structure and electrical system of the robot are assembled and debugged, and the robot numbers are sequentially , ; a laser radar installation space is reserved in the front slot position of the robot. Then, the Ouster laser radar is installed in the reserved position and connected with the NVIDIA Jetson Orin edge computing device inside the robot. The Ouster laser radar moves synchronously with the robot to realize laser scanning, data transmission, and inference processing of the point cloud collaborative three-dimensional recognition model.

[0096] Two robots , respectively carry Ouster laser radars to systematically collect point cloud data for aircraft manufacturing large-scale scenes according to processing, assembly, transfer docking, and other work tasks, and obtain point cloud data , ; at the same time, the pose information of each robot is obtained in real time through the motion capture system to record the spatial position and motion state of the robot during the collection process. The point cloud data and real-time pose information are used to construct a collaborative three-dimensional point cloud detection dataset for the two robots, and the point cloud data of each robot is mapped to the same coordinate system to facilitate data integration and subsequent processing.

[0097] The point cloud data collected is manually annotated using the point cloud annotation software SuperAnnotate. Specifically, for the point cloud data , , first, based on the real-time pose information of the robot obtained by the motion capture system, the coordinate system transformation operation is performed on , to map them to the same world coordinate system, thereby generating global collaborative point cloud data , containing point cloud information. Then, the global collaborative point cloud data is annotated, including three-dimensional bounding box annotation and class annotation of target objects in the scene to ensure accurate identification of different types of targets in three-dimensional space. Through the above annotation process, a structured point cloud annotation dataset can be formed, denoted as , which can be used as a high-quality basic data source for subsequent three-dimensional target detection model training and performance evaluation.

[0098] The annotated dataset is divided, and 70% of the data in the annotated dataset is randomly taken out for training as the training set, denoted as , and the annotated dataset Randomly select from the remaining data equivalent to 15% of the data is used for validation, as the validation set, denoted as . , label the dataset The remaining part is used as the test set, which is... This is used to test the generalization ability of the model.

[0099] In one embodiment, S300 includes:

[0100] S310: The input point cloud uses the LiDAR as the coordinate axis center, the robot's forward direction as the x-axis, the robot's leftward direction as the y-axis, and the robot's upward direction as the z-axis. The point cloud is mathematically represented, with each robot's collected point cloud data represented as an N×4 vector, containing... Where N is the number of point clouds, and the vector content includes the three-dimensional coordinates of the point clouds. and reflection intensity ;

[0101] S320: Different PointPillars point cloud feature extraction networks are used to process the point cloud data of the two robots. , Feature extraction is performed to obtain the corresponding agent features. , .

[0102] In one embodiment, S400 includes:

[0103] S410: Agent features are mapped through a shared spatial mapping layer. , Mapping to a shared feature space yields shared features. , ;

[0104] S420: Calculate shared features using the mutual information lower bound estimation method. , Lower bound estimate of mutual information between them;

[0105] S430: Transmits agent features through a unique spatial mapping layer , Mapping to different unique feature spaces yields unique features. , ;

[0106] S440: Calculate unique features using the mutual information upper bound estimation method. , The upper bound estimate of mutual information between them;

[0107] S450: constructing an unwinding loss function based on the lower and upper bound estimators of mutual information to realize the unwinding of shared features and unique features;

[0108] S460: inputting the shared features and unique features of the dual robots into the feature fusion module based on the attention mechanism, and generating agent fusion features for collaborative recognition through cross-attention and multi-head self-attention, splicing, pooling operation and three-dimensional convolution fusion 、 、 . .

[0109] In one embodiment, S410 is specifically:

[0110] ;

[0111] ;

[0112] wherein is a multi-agent shared space mapping layer, is a fully connected layer, is an activation function, is layer normalization, 、 is an agent feature, 、 is a shared feature;

[0113] S420 is specifically:

[0114] ;

[0115] wherein is the mutual information between and , is a lower bound estimator of mutual information, is a lower bound estimator of mutual information between , is the number of sample pairs participating in estimation for one forward calculation, is a trainable discriminant function, is a randomly rearranged sample of ;

[0116] S430 is specifically:

[0117] ;

[0118] ;

[0119] wherein 、 ​is a unique feature space of multi-agent, 、 is a unique feature.

[0120] In one embodiment, S440 is specifically:

[0121] ;

[0122] ;

[0123] ;

[0124] ;

[0125] wherein is the mutual information between and , is a mutual information upper bound estimation method, is a mutual information upper bound estimation quantity, is the number of sample pairs involved in estimation for one forward calculation, 、 are negative square error terms of positive sample pairs and negative sample pairs respectively, is the dimension of the feature vector, and are the mean vector and variance vector of the conditional distribution respectively, and are trainable neural networks for estimating and , is a target feature vector, and are the kth component of the mean vector and variance vector of the conditional distribution in the feature dimension d respectively, is the kth component of the target feature vector in the feature dimension d;

[0126] S450 is specifically:

[0127] ;

[0128] wherein is an unscrambling loss function, is a mutual information upper bound estimation quantity between , is a mutual information upper bound estimation quantity between , is a mutual information upper bound estimation quantity between , is a mutual information lower bound estimation quantity between the first three terms of the loss function are used for unscrambling unique features With and share features With distinction, the last term of the loss function is used to disentangle shared features With ;

[0129] S460 is specifically:

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] wherein, Cross-attention refers to a cross-attention mechanism, which uses shared features of the dual robots to interact at the feature level to obtain unique shared features , Multi-head self-attention mechanism, the unique shared features and unique features feature selection to obtain intermediate features , is a concatenation operation, and are max-pooling and average-pooling operations respectively, is the concatenated feature, is a feature fusion layer, is a three-dimensional convolution layer, is an activation function, is the agent fusion feature.

[0135] In one embodiment, S500 includes:

[0136] S510: input the agent fusion feature into a classification convolutional layer to predict the target classification score, and input the agent fusion feature into a regression convolutional layer to predict the three-dimensional detection frame regression parameter, specifically:

[0137] ;

[0138] ;

[0139] wherein, is used to predict the classification score, which is composed of a 1x1 two-dimensional convolutional layer, The regression for the three-dimensional bounding box is composed of a 1x1 two-dimensional convolutional layer;

[0140] S520: updating the model parameters by minimizing the total loss function including the classification loss, the regression loss and the feature unwinding loss, specifically:

[0141] ;

[0142] ;

[0143] ;

[0144] wherein, , , is the predicted bounding box classification loss, is the three-dimensional bounding box regression loss, is the unwinding loss function, is the total loss, are the predicted and ground truth values of the classification respectively, are the predicted and ground truth values of the regression respectively;

[0145] S530: repeating S520 to update the point cloud three-dimensional detection model parameters through the loss function until the model converges.

[0146] In one embodiment, S600 comprises:

[0147] S610: after completing the number of training batches, evaluating the trained three-dimensional detection model to obtain the model validation set, selecting the point cloud collaborative three-dimensional target recognition model with the best effect , inputting the test set into the optimal model for testing to obtain the test result to evaluate the generalization performance;

[0148] S620: deploying the optimal model to the NVIDIA Jetson Orin edge computing platform of the robot to realize collaborative three-dimensional target recognition of multiple targets such as aircraft large components, workers and working robots in a large range of scenes.

[0149] The application provides a double-robot cooperative three-dimensional target recognition method based on mutual information estimation feature unwinding, which is suitable for large-scale complex industrial scenes such as aircraft manufacturing, can perform high-precision three-dimensional detection on aircraft large components, workers and operation equipment, provides complete spatial structure and position information for a double-robot system, and effectively improves operation efficiency and safety. In view of the perspective difference and feature redundancy problems existing in multi-source point cloud data, the application constructs a feature unwinding and fusion model based on mutual information constraint. The model extracts shared features through a shared mapping structure, retains unique features in combination with a difference mapping structure, and then uses a mutual information estimation method to constrain the correlation between the shared features and the unique features, so as to generate a fusion feature representation that covers global semantics and retains local details, thereby enhancing the adaptability and robustness of three-dimensional detection in a complex structure environment. The method and system provided by the application can support multi-double-robot cooperative target detection in a three-dimensional environment, and can obtain the spatial position and three-dimensional structure information of key targets in real time, thereby assisting the robot to perform obstacle identification, obstacle avoidance, path planning and motion control and other tasks, significantly improving the operation efficiency of the manufacturing process such as assembly and transfer, and guaranteeing the safety and stability of production. In addition, the method is not only suitable for aircraft manufacturing tasks, but also has the potential to be extended to other industrial manufacturing scenes, and its high efficiency, accuracy and good expansibility will provide important technical support for the development of intelligent manufacturing, and promote the industrial manufacturing to the intelligent, automated and high-quality direction.

[0150] In one embodiment, a double-robot cooperative three-dimensional target recognition system based on mutual information estimation feature unwinding is also provided, comprising:

[0151] A data preprocessing module is configured to construct a double-robot cooperative three-dimensional perception platform, use the laser radar carried on the double-robot to scan and collect three-dimensional point cloud data of a large-scale aircraft manufacturing scene, obtain real-time pose information of the double-robot, unify the point cloud data collected by the double-robot to the same world coordinate system based on the real-time pose information, label the point cloud data after unification of the coordinate system, and divide the point cloud data into a training set, a verification set and a test set;

[0152] A point cloud cooperative three-dimensional target recognition model construction module is configured to construct a point cloud cooperative three-dimensional target recognition model, the model comprising a multi-source point cloud feature extraction module, a multi-agent feature unwinding and fusion module based on mutual information estimation, and a cooperative three-dimensional target classification and regression module.

[0153] The multi-source point cloud feature extraction module is configured to receive the training set and extract agent features of the point cloud data of the double-robot respectively.

[0154] The multi-agent feature disentanglement and fusion module based on mutual information estimation is configured to receive agent features of the dual robots, map the agent features to a shared feature space and two unique feature spaces in parallel, and obtain shared features and unique features; based on mutual information estimation, lower bounds of mutual information between the shared features and upper bounds of mutual information between the unique features are calculated; the disentangled shared features and unique features are fused to generate agent fusion features for an identification task;

[0155] The collaborative three-dimensional target classification and regression module is configured to receive the agent fusion features in parallel and output three-dimensional target classification and regression results; and model parameters are updated through a total loss function until the model converges.

[0156] The three-dimensional target identification module is configured to verify and test the trained model by using a verification set and a test set, and deploy a final model to an edge computing device of the robot to implement collaborative three-dimensional target identification of targets in a large-scale scene.

[0157] Specific limitations of the dual-robot collaborative three-dimensional target identification system based on feature disentanglement based on mutual information estimation can be seen in the limitations of the dual-robot collaborative three-dimensional target identification method based on feature disentanglement based on mutual information estimation in the foregoing, which will not be repeated here. Each module in the dual-robot collaborative three-dimensional target identification system based on feature disentanglement based on mutual information estimation can be realized by software, hardware, or a combination thereof, in whole or in part. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each module.

[0158] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the dual-robot collaborative three-dimensional target identification method based on feature disentanglement based on mutual information estimation when executing the computer program.

[0159] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0160] The foregoing has provided a detailed description of a dual-robot cooperative 3D target recognition method and system based on mutual information estimation feature untangling provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention, and the descriptions of the embodiments above are merely for the purpose of helping to understand the core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.

Claims

1. A dual-robot cooperative 3D target recognition method based on mutual information estimation and feature untangling, characterized in that, The method includes the following steps: S100: Construct a dual-robot collaborative 3D perception platform. Utilize the lidar mounted on the dual robots to scan and collect 3D point cloud data of a large-scale scene in aircraft manufacturing according to the task. At the same time, obtain the real-time pose information of the dual robots. Based on the real-time pose information, unify the point cloud data collected by the dual robots to the same world coordinate system. Label the point cloud data after unifying the coordinate system and divide it into training set, validation set and test set. S200: Construct a point cloud collaborative 3D target recognition model. The model includes a multi-source point cloud feature extraction module, a multi-agent feature unwrapping and fusion module based on mutual information estimation, and a collaborative 3D target classification and regression module. S300: Input the training set into the multi-source point cloud feature extraction module in the model to extract the agent features of the dual robot point cloud data respectively; S400: The agent features of the two robots are input into the multi-agent feature untangling and fusion module based on mutual information estimation in the model, and mapped in parallel to a shared feature space and two unique feature spaces to obtain shared features and unique features; based on mutual information estimation, the lower bound of mutual information between shared features and the upper bound of mutual information between unique features are calculated respectively; the untangled shared features and unique features are fused to generate agent fusion features for recognition tasks; The S400 includes: S410: Agent features are mapped through a shared spatial mapping layer. , Mapping to a shared feature space yields shared features. , ; S420: Calculate shared features using the mutual information lower bound estimation method. , Lower bound estimate of mutual information between them; S430: Transmits agent features through a unique spatial mapping layer , Mapping to different unique feature spaces yields unique features. , ; S440: Calculate unique features using the mutual information upper bound estimation method. , The upper bound estimate of mutual information between them; S450: Construct an untangling loss function based on the lower and upper bound estimates of mutual information to achieve untangling of shared and unique features; S460: Sharing features between two robots , With unique characteristics , The input is an attention-based feature fusion module, which performs feature interaction and selection through cross-attention and multi-head self-attention. The features are then fused through concatenation, pooling, and 3D convolution to generate agent-based fusion features for collaborative recognition. ; S410 specifically refers to: ； ； in For multi-agent shared space mapping layer, It is a fully connected layer. For activation function, For layer normalization, , Features of the intelligent agent , For shared features; The S420 specifically refers to: ； in for and Mutual information between them As a lower bound estimator for mutual information, for Lower bound estimator of mutual information between them The number of sample pairs involved in a single forward computation for estimation. For a trainable discriminant function, for Randomly rearranged samples; The S430 is specifically: ； ； in , It is a multi-agent unique feature space mapping layer. , As a unique feature; The S440 specifically refers to: ； ； ； ； in for and Mutual information between them For the upper bound of mutual information, for Upper bound estimate of inter-information information The number of sample pairs involved in a single forward computation for estimation. , These are the negative squared error terms for positive and negative sample pairs, respectively. The dimension of the feature vector. and These are the mean vector and variance vector of the conditional distribution, respectively. and Estimate respectively and Trainable neural networks, For the target feature vector, and Let be the k-th components of the mean and variance vectors of the conditional distribution along the feature dimension d, respectively. The k-th component of the target feature vector in feature dimension d; S500: Inputs the fused features of the agent into the collaborative 3D target classification and regression module in the model in parallel, and outputs the 3D target classification and regression results; updates the model parameters through the total loss function until the model converges; S600: The trained model is validated and tested using validation and test sets, and the final model is deployed to the robot's edge computing device to achieve collaborative 3D target recognition of targets in a wide range of scenes.

2. The method according to claim 1, characterized in that, S100 includes: S110: Install the lidar in the reserved position of the dual robots and connect it to the edge computing device inside the robot; S120: Utilizes the lidar carried by the dual robots to systematically collect point cloud data of the processing, assembly, or transfer docking operation areas in the aircraft manufacturing scene, and obtains the point cloud data of the dual robots. S130: The pose information of the two robots is acquired in real time through the motion capture system. Based on the real-time pose information, the point cloud data of the two robots are uniformly mapped to the same world coordinate system to generate global collaborative point cloud data. S140: Perform 3D bounding box annotation and category annotation on the global collaborative point cloud data to form a point cloud annotation dataset. ; for point cloud labeled datasets The dataset is divided into two parts, and 70% of the data is randomly selected as the training set. Randomly select from the remaining data equivalent to 15% of the data was used as the validation set. The remaining part is used as the test set. .

3. The method according to claim 2, characterized in that, The S300 includes: S310: The input point cloud uses the LiDAR as the coordinate axis center, the robot's forward direction as the x-axis, the robot's leftward direction as the y-axis, and the robot's upward direction as the z-axis. The point cloud is mathematically represented, with each robot's collected point cloud data represented as an N×4 vector, containing... Where N is the number of point clouds, and the vector content includes the three-dimensional coordinates of the point clouds. and reflection intensity ; S320: Different PointPillars point cloud feature extraction networks are used to process the point cloud data of the two robots. , Feature extraction is performed to obtain the corresponding agent features. , .

4. The method according to claim 3, characterized in that, The S450 is specifically: ； in For the untangling loss function, for Upper bound estimate of mutual information between them for Upper bound estimate of mutual information between them for Upper bound estimate of mutual information between them for The lower bound estimate of mutual information between them; the first three terms of the loss function are used to untangle unique features. and And its connection with shared features and The last term of the loss function is used to distinguish between shared features. and ; S460 specifically refers to: ； ； ； ； in, Cross-attention refers to a cross-attention mechanism that uses... Shared features of the two robots Perform feature-level interactions to obtain unique shared features. , For multi-head self-attention mechanisms, the unique shared feature is... With unique characteristics Feature selection is performed to obtain intermediate features. , It's a splicing operation. and These are max pooling and average pooling operations, respectively. Features after splicing It is a feature fusion layer. It is a three-dimensional convolutional layer. For activation function, This refers to the fusion characteristics of intelligent agents.

5. The method according to claim 4, characterized in that, The S500 includes: S510: Integrating Agent Features Input a classification convolutional layer to predict the target classification score, and fuse features of the agent. The regression convolutional layer is input to predict the regression parameters of the 3D detection box, specifically: ； ； in, It is used to predict classification scores and consists of a 1×1 two-dimensional convolutional layer. The regression for the 3D bounding box consists of a 1×1 2D convolutional layer; S520: The model parameters are updated by minimizing the total loss function, which includes classification loss, regression loss, and feature unwrapping loss. Specifically: ； ； ； in, , , For the classification loss of the predicted bounding box, For the regression loss of the 3D detection box, For the untangling loss function, For the total loss, These are the predicted values ​​for the classification and the actual ground values, respectively. These are the predicted values ​​from the regression and the actual ground values, respectively. S530: Repeat S520 to update the point cloud 3D detection model parameters using the loss function until the model converges.

6. The method according to claim 5, characterized in that, The S600 includes: S610: After completing the required number of training batches, evaluate the trained 3D detection model and... The validation set is used to validate the model, and the point cloud collaborative 3D target recognition model with the best performance is selected. Test set Input optimal model Conduct the test and obtain the test results. To evaluate generalization performance; S620: Optimal Model The NVIDIA Jetson Orin edge computing platform deployed on the robot enables collaborative 3D target recognition of multiple types of targets in a wide range of scenes.

7. A dual-robot cooperative 3D target recognition system based on mutual information estimation feature untangling, performing the method as described in any one of claims 1 to 6, characterized in that, include: The data preprocessing module is used to build a collaborative 3D perception platform for two robots. It uses the lidar mounted on the two robots to scan and collect 3D point cloud data of a large-scale scene in aircraft manufacturing. At the same time, it obtains the real-time pose information of the two robots. Based on the real-time pose information, it unifies the point cloud data collected by the two robots to the same world coordinate system. The point cloud data after unifying the coordinate system is labeled and divided into training set, validation set and test set. The point cloud collaborative 3D target recognition model construction module is used to construct a point cloud collaborative 3D target recognition model. The model includes a multi-source point cloud feature extraction module, a multi-agent feature unwrapping and fusion module based on mutual information estimation, and a collaborative 3D target classification and regression module. The multi-source point cloud feature extraction module is used to receive the training set and extract the agent features from the point cloud data of the two robots respectively. A multi-agent feature untangling and fusion module based on mutual information estimation is used to receive the agent features of two robots and map them in parallel to a shared feature space and two unique feature spaces to obtain shared features and unique features. Based on mutual information estimation, the lower bound of mutual information between shared features and the upper bound of mutual information between unique features are calculated respectively; the unwrapped shared features and unique features are fused to generate agent fusion features for recognition tasks. The collaborative 3D target classification and regression module is used to receive the fused features of the agent in parallel and output the 3D target classification and regression results; the model parameters are updated through the total loss function until the model converges. The 3D target recognition module is used to verify and test the trained model using validation and test sets, and deploy the final model to the robot's edge computing device to achieve collaborative 3D target recognition of targets in a wide range of scenes.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images