Multi-robot collaborative 3D target detection method based on visual state space model

Abstract

The present invention belongs to the field of collaborative three-dimensional perception of robots in large-scale manufacturing scenes, and specifically relates to a multi-robot collaborative three-dimensional target detection method based on a visual state space model, comprising: building a multi-robot laser radar three-dimensional perception platform to collect three-dimensional point cloud data in a large-scale scene of aircraft manufacturing and assembly; annotating the collected three-dimensional point cloud data to produce a three-dimensional point cloud detection data set, and dividing the three-dimensional point cloud detection data set; constructing a point cloud three-dimensional target detection model based on the visual state space model; substituting a training set into the point cloud three-dimensional target detection model for iterative training, and using a validation set for model verification to obtain an optimal point cloud three-dimensional target detection model; substituting a test set into the optimal point cloud three-dimensional target detection model for prediction to obtain a three-dimensional point cloud detection result, thereby realizing collaborative three-dimensional target detection by multiple robots in a large-scale scene.

Description

Technical Field

[0001] The present invention belongs to the field of robot collaborative three-dimensional perception in large-scale manufacturing scenarios, and specifically relates to a multi-robot collaborative three-dimensional target detection method based on a visual state space model. Background Art

[0002] In the field of aviation equipment manufacturing and assembly, especially for the manufacturing and assembly process of domestic large aircraft, high-precision and all-round quality control is the core element to ensure the comprehensive model and safe operation of the aircraft, which affects national defense security and national economic development. In the manufacturing and assembly process of large aircraft, it is key to ensure the high-precision assembly of large thin-walled components such as skin, wings, and fuselage. It requires real-time positioning of key points, holes, and surface features of local assembly parts, and monitoring of assembly force, overall deformation, inclination, and ground settlement. At present, the manufacturing and assembly of these complex parts mainly rely on manual labor, but with the development of robots and artificial intelligence technology, unmanned production and assembly have become a national strategic trend and an important demonstration of intelligent manufacturing. The deep integration of mobile robots and artificial intelligence technology can optimize the measurement, positioning, monitoring and other processes, reduce manpower and material input, and improve production efficiency and quality.

[0003] In the process of aircraft parts manufacturing and assembly, the realization of high-precision 3D target monitoring of mobile robots in a wide range of manufacturing scenes is the basis and key. The 3D target detection method based on deep learning, which has been widely used in recent years, still cannot meet the engineering requirements of high-precision 3D target detection of robots in large aircraft manufacturing and assembly scenes.

[0004] At present, LiDAR is mainly used as the spatial and position information sensor of mobile robots to perform point cloud scanning of large-scale manufacturing scenes, and extract point cloud features based on the deep learning algorithm framework to complete the three-dimensional target detection of mobile robots. However, there are certain limitations in its specific implementation process, such as: the perception field and detection range of a single robot are limited, making it difficult to perceive the entire aircraft manufacturing and assembly scene; point cloud information with a large amount of information and many types of targets will cause a huge computational burden on the neural network structure, or the point cloud information will be lost due to the acceleration of calculations through voxel processing; the target information of flat and slender aircraft skin components is easily reduced in the neural network as the network depth increases, which will cause the network to inaccurately locate the aircraft skin structure. Summary of the invention

[0005] The present invention provides a multi-robot collaborative 3D target detection method based on a visual state space model. By collecting 3D point cloud data on a constructed lidar 3D perception platform, a point cloud 3D target detection model based on the visual state space model is constructed to train, verify and test the collected data, so as to achieve high-precision 3D detection and high-speed reasoning of targets such as aircraft skin in a large-scale scene of multiple robots, quickly provide multiple robots with position and spatial information of aircraft skin, staff and operating robots, help multiple robots avoid obstacles, plan and control in a large-scale manufacturing scene, so that they can better complete operation and assembly tasks, thereby improving the efficiency and accuracy of robots in the manufacturing and assembly process, and promoting the transformation of domestic large aircraft manufacturing and assembly processes to intelligent and unmanned operations.

[0006] A multi-robot collaborative three-dimensional target detection method based on a visual state space model comprises the following steps:

[0007] Build a multi-robot LiDAR 3D perception platform to collect 3D point cloud data in a wide range of scenarios for aircraft manufacturing and assembly;

[0008] Annotate the collected 3D point cloud data, create a 3D point cloud detection data set, and divide the 3D point cloud detection data set into a training set, a validation set, and a test set;

[0009] Constructing a point cloud three-dimensional target detection model based on a visual state space model; the point cloud three-dimensional target detection model based on a visual state space model includes a point cloud encoding module, a feature extraction module, a feature fusion module, and a detection head;

[0010] Substitute the training set into the point cloud 3D object detection model based on the visual state space model for iterative training, and use the validation set to verify the model to obtain the optimal point cloud 3D object detection model based on the visual state space model;

[0011] The test set is substituted into the optimal point cloud 3D target detection model based on the visual state space model for prediction to obtain the 3D point cloud detection result. The 3D point cloud detection results of multiple robots are fused at the result level to realize collaborative 3D target detection of multiple robots in a large range of scenes.

[0012] By collecting three-dimensional point cloud data on the constructed lidar three-dimensional perception platform, a point cloud three-dimensional target detection model based on the visual state space model is constructed to train, verify and test the collected data, so as to achieve high-precision three-dimensional detection and high-speed reasoning of targets such as aircraft skin in large-scale multi-robot scenes, and quickly provide multiple robots with the position and spatial information of aircraft skin, staff and working robots, and help multiple robots to avoid obstacles, plan and control in large-scale manufacturing scenarios, so that they can better complete operation and assembly tasks, thereby improving the efficiency and accuracy of robots in the manufacturing and assembly process, and promoting the transformation of domestic large aircraft manufacturing and assembly processes to intelligent and unmanned operations.

[0013] Furthermore, the process of building a multi-robot lidar 3D perception platform to collect 3D point cloud data in a large range of scenes for aircraft manufacturing and assembly specifically includes the following steps:

[0014] Number multiple robots;

[0015] Install the repetitive scanning laser radar in front of the multi-robot and connect the repetitive scanning laser radar to the industrial computer inside the multi-robot;

[0016] Repeated scanning laser radars carried by multiple robots are used to collect 3D point cloud data for a large range of scenes in aircraft manufacturing and assembly, and the corresponding 3D point cloud data of multiple robots are obtained respectively.

[0017] Furthermore, the process of annotating the collected 3D point cloud data, preparing a 3D point cloud detection data set, and dividing the 3D point cloud detection data set into a training set, a validation set, and a test set specifically includes the following steps:

[0018] 50% of the 3D point cloud data are selected and annotated using the point cloud lightweight annotation software labelcloud to form an annotated data set; the remaining unannotated 3D point cloud data form an unannotated data set;

[0019] 70% of the labeled data set is divided into the training set, the remaining 30% of the labeled data set is divided into the validation set, and the unlabeled data set is used as the test set.

[0020] Furthermore, the process of the point cloud encoding module specifically includes the following steps:

[0021] With the repetitive scanning laser radar as the center of the coordinate axis, the robot's forward direction as the X-axis direction, the left direction as the Y-axis direction, and the upward direction as the Z-axis direction, the input point cloud is represented by a mathematical vector;

[0022] Select the coordinate range of the point cloud and divide the point cloud into multiple point cloud columns;

[0023] Taking the center of each point cloud column as the center point coordinate, calculate the relative coordinates of each point and the center of the point cloud column, and use it as a supplementary representation of the point cloud;

[0024] Combining the vector representation of the input point cloud and the vector representation of the supplemented point cloud to construct a vector representation of the point cloud;

[0025] Based on the vector representation of the constructed point cloud, a fully connected layer is used for encoding and output.

[0026] Furthermore, the process of the feature extraction module specifically includes the following steps:

[0027] The output result of the fully connected layer of the point cloud encoding module is feature mapped and output;

[0028] The output result of the feature map is input into the visual state space model for the first iterative training, and the output result of the first iterative training is input into the visual state space model again for the second iterative training and then output;

[0029] The output results of the feature map are input into the sparse convolutional neural network to extract local features and output after densification operation.

[0030] Furthermore, the process of inputting the output result of the feature map into the visual state space model for the first iterative training, and then inputting the output result into the visual state space model for the second iterative training and then outputting the result specifically includes the following steps:

[0031] The output result of the feature map is input into the first part of the visual state space model for initial iterative training and then output; the first part of the visual state space model refers to multi-directional feature modeling of low-level columnar point clouds based on a two-dimensional selective scanning model;

[0032] The output result of the initial iterative training of the first part of the visual state space model is input into the second part of the visual state space model for initial iterative training and then output; the second part of the visual state space model is based on a feedforward network and introduces feature extraction of a gated multilayer perceptron and a residual connection network;

[0033] The output result of the first iterative training of the second part of the visual state space model is inputted into the first part of the visual state space model for second iterative training and then outputted;

[0034] The output result of the second iterative training of the first part of the visual state space model is input into the second part of the visual state space model for second iterative training and then output.

[0035] The two-dimensional selective scanning model applied in the first part of the visual state space model can speed up training and reasoning, and increase the feature receptive field of the point cloud to obtain global features with contextual connections.

[0036] Furthermore, the process of the feature fusion module specifically includes the following steps:

[0037] The output results of the second iteration training in the visual state space model and the output results of the sparse convolutional neural network are combined and batch normalized.

[0038] The feature combination results are deeply fused with the batch normalization results to obtain the global and local features of a large-scale scene point cloud.

[0039] Furthermore, the process of the detection head specifically refers to classifying, regressing and predicting the direction of the acquired deep feature fusion results through different network layers.

[0040] By deeply fusing the global and local features of three-dimensional point cloud features in a large-scale scene, and inputting the features at different receptive field levels into the detection head for prediction, the network's attention to the target can be increased, thereby improving the accuracy of three-dimensional target detection.

[0041] Furthermore, the training set is substituted into the point cloud 3D object detection model based on the visual state space model for iterative training, and the validation set is used to verify the model. The process of obtaining the optimal point cloud 3D object detection model based on the visual state space model specifically includes the following steps:

[0042] The training set is input into the point cloud 3D object detection model, and the classification, regression and direction prediction scores of multiple robots are obtained through the visual state space model.

[0043] The loss function is used to update the model parameters of the point cloud 3D object detection model, and the training is iterated until the training batch is completed;

[0044] The trained point cloud 3D object detection model is evaluated and verified using a validation set to obtain the point cloud 3D detection model with the best effect.

[0045] Furthermore, the test set is substituted into the optimal point cloud 3D target detection model based on the visual state space model for prediction, and the 3D point cloud detection results are obtained. The 3D point cloud detection results of multiple robots are fused at the result level to realize the collaborative 3D target detection of multiple robots in a large-scale scene. The specific steps include the following:

[0046] Substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model to obtain the 3D point cloud detection result;

[0047] The obtained 3D point cloud detection results are converted from the lidar coordinate system to the virtual world coordinate system, and the 3D point cloud detection results of multiple robots are fused at the result level to realize collaborative 3D target detection of multiple robots in a large-scale scene.

[0048] By training the point cloud data obtained by multiple robots separately and finally converting them into the virtual world coordinate system for result-level fusion, multi-robot collaborative three-dimensional target detection is achieved in a large range of manufacturing scenarios.

[0049] The beneficial effects of the present invention are:

[0050] The present invention collects 3D point cloud data on a constructed lidar 3D perception platform, constructs a point cloud 3D target detection model based on the visual state space model, and trains, verifies and tests the collected data, so as to achieve high-precision 3D detection and high-speed reasoning of targets such as aircraft skin in a large-scale scene with multiple robots, quickly provide multiple robots with position and spatial information of aircraft skin, staff and operating robots, help multiple robots to avoid obstacles, plan and control in a large-scale manufacturing scene, so that they can better complete operation and assembly tasks, thereby improving the efficiency and accuracy of robots in the manufacturing and assembly process, and promoting the transformation of the manufacturing and assembly process of domestic large aircraft to intelligent and unmanned operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the present invention;

[0052] Figure 2 This is a schematic diagram of the full-scene lidar perception platform;

[0053] Figure 3 It is a schematic diagram of the structure of the point cloud 3D target detection model based on the visual state space model;

[0054] Figure 4 It is a schematic diagram of the structure of the visual state space model;

[0055] Figure 5 Schematic diagram of feature fusion structure;

[0056] Figure 6 A schematic diagram of the system structure. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0059] In addition, in the following description, specific details are provided to facilitate a thorough understanding of the examples, and a person of ordinary skill in the art can understand the specific meanings of the above terms in this application under specific circumstances.

[0060] Example 1

[0061] Figure 1 The method shown is a multi-robot collaborative 3D target detection method based on a visual state space model. By collecting 3D point cloud data on a built LiDAR 3D perception platform, a point cloud 3D target detection model based on a visual state space model is constructed to train, verify and test the collected data, so as to achieve high-precision 3D detection and high-speed reasoning of targets such as aircraft skins in a large-scale multi-robot scene, and quickly provide multi-robots with the position and spatial information of aircraft skins, staff and operating robots, so as to help multi-robots avoid obstacles, plan and control in a large-scale manufacturing scene, so that they can better complete operations and assembly tasks, thereby improving the efficiency and accuracy of robots in the manufacturing and assembly process, and promoting the transformation of domestic large aircraft manufacturing and assembly processes to intelligent and unmanned operations. The specific steps include the following:

[0062] S1: Build a multi-robot LiDAR 3D perception platform to collect 3D point cloud data in a wide range of aircraft manufacturing and assembly scenarios;

[0063] S11: Number multiple robots A i ;

[0064] Among them, A i represents the i-th robot, i = 1, 2, ..., M, i represents the number of robots, and M represents the total number of robots;

[0065] S12: Install the repetitive scanning laser radar in front of the multi-robot and connect the repetitive scanning laser radar with

[0066] Connect the industrial computers inside multiple robots;

[0067] S13: Use repeated scanning laser radars carried by multiple robots to collect 3D point cloud data for large-scale scenes of aircraft manufacturing and assembly, and obtain 3D point cloud data P corresponding to multiple robots i .

[0068] Figure 2 Schematic diagram of the multi-robot lidar 3D perception platform shown.

[0069] S2: annotate the collected 3D point cloud data, create a 3D point cloud detection data set, and divide the 3D point cloud detection data set into a training set, a validation set, and a test set;

[0070] The production process of the 3D point cloud detection dataset specifically includes:

[0071] S211: Select 50% of the 3D point cloud data and annotate them using the point cloud lightweight annotation software labelcloud to form the annotated data set D i_label ;

[0072] S212: The remaining unlabeled 3D point cloud data is used to form an unlabeled data set D i_unlabel .

[0073] Among them, the division of the 3D point cloud detection data set specifically includes:

[0074] S221: Divide 70% of the labeled data set into training set D i_train ;

[0075] S222: The remaining 30% of the labeled data set is divided into the validation set D i_val ;

[0076] S223: Use the unlabeled dataset as the test set D i_test .

[0077] S3: Construct a point cloud 3D target detection model based on the visual state space model; the point cloud 3D target detection model based on the visual state space model includes a point cloud encoding module, a feature extraction module, a feature fusion module, and a detection head;

[0078] Figure 3 The figure shows a schematic diagram of the structure of the point cloud three-dimensional target detection model based on the visual state space model.

[0079] Specifically, the process of the point cloud encoding module includes the following steps:

[0080] S311: With the repetitive scanning laser radar as the coordinate axis center, the robot's forward direction as the X-axis direction, the left direction as the Y-axis direction, and the upward direction as the Z-axis direction, the input point cloud is represented by a mathematical vector (x, y, z, r);

[0081] Among them, (x, y, z) represents the three-dimensional coordinates of the point cloud data, r represents the reflection intensity of the point cloud data; the mathematical vector representation size of the point cloud is N×4, and N represents the number of point clouds.

[0082] S312: Selecting a coordinate range of the point cloud and dividing the point cloud into a plurality of point cloud columns;

[0083] The coordinate range of the point cloud on the XYZ coordinate axis is specified as [(x d , x u ), (y d ,y u ), (z d , z u )], and divide the point cloud into columns based on the coordinate range, so that the specification of each point cloud column is (Δ x , Δ y , Δ z ).

[0084] S313: Taking the center of each point cloud column as the center point coordinate, calculate the relative coordinates of each point and the center of the point cloud column, and use them as the supplementary representation of the point cloud (x relative ,y relative , z relative );

[0085] The difference method is used to calculate the relative coordinates of each point and the center of the point cloud column.

[0086] S314: Combine the vector representation of the input point cloud and the vector representation of the supplementary point cloud to construct a vector representation of the point cloud (x, y, z, r, x relative ,y relative , z relative );

[0087] Among them, the vector representation size of the point cloud is N×7.

[0088] S315: Based on the vector representation of the constructed point cloud, a fully connected layer is used to encode and output F1.

[0089] Specifically, the process of the feature extraction module includes the following steps:

[0090] S321: Perform feature mapping on the output result F1 of the fully connected layer of the point cloud encoding module and output F2;

[0091] Among them, the expression for feature mapping is:

[0092] F2=subm(spconv(subm(spconv(Linear(subm(F1))))));

[0093] In the formula, F1 represents the output of the fully connected layer; subm() represents the submanifold sparse convolution layer; spconv() represents the sparse convolution layer. After the output of the fully connected layer passes through three submanifold sparse convolution layers and two sparse convolution layers, the output feature map F2 is output after the densification operation. Its size is (B, C, H, W), where B represents the number of batches, C represents the number of channels, H represents the length of the feature map, and W represents the width of the feature map.

[0094] S322: Input the output result F2 of the feature map into the visual state space model for the first iterative training, and then input the output result of the first iterative training into the visual state space model for the second iterative training and output F vssm ;

[0095] S3221: Input the output result F2 of the feature map into the first part of the visual state space model for initial

[0096] Output after iterative training;

[0097] S3222: Input the output result of the initial iterative training of the first part of the visual state space model into the second part of the visual state space model for initial iterative training, and output F vssm1 ;

[0098] S3223: Output result F of the first iteration training of the second part of the visual state space model vssm1 Re-enter

[0099] Output after the second iteration training to the first part of the visual state space model;

[0100] S3224: Input the output result of the second iterative training of the first part of the visual state space model into the second part of the visual state space model for second iterative training, and output F vssm .

[0101] In this embodiment, the first part of the visual state space model refers to the feature modeling of the low-level columnar point cloud in four directions based on the two-dimensional selective scanning model, that is, the SS2D model; its expression is:

[0102] F ss2d =Linear(SS2D(GeLU(DWConv(Linear(LN(F0))))));

[0103] Fvssm_1 =F0+F ss2d ;

[0104] In the formula, F ss2d Represents the output of the two-dimensional selective scanning model; Linear() represents the fully connected layer; LN(() represents layer normalization; SS2D() represents the two-dimensional selective scanning feature model; GeLU() represents the nonlinear activation function; DWConv() represents the depth separable convolution; F vssm_1 represents the output result of the first part of the first iteration training or the second iteration training; F0 represents the input of the visual state space model, which is specifically the input in the first iteration training process, that is, the output of the feature map F2, or the input in the second iteration training process, that is, the output F of the first iteration training process vssm1 ;

[0105] In this embodiment, the second part of the visual state space model is based on a feed-forward network, i.e., a FFN network.

[0106] Feature extraction of entry-level multilayer perceptron GMLP and residual connection network; its expression is:

[0107] F x =Linear(LN(F vssm_1 ));

[0108] F z =Linear(LN(F vssm_1 ));

[0109] F′0=Linear(GeLU(F x +DWConv(F x ))*F z )+F vssm_1 ;

[0110] In the formula, F x 、F z They represent the intermediate quantities in the gated multilayer perceptron process; F′0 represents the output result F in the first iterative training process of the second part vssm1 , or the output result of the second iteration training F vssm .

[0111] Figure 4 Shown is a schematic diagram of the structure of the visual state space model.

[0112] S323: Input the output result F2 of the feature map into the sparse convolutional neural network, extract the local features, and output F after densification operation cnn .

[0113] Among them, the expression for calculating the output of the sparse convolutional neural network is:

[0114] F cnn = subm(subm(spconv(F2)));

[0115] In the formula, F cnn The output of the sparse convolutional neural network is represented by subm(), and spconv() represents a sparse convolutional layer. After the output of the feature map F2 passes through two submanifold sparse convolutional layers and one sparse convolutional layer, the densification operation output F cnn .

[0116] Specifically, the process of the feature fusion module includes the following steps:

[0117] S331: The output result F of the second iteration training in the visual state space model vssm And the output of the sparse convolutional neural network F cnn Combine features and perform batch normalization;

[0118] Among them, the expression of feature combination is:

[0119] F c =F vssm +F cnn ;

[0120] In the formula, F c Represents the output result F of the second iterative training in the visual state space model vssm And the output of the sparse convolutional neural network F cnn feature combination.

[0121] Among them, the expression of batch normalization processing is:

[0122] F b =BN(conv2d(BN(conv2d(F c ))));

[0123] In the formula, F b Represents the result of batch normalization processing; BN() represents batch normalization processing; conv2d() represents a two-dimensional convolutional layer.

[0124] S332: Combine the features with the result F c And the batch normalization result F b Perform deep feature fusion to obtain global and local features of large-scale scene point clouds.

[0125] Among them, the expression of deep feature fusion is:

[0126] F f=block(concat(block(F b ), deconv(F c )));

[0127] In the formula, F f It represents the output result of deep feature fusion, which has contextual global features and detailed local features; block() represents a network composed of 6 two-dimensional convolutional layers; concat() represents the feature fusion operation, that is, the concatenate operation, which is used for tensor dimension splicing; deconv() represents the transposed convolution operation, which is used for upsampling.

[0128] Figure 5 Shown is a structural diagram of the feature fusion module.

[0129] Specifically, the process of the detection head refers to the fusion of the acquired deep features F f Classification, regression and direction prediction are performed respectively through different network layers, and the global and local features of the three-dimensional point cloud features in a large-scale scene are deeply integrated. Then, the features at different receptive field levels are input into the detection head for prediction, which can improve the network's attention to the target and improve the accuracy of three-dimensional target detection.

[0130] S341: Deep feature fusion result F f Make classification score predictions;

[0131] The expression for classification score prediction is:

[0132] cls pre =conv_cls(F f );

[0133] Where cls pre Represents the classification score prediction result; conv_cls() represents the predicted classification score function, which is composed of a 1×1 two-dimensional convolution layer;

[0134] S342: Deep feature fusion result F f Perform regression prediction of 3D detection box;

[0135] The expression for regression prediction of the 3D detection box is:

[0136] box pre =conv_box(F f );

[0137] In the formula, box pre Represents the regression prediction result of the three-dimensional detection box; conv_box() represents the regression prediction function of the three-dimensional detection box, which is composed of a 1×1 two-dimensional convolution layer;

[0138] S343: Deep feature fusion result F f Predict the direction of the 3D detection box;

[0139] The expression for predicting the direction of the 3D detection box is:

[0140] dir pre =conv_dir(F f );

[0141] In the formula, dir pre Represents the prediction result of the direction of the three-dimensional detection box; conv_dir() represents the prediction function of the direction of the three-dimensional detection box, which is composed of a 1×1 two-dimensional convolutional layer.

[0142] S4: Substitute the training set into the point cloud 3D object detection model based on the visual state space model for iterative training, and use the validation set to verify the model to obtain the optimal point cloud 3D object detection model based on the visual state space model;

[0143] S41: The training set D i_train Input into the point cloud 3D target detection model, and obtain the classification, regression and direction prediction scores of multiple robots through the visual state space model;

[0144] S42: using the loss function to update the model parameters of the point cloud 3D object detection model, and iterating the training until the training batch is completed;

[0145] Among them, the expression of the loss function is:

[0146]

[0147] In the formula, L represents the total loss of the model; N pos Indicates the number of positive samples; L cls represents the prediction box classification loss; w represents the prediction box classification loss weight; L box represents the 3D detection box regression loss; μ represents the 3D detection box regression loss weight; L_dir represents the direction prediction loss; λ represents the direction prediction loss weight.

[0148] In this embodiment, the prediction box classification loss weight w=1.0; the 3D detection box regression loss weight μ=2.0; and the direction prediction loss weight λ=0.2.

[0149] S43: Evaluate the trained point cloud 3D object detection model and use the validation set D i_val Carry out model verification to obtain the optimal point cloud 3D detection model.

[0150] S5: Substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model for prediction, and obtain the 3D point cloud detection results. Then, fuse the 3D point cloud detection results of multiple robots at the result level to realize collaborative 3D target detection of multiple robots in a large range of scenes.

[0151] S51: Test set D i_test Substitute it into the optimal point cloud 3D target detection model based on the visual state space model to obtain the 3D point cloud detection result Result i_test ;

[0152] S52: Convert the obtained three-dimensional point cloud detection results from the laser radar coordinate system to the virtual world coordinate system, and fuse the three-dimensional point cloud detection results of multiple robots at the result level to achieve collaborative three-dimensional target detection of multiple robots in a large-scale scene.

[0153] Example 2

[0154] like Figure 6 As shown, based on the same technical concept, this embodiment provides a system for a multi-robot collaborative three-dimensional target detection method based on a visual state space model, which includes a data acquisition module, a data labeling module, a model building module, a model training module, and a model detection module.

[0155] Specifically, the data acquisition module is used to build a multi-robot lidar 3D perception platform to collect 3D point cloud data in a wide range of scenarios for aircraft manufacturing and assembly;

[0156] Specifically, a data annotation module is used to annotate the collected 3D point cloud data, create a 3D point cloud detection data set, and divide the 3D point cloud detection data set into a training set, a validation set, and a test set;

[0157] Specifically, a model building module is used to build a point cloud 3D target detection model based on a visual state space model; the point cloud 3D target detection model based on a visual state space model includes a point cloud encoding module, a feature extraction module, a feature fusion module, and a detection head;

[0158] Specifically, the model training module is used to substitute the training set into the point cloud 3D object detection model based on the visual state space model for iterative training, and use the validation set to verify the model, so as to obtain the optimal point cloud 3D object detection model based on the visual state space model;

[0159] Specifically, the model detection module is used to substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model for prediction, obtain the 3D point cloud detection results, and fuse the 3D point cloud detection results of multiple robots at the result level to realize the collaborative 3D target detection of multiple robots in a large range of scenes.

[0160] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0161] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A multi-robot collaborative three-dimensional target detection method based on a visual state space model, characterized in that: The steps include: Build a multi-robot LiDAR 3D perception platform to collect 3D point cloud data in a wide range of scenarios for aircraft manufacturing and assembly; Annotate the collected 3D point cloud data, create a 3D point cloud detection data set, and divide the 3D point cloud detection data set into a training set, a validation set, and a test set; Constructing a point cloud three-dimensional target detection model based on a visual state space model; the point cloud three-dimensional target detection model based on a visual state space model includes a point cloud encoding module, a feature extraction module, a feature fusion module, and a detection head; Substitute the training set into the point cloud 3D object detection model based on the visual state space model for iterative training, and use the validation set to verify the model to obtain the optimal point cloud 3D object detection model based on the visual state space model; Substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model for prediction, and obtain the 3D point cloud detection results. Then, fuse the 3D point cloud detection results of multiple robots at the result level to achieve collaborative 3D target detection of multiple robots in a large range of scenes. The process of the feature extraction module specifically includes the following steps: The output result of the fully connected layer of the point cloud encoding module is feature mapped and output; The output result of the feature map is input into the visual state space model for the first iterative training, and the output result of the first iterative training is input into the visual state space model again for the second iterative training and then output; The output of the feature map is input into a sparse convolutional neural network to extract local features and output after densification operation; The process of inputting the output result of the feature map into the visual state space model for the first iterative training, and then inputting the output result into the visual state space model for the second iterative training and then outputting includes the following steps: The output result of the feature map is input into the first part of the visual state space model for initial iterative training and then output; the first part of the visual state space model refers to multi-directional feature modeling of low-level columnar point clouds based on a two-dimensional selective scanning model; Inputting the output result of the initial iterative training of the first part of the visual state space model into the second part of the visual state space model for initial iterative training and then outputting; The second part of the visual state space model is based on a feedforward network, introducing feature extraction of a gated multilayer perceptron and a residual connection network; The output result of the first iterative training of the second part of the visual state space model is inputted into the first part of the visual state space model for second iterative training and then outputted; The output result of the second iterative training of the first part of the visual state space model is input into the second part of the visual state space model for second iterative training and then output; The process of the feature fusion module specifically includes the following steps: The output results of the second iteration training in the visual state space model and the output results of the sparse convolutional neural network are combined and batch normalized. The feature combination results are deeply fused with the batch normalization results to obtain the global and local features of a large-scale scene point cloud.

2. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 1, characterized in that: The process of building a multi-robot LiDAR 3D perception platform to collect 3D point cloud data in a large range of aircraft manufacturing and assembly scenes includes the following steps: Number multiple robots; Install the repetitive scanning laser radar in front of the multi-robot and connect the repetitive scanning laser radar to the industrial computer inside the multi-robot; Repeated scanning laser radars carried by multiple robots are used to collect 3D point cloud data for a large range of scenes in aircraft manufacturing and assembly, and the corresponding 3D point cloud data of multiple robots are obtained respectively.

3. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 1, characterized in that: The process of annotating the collected 3D point cloud data, creating a 3D point cloud detection data set, and dividing the 3D point cloud detection data set into a training set, a validation set, and a test set specifically includes the following steps: 50% of the 3D point cloud data were selected and annotated using the point cloud lightweight annotation software Labelcloud to form an annotated data set; the remaining unannotated 3D point cloud data formed an unannotated data set; 70% of the labeled data set is divided into a training set, the remaining 30% of the labeled data set is divided into a validation set, and the unlabeled data set is used as a test set.

4. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 3, characterized in that: The process of the point cloud encoding module specifically includes the following steps: With the repetitive scanning laser radar as the center of the coordinate axis, the robot's forward direction as the X-axis direction, the left direction as the Y-axis direction, and the upward direction as the Z-axis direction, the input point cloud is represented by a mathematical vector; Select the coordinate range of the point cloud and divide the point cloud into multiple point cloud columns; Taking the center of each point cloud column as the center point coordinate, calculate the relative coordinates of each point and the center of the point cloud column, and use it as a supplementary representation of the point cloud; Combining the vector representation of the input point cloud and the vector representation of the supplemented point cloud to construct a vector representation of the point cloud; Based on the vector representation of the constructed point cloud, a fully connected layer is used for encoding and output.

5. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 1, characterized in that: The process of the detection head specifically refers to classifying, regressing and predicting the direction of the acquired deep feature fusion results through different network layers.

6. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 5, characterized in that: Substituting the training set into the point cloud 3D object detection model based on the visual state space model for iterative training, and using the validation set for model verification, the process of obtaining the optimal point cloud 3D object detection model based on the visual state space model specifically includes the following steps: The training set is input into the point cloud 3D object detection model, and the classification, regression and direction prediction scores of multiple robots are obtained through the visual state space model. The loss function is used to update the model parameters of the point cloud 3D object detection model, and the training is iterated until the training batch is completed; The trained point cloud 3D object detection model is evaluated and verified using a validation set to obtain the point cloud 3D detection model with the best effect.

7. The method for multi-robot collaborative three-dimensional target detection based on a visual state space model according to claim 6, characterized in that: Substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model for prediction, obtain the 3D point cloud detection result, and fuse the 3D point cloud detection results of multiple robots at the result level to realize the collaborative 3D target detection of multiple robots in a large range of scenes. The specific process includes the following steps: Substitute the test set into the optimal point cloud 3D target detection model based on the visual state space model to obtain the 3D point cloud detection result; The obtained 3D point cloud detection results are converted from the lidar coordinate system to the virtual world coordinate system, and the 3D point cloud detection results of multiple robots are fused at the result level to realize collaborative 3D target detection of multiple robots in a large-scale scene.

Images