A unified 3D object detection method for the entire scene of large assembly process

Abstract

A unified three-dimensional target detection method for the entire scene of a large assembly process includes: 1. constructing a full-scene laser radar perception platform to collect three-dimensional point cloud data; 2. making a detection data set for assembly manufacturing scenes; 3. constructing a three-dimensional target detection model; 4. inputting the data in the training set into the three-dimensional target detection model to obtain target classification results and target bounding box regression results; 5. constructing a total loss function, calculating the loss value of the target classification result and the target bounding box regression result, looping from 4 to 5 until the total loss function converges, updating the weights, and obtaining a trained three-dimensional target detection model; 6. testing the trained three-dimensional target detection model to obtain detection results. The present invention increases assembly efficiency, improves assembly quality, can timely discover and handle problems in the assembly process, avoids the safety hazards of traditional manual assembly processes in the manufacturing industry, and helps promote the rapid and high-quality development of industrial intelligent manufacturing.

Description

Technical Field

[0001] The present invention relates to the field of assembly manufacturing full-scene perception technology, and in particular to a unified three-dimensional target detection method for the full scene of a large-scale assembly process. Background Art

[0002] Large and complex components such as aircraft panels and aircraft skins are important components of major equipment, and ensuring their manufacturing quality is a key factor in determining the comprehensive performance and safe operation of the equipment. For example, in the assembly process of large thin-walled aircraft wings and fuselages (up to 39 meters long), it is necessary to detect the elements to be assembled, assembly robots, operators and other targets in the industrial assembly scene in real time, and provide environmental information to the robots to guide efficient and accurate assembly. Therefore, how to accurately and effectively perceive a variety of key information is a difficult point in the manufacturing of major equipment, and it also plays an irreplaceable basic supporting role in improving the manufacturing quality of major equipment. At present, information perception in the assembly and manufacturing process of large components relies heavily on manual labor, and there are prominent problems such as poor consistency and low efficiency. The development of special automatic detection equipment is extremely costly and has low utilization, and it is difficult to adapt to the perception requirements of multiple information in super-large manufacturing scenarios.

[0003] In response to the above problems, robots are coordinated with multi-source heterogeneous sensors to optimize the assembly process of large and complex parts, and play an important supporting role in high-quality development such as quality improvement, efficiency increase, cost reduction, and flexible production. At present, many three-dimensional target detection methods for industrial manufacturing scenarios have been proposed, but they are still not enough to meet the requirements of engineering applications such as full coverage of scenes and high efficiency of data annotation. Therefore, how to achieve high-precision three-dimensional target detection in large-scale assembly manufacturing scenarios and provide information guidance for assembly manufacturing has become an urgent problem to be solved in this field at this stage. However, it is difficult to meet the full coverage and high-precision requirements of large-scale assembly scenarios with a single laser radar, and the three-dimensional target detection technology that integrates multiple sensors can further improve the range coverage capability of refined three-dimensional perception in large-scale assembly scenarios. Therefore, it is very necessary to design a unified three-dimensional target detection method for the entire scene of large-scale assembly processes.

[0004] In recent years, deep learning-based 3D object detection has attracted more and more attention. Using multiple laser radars to study learning-based 3D object detection methods to assist large-scale assembly in industrial scenes is more challenging than traditional template matching. There are several difficulties in the specific implementation process:

[0005] 1. Multiple lidar sensors can cover the largest possible scene, but the data collected by different lidars have differences in range, collection angle, and target type. Directly learning features from these data will make it difficult for model training to converge.

[0006] 2. There are differences in the data structure of the data sets. The three-dimensional representations of the data collected by different lidar sensors for the same target in the assembly process are heterogeneous, which may lead to inconsistencies in the learned features.

[0007] 3. It will take a lot of time to label the data sets collected by different lidars, and the data collected by different lidars contain common areas and non-common areas, which makes the labeling process have a certain degree of repetitiveness. Summary of the invention

[0008] The present invention provides a unified three-dimensional target detection method for the entire scene of a large-scale assembly process to solve the technical problems mentioned in the background technology.

[0009] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0010] The present invention provides a unified three-dimensional object detection method for the entire scene of a large assembly process, comprising the following steps:

[0011] S1. Build a full-scene lidar perception platform to collect 3D point cloud data of large assemblies;

[0012] S2. Annotate the 3D point cloud data of the large assembly, create a detection data set for the assembly manufacturing scene, and divide the detection data set into a training set, a validation set, and a test set;

[0013] S3, build a unified 3D object detection model;

[0014] S4, input the data in the training set into a unified 3D object detection model to obtain object classification results and object bounding box regression results;

[0015] S5, construct a total loss function, calculate the loss value of the target classification result and the target bounding box regression result, loop S4 to S5, minimize the loss function, until the total loss function converges, use the validation set for verification, update the weights of the unified 3D target detection model through back propagation, and obtain the trained unified 3D target detection model;

[0016] S6. Use the test set to test the trained unified 3D object detection model to obtain the detection result.

[0017] Furthermore, the S1 specifically includes the following steps:

[0018] S11. First, four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene;

[0019] S12, then installing a repetitive scanning laser radar on the assembly robot, and moving synchronously with the assembly robot;

[0020] S13: The four non-repetitive scanning laser mines located on the ceiling collect a point cloud dataset corresponding to the assembly scene respectively and merge them into the first point cloud dataset. ;

[0021] S14: The repetitive scanning laser radar on the assembly robot collects a second point cloud data set of the assembly scene .

[0022] Furthermore, the unified three-dimensional target detection model includes a data alignment module, a three-dimensional backbone network, a feature statistics alignment module, a local information fusion module, and a multi-data detection head which are connected in sequence.

[0023] Furthermore, the S2 specifically includes the following steps:

[0024] S21. Use universal point cloud annotation software to annotate the first point cloud dataset. The data set after labeling is recorded as , the first point cloud dataset The unlabeled data samples are denoted as ; Set the ratio to 10%;

[0025] S22. Use universal point cloud annotation software to annotate the second point cloud dataset. All the data in are labeled, and the labeled data set is recorded as ;

[0026] S23. Divide the data sets separately and , dataset 80% of the data is used for training, denoted as , dataset 20% of the data is used for verification, denoted as , dataset 80% of the data is used for training, denoted as ; The dataset and Merge to get the training set; data set 20% of the data is used for verification, denoted as ; The dataset and Merge to get the validation set;

[0027] S24. Annotate the data set described in S21 10% of the data is recorded as , get the test set.

[0028] Furthermore, the S4 specifically includes the following steps:

[0029] S41. Dataset and dataset Input into the data alignment module and align the data set and The original point cloud space in the world coordinate system is converted to the point cloud space in the world coordinate system, and the point cloud within the set range is eliminated to obtain an updated non-repetitive scanning point cloud dataset. and updated repetitive scan point cloud dataset ;

[0030] S42, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the three-dimensional backbone network to obtain the first eigenvector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ;

[0031] The first feature vector is aligned using the feature statistics module and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector ;

[0032] S43, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the local information fusion module to obtain the first local geometric code , the second local geometry encoding ; The first normalized eigenvector With the second normalized eigenvector Input into the local information fusion module to obtain the first local feature encoding and the second local feature encoding , then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature ;

[0033] S44, fusing the first local information with features , the second local information fusion feature The first bird's-eye view feature vector , the second bird's-eye view feature vector After splicing, convolution, multi-layer perceptron and addition operations, they are input into the multi-data detection head to obtain the target classification results and target bounding box regression results.

[0034] Furthermore, the S41 specifically includes the following steps:

[0035] S411, the data set Input into the data alignment module, and use the hand-eye calibration method to align the data set Converted to the world coordinate system, the conversion relationship between the world coordinate system and the coordinate system of the non-repetitive scanning laser radar is shown in the following formula:

[0036] ;

[0037] in, Indicates Non-repetitive scanning LiDAR to world coordinate system The rotation of Indicates Non-repetitive scanning LiDAR to world coordinate system Translation of Indicates A coordinate system for a non-repeating scanning LiDAR;

[0038] S412, solve the data set after coordinate transformation by the following formula Each spatial point cloud in;

[0039] ;

[0040] in, Representation dataset Middle The original spatial point cloud of a non-repetitive scanning LiDAR, Represents the first A spatial point cloud of a non-repetitive scanning LiDAR;

[0041] Integrate the spatial point clouds of the four non-repetitive scanning lidars after coordinate transformation to obtain a non-repetitive scanning point cloud dataset after coordinate alignment ;

[0042] S413, use inertial sensors to record the movement changes during the assembly process, and obtain the integral of the movement state and , and the coordinate system of the repetitive scanning lidar is converted to the world coordinate system by the following formula:

[0043] ;

[0044] in, Represents repeated scanning of the laser radar to the world coordinate system Rotation of Represents repeated scanning of the laser radar to the world coordinate system Translation of Represents the coordinate system of the repetitive scanning lidar;

[0045] S414, use the following formula to solve the data set after coordinate transformation The spatial point cloud in the image is used to obtain a repeated scanning point cloud dataset. , ;

[0046] ;

[0047] in, Representation dataset The original space point cloud; It is the spatial point cloud of the repeated scanning laser radar in the world coordinate system;

[0048] S415, non-repetitive scanning point cloud dataset and repeated scan point cloud datasets Point clouds outside the set range are eliminated to obtain an updated non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset .

[0049] Furthermore, the S42 specifically includes the following steps:

[0050] S421. Updated non-repetitive scan point cloud dataset and updated repetitive scan point cloud dataset Voxelization is performed, and the voxel coordinates are obtained by calculating the mean center of the point coordinates within the voxel. The calculation process is shown in the following formula:

[0051] ;

[0052] in, Represents the coordinates of a point within a voxel , represents the number of points in a voxel, Represents the coordinates of the voxel ; represents voxel;

[0053] S422, the voxelized non-repetitive scanned point cloud dataset and updated repetitive scan point cloud dataset All are input into the 3D voxel backbone network for feature extraction to obtain the first feature vector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ;

[0054] S423, using the feature statistics alignment module and using the following formula to align the first feature vector and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector , the formula is as follows:

[0055] ;

[0056] in, Indicates The zero-mean unbiased feature of the dataset, Indicates The characteristics of the dataset, Represents the minimum value, that is, a value close to 0, to ensure the stability of numerical calculations; and Respectively represent The mean and variance of the features in the channel dimension of the dataset.

[0057] Furthermore, the S43 specifically includes the following steps:

[0058] S431, firstly update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Build the octree of point cloud respectively, and build the point according to Euclidean distance respectively. Neighborhood , point Neighborhood By point and its nearest points, and define the point about Neighborhood The local geometry of , as shown in the following formula:

[0059] ;

[0060] in, Point-to-point in the neighborhood The vector of are the coordinates of the points in the neighborhood, Represents the neighborhood Point to Point Positional relationship; It is a mathematical expression, indicating In a space of dimension kx6;

[0061] S432, then use a multi-layer perceptron to analyze the local geometric graph Encode and apply the maximum pooling function to aggregate The local geometric content of the neighborhood is used to obtain a non-repetitive scanned point cloud dataset. The first local geometric encoding , Repeated scanning point cloud dataset The second local geometric encoding ;

[0062] S433, according to Neighborhood , from the first normalized eigenvector With the second normalized eigenvector In the dictionary index point exist The eigenvector corresponding to the neighborhood , build about Local feature map of the neighborhood , as shown in the following formula:

[0063] ;

[0064] in Represents feature to feature within the neighborhood The vector of represents the eigenvector in the neighborhood, Represents the neighborhood Feature to Feature The position relationship; C represents the number of channels;

[0065] S434, then use the multi-layer perceptron to analyze the local feature map Encode and apply the maximum pooling function to aggregate The local feature content of the neighborhood is used to obtain the first local feature code and the second local feature encoding ;

[0066] S435, then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature , the formula is as follows:

[0067] .

[0068] Furthermore, the S44 specifically includes the following steps:

[0069] S441, the first bird's-eye view feature vector , the second bird's-eye view feature vector Fusion features with the first local information , the second local information fusion feature Concatenate and get the first eigenvector and the second eigenvector , as shown in the following formula:

[0070] ;

[0071] S442, for the first eigenvector and the second eigenvector The splicing is performed and the splicing results are input into a layer of convolution and a layer of multi-layer perceptron in turn to obtain the characteristics of the two data being tightly coupled. , the feature Input each layer into a different compression network to obtain the first compression feature and the second compression feature ; Then the first eigenvector and the second eigenvector The first compression feature and the second compression feature Add to get the first input vector of the input multi-data detection head and the second input vector , as shown in the following formula:

[0072] ;

[0073] in, represents convolution, represents a multi-layer perceptron, represents an arbitrary compression-excitation network;

[0074] S443: The first input vector and the second input vector The data are input into two different detection heads in the multi-data detection head respectively to obtain the target classification results and target bounding box regression results.

[0075] Furthermore, the total loss function is specifically as follows:

[0076] ;

[0077] in, is the total loss function, For the The classification loss of the detection head, For the The bounding box regression loss of each detection head, For the A detection head.

[0078] Beneficial effects of the present invention:

[0079] 1. The unified three-dimensional target detection method for the entire scene of a large-scale assembly process proposed in the present invention can detect elements such as robots, components, and personnel in the assembly process, and provide guidance for assembly robots, such as target positioning and obstacle avoidance.

[0080] 2. The present invention solves the problem of training difficulties caused by different models of lidar sensors through a data alignment module and a feature statistics alignment module. The data alignment module ensures that the data can be processed in the same coordinate system, which is beneficial to the data input of the network. The feature statistics alignment module ensures that the features generated by different data through the network are unbiased in mean and variance, which is beneficial to the convergence of the training process.

[0081] 3. The local information fusion module makes full use of point cloud information, improves the effectiveness of the network, and ensures that the network can accurately capture point cloud features from different installation positions and perspectives. When these features belong to the same target object, the features are represented by a small feature distance in the high-dimensional vector space;

[0082] 4. The multi-data detection head solves the problem of data set label differences and improves the robustness of the network. It ensures that when the network inputs different data sets, there is no need to pay extra attention to the content of the data set labels, and the training and verification of the network model can be completed efficiently.

[0083] 5. The detection method proposed in the present invention makes efficient use of point cloud information. Since multiple laser radars collect data in the same space, there is bound to be redundancy in the information on the assembly plant scene. Therefore, in the stage of making data sets, the patent of the present invention only marks 10% of the data from non-repetitive scanning laser radars and 100% of the data from repetitive scanning laser radars. This will reduce a lot of repetitive labeling work in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] Figure 1 is a flow chart of the present invention;

[0085] Figure 2 It is a framework diagram of the unified three-dimensional object detection model in the present invention;

[0086] Figure 3 Schematic diagram of the full-scenario laser radar perception platform in the present invention;

[0087] Figure 4 1 is a view of the laser mine of non-repetitive scanning in the present invention, wherein (a) to (d) are respectively the view of the 1st to 4th non-repetitive scanning laser mines;

[0088] Figure 5 Schematic diagram of repeated scanning laser radar data acquisition in the present invention. DETAILED DESCRIPTION

[0089] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0090] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0091] Reference Figure 1 The embodiment of the present application provides a unified three-dimensional target detection method for the entire scene of a large assembly process, including the following steps:

[0092] S1. Build a full-scene lidar perception platform to collect 3D point cloud data of large assemblies;

[0093] S2. Annotate the 3D point cloud data of the large assembly, create a detection data set for the assembly manufacturing scene, and divide the detection data set into a training set, a validation set, and a test set;

[0094] S3, build a unified 3D object detection model;

[0095] S4, input the data in the training set into a unified 3D object detection model to obtain object classification results and object bounding box regression results;

[0096] S5, construct a total loss function, calculate the loss value of the target classification result and the target bounding box regression result, loop S4 to S5, minimize the loss function, until the total loss function converges, use the validation set for verification, update the weights of the unified 3D target detection model through back propagation, and obtain the trained unified 3D target detection model;

[0097] S6. Use the test set to test the trained unified three-dimensional object detection model to obtain a detection result.

[0098] In some embodiments, reference Figures 3 to 5 , the S1 specifically comprises the following steps:

[0099] S11. First, four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene; the layout of the four non-repetitive scanning laser radars is as follows: Figure 4 As shown in (a) to (d);

[0100] S12, then a repetitive scanning laser radar is installed on the assembly robot, and moves synchronously with the assembly robot; the layout of the repetitive scanning laser radar is as follows Figure 3 As shown;

[0101] S13: The four non-repetitive scanning laser mines located on the ceiling collect a point cloud dataset corresponding to the assembly scene respectively and merge them into the first point cloud dataset. , ;in, They are point cloud datasets collected from the first non-repetitive scanning laser radar to the fourth non-repetitive scanning laser radar;

[0102] S14: The repetitive scanning laser radar on the assembly robot collects a second point cloud data set of the assembly scene . The schematic diagram of repeated scanning laser radar acquisition is shown in Figure 5 shown.

[0103] In some embodiments, reference Figure 2The unified 3D target detection model includes a data alignment module, a 3D backbone network, a feature statistics alignment module, a local information fusion module, a multi-data detection head (i.e. Figure 2 Multi-data detection head module in ).

[0104] In some embodiments, S2 specifically includes the following steps:

[0105] S21. Use universal point cloud annotation software to annotate the first point cloud dataset. 10% of the data is labeled, and the labeled data set is recorded as , the first point cloud dataset The unlabeled data samples are denoted as ;

[0106] S22. Use universal point cloud annotation software to annotate the second point cloud dataset. All the data in are labeled, and the labeled data set is recorded as ;

[0107] S23. Divide the data sets separately and , dataset 80% of the data is used for training, denoted as , dataset 20% of the data is used for verification, denoted as , dataset 80% of the data is used for training, denoted as ; The dataset and Merge to get the training set; data set 20% of the data is used for verification, denoted as ; The dataset and Merge to get the validation set;

[0108] S24. Annotate the dataset described in S21 10% of the data is recorded as , get the test set.

[0109] In some embodiments, reference Figure 2 , the S4 specifically comprises the following steps:

[0110] S41. Dataset and dataset Input into the data alignment module and align the data set and The original point cloud space in the world coordinate system is converted to the point cloud space in the world coordinate system, and the point cloud within the set range is eliminated to obtain an updated non-repetitive scanning point cloud dataset. and updated repetitive scan point cloud dataset ;

[0111] S42, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the three-dimensional backbone network to obtain the first eigenvector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ;

[0112] The first feature vector is aligned using the feature statistics module and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector ;

[0113] S43, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the local information fusion module to obtain the first local geometric code , the second local geometry encoding ; The first normalized eigenvector With the second normalized eigenvector Input into the local information fusion module to obtain the first local feature encoding and the second local feature encoding , then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature ;

[0114] S44, fusing the first local information with features , the second local information fusion feature The first bird's-eye view feature vector , the second bird's-eye view feature vector After splicing, convolution, multi-layer perceptron and addition operations, they are input into the multi-data detection head to obtain the target classification results and target bounding box regression results.

[0115] In some embodiments, the S41 specifically includes the following steps:

[0116] S411, the data set Input into the data alignment module, and use the hand-eye calibration method to align the data set Converted to the world coordinate system, the coordinate system of each non-repetitive scanning laser radar is , the world coordinate system is , the conversion relationship between the world coordinate system and the coordinate system of the non-repetitive scanning laser radar is shown in the following formula:

[0117] ;

[0118] in, Indicates Non-repetitive scanning LiDAR to world coordinate system The rotation of Indicates Non-repetitive scanning LiDAR to world coordinate system Translation of and It can be easily obtained through the software toolkit of hand-eye calibration algorithm and lidar;

[0119] S412, solve the data set after coordinate transformation by the following formula Each spatial point cloud in;

[0120] ;

[0121] in, Representation dataset Middle The original spatial point cloud of a non-repetitive scanning LiDAR, Represents the first A spatial point cloud of a non-repetitive scanning LiDAR;

[0122] Integrate the spatial point clouds of the four non-repetitive scanning lidars after coordinate transformation to obtain a non-repetitive scanning point cloud dataset after coordinate alignment , ;

[0123] S413. Since the laser radar of repeated scanning will follow the movement of the assembly robot, the laser radar coordinate system of repeated scanning (denoted as ) to the world coordinate system and The present invention uses an inertial sensor to record the motion changes during the assembly process, and obtains the integral of the motion state. and , and the coordinate system of the repetitive scanning lidar is converted to the world coordinate system by the following formula:

[0124] ;

[0125] in, Represents repeated scanning of the laser radar to the world coordinate system Rotation of Represents repeated scanning of the laser radar to the world coordinate system Translation of Represents the coordinate system of the repetitive scanning lidar;

[0126] S414, use the following formula to solve the data set after coordinate transformation The spatial point cloud in the image is used to obtain a repeated scanning point cloud dataset. , ;

[0127] ;

[0128] in, Representation dataset The original space point cloud; It is the spatial point cloud of the repeated scanning laser radar in the world coordinate system;

[0129] S415, non-repetitive scanning point cloud dataset and repeated scan point cloud datasets Point clouds outside the set range are eliminated to obtain an updated non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Ensuring that the point cloud data collected by lidars from different locations can each cover part of the assembly scene and eliminating irrelevant spatial locations will reduce the amount of calculation.

[0130] In some embodiments, the S42 specifically includes the following steps:

[0131] S421. In order to alleviate the problem of too much data and difficulty in calculation, the updated non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Voxelization is performed, and the voxel coordinates are obtained by calculating the mean center of the point coordinates within the voxel. The calculation process is shown in the following formula:

[0132] ;

[0133] in, Represents the coordinates of a point within a voxel , represents the number of points in a voxel, Represents the coordinates of the voxel ; represents voxel;

[0134] S422, the voxelized non-repetitive scanned point cloud dataset and updated repetitive scan point cloud dataset All are input into the 3D voxel backbone network for feature extraction to obtain the first feature vector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ;

[0135] S423, using the feature statistics alignment module and using the following formula to align the first feature vector and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector , the formula is as follows:

[0136] ;

[0137] in, Indicates The zero-mean unbiased feature of the dataset, Indicates The characteristics of the dataset, Represents the minimum value, that is, a value close to 0, to ensure the stability of numerical calculations; and Respectively represent The mean and variance of the features in the channel dimension of the dataset.

[0138] In some embodiments, the S43 specifically includes the following steps:

[0139] S431, firstly update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Build the octree of point cloud respectively, and build the point according to Euclidean distance respectively. Neighborhood , point Neighborhood By point and its nearest points, and define the point about Neighborhood The local geometry of , as shown in the following formula:

[0140] ;

[0141] in, Point-to-point in the neighborhood The vector of are the coordinates of the points in the neighborhood, Represents the neighborhood Point to Point Positional relationship; It is a mathematical expression, indicating In a space of dimension kx6;

[0142] S432, then use a multi-layer perceptron to analyze the local geometric graph Encode and apply the maximum pooling function to aggregate The local geometric content of the neighborhood is used to obtain a non-repetitive scanned point cloud dataset. The first local geometric encoding , Repeated scanning point cloud dataset The second local geometric encoding ;

[0143] Specifically, the local geometric map is analyzed by a multi-layer perceptron The encoding is mainly obtained through the following formula:

[0144] .

[0145] S433, according to Neighborhood , from the first normalized eigenvector With the second normalized eigenvector In the dictionary index point exist The eigenvector corresponding to the neighborhood , build about Local feature map of the neighborhood , as shown in the following formula:

[0146] ;

[0147] in Represents feature to feature within the neighborhood The vector of represents the eigenvector in the neighborhood, Represents the neighborhood Feature to Feature The position relationship; C represents the number of channels;

[0148] S434, then use the multi-layer perceptron to analyze the local feature map Encode and apply the maximum pooling function to aggregate The local feature content of the neighborhood is used to obtain the first local feature code and the second local feature encoding ;

[0149] Specifically, the local feature map is analyzed by a multi-layer perceptron The encoding is mainly calculated by the following formula:

[0150] .

[0151] S435, then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature , the formula is as follows:

[0152] .

[0153] In some embodiments, the S44 specifically includes the following steps:

[0154] S441, the first bird's-eye view feature vector , the second bird's-eye view feature vector Fusion features with the first local information , the second local information fusion feature Concatenate and get the first eigenvector and the second eigenvector , as shown in the following formula:

[0155] ;

[0156] S442, for the first eigenvector and the second eigenvector The splicing is performed and the splicing results are input into a layer of convolution and a layer of multi-layer perceptron in turn to obtain the characteristics of the two data being tightly coupled. , the feature Input each layer into a different compression network to obtain the first compression feature and the second compression feature ; Then the first eigenvector and the second eigenvector The first compression feature and the second compression feature Add to get the first input vector of the input multi-data detection head and the second input vector , as shown in the following formula:

[0157] ;

[0158] in, represents convolution, represents a multi-layer perceptron, represents an arbitrary compression-excitation network;

[0159] S443: The first input vector and the second input vector The data are input into two different detection heads in the multi-data detection head respectively to obtain the target classification results and target bounding box regression results.

[0160] In some embodiments, the total loss function is as follows:

[0161] ;

[0162] in, is the total loss function, For the The classification loss of the detection head, For the The bounding box regression loss of each detection head, For the A detection head.

[0163] The present invention increases assembly efficiency, improves assembly quality, can timely discover and handle problems in the assembly process, avoids safety hazards in traditional manual assembly processes in the manufacturing industry, and helps promote rapid and high-quality development of industrial intelligent manufacturing.

[0164] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A unified 3D object detection method for the entire scene of a large assembly process, characterized in that: The steps include: S1. Build a full-scene lidar perception platform to collect 3D point cloud data of large assemblies; The S1 specifically includes the following steps: S11. First, four non-repetitive scanning laser radars are installed and fixed at the four corners of the ceiling beam, with the scanning direction facing the assembly scene; S12, then installing a repetitive scanning laser radar on the assembly robot, and moving synchronously with the assembly robot; S13: The four non-repetitive scanning laser radars located on the ceiling collect a point cloud dataset corresponding to the assembly scene respectively and merge them into the first point cloud dataset. ; S14: The repetitive scanning laser radar on the assembly robot collects a second point cloud data set of the assembly scene ; S2. Annotate the 3D point cloud data of the large assembly, create a detection data set for the assembly manufacturing scene, and divide the detection data set into a training set, a validation set, and a test set; S3, build a unified 3D object detection model; S4, input the data in the training set into a unified 3D object detection model to obtain object classification results and object bounding box regression results; S5, construct a total loss function, calculate the loss value of the target classification result and the target bounding box regression result, loop S4 to S5, minimize the loss function, until the total loss function converges, use the validation set for verification, update the weights of the unified 3D target detection model through back propagation, and obtain the trained unified 3D target detection model; S6. Using the test set to test the trained unified 3D object detection model to obtain a detection result; The unified three-dimensional target detection model includes a data alignment module, a three-dimensional backbone network, a feature statistics alignment module, a local information fusion module, and a multi-data detection head which are connected in sequence.

2. The unified three-dimensional object detection method according to claim 1, characterized in that: The S2 specifically includes the following steps: S21. Use universal point cloud annotation software to annotate the first point cloud dataset. The data set after labeling is recorded as , the first point cloud dataset The unlabeled data samples are denoted as ; S22. Use universal point cloud annotation software to annotate the second point cloud dataset. All the data in are labeled, and the labeled data set is recorded as ; S23. Divide the data sets separately and , dataset 80% of the data is used for training, denoted as , dataset 20% of the data is used for verification, denoted as , dataset 80% of the data is used for training, denoted as ; The dataset and Merge to get the training set; data set 20% of the data is used for verification, denoted as ; The dataset and Merge to get the validation set; S24. Annotate the dataset described in S21 10% of the data is recorded as , get the test set.

3. The unified three-dimensional object detection method according to claim 2, characterized in that: The S4 specifically includes the following steps: S41. Dataset and dataset Input into the data alignment module and align the data set and The original point cloud space in the world coordinate system is converted to the point cloud space in the world coordinate system, and the point cloud within the set range is eliminated to obtain an updated non-repetitive scanning point cloud dataset. and updated repetitive scan point cloud dataset ; S42, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the three-dimensional backbone network to obtain the first eigenvector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ; The first feature vector is aligned using the feature statistics module and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector ; S43, update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Input into the local information fusion module to obtain the first local geometric code , the second local geometry encoding ; The first normalized eigenvector With the second normalized eigenvector Input into the local information fusion module to obtain the first local feature encoding and the second local feature encoding , then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature ; S44, fusing the first local information with features , the second local information fusion feature The first bird's-eye view feature vector , the second bird's-eye view feature vector After splicing, convolution, multi-layer perceptron and addition operations, they are input into the multi-data detection head to obtain the target classification results and target bounding box regression results.

4. The unified three-dimensional object detection method according to claim 3, characterized in that: The S41 specifically includes the following steps: S411, the data set Input into the data alignment module, and use the hand-eye calibration method to align the data set Converted to the world coordinate system, the conversion relationship between the world coordinate system and the coordinate system of the non-repetitive scanning laser radar is shown in the following formula: ； in, Indicates Non-repetitive scanning LiDAR to world coordinate system The rotation of Indicates Non-repetitive scanning LiDAR to world coordinate system Translation of Indicates A coordinate system for a non-repeating scanning LiDAR; S412, solve the data set after coordinate transformation by the following formula Each spatial point cloud in; ； in, Representation dataset Middle The original spatial point cloud of a non-repetitive scanning LiDAR, Represents the first A spatial point cloud of a non-repetitive scanning LiDAR; Integrate the spatial point clouds of the four non-repetitive scanning lidars after coordinate transformation to obtain a non-repetitive scanning point cloud dataset after coordinate alignment ; S413, use inertial sensors to record the movement changes during the assembly process, and obtain the integral of the movement state and , and the coordinate system of the repetitive scanning lidar is converted to the world coordinate system by the following formula: ； in, Represents repeated scanning of the laser radar to the world coordinate system Rotation of Represents repeated scanning of the laser radar to the world coordinate system Translation of Represents the coordinate system of the repetitive scanning lidar; S414, use the following formula to solve the data set after coordinate transformation The spatial point cloud in the image is used to obtain a repeated scanning point cloud dataset. , ; ； in, Representation dataset The original space point cloud; It is the spatial point cloud of the repeated scanning laser radar in the world coordinate system; S415, non-repetitive scanning point cloud dataset and repeated scan point cloud datasets Point clouds outside the set range are eliminated to obtain an updated non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset .

5. The unified three-dimensional object detection method according to claim 4, characterized in that: The S42 specifically includes the following steps: S421. Updated non-repetitive scan point cloud dataset and updated repetitive scan point cloud dataset Voxelization is performed, and the voxel coordinates are obtained by calculating the mean center of the point coordinates within the voxel. The calculation process is shown in the following formula: ； in, Represents the coordinates of a point within a voxel , represents the number of points in a voxel, Represents the coordinates of the voxel ; represents voxel; S422, the voxelized non-repetitive scanned point cloud dataset and updated repetitive scan point cloud dataset All are input into the 3D voxel backbone network for feature extraction to obtain the first feature vector and the second eigenvector , and then project the two eigenvectors onto the horizontal plane to obtain the first bird's-eye view eigenvectors and the second bird's-eye view feature vector ; S423, using the feature statistics alignment module and using the following formula to align the first feature vector and the second eigenvector Perform the alignment operation to obtain the first normalized feature vector after alignment With the second normalized eigenvector , the formula is as follows: ； in, Indicates The zero-mean unbiased feature of the dataset, Indicates The characteristics of the dataset, Represents the minimum value, that is, a value close to 0, which is used to ensure the stability of numerical calculations; and Respectively represent The mean and variance of the features in the channel dimension of the dataset.

6. The unified three-dimensional object detection method according to claim 5, characterized in that: The S43 specifically includes the following steps: S431, firstly update the non-repetitive scanning point cloud dataset and updated repetitive scan point cloud dataset Build the octree of point cloud respectively, and build the point according to Euclidean distance respectively. Neighborhood , point Neighborhood By point and its nearest points, and define the point about Neighborhood The local geometry of , as shown in the following formula: ； in, Point-to-point in the neighborhood The vector of are the coordinates of the points in the neighborhood, Represents the neighborhood Point to Point Positional relationship; It is a mathematical expression, indicating In a space of dimension kx6; S432, then use a multi-layer perceptron to analyze the local geometric graph Encode and apply the maximum pooling function to aggregate The local geometric content of the neighborhood is used to obtain a non-repetitive scanned point cloud dataset. The first local geometric encoding , Repeated scanning point cloud dataset The second local geometric encoding ; S433, according to Neighborhood , from the first normalized eigenvector With the second normalized eigenvector In the dictionary index point exist The eigenvector corresponding to the neighborhood , build about Local feature map of the neighborhood , as shown in the following formula: ； in Represents feature to feature within the neighborhood The vector of represents the eigenvector in the neighborhood, Represents the neighborhood Feature to Feature The position relationship; C represents the number of channels; S434, then use the multi-layer perceptron to analyze the local feature map Encode and apply the maximum pooling function to aggregate The local feature content of the neighborhood is used to obtain the first local feature code and the second local feature encoding ; S435, then encode the first local geometry and the first local feature encoding Perform stitching and encode the second local geometry and the second local feature encoding Splice and obtain the first local information fusion features , the second local information fusion feature , the formula is as follows: 。 7. The unified three-dimensional object detection method according to claim 6, characterized in that: The S44 specifically includes the following steps: S441, the first bird's-eye view feature vector , the second bird's-eye view feature vector Fusion features with the first local information , the second local information fusion feature Concatenate and get the first eigenvector and the second eigenvector , as shown in the following formula: ； S442, for the first eigenvector and the second eigenvector The splicing is performed and the splicing results are input into a layer of convolution and a layer of multi-layer perceptron in turn to obtain the characteristics of the two data being tightly coupled. , the feature Input each layer into a different compression network to obtain the first compression feature and the second compression feature ; Then the first eigenvector and the second eigenvector The first compression feature and the second compression feature Add to get the first input vector of the input multi-data detection head and the second input vector , as shown in the following formula: ； in, represents convolution, represents a multi-layer perceptron, represents any compression-excitation network; S443: The first input vector and the second input vector The data are input into two different detection heads in the multi-data detection head respectively to obtain the target classification results and target bounding box regression results.

8. The unified three-dimensional object detection method according to claim 1, characterized in that: The total loss function is as follows: ； in, is the total loss function, For the The classification loss of the detection head, For the The bounding box regression loss of each detection head, For the A detection head.

Images