Semi-supervised 3D object detection method, system and device based on digital twin

Abstract

The present invention discloses a semi-supervised 3D target detection method, system and device based on digital twins, which belongs to the field of computer vision, including receiving point clouds, inputting the point clouds into a preset teacher model and a student model for preprocessing, respectively obtaining the uncertain prediction results of the teacher module and the uncertain prediction results of the student module; performing pseudo-label screening and weight distribution on the uncertain prediction results of the teacher module obtained by preprocessing; using pseudo-labels and weight scores to supervise the unlabeled data of the student model, using Groud‑Truth to supervise the labeled data of the student model, and using IoU-guided NMS on the uncertain prediction results of the teacher module to obtain the final detection results. The present invention designs an uncertainty estimation method based on face perception and a pseudo-label screening strategy, which ultimately helps the model to accurately and efficiently locate object instances and identify object categories.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a semi-supervised 3D target detection method, system and device based on digital twins. Background Art

[0002] 3D object detection is a fundamental task in 3D scene understanding, which aims to predict the semantic label and spatial positioning box of each object in the point cloud scene. With the popularity of AR / VR, 3D indoor scanning, and autonomous driving, 3D object detection has become a key technology to promote scene understanding. In the past few decades, many fully supervised 3D object detection methods have been proposed, and significant progress has been made in this field. However, these methods rely on a large amount of carefully annotated 3D scene data, which is expensive and time-consuming to collect. In order to reduce the high annotation cost of fully supervised 3D object detection methods, semi-supervised methods that combine a small amount of labeled data with a large amount of unlabeled data for model training have received increasing attention.

[0003] At present, semi-supervised 3D object detection methods can be roughly divided into two categories: consistency constraint-based methods and pseudo-label-based methods. The core idea of ​​the consistency constraint-based method is to encourage consistency in the prediction results of data with different data augmentation. Specifically, the data will be input into the teacher model and the student model respectively through data augmentation, and the prediction results will be given for the data with different data augmentation respectively, and the consistency loss will be used for constraint. The pseudo-label-based method aims to select high-quality pseudo-labels from the model predictions of unlabeled data, and then use them in combination with labeled data for model training. The pseudo-label-based method uses global metric scores (IoU, classification confidence, and voting scores, etc.) to select pseudo-labels. However, pseudo-labels with higher global metric scores may not cover every side of the object well, while pseudo-labels with lower global metric scores can provide correct predictions for some faces. Therefore, a semi-supervised 3D object detection method based on digital twins is proposed. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a semi-supervised 3D target detection method based on digital twins, which optimizes the three processes of pseudo-label prediction, quality assessment, and screening to achieve high-precision 3D target detection results.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] In the first aspect, the present application proposes a semi-supervised 3D target detection method based on digital twins, comprising:

[0007] The point cloud is received and input into a preset teacher model and a student model for preprocessing, respectively, to obtain an uncertain prediction result of the teacher module and an uncertain prediction result of the student module, respectively, wherein the steps of the preprocessing are as follows:

[0008] Input point cloud, obtain candidate point P through point cloud feature extraction network PointNet, translation operation and voting mechanism prop The spatial location and characteristics of

[0009] Based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box;

[0010] Combined with the spatial position of the predicted bounding box, a face-based pooling operation is used to obtain the geometric features of the face;

[0011] The geometric features of the surface and the discrete probability distribution features are spliced ​​together, input into the multi-layer perceptron, and the uncertainty prediction results are output;

[0012] The uncertain prediction results of the teacher module obtained by preprocessing are subjected to pseudo-label screening and weight assignment;

[0013] Use pseudo labels and weight scores to supervise the unlabeled data of the student model, and use Groud-Truth to supervise the labeled data of the student model;

[0014] The IoU-guided NMS is used on the uncertain prediction results of the teacher module to obtain the final detection results.

[0015] In some embodiments, the input point cloud is processed through a point cloud feature extraction network PointNet and a translation operation and voting mechanism to obtain the spatial position and features of the candidate points;

[0016] Among them, the input point cloud is sampled by the farthest point to obtain the seed point P seed , seed point P seed Use k-nearest neighbors and multi-layer perceptron to aggregate the features of surrounding points and repeat it twice as the input point cloud for the next stage;

[0017] Use the last output seed point P seed The features of are used to predict the probability of it being a foreground object point and its distance from the center of the object, and its spatial position is translated to the center of the object. The translated seed point P seed It is called candidate point P prop .

[0018] In some embodiments, the use of a multi-layer perceptron to predict the category of an object in space and the spatial distribution of each face of a bounding box comprises the following steps:

[0019] We design a side-aware parameterization method to represent bounding boxes. Specifically, given a candidate point P prop Instead of predicting the offset to the center point and the size of the object, we predict the prop The distance to each side. Based on the observation that the predicted probability distribution can measure uncertainty, we modify the bounding box parameterization from a deterministic approach to a probabilistic approach. We calculate the distance of each face relative to the candidate point P prop The distance of the position is divided into discrete grids in space, and the probability of the current face falling in the grid is predicted. The spatial position of the face is obtained by the expectation of the distribution. The formula is as follows:

[0020]

[0021] where s i is the distance value represented by each grid, P is the predicted probability, is the predicted spatial position of the face;

[0022] In some embodiments, the surface-based pooling operation obtains the geometric features of the surface, including the following steps:

[0023] The face-based pooling operation requires the selection of face points P side , a face point is a virtual grid point that contains a specific face of an object. More specifically, for the front face of an object, we divide the width and height of the object into D segments. Subsequently, we generate 2×D×D grid points in front and behind the front face of the object. For each face point, we find its k nearest neighbors and start from the seed point P seed The feature propagation is distance-weighted interpolation to the surface point P side Then, we transform all the face points P side Input into a point cloud feature extraction network PointNet to obtain the geometric feature F geo .

[0024] In some embodiments, the geometric features of the surface and the discrete probability distribution features are spliced ​​together, input into a multi-layer perceptron, and the uncertainty prediction result is output, including the following steps:

[0025] The discrete probability distribution feature is the statistic of the discrete distribution of the surface. We select the average value of the k values ​​with the largest probability, the variance of the distribution, and all the probability values ​​of the discrete distribution as the distribution feature Fx of the surface. ist ;

[0026] The distribution characteristics of the surface F dist The geometric characteristics of the surface F geo splice together;

[0027] The fused features are then input into a multi-layer perceptron, and the uncertainty measure u of the output surface is obtained through the Sigmoid activation layer. s , the formula is as follows:

[0028] u s=Sigmoid(MLP(Cat(F geo ,F dist )))

[0029] The above calculation method can be used to obtain the uncertainty measure of the surface. In order to guide the training of the uncertainty estimation module, we use the absolute error between the predicted surface and the true surface as the uncertainty label. To constrain, the label calculation formula is as follows:

[0030]

[0031] where y s is the spatial position of the real surface, is the spatial position of the predicted surface, MIN is the minimum value, α is the scale factor, which is set to 4 in the present invention; further, the present invention uses the absolute error between the predicted surface and the real surface and the prediction uncertainty u s The mean square error between them is used as the uncertainty regression loss Lx ncer :

[0032]

[0033] where U is the uncertainty estimate for each side of the object U = {u s |∈B}.

[0034] In some embodiments, the pseudo-label screening and weight assignment include the following steps:

[0035] Pseudo-label filtering and weight assignment module: This module consists of three parts: category-specific filter, IoU-guided low half NMS strategy and face-aware weight assignment; category-specific filter uses category confidence, foreground confidence and IoU prediction value to filter out low-quality pseudo-labels; IoU-guided low half NMS strategy discards only half of the prediction results with low IoU through IoU-guided low half NMS; face-aware weight assignment uses the uncertainty of the face to assign weights to each face of the pseudo-label. The weight assignment formula is:

[0036]

[0037] where q s is the mass fraction (weight) of the face, α 2 is the scaling factor, and finally we weight the loss using the quality score:

[0038]

[0039] q B Yes s The average value of reflects the global localization quality of the bounding box. In this way, we reduce the interference of faces with poor localization in model training.

[0040] In some embodiments, the method of using pseudo labels and weight scores to supervise the unlabeled data of the student model and using Groud-Truth to supervise the labeled data of the student model includes the following steps:

[0041] Since we decouple the bounding box into a separate regression problem for each face, the general bounding box regression loss cannot handle the positioning problem proposed by this invention well. Therefore, we use the rotated IoU loss and the face-aware smoothed L1 loss in the face-aware network:

[0042]

[0043] Face-based smooth L1 loss L reg (s) focuses on the local positioning of each face, which is beneficial to the subsequent uncertainty prediction. Rotation IoU loss L IoU (B) Focuses on the global localization of bounding boxes, which is robust to changes in shape and scale.

[0044] For unlabeled data, we use pseudo labels as supervisory signals, weighted by quality scores:

[0045]

[0046] In some embodiments, in the inference phase, using IoU-guided NMS on the output of the teacher model comprises the following steps:

[0047] The detection model outputs the bounding box of the object, the category of the object, the uncertainty of each face of the object and the IoU prediction value of the object; the present invention uses the IoU prediction value of the object as an additional reference value of the non-maximum suppression algorithm, that is, multiplying the IoU prediction value of the object and the category confidence of the object as the total confidence of the object, which is input into the non-maximum suppression algorithm, thereby filtering out repeated low-quality prediction results and obtaining high-precision detection results.

[0048] In a second aspect, the present application also proposes a semi-supervised 3D object detection system based on surface uncertainty estimation, comprising:

[0049] A preprocessing module is used to receive point clouds, input the point clouds into a preset teacher model and a student model for preprocessing, and obtain the uncertain prediction results of the teacher module and the uncertain prediction results of the student module respectively; the preselection module includes a candidate point feature extraction module, a surface spatial distribution prediction module, a surface geometric feature acquisition module and a surface discrete distribution prediction module;

[0050] The candidate point feature extraction module is used to input the point cloud and obtain the candidate point P through the point cloud feature extraction network PointNet, translation operation and voting mechanism. prop The spatial location and characteristics of

[0051] The spatial distribution prediction module of the surface is used to predict the spatial distribution of the surface based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box;

[0052] The geometric feature acquisition module of the face is used to obtain the geometric features of the face by combining the spatial position of the predicted bounding box and using the face-based pooling operation;

[0053] The surface discrete distribution prediction module is used to combine the geometric features of the surface with the discrete probability distribution features, input them into the multi-layer perceptron, and output the uncertainty prediction results;

[0054] The uncertainty prediction module is used for pseudo-label screening and weight distribution module, which is used to perform pseudo-label screening and weight distribution on the uncertain prediction results of the teacher module obtained by preprocessing;

[0055] The output module is used to supervise the unlabeled data of the student model using pseudo labels and weight scores, and to supervise the labeled data of the student model using Groud-Truth.

[0056] On the third aspect, the present application also proposes a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, a semi-supervised 3D target detection method based on digital twins as described above is adopted.

[0057] In a fourth aspect, the present application also proposes a computer-readable storage medium, which stores a computer program. When the computer program is loaded and executed by a processor, a semi-supervised 3D target detection method based on digital twins as described above is adopted.

[0058] Beneficial effects of the present invention:

[0059] This patent proposes a semi-supervised 3D target detection method, system and device based on digital twins. In order to solve the problem that the pseudo-labels generated by the teacher model are of poor quality and are not conducive to model training, we designed a discrete distribution prediction module and an uncertainty prediction module for faces to ensure that the positioning problem of the object bounding box can be decoupled into the positioning problem of several faces, and the prediction reliability evaluation of each face can be given. In addition, we designed a pseudo-label screening and weight distribution module to suppress the interference of the incorrect positioning of faces in the pseudo-labels on the model training process, so as to obtain better target detection performance. Through the design of this patent, the problems of unreasonable pseudo-label selection in 3D targets and insufficient utilization of effective information in the data set are largely solved, ultimately helping the model to accurately and efficiently detect object instances and identify object categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with the accompanying drawings.

[0061] Figure 1 This is the block diagram of the semi-supervised 3D object detection system based on digital twins for this application;

[0062] Figure 2 This is a flow chart of the pooling operation and uncertainty prediction of the surface perception in the embodiment of the present application;

[0063] Figure 3 This is a flow chart of the semi-supervised 3D target detection method based on digital twins in this application. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0066] See also Figure 2-3 , this application proposes a semi-supervised 3D target detection method based on digital twins, including:

[0067] The point cloud is received and input into a preset teacher model and a student model for preprocessing, respectively, to obtain an uncertain prediction result of the teacher module and an uncertain prediction result of the student module, respectively, wherein the steps of the preprocessing are as follows:

[0068] Input point cloud, obtain candidate point P through point cloud feature extraction network PointNet, translation operation and voting mechanism prop The spatial location and characteristics of

[0069] Based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box;

[0070] Combined with the spatial position of the predicted bounding box, a face-based pooling operation is used to obtain the geometric features of the face;

[0071] The geometric features of the surface and the discrete probability distribution features are spliced ​​together, input into the multi-layer perceptron, and the uncertainty prediction results are output;

[0072] The uncertain prediction results of the teacher module obtained by preprocessing are subjected to pseudo-label screening and weight assignment;

[0073] Use pseudo labels and weight scores to supervise the unlabeled data of the student model, and use Groud-Truth to supervise the labeled data of the student model;

[0074] The IoU-guided NMS is used on the uncertain prediction results of the teacher module to obtain the final detection results.

[0075] In some embodiments, the input point cloud is processed through a point cloud feature extraction network PointNet and a translation operation and voting mechanism to obtain the spatial position and features of the candidate points;

[0076] Among them, the input point cloud is sampled by the farthest point to obtain the seed point P seed , seed point P seed Use k-nearest neighbors and multi-layer perceptron to aggregate the features of surrounding points and repeat it twice as the input point cloud for the next stage;

[0077] Use the last output seed point P seed The features of are used to predict the probability of it being a foreground object point and its distance from the center of the object, and its spatial position is translated to the center of the object. The translated seed point P seed It is called candidate point P prop .

[0078] In some embodiments, the use of a multi-layer perceptron to predict the category of an object in space and the spatial distribution of each face of a bounding box comprises the following steps:

[0079] We design a side-aware parameterization method to represent bounding boxes. Specifically, given a candidate point P prop Instead of predicting the offset to the center point and the size of the object, we predict the prop The distance to each side. Based on the observation that the predicted probability distribution can measure uncertainty, we modify the bounding box parameterization from a deterministic approach to a probabilistic approach. We calculate the distance of each face relative to the candidate point P prop The distance of the position is divided into discrete grids in space, and the probability of the current face falling in the grid is predicted. The spatial position of the face is obtained by the expectation of the distribution. The formula is as follows:

[0080]

[0081] where s i is the distance value represented by each grid, P is the predicted probability, is the predicted spatial position of the face;

[0082] In some embodiments, the surface-based pooling operation obtains the geometric features of the surface, including the following steps:

[0083] The face-based pooling operation requires the selection of face points P side , a face point is a virtual grid point that contains a specific face of an object. More specifically, for the front face of an object, we divide the width and height of the object into D segments. Subsequently, we generate 2×D×D grid points in front and behind the front face of the object. For each face point, we find its k nearest neighbors and start from the seed point P seed The feature propagation is distance-weighted interpolation to the surface point P side Then, we transform all the face points P side Input into a point cloud feature extraction network PointNet to obtain the geometric feature F geo .

[0084] In some embodiments, the geometric features of the surface and the discrete probability distribution features are spliced ​​together, input into a multi-layer perceptron, and the uncertainty prediction result is output, including the following steps:

[0085] The discrete probability distribution feature is the statistic of the discrete distribution of the surface. We select the average value of the k values ​​with the largest probability, the variance of the distribution, and all the probability values ​​of the discrete distribution as the distribution feature F of the surface. dist ;

[0086] The distribution characteristics of the surface F dist The geometric characteristics of the surface F geo splice together;

[0087] The fused features are then input into a multi-layer perceptron, and the uncertainty measure u of the output surface is obtained through the Sigmoid activation layer. s , the formula is as follows:

[0088] u s =Sigmoid(MLP(Cat(F geo ,F dist )))

[0089] The above calculation method can be used to obtain the uncertainty measure of the surface. In order to guide the training of the uncertainty estimation module, we use the absolute error between the predicted surface and the true surface as the uncertainty label. To constrain, the label calculation formula is as follows:

[0090]

[0091] where y s is the spatial position of the real surface, is the spatial position of the predicted surface, MIN is the minimum value, α is the scale factor, which is set to 4 in the present invention; further, the present invention uses the absolute error between the predicted surface and the real surface and the prediction uncertainty u s The mean square error between them is used as the uncertainty regression loss L uncer :

[0092]

[0093] where U is the uncertainty estimate for each side of the object U = {u s |∈B}.

[0094] In some embodiments, the pseudo-label screening and weight assignment include the following steps:

[0095] Pseudo-label filtering and weight assignment module: This module consists of three parts: category-specific filter, IoU-guided low half NMS strategy and face-aware weight assignment; category-specific filter uses category confidence, foreground confidence and IoU prediction value to filter out low-quality pseudo-labels; IoU-guided low half NMS strategy discards only half of the prediction results with low IoU through IoU-guided low half NMS; face-aware weight assignment uses the uncertainty of the face to assign weights to each face of the pseudo-label. The weight assignment formula is:

[0096]

[0097] where q s is the mass fraction (weight) of the face, α 2 is the scaling factor, and finally we weight the loss using the quality score:

[0098]

[0099] q B Yes s The average value of reflects the global localization quality of the bounding box. In this way, we reduce the interference of faces with poor localization in model training.

[0100] In some embodiments, the method of using pseudo labels and weight scores to supervise the unlabeled data of the student model and using Groud-Truth to supervise the labeled data of the student model includes the following steps:

[0101] Since we decouple the bounding box into a separate regression problem for each face, the general bounding box regression loss cannot handle the positioning problem proposed by this invention well. Therefore, we use the rotated IoU loss and the face-aware smoothed L1 loss in the face-aware network:

[0102]

[0103] Face-based smooth L1 loss L reg (s) focuses on the local positioning of each face, which is beneficial to the subsequent uncertainty prediction. Rotation IoU loss L IoU (B) Focuses on the global localization of bounding boxes, which is robust to changes in shape and scale.

[0104] For unlabeled data, we use pseudo labels as supervisory signals, weighted by quality scores:

[0105]

[0106] In some embodiments, in the inference phase, using IoU-guided NMS on the output of the teacher model comprises the following steps:

[0107] The detection model outputs the bounding box of the object, the category of the object, the uncertainty of each face of the object and the IoU prediction value of the object; the present invention uses the IoU prediction value of the object as an additional reference value of the non-maximum suppression algorithm, that is, multiplying the IoU prediction value of the object and the category confidence of the object as the total confidence of the object, which is input into the non-maximum suppression algorithm, thereby filtering out repeated low-quality prediction results and obtaining high-precision detection results.

[0108] An embodiment of the present application discloses a semi-supervised 3D target detection system based on surface uncertainty estimation. The 3D target is the semantic label and bounding box of each object in the system input point cloud scene. The system includes: a candidate point feature extraction module, a surface discrete distribution prediction module, an uncertainty prediction module and a pseudo-label screening and weight allocation module.

[0109] Yet another embodiment of the present invention provides a semi-supervised 3D object detection system based on surface uncertainty estimation, comprising:

[0110] A preprocessing module is used to receive point clouds, input the point clouds into a preset teacher model and a student model for preprocessing, and obtain the uncertain prediction results of the teacher module and the uncertain prediction results of the student module respectively; the preselection module includes a candidate point feature extraction module, a surface spatial distribution prediction module, a surface geometric feature acquisition module and a surface discrete distribution prediction module;

[0111] The candidate point feature extraction module is used to input the point cloud and obtain the candidate point P through the point cloud feature extraction network PointNet, translation operation and voting mechanism. prop The spatial location and characteristics of

[0112] The spatial distribution prediction module of the surface is used to predict the spatial distribution of the surface based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box;

[0113] The geometric feature acquisition module of the face is used to obtain the geometric features of the face by combining the spatial position of the predicted bounding box and using the face-based pooling operation;

[0114] The surface discrete distribution prediction module is used to combine the geometric features of the surface with the discrete probability distribution features, input them into the multi-layer perceptron, and output the uncertainty prediction results;

[0115] The uncertainty prediction module is used for pseudo-label screening and weight distribution module, which is used to perform pseudo-label screening and weight distribution on the uncertain prediction results of the teacher module obtained by preprocessing;

[0116] The output module is used to supervise the unlabeled data of the student model using pseudo labels and weight scores, and to supervise the labeled data of the student model using Groud-Truth.

[0117] The overall execution process of the system is as follows Figure 1 As shown in Figure 1, assume that the input point cloud has N points, each of which contains position (x, y, z) information. First, we use the candidate point feature extraction module based on the point cloud feature extraction network PointNet to encode the position information and obtain the candidate point feature F. prop and position P prop Next, we will obtain the candidate points P through feature aggregation propBy inputting into the discrete distribution prediction module of the face, we can get the probability distribution of the 6 faces of the object predicted by each candidate point, as well as the orientation angle and object category of the object. After that, we input the distribution of all faces of the object and the geometric features obtained by the face-based pooling operation into the uncertainty prediction module respectively, and we can get the uncertainty measure of each face of the object. Furthermore, we use the pseudo-label screening and weight distribution module to filter the output of the teacher model, and use the high-quality pseudo-label information for the training of the student model. The final supervision information of the student model is the GT constraints of the labeled data and the pseudo-label constraints of the teacher model of the unlabeled data; the parameters of the teacher model are exponentially and dynamically updated using the parameters of the student model. In the inference stage, the output of the teacher model will be output through the IoU-guided NMS to remove duplicate prediction boxes and output the final prediction result. The specific details of different modules are as follows:

[0118] Candidate point feature extraction module. The candidate point feature extraction module mainly consists of three parts: point feature extraction network module PointNet, translation operation and voting operation.

[0119] Point feature extraction network module PointNet: The purpose of the point feature extraction network module is to aggregate seed points P seed The features of the multi-scale feature extraction are mainly realized by using multi-layer perceptron and farthest point sampling;

[0120]

[0121] Among them, max is the maximum pooling, MLP is the multi-layer perceptron, FPS is the farthest point sampling, and k represents the K nearest neighbors of the seed point.

[0122] The translation operation is to move the seed point P seed Input the feedforward network and output the offset from the current position to the center of the object, that is, the candidate point position P is obtained prop :

[0123] P prop =P seed +F seed W 1 +B 1

[0124] The voting operation is to input the candidate point into the feedforward network and output the probability P of the candidate point being inside the object. in :

[0125] P in =F seed W 2 +B 2

[0126] Where W 1 , W2 is the weight of the linear layer, B 1 , B 2 is the bias of the linear layer.

[0127] Discrete distribution prediction module of surface. The discrete distribution prediction module of surface mainly consists of two parts: object position prediction module and category prediction module.

[0128] Object position prediction module: given a candidate point P prop Instead of predicting the offset to the center point and the size of the object, we predict the prop The distance to each side. Based on the observation that the predicted probability distribution can measure uncertainty, we modify the bounding box parameterization from a deterministic approach to a probabilistic approach. We calculate the distance of each face relative to the candidate point P prop The distance of the position is divided into discrete grids in space, and the probability of the current face falling into the grid is predicted:

[0129] P(s=s i )=MLP(F prop )

[0130] We then use the spatial probability distribution to calculate the expected distribution to get the spatial position of the face. The formula is as follows:

[0131]

[0132] where s i is the distance value represented by each grid, P is the predicted probability, is the predicted spatial position of the face;

[0133] Object category prediction module: given a candidate point P prop The features of , we predict the category of the object, the formula is as follows:

[0134] P cls =MLP(F prop )

[0135] That is, through the above modules, we can obtain the categories and spatial probability distribution of candidate points.

[0136] Uncertainty prediction module. The uncertainty prediction module mainly consists of two parts: the surface geometric feature extraction module and the uncertainty prediction network module.

[0137] Surface geometric feature extraction module: Need to select surface point P side, a face point is a virtual grid point that contains a specific face of an object. More specifically, for the front face of an object, we divide the width and height of the object into D segments. Subsequently, we generate 2×D×D grid points in front and behind the front face of the object. For each face point, we find its k nearest neighbors and start from the seed point P seed The feature propagation is distance-weighted interpolation to the surface point F side :

[0138] F side =∑F i seed W i

[0139] where F i seed is the feature of the i-th nearest neighbor, W i is the weight value of distance weighted interpolation. After obtaining the spatial position and features of the surface point, it is input into the point feature extraction network PointNet to aggregate the geometric features of the surface of the prediction box F geo :

[0140]

[0141] Uncertainty prediction network module: The discrete probability distribution feature is the statistic of the discrete distribution of the surface. We select the average value of the k values ​​with the largest probability, the variance of the distribution, and all probability values ​​of the discrete distribution as the distribution feature F of the surface. dist ;

[0142] The distribution characteristics of the surface F dist The geometric characteristics of the surface F geo spliced ​​together; then the fused features are input into the multi-layer perceptron, and the uncertainty measure u of the output surface is obtained after the Sigmoid activation layer s , the formula is as follows:

[0143] u s =Sigmoid(MLP(Cat(F geo ,F dist )))

[0144] The above calculation method can be used to obtain the uncertainty measure of the surface. In order to guide the training of the uncertainty estimation module, we use the absolute error between the predicted surface and the true surface as the uncertainty label. To constrain, the label calculation formula is as follows:

[0145]

[0146] where y s is the spatial position of the real surface, is the spatial position of the predicted surface, MIN is the minimum value, α is the scale factor, which is set to 4 in the present invention; further, the present invention uses the absolute error between the predicted surface and the real surface and the prediction uncertainty u s The mean square error between them is used as the uncertainty regression loss L uncer :

[0147]

[0148] where U is the uncertainty estimate for each side of the object U = {u s |∈B}.

[0149] Pseudo-label filtering and weight assignment module. The pseudo-label filtering and weight assignment module mainly consists of three parts: category-specific filter, IoU-guided low-half NMS module and face-aware weight assignment module.

[0150] Category-specific filter: It uses category confidence, foreground confidence, and IoU predictions to filter out low-quality pseudo labels. The thresholds for the category score, foreground score, and IoU score are denoted as τ cls , τ obj and τ IoU ; The selected pseudo-label needs to satisfy:

[0151] P cls >τ cls ,P obj >τ obj ,P IoU >τ IoU ;

[0152] IoU-guided low-half NMS strategy: Suppress the noise in pseudo-labels caused by repeated bounding box predictions through the IoU-guided low-half non-maximum suppression strategy. For a bunch of highly overlapping pseudo-labels, we only discard half of the proposals with low predicted IoU;

[0153] Face-aware weight assignment module: Use the uncertainty of the face to assign weights to each face of the pseudo-label. The weight assignment formula is:

[0154]

[0155] where q s is the mass fraction (weight) of the face, α 2 is the scaling factor, and finally we weight the loss using the quality score:

[0156]

[0157] q B Yes sThe average value of reflects the global localization quality of the bounding box. In this way, we reduce the interference of faces with poor localization in model training.

[0158] The present invention can be widely used in systems in the fields of autonomous driving, robotic grasping, augmented reality, etc. to accurately locate and identify objects in point cloud scenes. In practice, it can be installed in front-end devices, robots, and autonomous vehicles in the form of software to provide real-time object target detection; it can also be installed in the back-end server to provide large quantities of object positioning and recognition results for 3D point cloud scenes.

[0159] Table 1: Comparison of experimental results on the ScanNetV2 dataset

[0160]

[0161] As shown in Table 1, we compared our method with the best performing method on the ScanNetV2 dataset. The results show that our method achieves the best performance in both AP@50 and AP@25 when using different amounts of labeled data, which verifies the effectiveness of our method.

[0162] An embodiment of the present application also discloses a terminal device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the computer program, any one of the semi-supervised 3D target detection methods based on digital twins in the above embodiments is adopted.

[0163] The terminal device may be a computer device such as a desktop computer, a laptop computer or a cloud server, and the terminal device includes but is not limited to a processor and a memory. For example, the terminal device may also include input and output devices, a network access device and a bus.

[0164] Among them, the processor can adopt a central processing unit (CPU). Of course, according to actual usage, other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. can also be adopted. The general-purpose processor can adopt a microprocessor or any conventional processor, etc., and this application does not impose any restrictions on this.

[0165] Among them, the memory can be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device, or it can be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC), a secure digital card (SD) or a flash memory card (FC) equipped on the terminal device, etc., and the memory can also be a combination of an internal storage unit and an external storage device of the terminal device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or is to be output, and this application does not impose any restrictions on this.

[0166] Among them, through this terminal device, any one of the semi-supervised 3D target detection methods based on digital twins in the above embodiments is stored in the memory of the terminal device, and is loaded and executed on the processor of the terminal device for easy use.

[0167] An embodiment of the present application also discloses a computer-readable storage medium, and the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, any one of the semi-supervised 3D target detection methods based on digital twins in the above embodiments is adopted.

[0168] Among them, the computer program can be stored in a computer-readable medium, the computer program includes computer program code, the computer program code can be in the form of source code, object code, executable file or certain middleware, etc. The computer-readable medium includes any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the computer-readable medium includes but is not limited to the above-mentioned components.

[0169] Among them, through this computer-readable storage medium, any one of the semi-supervised 3D target detection methods based on digital twins in the above embodiments is stored in a computer-readable storage medium, and is loaded and executed on a processor to facilitate the storage and application of the above method.

[0170] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A semi-supervised 3D object detection method based on digital twins, characterized in that: include: The point cloud is received and input into a preset teacher model and a student model for preprocessing, respectively, to obtain an uncertain prediction result of the teacher module and an uncertain prediction result of the student module, respectively, wherein the steps of the preprocessing are as follows: Input point cloud, obtain candidate point P through point cloud feature extraction network PointNet, translation operation and voting mechanism prop The spatial location and characteristics of Based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box; Combined with the spatial position of the predicted bounding box, a face-based pooling operation is used to obtain the geometric features of the face; The geometric features of the surface and the discrete probability distribution features are spliced ​​together, input into the multi-layer perceptron, and the uncertainty prediction results are output; The uncertain prediction results of the teacher module obtained by preprocessing are subjected to pseudo-label screening and weight assignment; Use pseudo labels and weight scores to supervise the unlabeled data of the student model, and use Groud-Truth to supervise the labeled data of the student model; The IoU-guided NMS is used on the uncertain prediction results of the teacher module to obtain the final detection results.

2. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: Input point cloud, and obtain the spatial position and features of candidate points through point cloud feature extraction network PointNet, translation operation and voting mechanism, including the following steps: The input point cloud is sampled by the farthest point to obtain the seed point. The seed point uses k nearest neighbor and multi-layer perceptron to aggregate the features of the surrounding points. The seed point after aggregation is used as the input point cloud of the next stage. The above operation is repeated twice to obtain the final output seed point. Use the features of the last output seed point to predict the probability that the last output seed point is a foreground object point and the distance between the last output seed point and the object center, and translate the spatial position of the last output seed point to the object center; the translated seed point is called the candidate point P prop .

3. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: Use a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box, including the following: Given a candidate point P prop The features and positions of the candidate points P prop The distance to each edge; Based on the observation that the predicted probability distribution measures uncertainty, the bounding box parameterization is revised from a deterministic approach to a probabilistic approach; Each face is relative to the candidate point P prop The distance of the position is divided into discrete grids in space, and the probability of the current face falling in the grid is predicted. The spatial position of the face is obtained by the expectation of the distribution. The formula is as follows: where s i is the distance value represented by each grid, P is the predicted probability, is the predicted spatial position of the face.

4. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: The surface-based pooling operation obtains the geometric features of the surface, including the following steps: The face-based pooling operation requires selecting face points P side , Pastry P side is a virtual grid point containing a specific face of an object; specifically, for the front face of an object, the width and height of the object are divided into D segments; 2×D×D grid points are generated in front and behind the front face of the object; for each face point P side , find each face point P side The k nearest neighbors of the seed point P seed The feature propagation is distance-weighted interpolation to the surface point P side ; All the surface points P side Input into a point cloud feature extraction network PointNet to obtain the geometric feature F geo .

5. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: The geometric features and discrete probability distribution features of the surface are spliced ​​together, input into a multi-layer perceptron, and the uncertainty prediction result is output, including the following steps: The discrete probability distribution feature is the statistic of the discrete distribution of the surface. The average value of the k values ​​with the largest probability, the variance of the distribution, and all the probability values ​​of the discrete distribution are selected as the distribution feature F of the surface. dist ; The distribution characteristics of the surface F dist The geometric characteristics of the surface F geo splice together; The fused features are input into the multi-layer perceptron, and the uncertainty measure u of the output surface is obtained after the Sigmoid activation layer. s , the formula is as follows: u s =Sigmoid(MLP(Cat(F geo ,F dist ))) Get the uncertainty measure of the surface, using the absolute error between the predicted surface and the true surface as the uncertainty label Constraints are imposed to guide the training of the uncertainty estimation module. The label calculation formula is as follows: where y s is the spatial position of the real surface, is the spatial position of the predicted surface, MIN is the minimum value, α is the scale factor, and the absolute error between the predicted surface and the true surface is used. and the prediction uncertainty u s The mean square error between them is used as the uncertainty regression loss L uncer : where U is the uncertainty estimate for each side of the object U = {u s |∈B}.

6. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: The pseudo-label filtering and weighting are implemented by a pseudo-label filtering and weighting module, wherein the pseudo-label filtering and weighting module includes a category-specific filter, an IoU-guided low-semi-NMS strategy, and a face-aware weighting; the category-specific filter uses category confidence, foreground confidence, and IoU prediction values ​​to filter out low-quality pseudo-labels; The IoU-guided low half NMS strategy discards half of the prediction results with lower IoU through the IoU-guided low half NMS; the face-aware weight allocation uses the uncertainty of the face to assign weights to each face of the pseudo-label. The weight allocation formula is: where q s is the face quality score, α2 is the scale factor, and the loss is weighted using the quality score: q B Yes s The average value of , reflects the global localization quality of the bounding box.

7. The semi-supervised 3D target detection method based on digital twins according to claim 1, characterized in that: The method of using pseudo labels and weight scores to supervise the unlabeled data of the student model and using Groud-Truth to supervise the labeled data of the student model includes the following steps: Use rotated IoU loss and face-aware smooth L1 loss in the face-aware network: Use pseudo labels as supervisory signals for unlabeled data, weighted by quality scores:

8. A semi-supervised 3D object detection system based on surface uncertainty estimation, characterized in that: include: A preprocessing module is used to receive point clouds, input the point clouds into a preset teacher model and a student model for preprocessing, and obtain the uncertain prediction results of the teacher module and the uncertain prediction results of the student module respectively; the preselection module includes a candidate point feature extraction module, a surface spatial distribution prediction module, a surface geometric feature acquisition module and a surface discrete distribution prediction module; The candidate point feature extraction module is used to input the point cloud and obtain the candidate point P through the point cloud feature extraction network PointNet, translation operation and voting mechanism. prop The spatial location and characteristics of The spatial distribution prediction module of the surface is used to predict the spatial distribution of the surface based on the candidate point P prop Features and spatial position, using a multi-layer perceptron to predict the category of objects in space and the spatial distribution of each face of the bounding box; The geometric feature acquisition module of the face is used to obtain the geometric features of the face by combining the spatial position of the predicted bounding box and using the face-based pooling operation; The surface discrete distribution prediction module is used to combine the geometric features of the surface with the discrete probability distribution features, input them into the multi-layer perceptron, and output the uncertainty prediction results; The uncertainty prediction module is used for pseudo-label screening and weight distribution module, which is used to perform pseudo-label screening and weight distribution on the uncertain prediction results of the teacher module obtained by preprocessing; The output module is used to supervise the unlabeled data of the student model using pseudo labels and weight scores, and to supervise the labeled data of the student model using Groud-Truth.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on a processor. When the processor loads and executes the computer program, a semi-supervised 3D target detection method based on digital twins described in any one of claims 1 to 7 is adopted.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by the processor, a semi-supervised 3D target detection method based on digital twins as described in any one of claims 1 to 7 is adopted.

Images