Target detection model training method and device and target detection method and device

Abstract

The invention relates to a target detection model training method and device, and a target detection method and device, and is applied to the technical field of computer vision, and the model training method comprises the steps: extracting a 3D voxel feature map through a first 3D feature extraction network, and processing the 3D voxel feature map through a first open vocabulary classification head and a first regression head, obtaining a K-dimensional feature vector and a prediction bounding box, and determining regression loss based on a bounding box label and the prediction bounding box; processing the category label by using a text encoder to obtain a K-dimensional text vector; determining a first classification loss based on the similarity of the K-dimensional feature vector and the K-dimensional text vector; determining a first total loss based on the first classification loss and the regression loss, and adjusting network parameters based on the first total loss; and when a first convergence condition is satisfied, constructing a target detection model according to the text encoder, the obtained second 3D feature extraction network, the second regression head and the second open vocabulary classification head. And 3D open set detection can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a target detection model training method and device and a target detection method and device. BACKGROUND

[0002] Training of a 3D target detection model is a core technology in the fields of autonomous driving, robot navigation, and augmented reality, and aims to identify and locate objects in three-dimensional space from point cloud or RGB-D data.

[0003] In recent years, 3D detection methods based on deep learning have made significant progress. However, most methods can only identify pre-defined classes in the training set, and are difficult to adapt to open scenarios in the real world (such as unknown objects or new classes). Moreover, 3D labeling (3D bounding boxes in point cloud) requires a large amount of manual work, and different data sets have different labeling standards (mainly different labeled classes), making it difficult to jointly train multi-source data (data from different projects). SUMMARY

[0004] To solve the above technical problems, the present application provides a target detection model training method and device, a target detection method and device, an electronic device, a storage medium, and a computer program product.

[0005] According to a first aspect of the present application, a target detection model training method is provided, comprising: obtaining a plurality of first sample point cloud data, a bounding box label corresponding to each first sample point cloud data, and a class label corresponding to each bounding box label; for a single first sample point cloud data, inputting the first sample point cloud data into a first 3D feature extraction network to obtain a first 3D voxel feature map, and processing the first 3D voxel feature map using a first open vocabulary classification head to obtain a first K-dimensional feature vector; K is a positive integer; processing the first 3D voxel feature map using a first regression head to obtain a predicted bounding box, determining a regression loss based on the bounding box label and the predicted bounding box; processing the class label corresponding to the first sample point cloud data using a text encoder to obtain a first K-dimensional text vector; determining a first classification loss based on the similarity of the first K-dimensional feature vector and the first K-dimensional text vector; determining a first total loss based on the first classification loss and the regression loss, and adjusting network parameters in the first 3D feature extraction network, the first regression head, and the first open vocabulary classification head based on the first total loss; When the first convergence condition is met, a trained second 3D feature extraction network, a second regression head and a second open vocabulary classification head are obtained, and a target detection model is constructed according to the text encoder, the second 3D feature extraction network, the second regression head and the second open vocabulary classification head.

[0006] Optionally, the constructing the target detection model according to the text encoder, the second 3D feature extraction network, the second regression head and the second open vocabulary classification head comprises: a plurality of second sample point cloud data, a boundary box label corresponding to each second sample point cloud data and a class label corresponding to each boundary box label are obtained; N is a positive integer less than or equal to M; the target layer in the second open vocabulary classification head is copied into M target layers in parallel with the same weight coefficient to obtain a third open vocabulary classification head; M is an integer greater than 1; for a single second sample point cloud data, the second sample point cloud data is input into the second 3D feature extraction network to obtain a second 3D voxel feature map, and the second 3D voxel feature map is processed by the third open vocabulary classification head to obtain M second K-dimensional feature vectors; the N class labels of the single boundary box label corresponding to the second sample point cloud data are respectively processed by the text encoder to obtain N second K-dimensional text vectors; based on the similarity of the M second K-dimensional feature vectors and the N second K-dimensional text vectors, a second classification loss is determined; and based on the second classification loss, the weight coefficients corresponding to the M target layers in parallel in the third open vocabulary classification head and the network parameters in the M target layers are adjusted; When the second convergence condition is met, a trained fourth open vocabulary classification head is obtained, and a target detection model is constructed according to the text encoder, the second 3D feature extraction network, the second regression head and the fourth open vocabulary classification head.

[0007] Optionally, the processing the class label corresponding to the first sample point cloud data by the text encoder to obtain a first K-dimensional text vector comprises: when a single boundary box label corresponds to a plurality of class labels, any class label is processed by the text encoder to obtain a first K-dimensional text vector; or, each class label is processed by the text encoder, and the average value of the plurality of K-dimensional text vectors obtained is taken as the first K-dimensional text vector; or, the plurality of class labels are processed by the text encoder to obtain a first K-dimensional text vector.

[0008] Optionally, if N is 1, the determining the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors comprises: calculating the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vectors respectively, and determining the second classification loss based on the maximum similarity; if N is an integer greater than 1, the determining the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors comprises: for each second K-dimensional text vector, calculating the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector, obtaining M similarities; based on the M similarities corresponding to the N second K-dimensional text vectors respectively, using the Hungarian algorithm to obtain target similarities corresponding to the N second K-dimensional text vectors respectively, and determining the second classification loss based on the target similarities; wherein the sum of the target similarities corresponding to the N second K-dimensional text vectors is maximum, and the target similarities corresponding to the N second K-dimensional text vectors are calculated by the N second K-dimensional text vectors respectively and the second K-dimensional feature vectors output by different target layers.

[0009] According to a second aspect of the present application, a target detection method is provided, comprising: obtaining point cloud data to be detected and a target category; inputting the point cloud data to be detected and target category information into a target detection model to obtain a detection frame belonging to the target category; wherein the target detection model is trained based on the method of the first aspect.

[0010] According to a third aspect of the present application, a training device of a target detection model is provided, comprising: a training data set acquisition module for acquiring a plurality of first sample point cloud data, a boundary box label corresponding to each first sample point cloud data, and a category label corresponding to each boundary box label; a feature extraction module for inputting the first sample point cloud data into a first 3D feature extraction network to obtain a first 3D voxel feature map, and processing the first 3D voxel feature map using a first open vocabulary classification head to obtain a first K-dimensional feature vector; K is a positive integer; a regression loss determination module for processing the first 3D voxel feature map using a first regression head to obtain a predicted boundary box, and determining a regression loss based on the boundary box label and the predicted boundary box; a text encoding module for processing the category label corresponding to the first sample point cloud data using a text encoder to obtain a first K-dimensional text vector; a classification loss determination module configured to determine a first classification loss based on a similarity between the first K-dimensional feature vector and the first K-dimensional text vector; a model training module configured to determine a first total loss based on the first classification loss and the regression loss, and adjust network parameters in the first 3D feature extraction network, the first regression head and the first open-vocabulary classification head based on the first total loss; a target detection model generation module configured to obtain a trained second 3D feature extraction network, a second regression head and a second open-vocabulary classification head when a first convergence condition is met, and construct a target detection model according to the text encoder, the second 3D feature extraction network, the second regression head and the second open-vocabulary classification head.

[0011] Optionally, the target detection model generation module is specifically configured to obtain a trained second 3D feature extraction network and a second open-vocabulary classification head when the first convergence condition is met, copy a target layer in the second open-vocabulary classification head into M target layers in parallel with the same weight coefficient to obtain a third open-vocabulary classification head, where M is an integer greater than 1; and obtain a plurality of second sample point cloud data, a boundary box label corresponding to each second sample point cloud data and a class label corresponding to each boundary box label; N is a positive integer less than or equal to M; for a single second sample point cloud data, input the second sample point cloud data into the second 3D feature extraction network to obtain a second 3D voxel feature map, and process the second 3D voxel feature map using the third open-vocabulary classification head to obtain M second K-dimensional feature vectors; process N class labels of a single boundary box label corresponding to the second sample point cloud data using a text encoder to obtain N second K-dimensional text vectors; and determine a second classification loss based on a similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors, adjust weight coefficients corresponding to M target layers in parallel in the third open-vocabulary classification head and network parameters in the M target layers based on the second classification loss, obtain a trained fourth open-vocabulary classification head when a second convergence condition is met, and construct a target detection model according to the text encoder, the second 3D feature extraction network, the second regression head and the fourth open-vocabulary classification head.

[0012] Optionally, the text encoding module is specifically used to process any category label using a text encoder to obtain a first K-dimensional text vector when a single bounding box label corresponds to multiple category labels; or, to process each category label using a text encoder and use the average of the multiple K-dimensional text vectors obtained as the first K-dimensional text vector; or, to process multiple category labels using a text encoder to obtain the first K-dimensional text vector.

[0013] Optionally, if N is 1, the object detection model generation module is specifically used to determine the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors through the following steps: Calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector, and determine the second classification loss based on the maximum similarity; If N is an integer greater than 1, the object detection model generation module is specifically used to determine the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors through the following steps: For each second K-dimensional text vector, calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector to obtain M similarity scores; Based on the M similarities corresponding to the N second K-dimensional text vectors, the Hungarian algorithm is used to obtain the target similarities corresponding to the N second K-dimensional text vectors, and the second classification loss is determined based on the target similarities. Among them, the sum of the target similarities corresponding to the N second K-dimensional text vectors is the largest, and the target similarities corresponding to the N second K-dimensional text vectors are calculated by comparing the N second K-dimensional text vectors with the second K-dimensional feature vectors output by different target layers.

[0014] According to a fourth aspect of this application, a target detection device is provided, comprising: The data acquisition module is used to acquire the point cloud data to be detected and the target category; The target detection module is used to input the point cloud data to be detected and the target category information into the target detection model to obtain a detection box belonging to the target category; wherein, the target detection model is trained based on the method described in the first aspect.

[0015] According to a fifth aspect of this application, an electronic device is provided, comprising: a processor configured to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method described in the first or second aspect.

[0016] According to a sixth aspect of this application, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first or second aspect.

[0017] According to a seventh aspect of this application, a computer program product is provided that, when the computer program product is run on a computer, causes the computer to perform the method described in the first or second aspect.

[0018] The technical solution provided in this application has the following advantages compared with the prior art: The first 3D voxel feature map of point cloud data is extracted by a first 3D feature extraction network. The first 3D voxel feature map is then processed by a first regression head and a first open vocabulary classification head to obtain predicted bounding boxes and a first K-dimensional feature vector. The predicted bounding box is used to calculate the regression loss, and the similarity between the first K-dimensional feature vector and the category feature vector generated by the text encoder is calculated. The classification loss is then calculated based on the similarity. The network parameters in the first 3D feature extraction network, the first regression head, and the first open vocabulary classification head are adjusted based on these two losses to finally train an object detection model. This model is then used to perform open vocabulary classification and object detection. Compared to existing closed-set object detection algorithms, this approach solves the problem of difficulty in jointly training multi-source data (data from different projects) due to significant differences in labeled categories across different 3D detection datasets. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the target detection model in the embodiments of this application; Figure 2 This is a flowchart of a training method for an object detection model in an embodiment of this application; Figure 3 This is another flowchart of the training method for the target detection model in the embodiments of this application; Figure 4 This is a flowchart of a target detection method in an embodiment of this application; Figure 5This is a schematic diagram of a training device for the target detection model in an embodiment of this application; Figure 6 This is a schematic diagram of the target detection device in one embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. Detailed Implementation

[0022] To better understand the above-mentioned objectives, features, and advantages of this application, the solution of this application will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of this application, but this application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of this application, and not all embodiments.

[0024] Figure 1 This is a schematic diagram of the target detection model in this embodiment. The target detection model includes: a 3D feature extraction network, a regression head, an open vocabulary classification head, and a text encoder. When the input data is point cloud data, the 3D feature extraction network is used to extract 3D voxel feature maps from the point cloud data (e.g., LiDAR or RGB-D depth point cloud information). When the input data is point cloud data and RGB images, the 3D feature extraction network can specifically be a 3D-2D fusion feature extraction network, obtaining 3D voxel feature maps based on the camera's intrinsic and extrinsic parameters. Specifically, the RGB image can be converted into a form with the same dimensions as the 3D voxel grid, and then merged with the point cloud voxel grid.

[0025] The regression head is used to predict the location, size, and orientation of the target. The text encoder is a pre-trained model with fixed network parameters, used to encode a given arbitrary category name (such as the English words for "car" and "pedestrian") into a text vector. The open vocabulary classification head is used to map 3D voxel feature maps to multi-dimensional (e.g., 512-dimensional) vectors and calculate the similarity with the text vectors output by the text encoder, thus achieving open vocabulary classification.

[0026] In this embodiment, the 3D feature extraction network can be, for example, the VoxelNeXt backbone or the BEVFusion (Bird's-eye View Multimodal Fusion Network) backbone. VoxelNeXt (Voxel Next Generation Detector) is a deep learning model in the field of 3D object detection. The VoxelNeXt backbone is one of the core components of VoxelNeXt, used for feature extraction and hierarchical representation learning, providing high-quality feature support for subsequent 3D object detection, segmentation, and other tasks. The text encoder can be, for example, CLIP (Contrastive Language-Image Pre-training), Sentence-BERT, or a large language model.

[0027] See Figure 2 , Figure 2 This is a flowchart of a training method for an object detection model in an embodiment of this application, which may include the following steps: Step S202: Obtain multiple first sample point cloud data, the bounding box label corresponding to each first sample point cloud data, and the category label corresponding to each bounding box label.

[0028] Training data can consist only of point cloud data, or it can consist of both point cloud data and image data. If the training data includes image data, camera intrinsic and extrinsic parameters can be obtained beforehand. This explanation uses a training data consisting only of point cloud data as an example. The first sample point cloud data is the input data. The bounding box label and the category label corresponding to each bounding box label are the label data. The first sample point cloud data can contain multiple bounding boxes, each with corresponding category information (e.g., "car" or "pedestrian"). Therefore, the first sample point cloud data can correspond to multiple sets of bounding box labels, each with a corresponding category label.

[0029] Step S204: For a single first sample point cloud data, input the first sample point cloud data into the first 3D feature extraction network to obtain a first 3D voxel feature map, and process the first 3D voxel feature map using the first open vocabulary classification head to obtain a first K-dimensional feature vector. K is a positive integer, such as K = 512, etc.

[0030] The first 3D feature extraction network, the first open vocabulary classification head, and the first regression head are all untrained network structures. The network parameters in the first 3D feature extraction network, the first open vocabulary classification head, and the first regression head are trained using training data to finally obtain the object detection model.

[0031] Step S206: Process the first 3D voxel feature map using the first regression head to obtain the predicted bounding box. Determine the regression loss based on the bounding box label and the predicted bounding box.

[0032] The predicted bounding box can include: center point, length, width, height, rotation angle, etc. The regression loss can be the Smooth-L1 loss of VoxelNeXt to optimize the position / size prediction.

[0033] Step S208: Use a text encoder to process the category labels corresponding to the first sample point cloud data to obtain the first K-dimensional text vector.

[0034] As mentioned earlier, each bounding box label has a corresponding category label. A single bounding box label can correspond to one category label or multiple category labels. For example, if a bounding box contains "children," then the category label corresponding to the bounding box label could include "person," "children," etc. Optionally, when a single bounding box label corresponds to multiple category labels, a text encoder is used to process any one category label to obtain a first K-dimensional text vector; or, a text encoder is used to process each category label, and the average of the resulting multiple K-dimensional text vectors is used as the first K-dimensional text vector; or, a text encoder is used to process multiple category labels to obtain the first K-dimensional text vector. For example, CLIP can process both a single word and a sentence, ultimately outputting a 512-dimensional text vector.

[0035] Step S210: Determine the first classification loss based on the similarity between the first K-dimensional feature vector and the first K-dimensional text vector.

[0036] The higher the similarity between the first K-dimensional feature vector and the first K-dimensional text vector, the lower the first classification loss. Optionally, Focal Loss can be used for the first classification loss to address the class imbalance problem.

[0037] Step S212: Determine the first total loss based on the first classification loss and regression loss, and adjust the network parameters in the first 3D feature extraction network, the first regression head, and the first open vocabulary classification head based on the first total loss.

[0038] Step S214: When the first convergence condition is met, the trained second 3D feature extraction network, second regression head, and second open vocabulary classification head are obtained.

[0039] Step S216: Construct an object detection model based on the text encoder, the second 3D feature extraction network, the second regression head, and the second open vocabulary classification head.

[0040] When the first convergence condition is met, the first 3D feature extraction network is updated to the second 3D feature extraction network, the first regression head is updated to the second regression head, and the first open vocabulary classification head is updated to the second open vocabulary classification head. According to... Figure 1 The structure of the object detection model shown can be constructed based on a text encoder, a second 3D feature extraction network, a second regression head, and a second open vocabulary classification head.

[0041] The training method of the object detection model in this application embodiment extracts a first 3D voxel feature map from point cloud data through a first 3D feature extraction network. The first 3D voxel feature map is then processed using a first regression head and a first open vocabulary classification head to obtain a predicted bounding box and a first K-dimensional feature vector. The predicted bounding box is used to calculate the regression loss, and the similarity between the first K-dimensional feature vector and the category feature vector generated by the text encoder is calculated. A classification loss is then calculated based on the similarity. The network parameters in the first 3D feature extraction network, the first regression head, and the first open vocabulary classification head are adjusted based on these two losses to finally train the object detection model. This model is then used to achieve open vocabulary classification and object detection. Compared to existing closed-set object detection algorithms, this method solves the problem of difficulty in jointly training multi-source data (data from different projects) due to significant differences in labeled categories across different 3D detection datasets.

[0042] Because a single bounding box label can correspond to multiple category labels, such as both "woman" and "person," using all labels during open-set object detection training will result in image / point cloud features that are close to both the text features of "woman" and "person," but it's impossible for the matching degree to reach 100% on both sides. If the text encoder is insufficient and the difference between "woman" and "person" is significant, the final matching degree between image / point cloud features and text features may be low. Therefore, multi-label classification will lead to a low matching degree, and theoretically, a perfect match (match degree equal to 1) is impossible.

[0043] To address this technical problem, in this embodiment, the open vocabulary classification head can be changed from a single head to M heads. M is fixed, for example, 5, representing the maximum number of labels a single object can have. For example, the same object could be labeled "person," "man" (or "woman"), or "smoker," "not wearing a helmet," "jaywalker," etc. Typically, most other objects do not have as many labels as M. For instance, the last layer or several layers of the open vocabulary classification head can be copied into M parallel heads, denoted as M1, M2, M3, M4, and M5, with corresponding weights a1, a2, a3, a4, and a5, respectively. During matching, the M features are matched against each text feature, and the highest matching degree among the M is used. In this way, even with multiple labels (such as "woman" and "person"), multiple patterns can be learned.

[0044] During training, if an object corresponds to only one category label, the category with the highest matching degree among M1 to M5 is selected, and the loss is calculated based on this highest matching degree. In principle, this weight will have a higher matching degree, while the other M-1 category weights will not participate in training, potentially leading to lower overall training efficiency. Furthermore, in many cases, different datasets have different levels of labeling. For example, dataset 1 may only label "person," while dataset 2 may label "man" or "woman," a situation known as "generalized multi-labeling." Generally, the relationships between words are unknown; for example, "person" and "smoker" are related by the former including the latter. With a large amount of data and many label types, it is difficult to process each one individually. Therefore, it is advisable to first... Figure 2 The method in this embodiment yields a trained second 3D feature extraction network, a second regression head, and a second open-vocabulary classification head. Then, only the M replicated parallel network layers are trained, focusing on training data with narrowly defined multi-labels. This ensures that all network layers participate in training. Additionally, to prevent overfitting, single-label sample data can also be used in this step, although their weight can be appropriately reduced.

[0045] See Figure 3 , Figure 3 This is another flowchart illustrating the training method of the target detection model in this application. Figure 2 Following step S214 in the embodiment, the following steps are also included: Step S302: Obtain multiple second sample point cloud data, the bounding box label corresponding to each second sample point cloud data, and the category label corresponding to each bounding box label; N is a positive integer less than or equal to M.

[0046] In this embodiment of the application, the second sample point cloud data and the first sample point cloud data can be the same or different, that is, the same training set or different training sets can be used in the two training sessions.

[0047] Step S304: Copy the target layer in the second open vocabulary classification head into M parallel target layers with the same weight coefficient to obtain the third open vocabulary classification head; M is an integer greater than 1.

[0048] The target layer in the second open vocabulary classifier can be the last layer or the last few layers of the second open vocabulary classifier, resulting in a third open vocabulary classifier containing M classifiers. Thus, after processing the second 3D voxel feature map, M second K-dimensional feature vectors can be output.

[0049] Step S306: For a single second sample point cloud data, input the second sample point cloud data into the second 3D feature extraction network to obtain the second 3D voxel feature map, and use the third open vocabulary classification head to process the second 3D voxel feature map to obtain M second K-dimensional feature vectors.

[0050] Step S308: Use a text encoder to process the N category labels of a single bounding box label corresponding to the second sample point cloud data to obtain N second K-dimensional text vectors.

[0051] Step S310: Determine the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors; adjust the weight coefficients corresponding to the M target layers connected in parallel in the third open vocabulary classification head and the network parameters in the M target layers based on the second classification loss.

[0052] In this embodiment of the application, in addition to adjusting the weight coefficients and network parameters of the M target layers connected in parallel in the third open vocabulary classifier solely based on the second classification loss, the second classification loss and regression loss can also be combined to adjust the weight coefficients and network parameters of the M target layers connected in parallel in the third open vocabulary classifier.

[0053] In some embodiments, if N is 1, the similarity between M second-K-dimensional feature vectors and the second-K-dimensional text vector is calculated, and the second classification loss is determined based on the maximum similarity. If N is an integer greater than 1, it indicates a multi-class labeling scenario. For each second-K-dimensional text vector, the similarity between M second-K-dimensional feature vectors and the second-K-dimensional text vector is calculated, resulting in M ​​similarities. Based on the M similarities corresponding to the N second-K-dimensional text vectors (i.e., N×M similarities), the Hungarian algorithm is used to obtain the target similarities corresponding to the N second-K-dimensional text vectors, and the second classification loss is determined based on the target similarities. The sum of the target similarities corresponding to the N second-K-dimensional text vectors is maximized, and the target similarities corresponding to the N second-K-dimensional text vectors are calculated by comparing the N second-K-dimensional text vectors with the second-K-dimensional feature vectors output from different target layers. For example, when M is 5, the target similarities corresponding to the N second-K-dimensional text vectors correspond to N of M1, M2, M3, M4, and M5, and cannot be repeated.

[0054] For example, if an object has two category labels, the following table can be calculated, recording the matching degree between any two as shown in Table 1: Table 1 M1 M2 M3 M4 M5 A 0.7 0.01 0.8 0.01 0.03 B 0.01 0.02 0.6 0.3 0.02 Using the Hungarian algorithm, the matching method with the highest overall matching score is: A corresponds to M1, B corresponds to M3, with a total matching score of 1.3. The overall matching scores obtained by other matching methods are all less than 1.3.

[0055] Step S312: When the second convergence condition is met, the trained fourth open vocabulary classification head is obtained, and an object detection model is constructed based on the text encoder, the second 3D feature extraction network, the second regression head, and the fourth open vocabulary classification head.

[0056] The 3D open-set object detection method in this application, by designing multiple open-vocabulary classification heads, can achieve multi-class label object detection. Furthermore, through multi-level training, the trained object detection model achieves high accuracy.

[0057] See Figure 4 , Figure 4 This is a flowchart of a target detection method in an embodiment of this application, including the following steps: Step S402: Obtain the point cloud data to be detected and the target category.

[0058] Step S404: Input the point cloud data to be detected and the target category information into the target detection model to obtain the detection boxes belonging to the target category; wherein, the target detection model is based on Figure 2 Implementation examples or Figure 3The training was obtained using the method described in the example.

[0059] The target category is the category of the object to be detected from the point cloud data. The point cloud data and target category information are input into the target detection model, and processed by the text encoder, second 3D feature extraction network, second regression head and fourth open vocabulary classification head in the target detection model, which can output the detection box belonging to the target category.

[0060] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0061] This application also provides a training device for an object detection model, see [link to relevant documentation]. Figure 5 The training device 500 for the object detection model includes: The training dataset acquisition module 502 is used to acquire multiple first sample point cloud data, the bounding box label corresponding to each first sample point cloud data, and the category label corresponding to each bounding box label. The feature extraction module 504 is used to input the first sample point cloud data into the first 3D feature extraction network to obtain the first 3D voxel feature map for a single first sample point cloud data, and to process the first 3D voxel feature map using the first open vocabulary classification head to obtain the first K-dimensional feature vector; K is a positive integer. The regression loss determination module 506 is used to process the first 3D voxel feature map using the first regression head to obtain the predicted bounding box, and to determine the regression loss based on the bounding box label and the predicted bounding box. The text encoding module 508 is used to process the category label corresponding to the first sample point cloud data using a text encoder to obtain the first K-dimensional text vector; The classification loss determination module 510 is used to determine the first classification loss based on the similarity between the first K-dimensional feature vector and the first K-dimensional text vector; The model training module 512 is used to determine the first total loss based on the first classification loss and regression loss, and to adjust the network parameters in the first 3D feature extraction network, the first regression head and the first open vocabulary classification head based on the first total loss; The object detection model generation module 514 is used to obtain the trained second 3D feature extraction network, second regression head and second open vocabulary classification head when the first convergence condition is met, and to construct an object detection model based on the text encoder, the second 3D feature extraction network, the second regression head and the second open vocabulary classification head.

[0062] Optionally, the object detection model generation module 514 is specifically used to, when the first convergence condition is met, obtain the trained second 3D feature extraction network and the second open vocabulary classification head, and copy the target layer in the second open vocabulary classification head into M parallel target layers with the same weight coefficients to obtain the third open vocabulary classification head; M is an integer greater than 1; and, Acquire multiple second sample point cloud data, the bounding box label corresponding to each second sample point cloud data, and the category label corresponding to each bounding box label; N is a positive integer less than or equal to M; for a single second sample point cloud data, input the second sample point cloud data into a second 3D feature extraction network to obtain a second 3D voxel feature map, and process the second 3D voxel feature map using a third open vocabulary classification head to obtain M second K-dimensional feature vectors; use a text encoder to process the N category labels of a single bounding box label corresponding to the second sample point cloud data to obtain N second K-dimensional text vectors; and, The second classification loss is determined based on the similarity between M second-K-dimensional feature vectors and N second-K-dimensional text vectors. The weight coefficients of the M target layers connected in parallel in the third open vocabulary classifier head and the network parameters in the M target layers are adjusted based on the second classification loss. When the second convergence condition is met, the trained fourth open vocabulary classifier head is obtained. The target detection model is constructed based on the text encoder, the second 3D feature extraction network, the second regression head and the fourth open vocabulary classifier head.

[0063] Optionally, the text encoding module 508 is specifically used to process any category label using a text encoder to obtain a first K-dimensional text vector when a single bounding box label corresponds to multiple category labels; or, to process each category label using a text encoder and use the average of the multiple K-dimensional text vectors obtained as the first K-dimensional text vector; or, to process multiple category labels using a text encoder to obtain the first K-dimensional text vector.

[0064] Optionally, if N is 1, the object detection model generation module 514 is specifically used to determine the second classification loss based on the similarity between M second K-dimensional feature vectors and N second K-dimensional text vectors through the following steps: Calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector, and determine the second classification loss based on the maximum similarity. If N is an integer greater than 1, the object detection model generation module 514 is specifically used to determine the second classification loss based on the similarity between M second K-dimensional feature vectors and N second K-dimensional text vectors through the following steps: For each second K-dimensional text vector, calculate the similarity between M second K-dimensional feature vectors and the second K-dimensional text vector to obtain M similarity scores; Based on the M similarities corresponding to the N second-K-dimensional text vectors, the Hungarian algorithm is used to obtain the target similarity corresponding to the N second-K-dimensional text vectors, and the second classification loss is determined based on the target similarity. Among them, the sum of the target similarities corresponding to the N second K-dimensional text vectors is the largest, and the target similarities corresponding to the N second K-dimensional text vectors are calculated by comparing the N second K-dimensional text vectors with the second K-dimensional feature vectors output by different target layers.

[0065] This application also provides a target detection device, see [link to relevant documentation]. Figure 6 The target detection device 600 includes: Data acquisition module 602 is used to acquire the point cloud data to be detected and the target category; The target detection module 604 is used to input the point cloud data to be detected and the target category information into the target detection model to obtain the detection box belonging to the target category; wherein, the target detection model is based on Figure 2 Implementation examples or Figure 3 The training was obtained using the method described in the example.

[0066] The specific details of each module or unit in the above-mentioned device have been described in detail in the corresponding methods, so they will not be repeated here.

[0067] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0068] This application also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the training method or object detection method of the above-described object detection model.

[0069] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device in an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0070] like Figure 7 As shown, the electronic device may include: a processor 702, a communication interface 704, a memory 706, and a communication bus 708.

[0071] The processor 702, communication interface 704, and memory 706 communicate with each other via communication bus 708.

[0072] Communication interface 704 is used to communicate with other electronic devices or servers.

[0073] The processor 702 is used to execute program 710, specifically the relevant steps in the above method embodiments.

[0074] Specifically, program 710 may include program code that includes computer operation instructions.

[0075] The processor 702 may be a central processing unit, a specific integrated circuit, or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0076] Memory 706 is used to store program 710. Memory 706 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0077] Specifically, program 710 can be used to cause processor 702 to execute the steps in the above-described object detection model training method or object detection method embodiment.

[0078] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0079] In this embodiment of the application, a computer-readable storage medium is also provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the above-described training method or object detection method for the object detection model.

[0080] It should be noted that the computer-readable storage medium shown in this application can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, radio frequency, etc., or any suitable combination thereof.

[0081] In this embodiment of the application, a computer program product is also provided, which, when run on a computer, causes the computer to execute the training method or object detection method of the above-mentioned object detection model.

[0082] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0083] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A training method for an object detection model, characterized in that, include: Acquire multiple first sample point cloud data, the bounding box label corresponding to each first sample point cloud data, and the category label corresponding to each bounding box label; For a single first sample point cloud data, the first sample point cloud data is input into the first 3D feature extraction network to obtain the first 3D voxel feature map, and the first open vocabulary classification head is used to process the first 3D voxel feature map to obtain the first K-dimensional feature vector; K is a positive integer; The first 3D voxel feature map is processed using the first regression head to obtain the predicted bounding box. Based on the bounding box label and the predicted bounding box, the regression loss is determined. The first sample point cloud data is processed using a text encoder to obtain the first K-dimensional text vector. The first classification loss is determined based on the similarity between the first K-dimensional feature vector and the first K-dimensional text vector; A first total loss is determined based on the first classification loss and the regression loss, and the network parameters in the first 3D feature extraction network, the first regression head, and the first open vocabulary classification head are adjusted based on the first total loss. When the first convergence condition is met, the trained second 3D feature extraction network, second regression head, and second open vocabulary classification head are obtained, and an object detection model is constructed based on the text encoder, the second 3D feature extraction network, the second regression head, and the second open vocabulary classification head.

2. The method according to claim 1, characterized in that, The step of constructing an object detection model based on the text encoder, the second 3D feature extraction network, the second regression head, and the second open vocabulary classification head includes: Obtain multiple second sample point cloud data, the bounding box label corresponding to each second sample point cloud data, and the category label corresponding to each bounding box label; N is a positive integer less than or equal to M; The target layer in the second open vocabulary classification head is copied into M parallel target layers with the same weight coefficient to obtain the third open vocabulary classification head; M is an integer greater than 1. For a single second sample point cloud data, the second sample point cloud data is input into the second 3D feature extraction network to obtain a second 3D voxel feature map, and the second 3D voxel feature map is processed by the third open vocabulary classification head to obtain M second K-dimensional feature vectors. The text encoder is used to process the N category labels of a single bounding box label corresponding to the second sample point cloud data to obtain N second K-dimensional text vectors. Based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors, a second classification loss is determined; based on the second classification loss, the weight coefficients corresponding to the M parallel target layers in the third open vocabulary classification head and the network parameters in the M target layers are adjusted. When the second convergence condition is met, the trained fourth open vocabulary classification head is obtained, and an object detection model is constructed based on the text encoder, the second 3D feature extraction network, the second regression head, and the fourth open vocabulary classification head.

3. The method according to claim 1, characterized in that, The step of processing the category labels corresponding to the first sample point cloud data using a text encoder to obtain a first K-dimensional text vector includes: When a single bounding box label corresponds to multiple category labels, a text encoder is used to process any category label to obtain the first K-dimensional text vector; or... Each category label is processed using a text encoder, and the average of the resulting K-dimensional text vectors is used as the first K-dimensional text vector; or, A text encoder is used to process multiple category labels to obtain the first K-dimensional text vector.

4. The method according to claim 2, characterized in that, If N is 1, determining the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors includes: Calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector, and determine the second classification loss based on the maximum similarity; If N is an integer greater than 1, the step of determining the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors includes: For each second K-dimensional text vector, calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector to obtain M similarity scores; Based on the M similarities corresponding to the N second K-dimensional text vectors, the Hungarian algorithm is used to obtain the target similarities corresponding to the N second K-dimensional text vectors, and the second classification loss is determined based on the target similarities. Among them, the sum of the target similarities corresponding to the N second K-dimensional text vectors is the largest, and the target similarities corresponding to the N second K-dimensional text vectors are calculated by comparing the N second K-dimensional text vectors with the second K-dimensional feature vectors output by different target layers.

5. A target detection method, characterized in that, include: Acquire the point cloud data to be detected and the target category; The point cloud data to be detected and the target category information are input into the target detection model to obtain a detection box belonging to the target category; wherein the target detection model is trained based on the method described in any one of claims 1 to 4.

6. A training device for an object detection model, characterized in that, include: The training dataset acquisition module is used to acquire multiple first sample point cloud data, the bounding box label corresponding to each first sample point cloud data, and the category label corresponding to each bounding box label. The feature extraction module is used to input the first sample point cloud data into the first 3D feature extraction network to obtain the first 3D voxel feature map for a single first sample point cloud data, and to process the first 3D voxel feature map using the first open vocabulary classification head to obtain the first K-dimensional feature vector; K is a positive integer. The regression loss determination module is used to process the first 3D voxel feature map using the first regression head to obtain the predicted bounding box, and to determine the regression loss based on the bounding box label and the predicted bounding box. The text encoding module is used to process the category labels corresponding to the first sample point cloud data using a text encoder to obtain a first K-dimensional text vector; The classification loss determination module is used to determine the first classification loss based on the similarity between the first K-dimensional feature vector and the first K-dimensional text vector; The model training module is used to determine a first total loss based on the first classification loss and the regression loss, and to adjust the network parameters in the first 3D feature extraction network, the first regression head and the first open vocabulary classification head based on the first total loss; The object detection model generation module is used to obtain the trained second 3D feature extraction network, second regression head, and second open vocabulary classification head when the first convergence condition is met, and to construct an object detection model based on the text encoder, the second 3D feature extraction network, the second regression head, and the second open vocabulary classification head.

7. The apparatus according to claim 6, characterized in that, The target detection model generation module is specifically used to obtain the trained second 3D feature extraction network and the second open vocabulary classification head when the first convergence condition is met, and to copy the target layer in the second open vocabulary classification head into M parallel target layers with the same weight coefficients to obtain the third open vocabulary classification head. M is an integer greater than 1; and, Obtain multiple second sample point cloud data, the bounding box label corresponding to each second sample point cloud data, and the category label corresponding to each bounding box label; N is a positive integer less than or equal to M; For a single second sample point cloud data, the second sample point cloud data is input into the second 3D feature extraction network to obtain a second 3D voxel feature map, and the second 3D voxel feature map is processed by the third open vocabulary classification head to obtain M second K-dimensional feature vectors; the text encoder is used to process the N category labels of the single bounding box label corresponding to the second sample point cloud data to obtain N second K-dimensional text vectors. as well as, Based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors, a second classification loss is determined; based on the second classification loss, the weight coefficients corresponding to the M parallel target layers in the third open vocabulary classifier head and the network parameters in the M target layers are adjusted; when the second convergence condition is met, a trained fourth open vocabulary classifier head is obtained; an object detection model is constructed based on the text encoder, the second 3D feature extraction network, the second regression head, and the fourth open vocabulary classifier head.

8. The apparatus according to claim 6, characterized in that, The text encoding module is specifically used to process any category label using a text encoder when a single bounding box label corresponds to multiple category labels, to obtain a first K-dimensional text vector; Alternatively, a text encoder can be used to process each category label, and the average of the resulting multiple K-dimensional text vectors can be used as the first K-dimensional text vector. Alternatively, a text encoder can be used to process multiple category labels to obtain the first K-dimensional text vector.

9. The apparatus according to claim 7, characterized in that, If N is 1, the object detection model generation module is specifically used to determine the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors through the following steps: Calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector, and determine the second classification loss based on the maximum similarity; If N is an integer greater than 1, the object detection model generation module is specifically used to determine the second classification loss based on the similarity between the M second K-dimensional feature vectors and the N second K-dimensional text vectors through the following steps: For each second K-dimensional text vector, calculate the similarity between the M second K-dimensional feature vectors and the second K-dimensional text vector to obtain M similarity scores; Based on the M similarities corresponding to the N second K-dimensional text vectors, the Hungarian algorithm is used to obtain the target similarities corresponding to the N second K-dimensional text vectors, and the second classification loss is determined based on the target similarities. Among them, the sum of the target similarities corresponding to the N second K-dimensional text vectors is the largest, and the target similarities corresponding to the N second K-dimensional text vectors are calculated by comparing the N second K-dimensional text vectors with the second K-dimensional feature vectors output by different target layers.

10. A target detection device, characterized in that, include: The data acquisition module is used to acquire the point cloud data to be detected and the target category; The target detection module is used to input the point cloud data to be detected and the target category information into the target detection model to obtain a detection box belonging to the target category; wherein the target detection model is trained based on the method described in any one of claims 1 to 4.

Images