A 3D open world object detection method based on multi-prototype clustering

Abstract

The application discloses a 3D open world target detection method based on multi-prototype clustering, and relates to the technical field of 3D open world target detection. The method comprises the following steps: optimizing a neural network model according to an inter-class contrast loss function between region-of-interest features of different categories and an intra-class loss function between the region-of-interest features of each category and a plurality of different sub-class features of each category, and determining the optimized neural network model. The region-of-interest features of different categories are determined according to feature maps of a plurality of different categories of training data of a current task, candidate boxes are determined based on the feature maps, instance feature maps are obtained, and the region-of-interest features of different categories are obtained through the instance feature maps. The sub-class features are obtained according to the region-of-interest features. The optimized neural network model is used to search for sub-class features of a plurality of different categories that are difficult to label, the neural network model can be better optimized, and the detection result precision of unknown categories of targets is improved.

Description

Technical Field

[0001] This invention relates to the field of 3D open-world object detection technology, and in particular to a 3D open-world object detection method based on multi-prototype clustering. Background Technology

[0002] 3D object detection algorithms possess stronger spatial perception capabilities compared to 2D object detection algorithms, thus having broad application scenarios in fields such as autonomous driving and robotics. Currently, most mainstream 3D object detection algorithms are designed for closed worlds. When encountering test set data with a different distribution than the training set, their performance degrades significantly. Traditional methods employ semantic labeling for supervision to avoid performance degradation. However, this method often fails to capture the underlying data structure, thus failing to search for subclass features of the training set that are difficult to label but crucial in the application scenario. Consequently, it cannot better shape the feature space, ultimately leading to low accuracy in detecting targets of unknown categories. Summary of the Invention

[0003] The purpose of this invention is to provide a 3D open-world object detection method based on multi-prototype clustering, in order to solve the problem in the prior art that it is impossible to search for subclass features of the training set that are difficult to label but are important in the application scenario, thus failing to better shape the feature space and ultimately resulting in low accuracy of the detection results for targets of unknown categories.

[0004] To achieve the above objectives, the present invention provides the following solution:

[0005] A 3D open-world object detection method based on multi-prototype clustering includes:

[0006] Multiple training data of different categories for the current task are input into the backbone network of the neural network model to determine feature maps of multiple different categories; the training data includes point cloud data.

[0007] The feature maps of multiple different categories are input into a multi-detector region proposal network to extract candidate boxes of multiple different categories;

[0008] Based on the feature maps of multiple different categories and the candidate boxes of multiple different categories, determine multiple instance feature maps of different categories with uniform size;

[0009] Based on the feature maps of multiple instances of different categories with uniform size, the region of interest features of multiple different categories are determined;

[0010] The neural network model is trained using the features of the regions of interest of multiple different categories, and outputs multiple different sub-class features of multiple different categories to determine the trained neural network model;

[0011] The trained neural network model is optimized based on the inter-class contrast loss function between the region of interest features of different categories and the intra-class contrast loss function between the region of interest features of each category and the multiple different sub-class features of each category, thereby determining the optimized neural network model; the optimized neural network model is used to search for multiple different sub-class features in the application scenario.

[0012] The test data of the current task is input into the optimized neural network model to determine the category of the target to be detected.

[0013] Optionally, multiple feature maps of different categories are input into a multi-detector region proposal network to extract multiple candidate boxes of different categories, specifically including:

[0014] Multiple feature maps of different categories are input into a multi-detector region proposal network, and localization loss function, classification loss function and orientation loss function are used to train multiple feature maps of different categories to determine multiple candidate boxes of different categories.

[0015] Optionally, multiple feature maps of different categories are input into a multi-detector region proposal network, and localization loss function, classification loss function, and orientation loss function are used to train the multiple feature maps of different categories to determine multiple candidate boxes of different categories, specifically including:

[0016] The feature maps are input into a multi-detector region proposal network. Based on the average loss value of the feature maps of different categories in each round, the loss update weights of the feature maps of different categories are determined. The loss update weights include intensity weights and rate weights.

[0017] Based on the localization loss function, the classification loss function, the orientation loss function, and the loss update weights of the various training data, multiple feature maps of different categories are trained to determine multiple candidate boxes of different categories.

[0018] Optionally, the intensity weight is:

[0019]

[0020] in, Let C be the intensity weight of category c in round t. cur Where is the number of categories for the current task, and T is the temperature coefficient that controls the gradient balance intensity. Let be the average loss value of category c at round t-1. Let be the average loss value of category j at round t-1, j = 1, 2, ..., C. cur .

[0021] Optionally, the rate weight is:

[0022]

[0023] in, C represents the rate weight of category c in round t. cur R represents the number of categories for the current task, T is the temperature coefficient controlling the gradient balance intensity, and R... c (t-1) represents the relative rate of decrease in the loss of category c at round t-1, R j (t-1) represents the relative rate of decrease in loss for category j at round t-1, where j = 1, 2, ..., C cur .

[0024] Optionally, the loss update weights are:

[0025]

[0026] Among them, a c (t) represents the loss update weight for category c in round t, and ρ is the momentum coefficient, which controls the tradeoff between loss intensity and loss rate.

[0027] Optionally, the inter-class comparison loss function is:

[0028]

[0029] in, Let τ be the inter-class contrast loss function, τ be the temperature coefficient, and f be any feature of the region of interest. For features belonging to the same subclass as f, the region of interest features, This refers to the region of interest features that belong to a different subclass than f.

[0030] Optionally, the intra-class contrastive loss function is:

[0031]

[0032] in, p is the intra-class contrastive loss function. + For the subclass features of f, For the h-th subclass feature of category c

[0033] Optionally, the trained neural network model is optimized based on the inter-class contrast loss function between features of different categories of regions of interest, and the intra-class contrast loss function between features of each category of regions of interest and features of multiple different subclasses of each category, to determine the optimized neural network model, specifically including:

[0034] The total loss function is determined based on the inter-class contrast loss function between the region of interest features of each different category, and the intra-class contrast loss function between the region of interest features of each category and the multiple different sub-class features of each category.

[0035] The parameters of the trained neural network model are optimized using the total loss function to determine the optimized neural network model; the parameters include the parameters of the multi-detector region proposal network, the parameters of the inter-class contrast loss function, and the parameters of the intra-class contrast loss function.

[0036] Optionally, the total loss function is:

[0037]

[0038] in, Let β1, β2, and β3 be the total loss function, and let β1, β2, and β3 be hyperparameters.

[0039] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0040] The 3D open-world object detection method based on multi-prototype clustering provided by this invention optimizes the trained neural network model by using inter-class contrast loss functions between region-of-interest (ROI) features of different categories and intra-class loss functions between ROI features of each category and multiple different sub-class features of each category. The optimized neural network model is then determined. ROI features of different categories are obtained by determining multiple feature maps of different categories based on training data of multiple categories for the current task. Candidate boxes are then determined based on these feature maps to obtain instance feature maps. Sub-class features are obtained based on ROI features. The optimized neural network model then searches for different sub-class features obtained from training data of multiple categories that are difficult to label but are important in the application scenario. This better optimizes the neural network model and improves the accuracy of detecting targets of unknown categories. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating the 3D open-world object detection method based on multi-prototype clustering provided by this invention;

[0043] Figure 2This is a flowchart illustrating the working process of the 3D open-world object detection method based on multi-prototype clustering provided by the present invention.

[0044] Figure 3 The flowchart illustrates the process of training the neural network model in the 3D open-world object detection method based on multi-prototype clustering provided by this invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The purpose of this invention is to provide a 3D open-world object detection method based on multi-prototype clustering. This method aims to search for different sub-class features derived from training data of multiple different categories—features that are difficult to label but crucial in the application scenario—through an optimized neural network model. This allows for better optimization of the neural network model, leading to a better shaping of the feature space and further improving the detection accuracy for targets of unknown categories. Furthermore, this invention addresses the problem of long-tailed data distribution through a multi-detector design and a loss balancing strategy.

[0047] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0048] like Figure 1 As shown, the 3D open-world object detection method based on multi-prototype clustering provided by this invention includes:

[0049] Step 101: Input training data of multiple different categories for the current task into the backbone network of the neural network model to determine feature maps of multiple different categories. The training data includes point cloud data.

[0050] Step 102: Input feature maps of multiple different categories into the multi-detector region proposal network to extract candidate boxes of multiple different categories.

[0051] Step 103: Based on the feature maps of multiple different categories and the candidate boxes of multiple different categories, determine multiple instance feature maps of different categories with uniform size.

[0052] Step 104: Based on the feature maps of multiple instances of different categories with uniform size, determine the features of multiple regions of interest for different categories.

[0053] Step 105: Train the neural network model using the features of the multiple different categories of regions of interest, output multiple different sub-class features of multiple different categories, and determine the trained neural network model.

[0054] Step 106: Optimize the trained neural network model based on the inter-class contrast loss function between the region of interest features of different categories and the intra-class contrast loss function between the region of interest features of each category and multiple different sub-class features of each category, and determine the optimized neural network model; the optimized neural network model is used to search for multiple different sub-class features in the application scenario.

[0055] Step 107: Input the test data of the current task into the optimized neural network model to determine the category of the target to be detected.

[0056] Step 102 specifically includes:

[0057] Multiple feature maps of different categories are input into a multi-detector region proposal network, and localization loss function, classification loss function and orientation loss function are used to train the feature maps of multiple different categories to determine multiple candidate boxes of different categories.

[0058] Furthermore, feature maps of multiple different categories are input into a multi-detector region proposal network (MPNR), and localization loss function, classification loss function, and orientation loss function are used to train the feature maps of multiple different categories to determine candidate boxes of multiple different categories. Specifically, this includes: inputting feature maps into the MPNR; determining loss update weights for multiple different categories of feature maps based on the average loss value of the feature maps of multiple different categories in each round; loss update weights include intensity weights and rate weights; and training the feature maps of multiple different categories based on the localization loss function, classification loss function, orientation loss function, and loss update weights of various training data to determine candidate boxes of multiple different categories.

[0059] Furthermore, the intensity weight is:

[0060] in, Let C be the intensity weight of category c in round t. cur Where is the number of categories for the current task, and T is the temperature coefficient that controls the gradient balance intensity. Let be the average loss value of category c at each round t-1. Let be the average loss value of category j at round t-1, j = 1, 2, ..., C. cur .

[0061] Furthermore, the rate weight is:

[0062] in, C represents the rate weight of category c in round t. cur R represents the number of categories for the current task, T is the temperature coefficient controlling the gradient balance intensity, and R... c (t-1) represents the relative rate of decrease in the loss of category c at round t-1, R j (t-1) represents the relative rate of decrease in loss for category j at round t-1, where j = 1, 2, ..., C cur .

[0063] Furthermore, the loss update weights are:

[0064] Among them, a c (t) represents the loss update weight for category c in round t, and ρ is the momentum coefficient, which controls the tradeoff between loss intensity and loss rate.

[0065] Furthermore, the inter-class contrast loss function is:

[0066]

[0067] in, Let τ be the inter-class contrast loss function, τ be the temperature coefficient, and f be any feature of the region of interest. For features belonging to the same subclass as f, the region of interest features, This refers to the region of interest features that belong to a different subclass than f.

[0068] Furthermore, the intra-class contrastive loss function is:

[0069] in, p is the intra-class contrastive loss function. + For the subclass features of f, This is the prototype feature of the h-th subclass of category c.

[0070] Furthermore, based on the inter-class contrast loss function between different categories of region of interest features, and the intra-class contrast loss function between each category of region of interest features and multiple different sub-class features of each category, the trained neural network model is optimized to determine the optimized neural network model. Specifically, this includes: determining the total loss function based on the inter-class contrast loss function between different categories of region of interest features, and the intra-class contrast loss function between each category of region of interest features and multiple different sub-class features of each category; using the total loss function to optimize the parameters of the trained neural network model to determine the optimized neural network model; the parameters include the parameters of the multi-detector region proposal network, the parameters of the inter-class contrast loss function, and the parameters of the intra-class contrast loss function.

[0071] Furthermore, the total loss function is:

[0072] in, Let β1, β2, and β3 be the total loss function, and let β1, β2, and β3 be hyperparameters.

[0073] like Figure 2 As shown below, the specific detection steps of the 3D open-world object detection method based on multi-prototype clustering provided by this invention will be described in detail.

[0074] First, training data from multiple different categories for the current task are input into the backbone network of the neural network model to determine feature maps for multiple different categories. The training data includes images, videos, and point cloud data.

[0075] Second, feature maps of multiple different categories are input into a multi-detector region proposal network (RPN) to extract candidate boxes of multiple different categories.

[0076] Specifically, for the design of the multi-detector region proposal network, a design scheme is adopted to assign a corresponding detector head to the feature map of each category, and the feature map of each category uses the localization loss function L. loc Classification loss function L cls and direction loss function L dir Three types of loss functions are used for training to generate multiple candidate boxes of different categories. Furthermore, to better capture the general properties of the foreground object, an objectivity loss function L is also introduced. obj This loss function estimates the objectivity of each region based on the overlap between the proposed Intersection over Union (IoU) and the ground truth. The objectivity loss function L... obj It trains uniformly on training data for all categories.

[0077] Third, assign loss update weights to the detection heads for each category, with L for each category... loc L cls and L dir The gradient-balanced RPN loss is calculated by dynamically updating the weights according to the class. Since the average loss change per round for each class reflects the training performance of that class, the convergence rate and strength of the average loss value are used as indicators for allocating the loss weights. This maintains a relatively balanced update state for imbalanced classes.

[0078] Assumption Let represent the average loss value of category c at each round t. The strength weight and rate weight of category c are defined to capture the loss update progress of category c, as follows:

[0079]

[0080] in, Let C be the intensity weight of category c in round t. cur Where is the number of categories for the current task, and T is the temperature coefficient that controls the gradient balance intensity. Let be the average loss value of category c at each round t-1. Let be the average loss value of category j at round t-1, j = 1, 2, ..., C. cur .

[0081]

[0082] in, C represents the rate weight of category c in round t. cur R represents the number of categories for the current task, T is the temperature coefficient controlling the gradient balance intensity, and R... c (t-1) represents the relative rate of decrease in loss for category c at round t-1. Let be the relative rate of decrease in loss for category j at round t-1, where j = 1, 2, ..., C. cur .

[0083] Specifically, L c (t) is averaged through multiple iterations to reduce instability due to stochastic gradient descent and random selection of training data.

[0084] The total loss update weights obtained by combining the above intensity weights and rate weights are as follows:

[0085]

[0086] Among them, a c (t) represents the loss update weight for category c in round t, and ρ is the momentum coefficient, which controls the tradeoff between loss intensity and loss rate.

[0087] Fourth, based on the feature maps of multiple different categories and the candidate boxes of multiple different categories, determine the instance feature maps of multiple categories with uniform size.

[0088] Fifth, based on the feature maps of multiple instances of different categories with uniform size, determine the features of multiple regions of interest (ROI) for different categories.

[0089] Sixth, input multiple ROI features of different categories into the neural network model, output multiple different subclass features of different categories, determine the trained neural network model, and perform subclass clustering.

[0090] Assigning fine-grained semantic labels to features of different categories without explicit subclass label supervision can be viewed as a special distribution matching problem. For each semantic label category c∈C, where C is the total number of categories, assumed to contain H potential subclasses. For each batch of ROI features, the c-category features are selected. Each ROI feature. When used separately The two vectors β and α represent the H-dimensional vectors. Given the distribution of features and H subclasses, all feasible solutions to the allocation matrix Q in the transport polyhedron can be represented as follows:

[0091]

[0092] The coupling matrix (or joint probability) mapping α to β can be quantified by optimizing the following problem:

[0093]

[0094]

[0095] Among them, 1 H and Q is a vector of all ones with the corresponding dimension; c Let Q be the solution space; α is Q c Projected to the edge of its row; β is Q c Projected onto the edge of its column; Q cT For Q c transpose; The second-to-last layer of features f before the classification head c and subclass prototype feature P c The similarity matrix between them.

[0096] OT(α,β) is the formula for the neural network model. Besides considering the feasible solution U(α,β) of the allocation matrix Q, OT(α,β) also takes into account the constraint H(Q) of the entropy regularization term. c ), where ε is the adjustment parameter for the entropy regularization term; the entire process of OT(α,β) can be solved using the efficient Sinkhorn algorithm. The OT(α,β) obtained through the above steps will serve as the clustering condition for the entire sub-clustering. The parameters of the neural network model are shown in Table 1.

[0097] Table 1. Parameters of Neural Network Model

[0098] <![CDATA[D dot ()]]> dot product H() Entropy regularization term constraint <![CDATA[S c ]]> Cost matrix <![CDATA[Q c ]]> Solution space <![CDATA[Q cT ]]> transpose of solution space

[0099] After completing the clustering operation for subclasses, momentum updates are needed for the subclass prototypes to accelerate the convergence of the clustering algorithm and improve its stability. The subclass prototypes are updated using a weighted average of the ROI features. The weights of the ROI features can be calculated using the prediction entropy as follows:

[0100]

[0101] in, Let the entropy weight be the feature of the i-th ROI in category c. Let i be the predicted entropy of the i-th ROI feature in category c, where i = 1, 2, ..., N, and N is the number of ROI features in category c.

[0102] In each batch of training, the prototype of the h-th subclass of class c is updated as follows:

[0103]

[0104] Where μ is the momentum coefficient. To assign to the prototype of the subclass The number of ROIs, k j The index of the j-th ROI feature assigned to subclass h in category c, j = 1, 2, ... Let the entropy weight be the feature of the j-th ROI in category c. For the kth class of category c j One ROI feature.

[0105] Based on the above publicly available information, it can be seen that, compared with the mean update method, the subclass prototype update, which takes into account the importance of ROI features, is more conducive to extracting representative prototype features.

[0106] By using intra-class contrastive loss function and inter-class contrastive loss function, the cluster space of features of the same category is made more compact, while increasing the difference between cluster spaces of different categories.

[0107] Two contrastive losses are used to encourage intra-class diversity. The inter-class contrastive loss function L... pff Used to characterize the relationships between ROI features, and defined as follows:

[0108]

[0109] in, Let τ be the inter-class contrast loss function, τ be the temperature coefficient, and f be any feature of the region of interest. For features belonging to the same subclass as f, the region of interest features, This refers to the region of interest features that belong to a different subclass than f.

[0110] Inter-class contrast loss is used to reduce the distance between features belonging to the same class and increase the distance between features from different classes.

[0111] Intra-class contrastive loss function L bfp The relationship between ROI features and subclass prototypes is defined as follows:

[0112]

[0113] in, p is the intra-class contrastive loss function. + For the subclass features of f, This is the prototype feature of the h-th subclass of category c.

[0114] Furthermore, the constraints of semantic label categories are equally important. Therefore, semantic label categories are also used to impose contrast constraints on different ROI features.

[0115] The total loss function for contrastive learning is:

[0116] in, Let β1, β2, and β3 be the total loss function, and let β1, β2, and β3 be hyperparameters used to balance different losses.

[0117] By optimizing and updating the total loss function through the above comparison, the optimized neural network can be determined. This allows for further shaping of the feature space for multi-prototype learning. Using the feature space of prototype learning, discriminative embedding representations can be obtained. Finally, the bounding boxes and corresponding categories of the target to be detected are determined through bounding box regression loss and classification loss.

[0118] like Figure 3 As shown, the specific steps for training a neural network model using the KITTI dataset as training data are as follows:

[0119] Step 301: Input the training data for the current task.

[0120] Step 302: Extract backbone features using the backbone network of the voxel-RCNN object detector.

[0121] Specifically, the voxel-RCNN object detector is used as the backbone of the overall network, and the feature maps of the input samples are extracted from the training data of the current task through the backbone network;

[0122] Step 303: The extracted feature map is transmitted to the multi-detector RPN network to extract candidate boxes.

[0123] Specifically, the feature maps are fed into a multi-detector RPN network to extract candidate boxes. The loss update of each detector in the RPN network is dynamically weighted according to the loss strength and update rate, thereby balancing the convergence of RPN losses for different categories and alleviating the long-tail distribution problem to some extent.

[0124] Step 304: Transfer the candidate bounding boxes and feature maps to the ROI to obtain instance feature maps of uniform size.

[0125] Instance feature map Instance feature map f roi ROI features are obtained after global pooling.

[0126] Step 305: The ROI features are transmitted to the neural network model for clustering operations, resulting in a trained neural network model. The trained neural network model can assign fine-grained semantic labels to the ROI features, search for difficult-to-label subclass patterns, thereby better shaping the feature space, and simultaneously updating the subclass centers with weights.

[0127] Step 306: Optimize the trained neural network model parameters by updating the intra-class contrast loss and inter-class contrast loss. This improves the clustering performance of the optimized neural network model, thereby better shaping the feature space and better detecting unknown categories of targets. Then, the bounding boxes and corresponding categories of the targets to be detected are determined using bounding box regression loss and classification loss.

[0128] Step 307: Save subclass features for each category as replay examples. Fine-tune the network using replay examples from all old tasks. Skip this step if it is a base task.

[0129] After training and optimization for the current task are completed, the neural network model can efficiently detect the categories of the current and previous tasks, and correctly label unknown categories. Repeat the above steps for all tasks until the training for the last task is complete.

[0130] The 3D open-world object detection method based on multi-prototype clustering provided by this invention has broad application scenarios in fields such as autonomous driving and robotics. In the field of autonomous driving, the training data specifically includes data from sensors such as onboard millimeter-wave radar and lidar. The actual parameters, and the targets to be detected, specifically include road boundaries, vehicles, pedestrians, and unseen obstacles. In the field of robotics, the training data specifically includes data from onboard ultrasonic radar, lidar, and other sensors, as well as robot state data. The actual parameters, and the targets to be detected, specifically include objects to be grasped and obstacles.

[0131] In summary, this invention not only addresses the long-tail distribution problem often found in point cloud datasets, and the significant increase in computational costs associated with traditional data augmentation methods, but also employs a multi-detector design to dynamically balance class difference information, balancing the loss update gradient based on the loss intensity and update rate of each class in the KITTI dataset. Furthermore, it addresses the issue of insufficient fine-grained semantic information in point cloud data, making accurate semantic understanding of point cloud scenes difficult. This results in the ability to identify targets of known categories but not those of location categories. This invention utilizes multi-prototype feature representation learning, obtaining discriminative subclass feature embeddings through clustering algorithms to discover the intrinsic differences between different categories. By capturing the inherent underlying data characteristics of point cloud data through clustering, the feature space is better shaped, leading to better detection of targets of unknown categories. Ultimately, this achieves a high-performance 3D open-world object detection algorithm capable of handling the complex and ever-changing real-world autonomous driving environment.

[0132] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0133] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D open-world object detection method based on multi-prototype clustering, characterized in that, include: The training data of multiple different categories for the current task are input into the backbone network of the neural network model to determine the feature maps of multiple different categories. The training data includes point cloud data; The feature maps of multiple different categories are input into a multi-detector region proposal network to extract candidate boxes of multiple different categories, specifically including: The feature maps are input into a multi-detector region proposal network. Based on the average loss value of the feature maps of different categories in each round, the loss update weights of the feature maps of different categories are determined. The loss update weights include intensity weights and rate weights. Based on the localization loss function, classification loss function, orientation loss function, and the loss update weights of the feature maps of different categories, multiple feature maps of different categories are trained to determine multiple candidate boxes of different categories. The intensity weight is: ； in, For category Intensity weight, This represents the number of categories for the current task. To control the temperature coefficient of gradient equilibrium intensity, For category In the round The average loss value at that time, For category In the round The average loss value at that time, j =1, 2, ..., ; The rate weight is: ； in, For category rate weights, This represents the number of categories for the current task. To control the temperature coefficient of gradient equilibrium intensity, In the round Time Category The relative rate of decrease in losses, In the round Time Category The relative rate of decrease in losses, j =1, 2, ..., ; The loss update weight is: ； in, For category In the round Update the weights based on the loss. The momentum coefficient controls the trade-off between loss intensity and loss rate. Based on the feature maps of multiple different categories and the candidate boxes of multiple different categories, determine multiple instance feature maps of different categories with uniform size; Based on the feature maps of multiple instances of different categories with uniform size, the region of interest features of multiple different categories are determined; The neural network model is trained using the features of the regions of interest of multiple different categories, and outputs multiple different sub-class features of multiple different categories to determine the trained neural network model; The trained neural network model is optimized based on the inter-class contrast loss function between the region of interest features of different categories and the intra-class contrast loss function between the region of interest features of each category and the multiple different sub-class features of each category, thereby determining the optimized neural network model; the optimized neural network model is used to search for multiple different sub-class features in the application scenario. The inter-class comparison loss function is: ； in, For inter-class comparison loss function, For temperature coefficient, f For any region of interest, To and Regions of interest features belonging to the same subclass of features, To and Regions of interest features belonging to different subclasses; The intra-class contrast loss function is: ； in, For intra-class contrastive loss function, for Subclass characteristics, For category c The h Individual subclass prototype characteristics; The test data of the current task is input into the optimized neural network model to determine the category of the target to be detected.

2. The 3D open-world object detection method based on multi-prototype clustering according to claim 1, characterized in that, The trained neural network model is optimized based on the inter-class contrast loss function between features of different categories of regions of interest, and the intra-class contrast loss function between features of each category of regions of interest and features of multiple different subclasses of each category. The optimized neural network model is determined by optimizing the trained neural network model, specifically including: The total loss function is determined based on the inter-class contrast loss function between the region of interest features of each different category, and the intra-class contrast loss function between the region of interest features of each category and the multiple different sub-class features of each category. The parameters of the trained neural network model are optimized using the total loss function to determine the optimized neural network model; the parameters include the parameters of the multi-detector region proposal network, the parameters of the inter-class contrast loss function, and the parameters of the intra-class contrast loss function.

3. The 3D open-world object detection method based on multi-prototype clustering according to claim 2, characterized in that, The total loss function is: ； in, For the total loss function, , , This is a hyperparameter.

Images