A distributed out-of-distribution target detection method and device
Patent Information
- Application Number
- CN202410585003.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-11
- Publication Date
- 2026-09-08
- Estimated Expiration
- 2044-05-11
AI Technical Summary
其问题在于很难找到一个最优的距离来采样离群点，采样时可能会将可见类别的特征值当作OOD特征值进行采样导致训练的分类器表现不够好
[0049] Compared to existing technologies, the method disclosed in this invention collects category prototype features during the training process, transforming the problem of out-of-distribution OOD data detection into a problem of comparing similarity with prototype features, which significantly improves the detection performance and stability of out-of-distribution OOD detectors.
Smart Images
Figure CN118298163B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of deep learning target detection, and relates to a method and apparatus for detecting out-of-distribution targets. Background Technology
[0002] There is often a gap between the training and real-world applications of artificial intelligence systems. Training typically involves a subset of categorized data, while in real-world applications, models encounter a much wider range of data, including unknown categories. This can lead to unreliable predictions on out-of-distribution (OOD) data, causing serious problems. For example, misidentifying OOD data as passable areas in autonomous driving could result in traffic accidents. To address these issues, the field of out-of-distribution object detection has emerged and attracted significant research attention.
[0003] This task aims to construct an object detector capable of detecting out-of-order (OOD) data by integrating an OOD detector into existing object detectors. The main challenge lies in obtaining a robust OOD detector with good classification performance. During training, since OOD data is invisible, some existing methods attempt to generate OOD data to train the OOD detector. Current methods for generating OOD data include generating OOD feature vectors in the feature space and generating adversarial examples from the original input image. The method of generating OOD data in the feature space samples outlier features that do not belong to any class using a distance parameter, which are then used as negative examples to train the classifier. The problem is that it is difficult to find an optimal distance for sampling outliers, and sampling may mistakenly use visible class features as OOD features, leading to poor classifier performance. In the adversarial example method, the generated adversarial examples have an extreme feature distribution, resulting in significant differences in performance across different datasets and unstable performance. Summary of the Invention
[0004] To address the shortcomings of existing technologies in out-of-distribution (OOD) target detection performance, this invention proposes an out-of-distribution target detection method and apparatus. This method leverages the strong class representativeness of prototype features, determining whether the target belongs to out-of-distribution (OOD) data by calculating the similarity between the features of the target and prototype features of different categories.
[0005] To achieve the above technical objectives, this invention proposes an out-of-distribution target detection method, comprising the following steps:
[0006] The image to be detected is input into the target detection model, and forward propagation is performed on the backbone network, image encoder and feature decoder in the target detection model to extract multiple query features with visual information.
[0007] The query features are input into the low-dimensional feature vector generation module for forward propagation to generate a low-dimensional feature vector containing semantic information.
[0008] The query features are input into the classification prediction module for forward propagation to obtain the target category;
[0009] The query features are input into the bounding box prediction module for forward propagation to obtain the target bounding box coordinates.
[0010] Calculate the similarity between the low-dimensional feature vector and the prototype feature;
[0011] The out-of-distribution target detection result is obtained based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates.
[0012] In one embodiment of the out-of-distribution target detection method of the present invention, the step of inputting the query features into the classification prediction module for forward propagation to obtain the target category further includes:
[0013] The query features are forward propagated in the classification prediction module to obtain a category logical score. The category logical score is normalized using a first nonlinear function, and the branch corresponding to the highest score is selected as the target category.
[0014] In one embodiment of the out-of-distribution target detection method of the present invention, the step of calculating the similarity between the low-dimensional feature vector and the prototype feature further includes:
[0015] The low-dimensional feature vector and prototype feature are concatenated and input into the feature similarity module for forward propagation to generate a similarity score;
[0016] Calculate the cosine similarity between the low-dimensional feature vector and the prototype feature to obtain the first cosine similarity, and then exponentialize the first cosine similarity to obtain the cosine similarity index result.
[0017] In one embodiment of the out-of-distribution target detection method of the present invention, the step of obtaining the out-of-distribution target detection result based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates further includes:
[0018] The target energy is obtained by multiplying the cosine similarity index result by the similarity score;
[0019] Select the energy score corresponding to the target category from the target energy, and combine the energy score with the target bounding box coordinates to obtain the out-of-distribution target detection result.
[0020] In one embodiment of the out-of-distribution target detection method of the present invention, the training of the feature similarity module is performed through two branches: the low-dimensional feature vector and the negative embedding feature.
[0021] In one embodiment of the out-of-distribution target detection method of the present invention, the step of training the feature similarity module through the low-dimensional feature vector branch further includes:
[0022] The low-dimensional feature vector and prototype feature are concatenated and input into the feature similarity module to generate a similarity score;
[0023] The target category and target bounding box coordinates are processed using the Hungarian algorithm to obtain the Hungarian algorithm output;
[0024] The feature similarity module is trained using the output of the Hungarian algorithm.
[0025] In one embodiment of the out-of-distribution target detection method of the present invention, the step of training the feature similarity module through the negative embedding feature branch further includes:
[0026] The negative embedding feature and the prototype feature are concatenated, and the concatenated result is input into the multilayer perceptron of the feature similarity module for forward propagation to obtain the negative similarity score;
[0027] Select the negative embedding features corresponding to negative similarity scores less than the second threshold, and train the feature similarity module.
[0028] In one embodiment of the out-of-distribution target detection method of the present invention, the step of generating the negative embedding feature further includes:
[0029] During the training of the target detection model, the cosine similarity between the low-dimensional feature vector and the prototype feature is calculated to obtain the second cosine similarity.
[0030] The difference between the numerical value 1 and the second cosine similarity is normalized using the second nonlinear function to obtain the negative embedding weight;
[0031] The negative embedding weights are multiplied by the prototype features, and the result of the multiplication is added to the low-dimensional feature vector to obtain the negative embedding features.
[0032] In one embodiment of the out-of-distribution target detection method of the present invention, the step of exponentializing the first cosine similarity to obtain the cosine similarity index result further includes:
[0033] The first cosine similarity is used as the exponent, and the natural constant e is used as the base for the exponential operation.
[0034] In one embodiment of the out-of-distribution target detection method of the present invention, the low-dimensional feature vector generation module, the classification prediction module, and the bounding box generation module all include a multilayer perceptron, and the weights of the multilayer perceptrons of the low-dimensional feature vector generation module, the classification prediction module, and the bounding box generation module are continuously updated during the training of the target detection model.
[0035] In one embodiment of the out-of-distribution target detection method of the present invention, the step of updating the prototype features further includes: continuously updating the prototype features according to the output of the Hungarian algorithm during the training process of the target detection model.
[0036] In one embodiment of the out-of-distribution target detection method of the present invention, the steps of concatenating the low-dimensional feature vector and the prototype feature, and concatenating the negative embedding feature and the prototype feature, further include:
[0037] Each of the low-dimensional feature vectors and negative embedding features is expanded to the same number of categories as the prototype features through a copy operation.
[0038] In one embodiment of the out-of-distribution target detection method of the present invention, the step of obtaining the out-of-distribution target detection result by combining the energy score with the target bounding box coordinates further includes:
[0039] The energy score is compared with a first threshold, and then the object in the target bounding box coordinates is determined to be an out-of-distribution target based on the comparison result.
[0040] The present invention also discloses an off-distribution target detection device for implementing any of the above methods, comprising:
[0041] The query feature extraction module is used to input the image to be detected into the target detection model, and perform forward propagation on the backbone network, image encoder and feature decoder in the target detection model to extract multiple query features with visual information.
[0042] The low-dimensional feature vector generation module is used to perform forward propagation based on the query features to generate a low-dimensional feature vector containing semantic information.
[0043] The classification prediction module is used to perform forward propagation based on the query features to obtain the target category;
[0044] The bounding box prediction module is used to perform forward propagation based on the query features to obtain the coordinates of the target bounding box;
[0045] A similarity module is used to calculate the similarity between the low-dimensional feature vector and the prototype feature;
[0046] The out-of-distribution target detection module is used to obtain out-of-distribution target detection results based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates.
[0047] The present invention also discloses an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory further includes the aforementioned device.
[0048] The present invention also discloses a storage medium for storing a computer control program, characterized in that the computer control program is used to perform the steps of any of the methods described above.
[0049] Compared to existing technologies, the method disclosed in this invention collects category prototype features during the training process, transforming the problem of out-of-distribution OOD data detection into a problem of comparing similarity with prototype features, which significantly improves the detection performance and stability of out-of-distribution OOD detectors.
[0050] In addition, by providing a negative embedding generator, this invention reduces the reliance on invisible data. At the same time, this invention uses prototype feature similarity to detect OOD data more conveniently and achieves better OOD detection results than other methods.
[0051] To further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are for reference and illustration only and are not intended to limit the present invention. Attached Figure Description
[0052] Figure 1 A flowchart illustrating an off-target detection method according to an embodiment of the present invention is shown.
[0053] Figure 2 A diagram illustrating the overall architecture of an off-site target detection method according to an embodiment of the present invention is shown.
[0054] Figure 3 A schematic diagram of the feature similarity module in one embodiment of the present invention is shown.
[0055] Figure 4 A schematic diagram of the negative embedding generator in one embodiment of the present invention is shown.
[0056] Figure 5 A block diagram illustrating an external target detection device according to an embodiment of the present invention is shown.
[0057] Figure 6 A block diagram illustrating an electronic device according to an embodiment of the present invention is shown.
[0058] In the attached figures, the following labels are used:
[0059] Steps S1-S8…
[0060] 1…Query Features
[0061] 2…low-dimensional feature vectors
[0062] 3…Target Category
[0063] 4…Bounding box coordinates
[0064] 5…Prototype Features
[0065] 9…Negative embedding features
[0066] 10…Out-of-distribution target detection device
[0067] 11… Query Feature Extraction Module
[0068] 12…Low-dimensional feature vector generation module
[0069] 13…Classification Prediction Module
[0070] 14…Bounding Box Prediction Module
[0071] 15… Similarity Module
[0072] 19… Negative Embedding Generator
[0073] 151… Feature Similarity Module
[0074] 16…Out-of-distribution target detection module
[0075] 100…electronic devices
[0076] 110… processor
[0077] 120…memory
[0078] 130… Network Interface
[0079] 140…Internal Bus Detailed Implementation
[0080] The following specific embodiments, in conjunction with the accompanying drawings, illustrate the implementation methods disclosed in this invention. Those skilled in the art can understand the advantages and effects of this invention from the content disclosed in this specification. However, the following disclosure is not intended to limit the scope of protection of this invention. Without departing from the spirit of the invention, those skilled in the art can implement this invention with other different embodiments based on different viewpoints and applications.
[0081] For clarity, the figures shown in this invention are simplified schematic diagrams illustrating the basic structure of the invention. Therefore, the structures shown in the figures are not drawn to scale according to the actual shape and size of the implementation. For example, the dimensions of certain components have been enlarged for ease of explanation.
[0082] Furthermore, it should be understood that when a component such as a layer, film, region, or substrate is referred to as being "on" or "connected" to another component, it may be directly on or connected to the other component, or an intermediate component may also be present. Conversely, when a component is referred to as being "directly on" or "directly connected" to another component, no intermediate component exists. As used herein, "connection" can refer to physical and / or electrical connections. Moreover, "electrical connection" or "coupling" can refer to the presence of other components between the two components.
[0083] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having the same meaning as they have in the context of the relevant technology and this invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined herein.
[0084] Furthermore, it should be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various components, parts, regions, layers, and / or portions, these components, parts, regions, and / or portions should not be limited by these terms. These terms are used only to distinguish one component, part, region, layer, or portion from another. Therefore, the “first component,” “part,” “region,” “layer,” or “part” discussed below may be referred to as a second component, part, region, layer, or portion without departing from the teachings of this document.
[0085] It should be understood that references to "an embodiment," "embodiment," "example embodiment," etc., in the specification refer to the described embodiment, indicating that it may include specific features, structures, or characteristics, but does not necessarily include these specific features, structures, or characteristics. Furthermore, such expressions do not refer to the same embodiment. Moreover, when describing specific features, structures, or characteristics in conjunction with embodiments, whether or not explicitly described, it is indicated that incorporating such features, structures, or characteristics into other embodiments is within the knowledge of those skilled in the art.
[0086] The specification and subsequent claims use certain terms to refer to specific modules, components, or parts. Those skilled in the art will understand that users or manufacturers may use different names or terms to refer to the same module, component, or part. This specification and subsequent claims do not distinguish modules, components, or parts by differences in name, but rather by differences in function. The terms "comprising" and "including" used throughout the specification and subsequent claims are open-ended and should be interpreted as "including but not limited to." Furthermore, the term "connection" here includes any direct and indirect electrical connection means. Indirect electrical connection means include connections through other means.
[0087] Furthermore, in the following description and claims, numerous terms will be referenced, which should be defined as having the following meanings. The singular forms “a” and “the” include plural referents, unless the context clearly specifies otherwise. “Optional” or “optionally” indicates that an event or situation subsequently described may or may not occur, and the description includes both the scenario where the event occurs and the scenario where the event does not occur.
[0088] Terminology Explanation:
[0089] The RT-DERT model (DETRs Beat YOLOs on Real-time Object Detection) is a real-time object detection network model consisting of a backbone network, an image encoder, and a feature decoder. It exhibits excellent object detection performance.
[0090] Backbone network: Uses convolutional neural networks to extract multi-resolution feature maps.
[0091] Multilayer Perceptron (MLP): A neural network structure that typically consists of an input layer, one or more hidden layers, and an output layer, forming a feedforward neural network.
[0092] Image Encoder: Uses CNN and Transformer structures to extract image features.
[0093] Feature Decoder: Decodes the features output by the encoder.
[0094] Projection Head: Projects query features onto a low-dimensional feature vector.
[0095] Prototypes: Low-dimensional feature vectors of different categories are collected as prototype features during training.
[0096] The Similarity Module concatenates the input feature vector with the prototype feature, and then uses a multilayer perceptron to calculate the similarity between the two features.
[0097] Negative Embedding Generator: Generates negative embeddings in the feature space based on low-dimensional feature vectors and prototype features.
[0098] Please see Figure 1 This invention proposes an out-of-distribution (OOD) target detection method. This method leverages the strong class representativeness of prototype features (5) to determine whether the target belongs to out-of-distribution (OOD) data by calculating the similarity between the features of the target and prototype features (5) of different categories.
[0099] To achieve the above technical objectives, an off-distribution target detection method includes the following steps:
[0100] Step S1: Input the image to be detected into the target detection model, perform forward propagation on the backbone network, image encoder and feature decoder in the target detection model, and extract multiple query features with visual information 1;
[0101] Step S2: Input the query feature 1 into the low-dimensional feature vector generation module 12 for forward propagation to generate a low-dimensional feature vector 2 containing semantic information;
[0102] Step S3: Input the query feature 1 into the classification prediction module for forward propagation to obtain the target category 3;
[0103] Step S4: Input the query feature 1 into the bounding box prediction module for forward propagation to obtain the target bounding box coordinates 4;
[0104] Step S5: Calculate the similarity between the low-dimensional feature vector 2 and the prototype feature 5;
[0105] Step S6: Obtain the out-of-distribution target detection result based on the similarity between the low-dimensional feature vector 2 and the prototype feature 5, the target category 3, and the target bounding box coordinates 4.
[0106] See Figure 2 The diagram shows the overall architecture of the out-of-distribution target detection method and Figure 3The diagram shows the structure of the feature similarity module. This object detection model is based on the RT-DETR network structure, and the backbone network of the object detection model is a ResNet-50 network structure pre-trained on the MS-COCO dataset. After the image passes through the backbone network, image encoder, and feature decoder, it outputs multiple query features 1. The classification prediction module (also known as the classification prediction head) performs forward propagation on each query feature 1 and outputs the target class 3 score. ,in This represents the predicted score for each known category, and `num_class` indicates the total number of known categories. (Selection) The branch with the highest score is designated as the target category 3. The bounding box prediction module (also known as the bounding box prediction head) performs forward propagation on each query feature 1 and outputs the target bounding box coordinates 4. ,in This indicates the target bounding box position at coordinate 4. This represents a four-dimensional real tuple. A low-dimensional feature vector generation module 12 (also called a projection head) projects each query feature 1 from a high dimension into a low-dimensional feature vector 2. During the training phase, the low-dimensional feature vector 2 and the prototype feature 5 are fed into the feature similarity module 151 and the negative embedding generator 19, respectively. The low-dimensional feature vector 2 is processed by the similarity score output by the feature similarity module 151, while the negative embedding feature 9 generated by the negative embedding generator 19 is processed by the feature similarity module 151 to output a negative similarity score. Then, the Hungarian algorithm is used to sample a portion of the similarity scores to train the feature similarity module 151. A portion of the negative similarity scores below a second threshold are selected from the negative similarity scores to train the feature similarity module 151. During the testing phase of the target detection model, the low-dimensional feature vector 2 and the prototype feature 5 are concatenated and input into the feature similarity module 151 for forward propagation to generate a similarity score. The cosine similarity between the low-dimensional feature vector 2 and the prototype feature 5 is calculated. The cosine similarity between the low-dimensional feature vector 2 and the prototype feature 5 and the similarity score are multiplied to obtain the target energy E. The energy score corresponding to the target category 3 is selected from the target energy E. The energy score is compared with the first threshold. Then, the out-of-distribution target detection result is generated based on the bounding box coordinates 4.
[0107] In one embodiment of the out-of-distribution target detection method of the present invention, the step of inputting the query feature 1 into the classification prediction module for forward propagation to obtain the target category 3 further includes:
[0108] The query feature 1 is forward propagated in the classification prediction module to obtain a category logical score. The category logical score is normalized using a first nonlinear function, and the branch corresponding to the highest score is selected as the target category 3.
[0109] In one embodiment, the first nonlinear function is the Sigmoid function, but other nonlinear functions may also be used, and the present invention is not limited thereto.
[0110] In one embodiment of the out-of-distribution target detection method of the present invention, the step of calculating the similarity between the low-dimensional feature vector 2 and the prototype feature 5 further includes:
[0111] The low-dimensional feature vector 2 and the prototype feature 5 are concatenated and input into the feature similarity module 151 for forward propagation to generate a similarity score.
[0112] Calculate the cosine similarity between the low-dimensional feature vector 2 and the prototype feature 5 to obtain the first cosine similarity, and then perform exponential processing on the first cosine similarity to obtain the cosine similarity index result.
[0113] In one embodiment of the out-of-distribution target detection method of the present invention, the step of obtaining the out-of-distribution target detection result based on the similarity between the low-dimensional feature vector 2 and the prototype feature 5, the target category 3, and the target bounding box coordinates 4 further includes:
[0114] The target energy is obtained by multiplying the cosine similarity index result by the similarity score;
[0115] Select the energy score corresponding to the target category 3 from the target energy, and combine the energy score with the target bounding box coordinates 4 to obtain the out-of-distribution target detection result.
[0116] In one embodiment of the out-of-distribution target detection method of the present invention, the training of the feature similarity module 151 is performed through two branches: the low-dimensional feature vector 2 and the negative embedding feature 9.
[0117] In one embodiment of the out-of-distribution target detection method of the present invention, the step of training the feature similarity module 151 through the low-dimensional feature vector 2 branch further includes:
[0118] The low-dimensional feature vector 2 and the prototype feature 5 are concatenated and input into the feature similarity module 151 to generate a similarity score;
[0119] The target category 3 and the target bounding box coordinates 4 are processed using the Hungarian algorithm to obtain the output of the Hungarian algorithm;
[0120] The feature similarity module 151 is trained using the output of the Hungarian algorithm.
[0121] In one embodiment of the out-of-distribution target detection method of the present invention, the step of training the feature similarity module 151 through the negative embedding feature 9 branch further includes:
[0122] The negative embedding feature 9 and the prototype feature 5 are concatenated, and the concatenated result is input into the multilayer perceptron of the feature similarity module 151 for forward propagation to obtain the negative similarity score.
[0123] Select the negative embedding feature 9 corresponding to the negative similarity score being less than the second threshold, and train the feature similarity module 151.
[0124] Please see Figure 4 In one embodiment of the out-of-distribution target detection method of the present invention, the step of generating the negative embedding feature 9 further includes:
[0125] During the training of the target detection model, the cosine similarity between the low-dimensional feature vector 2 and the prototype feature 5 is calculated to obtain the second cosine similarity.
[0126] The difference between the numerical value 1 and the second cosine similarity is normalized using the second nonlinear function to obtain the negative embedding weight;
[0127] The negative embedding weight is multiplied by the prototype feature 5, and the result of the multiplication is added to the low-dimensional feature vector 2 to obtain the negative embedding feature 9.
[0128] In one embodiment of the out-of-distribution target detection method of the present invention, the step of exponentializing the first cosine similarity to obtain the cosine similarity index result further includes:
[0129] The first cosine similarity is used as the exponent, and the natural constant e is used as the base for the exponential operation.
[0130] Specifically, in one embodiment of the present invention, the cosine similarity between the low-dimensional feature vector 2 and the prototype feature 5 is calculated. Then, use formula (1) to generate the weights:
[0131] (1)
[0133] In formula (1), T is a manually set empirical value, the second nonlinear function is the Softmax function, and then the weights are... Multiplying the prototype feature 5 by the result and adding it to the low-dimensional feature vector 2, the output is the negative embedding feature 9.
[0134] Furthermore, the negative embedding feature 9 is only used in the training phase to train the feature similarity module 151. In one embodiment, negative embedding features 9 with values less than a second threshold are selected to participate in the training of the feature similarity module 151, where the second threshold is 0.3.
[0135] It should be noted that other nonlinear functions can also be used as the second nonlinear function, and this invention is not limited thereto.
[0136] In one embodiment of the out-of-distribution target detection method of the present invention, the low-dimensional feature vector generation module 12, the classification prediction module 13, and the bounding box generation module 14 all include a multilayer perceptron, and the weights of the multilayer perceptrons of the low-dimensional feature vector generation module 12, the classification prediction module 13, and the bounding box generation module 14 are continuously updated during the training of the target detection model.
[0137] In one embodiment of the out-of-distribution target detection method of the present invention, the step of updating the prototype feature 5 further includes: continuously updating the prototype feature 5 according to the output of the Hungarian algorithm during the training process of the target detection model.
[0138] In one embodiment of the out-of-distribution target detection method of the present invention, the steps of concatenating the low-dimensional feature vector 2 and the prototype feature 5, and concatenating the negative embedding feature 9 and the prototype feature 5, further include:
[0139] Each of the low-dimensional feature vectors 2 and negative embedding features 9 is expanded to the same number of categories as the prototype feature 5 through a copy operation.
[0140] In one embodiment of the out-of-distribution target detection method of the present invention, the step of obtaining the out-of-distribution target detection result by combining the energy score with the target bounding box coordinates further includes:
[0141] The energy score is compared with a first threshold to determine whether the object in the target bounding box at coordinate 4 belongs to an out-of-distribution target.
[0142] The present invention also discloses an off-target detection device 10 for implementing any of the above methods, comprising:
[0143] The query feature extraction module 11 is used to input the image to be detected into the target detection model, perform forward propagation on the backbone network, image encoder and feature decoder in the target detection model, and extract multiple query features 1 with visual information.
[0144] The low-dimensional feature vector generation module 12 is used to perform forward propagation based on the query feature 1 to generate a low-dimensional feature vector 2 containing semantic information.
[0145] The classification prediction module 13 is used to perform forward propagation based on the query feature 1 to obtain the target category 3;
[0146] Bounding box prediction module 14 is used to perform forward propagation based on the query feature 1 to obtain the target bounding box coordinates 4;
[0147] Similarity module 15 is used to calculate the similarity between the low-dimensional feature vector 2 and the prototype feature 5;
[0148] The out-of-distribution target detection module 16 is used to obtain the out-of-distribution target detection result based on the similarity between the low-dimensional feature vector 2 and the prototype feature 5, the target category 3, and the target bounding box coordinates 4.
[0149] The present invention also discloses an electronic device 100, including a memory 120, a processor 110 and a network interface 130, wherein the memory 120, the processor 110 and the network interface 130 are coupled through an internal bus 140, wherein the memory 120 further includes the aforementioned device.
[0150] Processor 110 may include a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 110 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0151] The present invention also discloses a storage medium for storing a computer control program, characterized in that the computer control program is used to perform the steps of any of the methods described above.
[0152] The aforementioned computer program may be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0153] Performance verification:
[0154] To verify the effectiveness of the prototype feature-based out-of-distribution object detection method, this embodiment conducted experiments using Pascal VOC and BDD100K as in-distribution datasets and MS-COCO and OpenImages as out-of-distribution datasets, and compared them with some OOD object detection methods. As shown in Table 1, the experimental results demonstrate the effectiveness of the present invention. On the Pascal VOC in-distribution dataset and MS-COCO out-of-distribution dataset, compared with SAFE-RegNext4.0, this embodiment achieved a nearly 4.6% increase in mAP on the in-distribution dataset, and a nearly 16% decrease and an 8% increase in the OOD test metrics, respectively. On the BDD100K in-distribution dataset and MS-COCO out-of-distribution dataset, FPR95 and AUROC achieved a nearly 9% decrease and a nearly 4% increase, respectively. These results indicate that the prototype feature-based out-of-distribution object detection method exhibits good and stable performance when detecting out-of-distribution data.
[0155] Table 1
[0156]
[0157] Compared to existing technologies, the method disclosed in this invention collects category prototype features during the training process, transforming the problem of out-of-distribution OOD data detection into a problem of comparing similarity with prototype features, which significantly improves the detection performance and stability of out-of-distribution OOD detectors.
[0158] In addition, by providing a negative embedding generator, this invention reduces the reliance on invisible data. At the same time, this invention uses prototype feature similarity to detect OOD data more conveniently and achieves better OOD detection results than other methods.
[0159] The above-disclosed content is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of the patent application of the present invention. Therefore, any equivalent technical changes made based on the contents of the present invention specification and drawings fall within the scope of the patent application of the present invention.
Claims
1. A method for detecting out-of-distribution targets, characterized in that, Includes the following steps: The image to be detected is input into the target detection model, and forward propagation is performed on the backbone network, image encoder and feature decoder in the target detection model to extract multiple query features with visual information. The query features are input into the low-dimensional feature vector generation module for forward propagation to generate a low-dimensional feature vector containing semantic information. The query features are input into the classification prediction module for forward propagation to obtain the target category; The query features are input into the bounding box prediction module for forward propagation to obtain the target bounding box coordinates. Calculate the similarity between the low-dimensional feature vector and the prototype feature; The out-of-distribution target detection result is obtained based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates; where, The step of calculating the similarity between the low-dimensional feature vector and the prototype feature further includes: The low-dimensional feature vector and prototype feature are concatenated and input into the feature similarity module for forward propagation to generate a similarity score; Calculate the cosine similarity between the low-dimensional feature vector and the prototype feature to obtain the first cosine similarity, and then exponentialize the first cosine similarity to obtain the cosine similarity index result. Training the feature similarity module is performed through two branches: the low-dimensional feature vector and the negative embedding feature. The step of training the feature similarity module through the low-dimensional feature vector branch further includes: The low-dimensional feature vector and prototype feature are concatenated and input into the feature similarity module to generate a similarity score; The target category and target bounding box coordinates are processed using the Hungarian algorithm to obtain the Hungarian algorithm output; The feature similarity module is trained using the output of the Hungarian algorithm. The step of training the feature similarity module through the negative embedding feature branch further includes: The negative embedding feature and the prototype feature are concatenated, and the concatenated result is input into the multilayer perceptron of the feature similarity module for forward propagation to obtain the negative similarity score; Select the negative embedding features corresponding to negative similarity scores less than the second threshold, and train the feature similarity module.
2. The method according to claim 1, characterized in that, The step of inputting the query features into the classification prediction module for forward propagation to obtain the target category further includes: The query features are forward propagated in the classification prediction module to obtain a category logical score. The category logical score is normalized using a first nonlinear function, and the branch corresponding to the highest score is selected as the target category.
3. The method according to claim 1, characterized in that, The step of obtaining the out-of-distribution target detection result based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates further includes: The target energy is obtained by multiplying the cosine similarity index result by the similarity score; Select the energy score corresponding to the target category from the target energy, and combine the energy score with the target bounding box coordinates to obtain the out-of-distribution target detection result.
4. The method according to claim 1, characterized in that, The step of generating the negative embedding feature further includes: During the training of the target detection model, the cosine similarity between the low-dimensional feature vector and the prototype feature is calculated to obtain the second cosine similarity. The difference between the numerical value 1 and the second cosine similarity is normalized using the second nonlinear function to obtain the negative embedding weight; The negative embedding weights are multiplied by the prototype features, and the result of the multiplication is added to the low-dimensional feature vector to obtain the negative embedding features.
5. The method according to claim 1, characterized in that, The step of exponentializing the first cosine similarity to obtain the cosine similarity index result further includes: The first cosine similarity is used as the exponent, and the natural constant e is used as the base for the exponential operation.
6. The method according to claim 1, characterized in that, The low-dimensional feature vector generation module, classification prediction module, and bounding box generation module all include a multilayer perceptron. The weights of the multilayer perceptrons in each of the low-dimensional feature vector generation module, classification prediction module, and bounding box generation module are continuously updated during the training of the target detection model.
7. The method according to claim 1, characterized in that, The step of updating the prototype features further includes: continuously updating the prototype features based on the output of the Hungarian algorithm during the training of the target detection model.
8. The method according to claim 1, characterized in that, The steps of concatenating the low-dimensional feature vector and the prototype feature, and concatenating the negative embedding feature and the prototype feature, further include: Each of the low-dimensional feature vectors and negative embedding features is expanded to the same number of categories as the prototype features through a copy operation.
9. The method according to claim 3, characterized in that, The step of combining the energy score with the target bounding box coordinates to obtain the out-of-distribution target detection result further includes: The energy score is compared with a first threshold, and then the object in the target bounding box coordinates is determined to be an out-of-distribution target based on the comparison result.
10. An off-target detection device, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: The query feature extraction module is used to input the image to be detected into the target detection model, and perform forward propagation on the backbone network, image encoder and feature decoder in the target detection model to extract multiple query features with visual information. The low-dimensional feature vector generation module is used to perform forward propagation based on the query features to generate a low-dimensional feature vector containing semantic information. The classification prediction module is used to perform forward propagation based on the query features to obtain the target category; The bounding box prediction module is used to perform forward propagation based on the query features to obtain the coordinates of the target bounding box; A similarity module is used to calculate the similarity between the low-dimensional feature vector and the prototype feature; The out-of-distribution target detection module is used to obtain out-of-distribution target detection results based on the similarity between the low-dimensional feature vector and the prototype feature, the target category, and the target bounding box coordinates.
11. An electronic device, characterized in that, It includes a memory and a processor, the memory being coupled to the processor; wherein the memory further includes the apparatus as described in claim 10.
12. A storage medium for storing a computer control program, characterized in that, The computer control program is used to perform the steps of the method as described in any one of claims 1 to 9.