A method and system for detecting out-of-distribution targets based on explicit foreground feature modeling

Abstract

The application discloses a distribution-out target detection method and system based on explicit foreground feature modeling, belongs to the technical field of computer vision, and solves the problem that the existing region-level distribution-out target detection method is difficult to accurately distinguish and locate unknown distribution-out targets, is prone to misjudging unknown targets as known categories or ignoring them as backgrounds due to the lack of perception and explicit constraint on foreground feature distribution in an open scene, and further leads to insufficient detection reliability and accuracy. Through constructing a context decoder, an in-distribution foreground feature explicit constraint target function and an out-of-distribution foreground feature explicit constraint target function, two-stage training is performed on the detection model. Based on the trained model, a prediction energy score is calculated, and out-of-distribution samples are determined accordingly. The application is suitable for target detection tasks highly related to safety in open scenes such as automatic driving, security monitoring and industrial quality inspection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to an out-of-distribution target detection method based on explicit foreground feature modeling. Background Technology

[0002] Object-level out-of-distribution detection aims to identify unknown targets outside the training categories in open environments, while maintaining the accuracy of in-distribution target detection and reducing security risks. Unlike image-level out-of-distribution detection, object-level out-of-distribution detection not only needs to determine whether the target belongs to an out-of-distribution category, but also needs to complete foreground localization and category determination at the region level. Therefore, out-of-distribution targets should not be simply regarded as anomalous inputs, but should be regarded as unknown foreground entities with spatial structure.

[0003] Existing target-level out-of-distribution detection methods primarily suppress the response of out-of-distribution regions to in-distribution categories through representation learning and regularization strategies. VOS (Very-Out-of-Distribution Targets) formalized this task for the first time using an unknown perception learning framework, synthesizing virtual outliers near the decision boundary of the feature space and utilizing energy-based uncertainty to identify out-of-distribution targets. Subsequent methods mostly follow the virtual outlier regularization paradigm, constructing pseudo-out-of-distribution features through generation, mixing, perturbation, or diffusion modeling to alleviate the problem of insufficient unknown data supervision during the training phase. However, the coverage of pseudo-outliers is limited by the finite in-distribution training distribution, making it difficult to fully represent the unbounded and complex unknown target space in the open world. Therefore, real out-of-distribution targets may still be mapped into the in-distribution feature space, resulting in feature collapse and causing unknown targets to be misclassified as known categories or suppressed as background.

[0004] To alleviate the problem of insufficient coverage of spurious outliers, some methods introduce real-world outlier exposure, incorporating more diverse out-of-distribution samples into detector training. These methods can provide richer evidence of unknown samples, but most still rely on response suppression regularization to reject out-of-distribution inputs. For methods containing... The distribution within the category and the first For a target detector of a background category, simply reducing the in-distribution category response in out-of-distribution regions does not guarantee that its foreground attributes will be preserved. When out-of-distribution regions cannot establish an effective match with any known category, their features may be further pushed towards the background category, resulting in missed detection of unknown foreground targets.

[0005] In recent years, large-scale visual language models have acquired open-vocabulary semantic priors through large-scale image-text alignment learning, providing a new technical path for out-of-distribution target recognition. Existing visual language model methods typically treat this as a post-processing module, performing text matching or semantic re-scoring on candidate regions already generated by the detector. While this paradigm can enhance the semantic discriminative ability of candidate regions, it still relies on the foreground localization ability of the base detector. Once an out-of-distribution target is suppressed to background during the candidate region generation stage, the subsequent visual language model cannot recover the target region, ultimately leading to the out-of-distribution target being ignored. Therefore, it is necessary to introduce the dense semantic priors derived from the visual language model into the detection process earlier, enhancing the foreground evidence of unknown regions before candidate region generation.

[0006] Despite the initial progress made in foreground-background decomposition methods, the following shortcomings still exist:

[0007] First, existing methods generally achieve out-of-distribution target rejection by suppressing in-distribution class responses to out-of-distribution features. However, target-level out-of-distribution detection inherently involves... The distribution within the category and the first This is a regional classification problem involving background categories. While simple response suppression can reduce the probability of out-of-distribution regions being misclassified as known categories, it may also weaken their foreground confidence, causing out-of-distribution foreground features to be absorbed into the background category, thus resulting in missed detections of unknown targets.

[0008] Second, existing advanced visual language modeling methods typically use a large visual language model as a post-processing module for text matching or semantic re-scoring of candidate regions generated by the detector. This paradigm still relies on the foreground localization capabilities of the base detector. Once out-of-distribution targets are suppressed as background during the candidate region generation stage, the subsequent large visual language model will be unable to recover them, ultimately leading to the ignoring of out-of-distribution targets.

[0009] Third, existing methods lack explicit modeling of the relationships between foreground, background, and out-of-distribution features. Background regions are typically highly heterogeneous and lack a unified semantic meaning. If only implicit classification boundaries are relied upon for distinction, out-of-distribution foreground features are easily affected by background manifold interference during training. Therefore, it is necessary to impose explicit constraints on the relationships between in-distribution and out-of-distribution features, establish independent discrimination boundaries for unknown foreground targets, and reduce the risk of them being absorbed by background categories. Summary of the Invention

[0010] This invention provides a method and system for detecting out-of-distribution targets based on explicit foreground feature modeling. The purpose is to solve the problem that existing target detection methods in open scenes lack explicit modeling and constraints on the distribution of foreground features, making it difficult to accurately distinguish and locate unknown out-of-distribution targets and easily misclassifying unknown targets as known categories, thus leading to insufficient detection reliability and accuracy.

[0011] Firstly, the purpose of this invention is to provide an out-of-distribution target detection method based on explicit foreground feature modeling, comprising the following steps:

[0012] S1: Obtain the in-distribution training dataset containing in-distribution samples, and the out-of-distribution auxiliary dataset containing out-of-distribution samples;

[0013] S2: Construct a target-level out-of-distribution detection model, which includes a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence;

[0014] S3: Read the in-distribution training dataset, construct corresponding text prompt features based on the in-distribution category names, input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model, and perform the first stage of training on the model by explicitly constraining the objective function through the in-distribution foreground features;

[0015] S4: Input the in-distribution training dataset, the out-of-distribution auxiliary dataset, and the text prompt features into the model that has completed the first stage of training, and perform the second stage of joint training on the model by combining the explicit constraint objective function of the in-distribution foreground features with the explicit constraint objective function of the out-of-distribution foreground features; the explicit constraint objective function of the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features to belong to any in-distribution Gaussian component;

[0016] S5: Input the image to be detected into the trained target-level out-of-distribution detection model, and calculate the energy score of the corresponding target prediction distribution based on the predicted logit output by the detection head;

[0017] S6: Compare the energy score with a preset judgment threshold. If the energy score is higher than the preset judgment threshold, the corresponding image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

[0018] Furthermore, a preferred embodiment is provided: the multimodal feature extractor includes a visual encoding branch and a text encoding branch; the visual encoding branch is used to extract multi-layer, multi-scale visual features of the input image and a global image feature vector, and the text encoding branch is used to encode the text prompt features and output text feature vectors corresponding to each preset category.

[0019] Furthermore, a preferred solution is provided: the context decoder uses the text feature vector as the query sequence and the global image feature vector as the context sequence, models the relationship between text semantics and visual foreground regions through multiple Transformer decoding layers, and outputs enhanced text features that fuse the current image visual context.

[0020] Furthermore, a preferred solution is provided: the processing procedure of the feature pyramid module includes:

[0021] Calculate the spatial similarity between the enhanced text features and the global image feature vector, and generate dense semantic response features that characterize the degree of semantic matching between each spatial region and category;

[0022] Guided by the dense semantic response features, the multi-layer, multi-scale visual features are subjected to layer-by-layer semantic refinement and top-down feature fusion to output a semantically enhanced multi-scale feature map.

[0023] Furthermore, a preferred solution is provided: the region candidate network generates candidate target regions on the semantically enhanced multi-scale feature map, and outputs the foreground confidence and bounding box regression offset of each candidate region;

[0024] The region of interest extraction module performs feature alignment operations on each candidate region to extract region-level features of a fixed size.

[0025] Furthermore, a preferred embodiment is provided: the objective function for explicitly constraining the foreground features within the distribution uses a Gaussian mixture distribution to model the feature distribution of the foreground region within the distribution, constraining the foreground features within the distribution to cluster within the distribution range of the corresponding Gaussian components.

[0026] Furthermore, a preferred embodiment is provided: the out-of-distribution foreground feature explicitly constrains the objective function to reduce the probability of the out-of-distribution foreground feature attributing to any in-distribution Gaussian component.

[0027] Secondly, the purpose of this invention is to propose an out-of-distribution target detection system based on explicit foreground feature modeling, for implementing the out-of-distribution target detection method based on explicit foreground feature modeling described in any one or more of the above-mentioned schemes, the system comprising:

[0028] Data acquisition unit: used to acquire the in-distribution training dataset containing in-distribution samples, and the out-of-distribution auxiliary dataset containing out-of-distribution samples;

[0029] Model building unit: used to build a target-level out-of-distribution detection model, the model including a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence;

[0030] The first training unit is used to read the in-distribution training dataset, construct corresponding text prompt features based on the in-distribution category names, input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model, and perform the first stage of training on the model by explicitly constraining the objective function through the in-distribution foreground features.

[0031] The second training unit is used to input the in-distribution training dataset, the out-of-distribution auxiliary dataset, and the text prompt features into the model that has completed the first stage of training. The second stage of joint training is performed on the model by combining the explicit constraint objective function of the in-distribution foreground features with the explicit constraint objective function of the out-of-distribution foreground features. The explicit constraint objective function of the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features being assigned to any in-distribution Gaussian component.

[0032] Feature calculation unit: used to input the image to be detected into the trained target-level out-of-distribution detection model, and calculate the energy score of the corresponding target prediction distribution based on the predicted logit output by the detection head;

[0033] Result determination unit: used to compare the energy score with a preset determination threshold. If the energy score is higher than the preset determination threshold, the corresponding image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

[0034] Thirdly, the present invention aims to provide a computer device, the computer device including a memory and a processor, the memory storing a computer program, and when the processor runs the computer program stored in the memory, the processor executes the out-of-distribution target detection method based on explicit foreground feature modeling according to any one or more of the above-described schemes.

[0035] Fourthly, the present invention aims to provide a computer-readable storage medium for storing a computer program that executes the out-of-distribution target detection method based on explicit foreground feature modeling described in any one or more of the above-described schemes.

[0036] Compared with the prior art, the advantages of the present invention are:

[0037] (1) By setting up a context decoder, the visual features are semantically enhanced by utilizing the cross-modal interaction relationship between text features and visual features, thereby reconstructing the original image-text matching method in CLIP into a pixel-text matching problem. The obtained pixel-text score map is used to guide the preservation of foreground perception features, so that the model can more fully retain the effective response of the target foreground region in the feature extraction and semantic enhancement stages.

[0038] (2) By adopting a Gaussian mixture distribution to explicitly reshape the in-distribution foreground features, the model can learn clearer and more discriminative in-distribution foreground feature representations. At the same time, by applying negative learning constraints to outlier data, the out-of-distribution foreground features are explicitly regularized, thereby keeping the out-of-distribution foreground features away from the in-distribution foreground feature distribution and background feature manifold, avoiding them from being misclassified as in-distribution categories or incorrectly absorbed into the background category.

[0039] (3) By combining in-distribution foreground feature modeling, out-of-distribution foreground feature constraints and energy score-based confidence assessment, this invention can improve out-of-distribution target detection performance while preserving out-of-distribution target foreground regions and preventing them from being absorbed into the background category.

[0040] This invention is applicable to target detection tasks that are highly related to safety in open scenarios such as autonomous driving, security monitoring, and industrial quality inspection. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a specific implementation of the out-of-distribution target detection method based on explicit foreground feature modeling as described in this invention.

[0043] Figure 2 This is a diagram of the out-of-distribution target detection structure based on explicit foreground feature modeling as described in this invention. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0045] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0047] Implementation Method 1

[0048] This embodiment proposes an out-of-distribution target detection method based on explicit foreground feature modeling, including the following steps:

[0049] S1: Obtain the in-distribution training dataset containing in-distribution samples and the out-of-distribution auxiliary dataset containing out-of-distribution samples;

[0050] S2: Construct a target-level out-of-distribution detection model, which includes a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence;

[0051] S3: Construct an explicit modeling objective function for in-distribution foreground features, read the in-distribution training dataset, and construct corresponding text prompt features based on the in-distribution category names. Input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model based on explicit modeling of foreground features, and optimize and train the target-level out-of-distribution detection model using the objective function explicitly constrained by the in-distribution foreground features.

[0052] S4: Construct an objective function with explicit constraints on out-of-distribution foreground features, read the in-distribution training dataset and the out-of-distribution auxiliary dataset containing out-of-distribution samples, input the in-distribution training dataset, the out-of-distribution auxiliary dataset and the text prompt features into the target-level out-of-distribution detection model based on explicit foreground feature modeling, and jointly optimize and train the target-level out-of-distribution detection model using the objective function with explicit constraints on in-distribution foreground features and the objective function with explicit constraints on out-of-distribution foreground features.

[0053] S5: Obtain the target-level out-of-distribution detection model based on explicit foreground feature modeling from the image to be detected, and calculate the energy score of the predicted distribution based on the feature vector generated by the model;

[0054] S6: Based on the calculated energy score, a confidence assessment is performed. If the score is greater than the preset threshold, the image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

[0055] Implementation Method 2

[0056] This embodiment is a further explanation of the target-level out-of-distribution detection model based on explicit foreground feature modeling in the out-of-distribution target detection method based on explicit foreground feature modeling described in Embodiment 1.

[0057] The construction steps of the target-level out-of-distribution detection model based on explicit foreground feature modeling are as follows:

[0058] S21: Constructing a multimodal feature extractor:

[0059] Let the multimodal input image be... The text information is .in, It is a series of learnable text context vectors. For distributed datasets The first object category The class name encoding of each category, The length of the learnable text context vector. For fixed embedding dimensions. The image... and text information The input is fed into a multimodal feature extractor, denoted as [missing information - likely a typo]. ,in Learnable parameters are extracted for multimodal feature extraction. The multimodal feature extractor includes a visual coding branch and a text coding branch, used to extract visual features of the image and semantic features of the text, respectively. After passing through the multimodal feature extractor, multi-layer, multi-scale visual features are obtained. Image feature vector and text feature vectors .in, is the training size in a batch, and is the output dimension of the text features. The output dimension for text features. Indicates visual feature hierarchy, and They represent the first The spatial height and spatial width of a layer's visual features are used to characterize the scale of the visual features at that layer, and different layers... and Corresponding to visual representations at different resolutions. In this embodiment, The image feature vector The text feature vector is used to represent the visual semantic response of an input image at different spatial locations. Used to represent the text semantic representation corresponding to each category within the distribution, providing input for subsequent context decoding and pixel-text matching.

[0060] S22: Construct the context decoder:

[0061] The image feature vector output by the multimodal feature extractor and text feature vectors In the input context decoder, the context decoder is denoted as... This results in enhanced text information. The context decoder includes a context information decoding layer, a text input layer, a visual input layer, and a text output layer. The text input layer consists of a batch normalization layer and a fully connected layer, used to process the text feature vector. Normalization and dimension mapping are performed; the visual input layer consists of a batch normalization layer and a fully connected layer, used to normalize and map image feature vectors. Normalization and dimensionality mapping are performed; the context information decoding layer consists of three Transformer decoding layers, used to model the relationship between text semantics and visual foreground features using text features as query information and image features as context information; the text output layer consists of fully connected layers, used to map the text features after context interaction into enhanced text information. Through the above context decoding process, the enhanced text information It can combine visual context information in the current input image to make the categorical text features pay more attention to the foreground target region in the image.

[0062] S23: Constructing a feature pyramid network:

[0063] Enhanced text features output by the context decoder Image feature vectors obtained from multimodal feature extractors Similarity calculations are performed to obtain the text-weighted region-level features, and the formula is as follows:

[0064] , (1)

[0065] in, This represents element-wise multiplication. This indicates summation along the feature dimension. This is a dense semantic response feature used to characterize the degree of matching between each spatial region in the image and the semantics of the category text. Subsequently, the dense semantic response feature... To guide the analysis of multi-layer visual features output by the multimodal feature extractor By refining the information layer by layer, we obtain semantically enhanced multi-scale features:

[0066] , (2)

[0067] in, Representation of feature pyramid network, This represents the learnable parameters in the feature pyramid network. , Indicates the first Layer semantic enhancement features.

[0068] Specifically, for the last layer of visual features First, the dense semantic response features Adjust to the same spatial dimensions as the visual features of that layer to obtain reuse The visual features of this layer are semantically weighted. Then, through lateral connections and top-down feature fusion, semantically enhanced multi-scale features are obtained. It is used to simultaneously preserve the local spatial structure information of the target and the foreground semantic priors provided by the vision-language model.

[0069] S24: Constructing the regional candidate network:

[0070] Multi-scale features with semantic enhancement Input the region candidate network, and denote the region candidate network as: .in, This represents the learnable parameters in the region candidate network. The region candidate network is used to generate candidate target regions on the semantically enhanced multi-scale feature map and outputs the foreground confidence and bounding box regression offset of the candidate regions. Specifically, for the ... Layer semantic enhancement features Multiple anchor frames of different sizes and aspect ratios are pre-set at each spatial location. ,in, This indicates the position corresponding to the first An anchor frame. Through convolutional layers... After feature mapping, the foreground confidence of the anchor boxes are obtained respectively. and bounding box regression offset , represented as:

[0071] , (3)

[0072] in, express Layer Each anchor box represents the confidence level of the foreground target. This represents the coordinate offset of the anchor frame relative to the actual target bounding box. The offset is then regressed based on the bounding box. For the corresponding anchor frame Position correction is performed to obtain the initial candidate region. : ,in, This indicates the decoding operation that generates candidate regions based on anchor frames and regression offsets. Subsequently, based on foreground confidence... All initial candidate regions are sorted, and non-maximum suppression is used to remove redundant candidate regions with overlap exceeding a preset threshold, resulting in the final candidate region set: .in, Indicates the first Candidate regions, This indicates the number of candidate regions retained.

[0073] S25: Constructing a Region of Interest Extraction Network:

[0074] The set of candidate regions output by the region candidate network Multi-scale features with semantic enhancement Input the region of interest extraction module to perform feature alignment and fixed-size feature extraction on each candidate region to obtain region-level features:

[0075] , (4)

[0076] in, Indicates the first The region-level features corresponding to each candidate region Indicates the first Candidate regions, This indicates the number of candidate regions. Further, the region-level features... Input detection head, the detection head is denoted as .in, This represents the learnable parameters in the detection head. The detection head performs classification prediction and bounding box regression on region-level features to obtain the category prediction results and bounding box regression results for the candidate regions. .in, Indicates the first The category prediction logit for each candidate region Indicates the first The bounding box regression offsets corresponding to each candidate region. For candidate regions After positional correction, the predicted bounding box is obtained: .in, Indicates the first The predicted bounding boxes corresponding to each candidate region.

[0077] Implementation Method 3

[0078] This embodiment is a further example illustrating the objective function of explicit foreground feature modeling in the out-of-distribution target detection method based on explicit foreground feature modeling described in Embodiment 1.

[0079] The steps for constructing the objective function for explicit modeling of the in-distribution foreground features are as follows:

[0080] S31. Obtain regional features within the distribution:

[0081] For the distributed region-level features obtained by the region of interest extraction network Based on their corresponding category labels and foreground markers, foreground features within the distribution are filtered. Let the first... The category labels for the candidate regions are: The current detection task has the following in-distribution class set: Then it will satisfy The candidate regions are considered as in-distribution foreground regions and denoted as the in-distribution foreground feature set. .

[0082] S32. Explicitly characterize the multimodal distribution structure of in-distribution foreground features. The set of in-distribution foreground features... The model is a Gaussian mixture distribution, denoted as:

[0083] , (5)

[0084] in, Indicates the number of Gaussian components. Indicates the first The mixing weights of Gaussian components, and They represent the first The mean vector and covariance matrix of each Gaussian component are calculated. Further, the regional features are calculated. The posterior responses belonging to each Gaussian component are denoted as:

[0085] , (6)

[0086] in, Representation of features The corresponding Gaussian component label, Representing regional characteristics Belongs to the The posterior responses of each Gaussian component are obtained. Based on these posterior responses, an explicit modeling objective function for the in-distribution foreground features is constructed such that the in-distribution foreground features can be stably constrained within the corresponding Gaussian mixture foreground distribution.

[0087] , (7)

[0088] in, Indicates the number of foreground features within the distribution. Indicates the number of Gaussian components. This represents the indicator function. To further encourage the extracted features to follow a Gaussian distribution, a regularization term is introduced to quantify the degree to which the features conform to the Gaussian distribution:

[0089] , (8)

[0090] in, This is used to constrain the foreground features within the distribution to cluster towards the corresponding Gaussian component center.

[0091] S33. The overall objective function obtained from the first stage of training can be expressed as:

[0092] , (9)

[0093] in, Indicates the target detection loss. and and represent the weight coefficients of the explicit modeling objective function for foreground features within the distribution and the Gaussian mixture regularization term, respectively. By minimizing the objective function, the model can learn the probability distribution boundary of foreground features within the distribution while simultaneously learning the ability to classify and locate objects within the distribution. This allows the foreground features to stably cluster in the corresponding Gaussian mixture foreground distribution, thus providing a distribution constraint basis for subsequently distinguishing unknown foreground features from known foreground features.

[0094] Implementation Method 4

[0095] This embodiment is a further illustrative example of the objective function of the out-of-distribution foreground feature explicit constraint in the out-of-distribution target detection method based on explicit foreground feature modeling described in Embodiment 1.

[0096] The process of constructing the objective function for the explicit constraint of the out-of-distribution foreground features is as follows:

[0097] S41. Obtain out-of-distribution regional features:

[0098] After completing the first stage of training in S3, continue using the aforementioned in-distribution training dataset. External auxiliary datasets In addition to textual prompt features, the foreground preservation detection model is jointly trained in the second stage. To reflect the stages of the training process, the total number of training rounds is [number missing]. In the case of, the former Each epoch uses only the in-distribution training dataset. Train the model; when the number of training epochs meets the requirements... At that time, the external auxiliary dataset will be distributed. Introducing the training process and comparing it with the in-distribution training dataset. They jointly participate in updating the model parameters, among which, Indicates the current training round number. This indicates the starting round number for introducing off-distribution auxiliary data.

[0099] Specifically, in the second phase of joint training, the in-distribution samples External auxiliary samples And a text-based feature-based foreground preservation detection model was used to obtain in-distribution and out-of-distribution region-level features, respectively. For out-of-distribution auxiliary data... The off-distribution candidate regions are obtained by sequentially inputting the feature extractor, context decoder, feature pyramid module, and region candidate network. For each out-of-distribution candidate region Region-level features are extracted using region-of-interest alignment operations, resulting in... .in, This represents the semantically enhanced features corresponding to out-of-distribution data. Indicates the first Regional-level features of candidate regions outside the distribution.

[0100] S42. Constructing out-of-distribution feature pseudo-labels:

[0101] For all region-level features obtained from out-of-distribution data, out-of-distribution pseudo-labels are uniformly assigned based on the negative learning approach. Let the set of out-of-distribution region-level features be... Then for any Build length is Negative learning pseudo-label vector in, This represents the number of categories within the distribution. The pseudo-label vector is composed of a random binary sequence and is used to specify the within-distribution categories that need to be excluded by the current out-of-distribution region-level features. When When, it indicates the first Within each distribution, the category is selected as the negative category; when When, it indicates the first Each category within a distribution does not participate in the current negative learning constraint;

[0102] S43. Construct an explicit modeling objective function for out-of-distribution foreground features:

[0103] Considering that out-of-distribution foreground region-level features obtained by the region of interest extraction network should not be absorbed into any in-distribution foreground component, an out-of-distribution foreground preservation constraint is constructed by forcing the output probability. Approaching zero:

[0104] a) of

[0105] , (12)

[0106] in, Indicates the number of foreground features outside the distribution. Indicates category Serves as a supplementary label. This is the current complementary tag set.

[0107] To further avoid confusion between in-distribution and out-of-distribution features, this implementation proposes a regularization technique tailored for out-of-distribution features. This technique pushes features away from Gaussian distributions of classes to which they do not belong:

[0108] , (13)

[0109] By minimizing the objective function, out-of-distribution features are spatially inferred into ID feature regions and their natural clustering is encouraged. This reduces the probability of out-of-distribution foreground features being assigned to any in-distribution Gaussian component while decreasing their absorption into the first distribution. Risks in each background category.

[0110] S44. The joint object detection loss, the in-distribution explicit foreground feature modeling objective function, the in-distribution Gaussian mixture regularization term, the out-of-distribution explicit foreground feature constraint objective function, and the out-of-distribution feature regularization term, together with the overall objective function for the second-stage joint training, can be expressed as:

[0111] , (14)

[0112] in, Indicates the target detection loss. and represents the weight coefficients of the objective function for explicit modeling of foreground features within the distribution and the Gaussian mixture regularization term, respectively. and and represent the weight coefficients of the explicit constraint objective function for out-of-distribution foreground features and the regularization term for out-of-distribution features, respectively. By minimizing the overall objective function of the second-stage joint training, the model can further enhance the clustering structure of in-distribution foreground features while maintaining the in-distribution object classification and localization capabilities, and suppress the aggregation of out-of-distribution foreground features into the in-distribution category feature space or background feature space.

[0113] Implementation Method 5

[0114] This embodiment proposes an out-of-distribution target detection system based on explicit foreground feature modeling, used to implement the out-of-distribution target detection methods based on explicit foreground feature modeling described in embodiments one to four. The system includes:

[0115] Data acquisition unit: used to acquire the in-distribution training dataset containing in-distribution samples, and the out-of-distribution auxiliary dataset containing out-of-distribution samples;

[0116] Model building unit: used to build a target-level out-of-distribution detection model, the model including a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence;

[0117] The first training unit is used to read the in-distribution training dataset, construct corresponding text prompt features based on the in-distribution category names, input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model, and perform the first stage of training on the model by explicitly constraining the objective function through the in-distribution foreground features.

[0118] The second training unit is used to input the in-distribution training dataset, the out-of-distribution auxiliary dataset, and the text prompt features into the model that has completed the first stage of training. The second stage of joint training is performed on the model by combining the explicit constraint objective function of the in-distribution foreground features with the explicit constraint objective function of the out-of-distribution foreground features. The explicit constraint objective function of the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features being assigned to any in-distribution Gaussian component.

[0119] Feature calculation unit: used to input the image to be detected into the trained target-level out-of-distribution detection model, and calculate the energy score of the corresponding target prediction distribution based on the predicted logit output by the detection head;

[0120] Result determination unit: used to compare the energy score with a preset determination threshold. If the energy score is higher than the preset determination threshold, the corresponding image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

[0121] Implementation Method Six

[0122] This embodiment is a further illustrative example of the out-of-distribution target detection method and system based on explicit foreground feature modeling described in the above embodiments.

[0123] The method described in this embodiment performs the steps of the methods described in Embodiments 1 to 4, wherein:

[0124] Step 1: Obtain the in-distribution training dataset containing in-distribution samples. ,in Indicates the first Target samples within a distribution, This represents the corresponding set of target annotations, where... Indicates the first The first image The bounding box of each target. The category label representing the target. Indicates the first Number of targets in the image This indicates the number of target categories within the distribution; similarly, an auxiliary dataset outside the distribution is obtained. ,in, Indicates the first Out-of-distribution auxiliary samples, This represents the total number of out-of-distribution auxiliary samples. These out-of-distribution auxiliary samples do not belong to the in-distribution category set and are used to provide out-of-distribution feature constraints during the training phase. In this embodiment, the PASCAL VOC dataset is used as the in-distribution training dataset, and the ImageNet1k validation set is used as the out-of-distribution auxiliary training dataset. The PASCAL VOC dataset provides category and bounding box annotations for in-distribution targets, while the ImageNet1k validation set provides auxiliary out-of-distribution samples that do not belong to the in-distribution category set, thereby enhancing the model's ability to constrain out-of-distribution features.

[0125] In this embodiment, the multimodal feature extractor adopts a CLIP-ResNet50 structure, including an image encoding branch and a text encoding branch. The image encoding branch is used to extract multi-layer visual semantic features from the input image and performs attention pooling on the final layer of visual features to obtain a global image feature vector. The text encoding branch is used to extract text prompt features. The multi-layer visual features The image feature vector is used to provide multi-scale visual input for subsequent feature pyramid networks. The text feature vector is used to represent the visual semantic response of an input image at different spatial locations. Used to represent the text semantic representation corresponding to each category within the distribution, providing input for subsequent context decoding and pixel-text matching.

[0126] In this embodiment, the region candidate network includes a shared convolutional layer, a foreground / background classification branch, and a bounding box regression branch. The shared convolutional layer extracts the local spatial features needed for candidate region generation, the foreground / background classification branch predicts whether each anchor box contains the target foreground, and the bounding box regression branch predicts the positional offset of the anchor box relative to the true target bounding box. This is achieved through multi-scale features after semantic enhancement. Constructing a regional candidate network allows the candidate region generation process to utilize both local spatial structure information and foreground semantic priors, thereby improving the recall rate of candidate target regions and reducing the interference of background regions and irrelevant regions outside the distribution on subsequent detection results.

[0127] In this embodiment, the region of interest extraction module employs the RoIAlign operation to avoid positional deviations caused by quantization rounding when mapping candidate regions to feature maps, thereby obtaining spatially aligned region-level features. The detection head includes two fully connected shared layers for processing region-level features, a classification branch for class prediction, and a regression branch for bounding box regression. The classification branch outputs the predicted logit of the candidate region belonging to each in-distribution target class, while the bounding box regression branch outputs the bounding box position offset corresponding to the candidate region. Through the region of interest extraction module and the detection head, the candidate regions generated by the region candidate network can be further transformed into target detection results with class semantics and positional coordinates, providing a region-level prediction foundation for subsequent in-distribution foreground feature modeling and out-of-distribution sample determination based on energy scores.

[0128] S5 in this embodiment specifically includes: acquiring the image to be detected and inputting it into a trained foreground-preserving detection model; calculating the energy score of the predicted distribution based on the feature vector generated by the detection model. Acquiring the image to be detected... and the image to be detected The trained foreground preservation detection model is input. This model sequentially passes through a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head to perform feature extraction, candidate region generation, and category prediction on the image to be detected, obtaining the region-level feature vector corresponding to the image to be detected.

[0129] , (15)

[0130] in, This represents the feature mapping function in the trained foreground-preserving detection model. This represents the model parameters after training. This represents the feature vector generated by the foreground preservation detection model.

[0131] Furthermore, the feature vector Input the detection head and obtain the corresponding predicted logit:

[0132] , (16)

[0133] in, Indicates the detection head, Indicates the parameters of the detection head. , This indicates that the image to be detected or the candidate region belongs to the first... The predicted logit for each class within a distribution. This indicates the number of target categories within the distribution. Simultaneously, the detection head also outputs bounding box regression results to obtain the predicted bounding boxes of the target candidate regions:

[0134] , (17)

[0135] in, This represents the candidate regions generated by the region candidate network. This represents the bounding box regression offset output by the detection head. This represents the predicted bounding box.

[0136] Therefore, the trained foreground-preserving detection model can obtain the predicted class logit, predicted bounding box, and corresponding region-level feature vector of the image to be detected, providing input for subsequent confidence evaluation and out-of-distribution sample determination based on the predicted logit energy score. Further, the predicted logit... Performing Softmax normalization yields the probability responses of the detected image or candidate region belonging to each category within its respective distribution: , (18)

[0137] in, This indicates that the image to be detected or the candidate region belongs to the first... The predicted probability of each class within a distribution. Indicates the first The predicted logit for each class within a distribution. This represents the number of target categories within the distribution. Based on the predicted probabilities of each category within the distribution, the in-distribution precision (mAP) can be further calculated.

[0138] S6 in this embodiment specifically includes: evaluating the credibility based on the calculated predicted logit energy; if the score is greater than a preset threshold, the image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample. The energy score is calculated based on the predicted logit.

[0139] , (19)

[0140] in, This represents the energy score corresponding to the image or candidate region to be detected. This represents the temperature parameter. The calculated energy score... With preset threshold Compare, if the following conditions are met: If the confidence level is low, it indicates that the image to be detected or the candidate region has a low confidence level in matching the in-distribution category set, and the image to be detected is determined to be an out-of-distribution sample; otherwise, the image to be detected is determined to be an in-distribution sample.

[0141] In this embodiment, the preset threshold The energy score distribution of in-distribution and out-of-distribution samples in the validation set can be used to determine the threshold. For example, an energy value that maintains a preset recall rate for in-distribution samples or reduces the false positive rate of in-distribution samples below a preset percentage can be selected as the threshold. Since in-distribution samples typically have a higher predictive logit response for their in-distribution categories, their energy scores are relatively low; while out-of-distribution samples have a weaker match with the in-distribution category set, their predictive logit response is lower, and their energy scores are relatively higher. Therefore, by comparing the energy scores with the preset threshold, the credibility of in-distribution and out-of-distribution samples can be assessed and determined.

[0142] To better verify the technical effectiveness of this invention, this embodiment first uses PASCAL VOC and Berkeley DeepDrive-100k as in-distribution datasets, and MS-COCO and OpenImages as out-of-distribution datasets, respectively. FPR95, AUROC, and mAP(ID) are used as evaluation metrics to assess the target-level out-of-distribution detection performance. Specifically, a lower FPR95 indicates a lower proportion of out-of-distribution samples being misclassified as in-distribution samples while maintaining a high in-distribution recall rate; a higher AUROC indicates a stronger ability of the model to distinguish between in-distribution and out-of-distribution samples; and mAP(ID) is used to evaluate the model's detection performance for in-distribution targets. As shown in Table 1, the method can effectively distinguish out-of-distribution samples while maintaining in-distribution detection performance.

[0143] Table 1. Regional-level out-of-region detection performance evaluation

[0144]

[0145] Secondly, OOD localization experiments were conducted to further verify the localization capability of the proposed method for out-of-distribution target regions or unknown target regions, as shown in Tables 1 and 2. In this experiment, the COCO-EXTEND and COCO-MIX datasets were used for evaluation, and FPR95, AUROC, U-F1, U-PRE, U-REC, AOSE, and WI were used as evaluation metrics. U-F1 represents the overall performance of unknown target detection, U-PRE represents the precision of unknown target detection, and U-REC represents the recall of unknown target detection. AOSE measures the number of unknown targets misidentified as known categories, and WI measures the degree of interference of unknown targets with the detection results of known categories. Generally, lower FPR95, AOSE, and WI indicate fewer misclassifications of unknown targets; higher AUROC, U-F1, U-PRE, and U-REC indicate stronger model recognition and localization capabilities for unknown target regions. As shown in Table 2, the proposed method can achieve good results in the detection and localization of unknown targets on both the COCO-EXTEND and COCO-MIX datasets. This indicates that by introducing explicit modeling of in-distribution foreground features and explicit constraints of out-of-distribution foreground features, the model can not only distinguish whether an image or candidate region belongs to an out-of-distribution sample, but also further locate the spatial region where out-of-distribution targets or unknown targets are located, thereby improving the reliability of target detection results in open scenes.

[0146] Table 2 Performance Evaluation of Off-Distribution Target Localization

[0147]

[0148] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An out-of-distribution target detection method based on explicit foreground feature modeling, characterized in that, Includes the following steps: S1: Obtain the in-distribution training dataset containing in-distribution samples, and the out-of-distribution auxiliary dataset containing out-of-distribution samples; S2: Construct a target-level out-of-distribution detection model, which includes a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence; S3: Read the in-distribution training dataset, construct corresponding text prompt features based on the in-distribution category names, input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model, and perform the first stage of training on the model by explicitly constraining the objective function through the in-distribution foreground features; S4: Input the in-distribution training dataset, the out-of-distribution auxiliary dataset, and the text prompt features into the model that has completed the first stage of training, and perform the second stage of joint training on the model by combining the explicit constraint objective function of the in-distribution foreground features with the explicit constraint objective function of the out-of-distribution foreground features; the explicit constraint objective function of the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features to belong to any in-distribution Gaussian component; S5: Input the image to be detected into the trained target-level out-of-distribution detection model, and calculate the energy score of the corresponding target prediction distribution based on the predicted logit output by the detection head; S6: Compare the energy score with a preset judgment threshold. If the energy score is higher than the preset judgment threshold, the corresponding image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

2. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 1, characterized in that, The multimodal feature extractor includes a visual encoding branch and a text encoding branch; the visual encoding branch is used to extract multi-layer, multi-scale visual features of the input image and global image feature vectors, and the text encoding branch is used to encode the text prompt features and output text feature vectors corresponding to each preset category.

3. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 2, characterized in that, The context decoder uses the text feature vector as the query sequence and the global image feature vector as the context sequence. It models the relationship between text semantics and visual foreground regions through multiple Transformer decoding layers and outputs enhanced text features that fuse the current image visual context.

4. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 3, characterized in that, The processing procedure of the feature pyramid module includes: Calculate the spatial similarity between the enhanced text features and the global image feature vector, and generate dense semantic response features that characterize the degree of semantic matching between each spatial region and category; Guided by the dense semantic response features, the multi-layer, multi-scale visual features are subjected to layer-by-layer semantic refinement and top-down feature fusion to output a semantically enhanced multi-scale feature map.

5. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 4, characterized in that, The region candidate network generates candidate target regions on the semantically enhanced multi-scale feature map and outputs the foreground confidence and bounding box regression offset of each candidate region. The region of interest extraction module performs feature alignment operations on each candidate region to extract region-level features of a fixed size.

6. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 1, characterized in that, The objective function for the explicit constraint of the in-distribution foreground features uses a Gaussian mixture distribution to model the feature distribution of the in-distribution foreground region, constraining the in-distribution foreground features to cluster within the distribution range of the corresponding Gaussian components.

7. The out-of-distribution target detection method based on explicit foreground feature modeling according to claim 1, characterized in that, The explicit constraint objective function for the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features being assigned to any in-distribution Gaussian component.

8. An out-of-distribution target detection system based on explicit foreground feature modeling, characterized in that, The system is used to implement the out-of-distribution target detection method based on explicit foreground feature modeling as described in any one of claims 1-7, the system comprising: Data acquisition unit: used to acquire the in-distribution training dataset containing in-distribution samples, and the out-of-distribution auxiliary dataset containing out-of-distribution samples; Model building unit: used to build a target-level out-of-distribution detection model, which includes a multimodal feature extractor, a context decoder, a feature pyramid module, a region candidate network, a region of interest extraction module, and a detection head connected in sequence; The first training unit is used to read the in-distribution training dataset, construct corresponding text prompt features based on the in-distribution category names, input the in-distribution training dataset and the text prompt features into the target-level out-of-distribution detection model, and perform the first stage of training on the model by explicitly constraining the objective function through the in-distribution foreground features. The second training unit is used to input the in-distribution training dataset, the out-of-distribution auxiliary dataset, and the text prompt features into the model that has completed the first stage of training. The second stage of joint training is performed on the model by combining the explicit constraint objective function of the in-distribution foreground features with the explicit constraint objective function of the out-of-distribution foreground features. The explicit constraint objective function of the out-of-distribution foreground features is used to reduce the probability of the out-of-distribution foreground features being assigned to any in-distribution Gaussian component. Feature calculation unit: used to input the image to be detected into the trained target-level out-of-distribution detection model, and calculate the energy score of the corresponding target prediction distribution based on the predicted logit output by the detection head; Result determination unit: used to compare the energy score with a preset determination threshold. If the energy score is higher than the preset determination threshold, the corresponding image to be detected is determined to be an out-of-distribution sample; otherwise, it is determined to be an in-distribution sample.

9. A computer device, characterized in that, The computer device includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the out-of-distribution target detection method based on explicit foreground feature modeling according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program that executes the out-of-distribution target detection method based on explicit foreground feature modeling as described in any one of claims 1-7.

Images