A method and system for detecting out-of-distribution data

Abstract

The application provides a distribution out-of-distribution data detection method and system, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining a semantic label of image data to be processed; when the semantic label belongs to a vertical field, inputting the image data to be processed into a first detection model to obtain a distribution out-of-distribution detection score of the image data to be processed output by the first detection model; wherein the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples and learnable tokens; and comparing the distribution out-of-distribution detection score with a preset threshold to obtain a distribution out-of-distribution detection result. The application utilizes the first detection model combined with the image data samples, the corresponding training semantic labels and the learnable tokens when the semantic label belongs to the vertical field, compares the distribution out-of-distribution detection score output by the first detection model with the preset threshold, detects the distribution out-of-distribution data, and thus improves the accuracy of the detection of the distribution out-of-distribution data in the vertical field data.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for detecting out-of-distribution data. Background Technology

[0002] In recent years, with the in-depth application of artificial intelligence technology in various industries, data-driven detection models have become an important part of ensuring system security and stable business operation.

[0003] In existing technologies, most methods do not fully utilize the semantic labels of image data during training and lack model customization mechanisms for different domain data. This results in detection models exhibiting insufficient generalization ability and low discrimination accuracy when processing images with specific business semantics in vertical domains. Due to the failure to effectively introduce semantic labels to classify data into domains, the detection models show low accuracy when faced with specialized semantic descriptions.

[0004] Therefore, improving the accuracy of testing has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides a method, system, electronic device, and storage medium for detecting distributed data, in order to overcome the deficiencies in the prior art and improve the accuracy of detection.

[0006] This invention provides a method for detecting out-of-distribution data, comprising the following steps: Obtain the semantic labels of the image data to be processed; When the semantic label belongs to a vertical domain, the image data to be processed is input into the first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained from image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; The out-of-distribution detection score is compared with a preset threshold to obtain the out-of-distribution detection result.

[0007] The out-of-distribution data detection method provided by the present invention further includes a training method for the first detection model: Acquire multiple image data samples and training semantic labels corresponding to each image data sample; A preset number of learnable tokens are inserted into multiple Transformer blocks of the visual encoder and text encoder of the contrastive learning multimodal model to obtain a multimodal model structure containing the learnable tokens. For any of the image data samples, the visual encoder extracts features from the image data sample to obtain training image features, and the text encoder extracts features from the corresponding training semantic labels to obtain text semantic features. A loss function is constructed based on all the training image features and all the text semantic features; Based on the loss function, the multimodal model structure is trained by backpropagation until the model converges, thus obtaining the first detection model.

[0008] According to the out-of-distribution data detection method provided by the present invention, the step of constructing a loss function based on all the training image features and all the text semantic features includes: Perform group normalization and convolution operations on any of the training image features to obtain optimized training image features, and perform layer normalization and linear mapping operations on any of the text semantic features to obtain optimized text semantic features. The optimized training image features and the optimized text semantic features are mapped to a common feature space; Based on the optimized training image features of all images labeled with the same training semantic label, the mean value of the image features of that training semantic label is calculated, and the visual matching score is calculated based on the optimized training image features and the mean value of the image features. Based on the optimized text semantic features of all training semantic labels, the mean semantic text feature of the training semantic label is calculated, and the text matching score is calculated based on the optimized text semantic features and the mean semantic text feature. A loss function is constructed based on the visual matching score and the text matching score to constrain the matching degree between the optimized training image features and the optimized text semantic features, thereby improving the discriminative power of in-distribution and out-of-distribution samples in the common feature space.

[0009] According to the out-of-distribution data detection method provided by the present invention, the step of constructing a loss function based on the visual matching score and the text matching score includes: Based on the visual matching score and the corresponding training semantic label, calculate the cross-entropy loss of the visual branch; Based on the text matching score and the corresponding training semantic label, calculate the cross-entropy loss of the text branch; The cross-entropy loss function is obtained by adding the cross-entropy loss of the visual branch to the cross-entropy loss of the text branch.

[0010] According to the out-of-distribution data detection method provided by the present invention, the step of inputting the image data to be processed into a first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model includes: The image data to be processed is input into the first detection model to obtain the image feature vector of the image data to be processed; The vector database is used to retrieve multiple positive sample feature vectors that have the highest similarity to the image feature vectors, forming a similarity group feature vector. The vector database is used to retrieve multiple negative sample feature vectors with the lowest similarity to the image feature vectors, forming a dissimilar group feature vector. Using radial basis function kernel function, a Gram matrix is ​​constructed based on the feature vectors of the similar group and the feature vectors of the dissimilar group, and the Gram matrix is ​​centered to obtain a centered kernel matrix; The out-of-distribution detection score of the image data to be processed is calculated based on the kernel matrix.

[0011] According to the out-of-distribution data detection method provided by the present invention, the step of calculating the out-of-distribution detection score of the image data to be processed based on the kernel matrix includes: Calculate the similarity score between the image data to be processed and the feature vectors of the dissimilar groups based on the kernel matrix; The similarity scores are normalized to obtain normalized similarity scores; Based on the normalized similarity scores, calculate the mean of the dissimilar groups and the variance of the dissimilar groups; The Z-score is used to standardize the mean and variance of the dissimilar groups to obtain the out-of-distribution detection score of the image data to be processed.

[0012] The out-of-distribution data detection method provided by the present invention further includes: When the semantic label belongs to the horizontal domain, the image data to be processed is input into the second detection model to obtain the out-of-distribution detection score of the image data to be processed output by the second detection model; wherein, the second detection model is a zero-shot learning model.

[0013] According to the out-of-distribution data detection method provided by the present invention, the step of comparing the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result includes: When the out-of-distribution detection score is greater than the preset threshold, the image data to be processed is determined to be in-distribution image data; When the out-of-distribution detection score is less than or equal to the preset threshold, the image data to be processed is determined to be out-of-distribution image data.

[0014] The present invention also provides an off-distribution data detection system, comprising the following modules: The acquisition module is used to acquire semantic tags for the image data to be processed; The first processing module is used to input the image data to be processed into the first detection model when the semantic label belongs to a vertical domain, and obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; The second processing module is used to compare the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the out-of-distribution data detection method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the out-of-distribution data detection method as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the out-of-distribution data detection method as described above.

[0018] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: By first acquiring semantic labels from the image data to be processed, the system can determine the domain to which the image data belongs based on the labels, thus selecting a more targeted detection model when vertical domain data appears. After determining that the semantic labels belong to the vertical domain, the image data to be processed is input into the first detection model, significantly improving the out-of-distribution detection accuracy for vertical domain scenarios, thereby avoiding the misjudgment or missed detection problems that general detection models may encounter in specific domains. By training the first detection model in combination with image data samples, corresponding training semantic labels, and learnable tokens during the training phase, the model has a higher alignment capability in the joint representation of visual and textual features, thereby enhancing the model's feature extraction and discrimination performance for vertical domain data. By comparing the out-of-distribution detection score output by the first detection model with a preset threshold, rapid identification of out-of-distribution data can be achieved through threshold determination, effectively distinguishing business-related data from business-irrelevant data and improving detection accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts of the off-distribution data detection method provided by the present invention.

[0021] Figure 2 This is the second flowchart of the off-distribution data detection method provided by the present invention.

[0022] Figure 3 This is the third flowchart of the off-distribution data detection method provided by the present invention.

[0023] Figure 4 This is the fourth flowchart of the off-distribution data detection method provided by the present invention.

[0024] Figure 5 This is the fifth flowchart of the off-distribution data detection method provided by the present invention.

[0025] Figure 6 This is the sixth flowchart of the off-distribution data detection method provided by the present invention.

[0026] Figure 7 This is a schematic diagram of the structure of the distributed off-grid data detection system provided by the present invention.

[0027] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] It should be noted that in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0030] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0031] The following is combined Figures 1-8 This invention describes the distributed data detection method, system, electronic device, and storage medium provided by the present invention.

[0032] Figure 1 This is one of the flowcharts of the out-of-distribution data detection method provided by the present invention, such as... Figure 1 As shown, including but not limited to the following steps: Step 101: Obtain the semantic labels of the image data to be processed.

[0033] In this embodiment, the semantic tags of the image data to be processed are first obtained. The semantic tags are textual descriptions of the domain or category to which the image data content belongs.

[0034] In practical business applications, image data from different domains exhibit significantly different semantic distribution characteristics. This is especially true between images from vertical domains (such as medical, industrial, or specific business scenarios) and horizontal domains (such as general natural scenes), where there are often large differences in data distribution and low generalization performance of out-of-distribution data detection methods. Therefore, before performing out-of-distribution data detection, it is necessary to first clarify the semantic labels of the image data to be processed, thereby determining the domain to which the image data belongs.

[0035] In practice, text classification models (such as BERT, RoBERTa, and other pre-trained language models) or predefined domain lexicons can be used. Through semantic retrieval, classification model inference, or keyword matching, the text information corresponding to the image data to be processed can be automatically analyzed, and then the semantic labels of the image data to be processed can be extracted to determine the specific domain corresponding to the image data.

[0036] By acquiring and analyzing the semantic tags of the image data to be processed in this embodiment, the specific domain corresponding to the image data to be processed can be accurately determined. This provides a clear basis for selecting a suitable out-of-distribution data detection model in subsequent steps, thereby improving the detection accuracy, generalization ability and computational efficiency of subsequent out-of-distribution detection steps, and ensuring the effectiveness and reliability of out-of-distribution data detection.

[0037] Step 102: When the semantic label belongs to the vertical domain, the image data to be processed is input into the first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples, and learnable tokens.

[0038] In this embodiment, when it is determined through step 101 that the semantic label of the image data to be processed belongs to a vertical domain, the method inputs the image data to be processed into the first detection model to obtain the out-of-distribution detection score of the image data to be processed. The first detection model is trained using specific image data samples, the corresponding training semantic labels of the image data samples, and learnable tokens.

[0039] In practical applications, different vertical domains (such as healthcare, industrial manufacturing, or specific business scenarios) have unique data distribution characteristics, with more concentrated data semantics and more explicit domain knowledge. Therefore, traditional, general-purpose zero-shot learning models often perform poorly on out-of-distribution detection tasks within vertical domains, failing to meet accuracy and generalization requirements. In view of this, this embodiment introduces a first detection model specifically optimized and trained for the data characteristics of vertical domains, effectively improving the performance of out-of-distribution detection tasks within vertical domains.

[0040] In subsequent embodiments, the training method and practical application method of the first detection model will be described in detail.

[0041] In one possible implementation, refer to Figure 2 , Figure 2 This is the second flowchart of the out-of-distribution data detection method provided by the present invention. The training method of the first detection model specifically includes steps 201-205: Step 201: Obtain multiple image data samples and the training semantic labels corresponding to each image data sample.

[0042] In this embodiment, to effectively train the first detection model, it is first necessary to acquire multiple image data samples and the corresponding training semantic labels for each image data sample. The image data samples provide rich and diverse visual information for subsequent model training, while the training semantic labels are used to clarify the specific category or semantic domain represented by these image data samples, thereby enabling the model to effectively learn the inherent mapping relationship between visual features and semantic labels.

[0043] In practice, the acquisition of training semantic labels can be accomplished through manual annotation, automatic classification tools, or matching with domain knowledge bases. For example, in an industrial production environment, image data samples of production lines for different types of products can be collected, and these image data can be classified and labeled according to product type, appearance characteristics, or process stage to generate training semantic labels that correspond one-to-one with the image data samples. As another example, in the transportation sector, various vehicle image data samples can be acquired from monitoring or acquisition equipment, and these images can be semantically labeled according to information such as vehicle type, brand, model, or purpose to form training semantic labels.

[0044] Step 202: Insert a preset number of learnable tokens into multiple Transformer blocks of the visual encoder and text encoder of the contrastive learning multimodal model to obtain a multimodal model structure containing learnable tokens.

[0045] In this embodiment, to further improve the recognition performance of the first detection model on out-of-distribution data in the vertical domain, this method inserts a preset number of learnable tokens into the visual encoder and text encoder of the contrastive learning multimodal model (e.g., the CLIP model), thereby constructing a multimodal model structure containing learnable tokens. This effectively improves the model's feature representation ability for vertical domain data and the accuracy of image-text feature mapping.

[0046] Specifically, in the implementation of the visual encoder, this embodiment first introduces b learnable tokens into the visual encoder, represented as... In the Transformer architecture of a visual encoder, starting with the first Transformer block V1, each Transformer block V... i New learnable tokens are introduced into the input sequences, specifically represented as , where square brackets This indicates a feature concatenation operation. For category tokens, For image block features, The introduced learnable tokens are executed in this manner until the J-th Transformer block V. J In the Jth Transformer block V J The subsequent Transformer block V J+1 To V K In the middle, the model continues to calculate the output feature sequence using the same concatenation method, that is... .

[0047] Similarly, in the implementation of the text encoder, this embodiment also introduces b learnable tokens. These learnable tokens are then compared with the original fixed input tokens. Together, they serve as the initial input sequence for the text encoder, i.e., the input features are... In the Transformer structure of the text encoder, starting from the first Transformer block L1, each Transformer block L... i A new learnable token is introduced, represented as , where square brackets Similarly, representing a concatenation operation, Pi is the learnable token sequence corresponding to the i-th layer, and Wi is the text token sequence. This method of introducing learnable tokens continues up to the J-th Transformer block L. J In the Jth Transformer block L J After that, the subsequent Transformer block L J+1 Continue performing the above calculations.

[0048] Step 203: For any image data sample, the visual encoder extracts features from the image data sample to obtain training image features, and the text encoder extracts features from the corresponding training semantic labels to obtain text semantic features.

[0049] In this embodiment, in order to achieve an accurate feature mapping relationship between image data and corresponding training semantic labels, thereby improving the out-of-distribution detection capability of the first detection model, this method further extracts features from the image data samples and the corresponding training semantic labels respectively to obtain effective training image features and text semantic features.

[0050] In the specific implementation process, for any image data sample, this embodiment first inputs it into the visual encoder containing learnable tokens constructed in step 202. Through multiple Transformer blocks in the visual encoder, the input image data sample is subjected to step-by-step feature extraction and feature transformation, and finally the high-dimensional visual feature expression corresponding to the image data sample is obtained, that is, the training image feature.

[0051] Simultaneously, in this embodiment, the training semantic labels corresponding to the image data sample are input into the text encoder containing learnable tokens constructed in step 202. Multiple Transformer blocks in the text encoder extract and transform the semantic features of the input training semantic labels, ultimately obtaining the text semantic features corresponding to the image data sample. These text semantic features can clearly express the key information of the image data sample in the semantic dimension.

[0052] By extracting training image features and text semantic features using visual encoders and text encoders respectively, an accurate and reliable feature foundation can be provided for subsequent optimization of feature representation, construction of loss function, and feature space mapping.

[0053] Step 204: Construct a loss function based on all training image features and all text semantic features.

[0054] In this embodiment, in order to further improve the generalization ability and accuracy of the first detection model in the vertical domain out-of-distribution data detection task, after completing feature extraction of all image data samples and all corresponding training semantic labels to obtain training image features and text semantic features, this method further utilizes these features to construct a specially designed loss function to optimize the matching degree of visual and text features, and significantly enhance the feature discrimination ability of the model.

[0055] The specific implementation method of step 204 will be described in detail in subsequent embodiments.

[0056] Step 205: Based on the loss function, perform backpropagation training on the multimodal model structure until the model converges, and obtain the first detection model.

[0057] In this embodiment, in order to enable the first detection model to accurately and efficiently perform the task of detecting out-of-distribution data in the vertical domain, this method uses the loss function constructed in step 204 to perform backpropagation training on the multimodal model structure until the model converges, thereby obtaining the trained and optimized first detection model.

[0058] Specifically, this embodiment first calculates the cross-entropy loss of the visual branch and the cross-entropy loss of the text branch based on the loss function constructed in step 204. The cross-entropy loss of the visual branch is calculated using the difference between the visual matching score and the corresponding training semantic label, effectively constraining the matching relationship between the optimized training image features and the corresponding semantic label. The cross-entropy loss of the text branch is calculated using the difference between the text matching score and the corresponding training semantic label, constraining the matching relationship between the optimized text semantic features and the corresponding semantic label.

[0059] Next, this embodiment adds the cross-entropy loss of the visual branch to the cross-entropy loss of the text branch to obtain the final cross-entropy loss function, which is then used as the optimization objective during model training. Then, the gradient backpropagation algorithm is used to propagate the error information of the loss function back layer by layer to the visual encoder and text encoder of the multimodal model structure. By continuously adjusting and updating the network parameters in the model (especially the learnable token parameters in the Transformer block), the model's prediction error is continuously reduced until the model's loss function converges to a stable state.

[0060] Through this backpropagation training process, the model can fully learn the inherent correlation between training image features and text semantic features, enabling the optimized multimodal model structure to exhibit higher feature representation and generalization capabilities on vertical domain data. This effectively improves the accuracy, stability, and robustness of the first detection model in distinguishing between in-distribution and out-of-distribution data in subsequent practical applications, ultimately achieving more accurate and reliable out-of-distribution data detection results.

[0061] In one possible implementation, refer to Figure 3 , Figure 3 This is the third flowchart of the out-of-distribution data detection method provided by the present invention. Step 204 specifically includes steps 301-305: Step 301: Perform group normalization and convolution operations on any training image feature to obtain optimized training image features, and perform layer normalization and linear mapping operations on any text semantic feature to obtain optimized text semantic features.

[0062] In this embodiment, in order to further improve the accuracy and robustness of feature representation in the first detection model, so as to more effectively distinguish between in-distribution data and out-of-distribution data in the vertical domain, this method performs feature optimization processing on the training image features and text semantic features extracted in step 203 respectively, so as to obtain more accurate and stable feature representation.

[0063] Specifically, for any training image feature, this embodiment employs group normalization and convolution operations to optimize its features, effectively reducing noise and redundant information in the image features. Group normalization divides the training image features into multiple feature channel groups and normalizes each group separately, making the feature distribution more stable. Convolution operations further extract local spatial structure information from the training image features, enhancing the features' ability to express fine-grained image content, ultimately obtaining optimized training image features. .

[0064] Similarly, for any text semantic feature, this embodiment employs layer normalization and linear mapping operations for optimization. Layer normalization effectively balances text features of different dimensions, reduces the bias between feature dimensions, and makes the text semantic features more stable and consistent. Subsequently, the linear mapping operation further projects the text semantic features onto a more suitable feature representation space, thereby obtaining the optimized text semantic features. This enhances the semantic alignment between features and image features.

[0065] By performing the aforementioned feature optimization operations on the training image features and text semantic features respectively, this embodiment significantly improves the accuracy and stability of feature representation, providing a more reliable feature foundation for subsequently mapping these features to a common feature space and constructing a loss function.

[0066] Step 302: Map the optimized training image features and optimized text semantic features to a common feature space.

[0067] In this embodiment, in order to accurately measure and optimize the matching degree between training image features and text semantic features, and further improve the generalization performance of the first detection model and the detection accuracy of out-of-distribution data, this method maps the optimized training image features and optimized text semantic features in step 301 into a common feature space, so as to facilitate more accurate and direct feature comparison and matching calculation.

[0068] In the specific implementation, this embodiment designs corresponding mapping functions (such as linear mapping or MLP multilayer perceptron mapping structure) for the optimized training image features and optimized text semantic features, respectively. These mapping functions are then used to project visual features and text features into a unified, low-dimensional common feature space. In this common feature space, feature representations from different modalities (visual and textual) can be directly compared, thereby more accurately capturing and measuring the matching relationship between visual features and corresponding semantic features.

[0069] The reason for mapping visual features and text features to a common feature space is that the original feature spaces of image data and text data usually differ significantly, making it difficult to accurately compare or calculate the similarity or matching degree between features. The common feature space mapping method in this embodiment effectively eliminates the differences between visual features and text semantic features, enabling the measurement of feature matching degree at a unified scale.

[0070] Step 303: Calculate the mean value of the image features of the training semantic label based on the optimized training image features of all images labeled with the same training semantic label, and calculate the visual matching score based on the optimized training image features and the mean value of the image features.

[0071] In this embodiment, in order to accurately measure the degree of matching between the image data to be processed and its corresponding semantic category in the dimension of visual features, this method further calculates the mean value of the image features of each training semantic label based on the optimized training image features mapped to the common feature space in step 302, and calculates the visual matching score accordingly, thereby achieving a fine characterization of the relationship between visual features and semantic categories.

[0072] Specifically, this embodiment first obtains the mean image feature value for each training semantic label by calculating the average value of the optimized training image features labeled with the same training semantic label. As a visual feature prototype of this semantic category, it effectively represents the typical feature expression of image data of this category.

[0073] In calculating the visual matching score, this embodiment uses the following formula: ; in, This represents the optimized training image features mapped to the common feature space in step 302. Represents training semantic labels The corresponding image feature mean, The preset temperature coefficient is typically used to adjust the hardness or softness of the feature space to control the sensitivity and smoothness of the feature distribution. cos() represents the cosine similarity calculation, which is used to measure the similarity between the optimized training image features and the mean of the image features.

[0074] The visual matching score calculated using the above method This study clearly quantifies the matching relationship between each image data feature and the prototype of each category feature, enabling the model to effectively distinguish the differences between visual features in different semantic categories and improving the discriminative power of visual features in subsequent loss function calculation and model optimization.

[0075] Step 304: Calculate the mean semantic text feature of the training semantic label based on the optimized text semantic features of all labels with the same training semantic label, and calculate the text matching score based on the optimized text semantic features and the mean semantic text feature.

[0076] In this embodiment, in order to further achieve accurate quantification of the matching relationship between the text semantic features of the image data to be processed and the corresponding semantic categories, this method calculates the mean of the semantic text features of each training semantic label based on the optimized text semantic features mapped to the common feature space in step 302, and further calculates the text matching score accordingly, thereby finely characterizing the association between text features and semantic categories.

[0077] Specifically, this embodiment first calculates the mean semantic text feature of each training semantic label by averaging the optimized text semantic features of all labeled with the same training semantic label. As a prototype of text features for this semantic category, it accurately represents the typical feature expression of text semantic data of this category.

[0078] Subsequently, this embodiment uses the following formula to calculate the text matching score: ; in, The optimized text semantic features mapped to the common feature space in step 302. To train semantic labels The corresponding semantic text feature mean, The preset temperature coefficient is used to adjust the sensitivity of feature similarity calculation. cos() represents cosine similarity calculation, which measures the similarity between the optimized text semantic features and the mean of the corresponding semantic text features.

[0079] The text matching score calculated using the above method This study clearly quantifies the matching relationship between text semantic features and semantic category feature prototypes, enabling the model to effectively capture semantic differences between different categories, thereby providing an accurate and stable computational foundation for the subsequent construction of the loss function.

[0080] Step 305: Construct a loss function based on the visual matching score and the text matching score to constrain the matching degree between the optimized training image features and the optimized text semantic features, thereby improving the discriminative power of in-distribution and out-of-distribution samples in the common feature space.

[0081] In this embodiment, in order to effectively constrain the matching degree of training image features and text semantic features in the common feature space, thereby improving the ability of the first detection model to distinguish data outside the vertical domain, this method constructs a specific loss function based on the visual matching score and text matching score calculated in steps 303 and 304 respectively, so as to accurately measure and further optimize the matching relationship between visual and text features.

[0082] In one possible implementation, refer to Figure 4 , Figure 4 This is the fourth flowchart of the out-of-distribution data detection method provided by the present invention. Step 305 specifically includes steps 401-403: Step 401: Calculate the cross-entropy loss of the visual branch based on the visual matching score and the corresponding training semantic label.

[0083] Step 402: Calculate the cross-entropy loss of the text branch based on the text matching score and the corresponding training semantic label.

[0084] Step 403: Add the cross-entropy loss of the visual branch to the cross-entropy loss of the text branch to obtain the cross-entropy loss function.

[0085] In this embodiment, in order to more accurately optimize the matching relationship between visual features and text semantic features, thereby enhancing the generalization performance of the first detection model on vertical domain data and the detection accuracy of out-of-distribution data, this method further constructs a specific cross-entropy loss function based on the visual matching score and text matching score obtained in step 305, so as to clearly quantify and optimize the model's learning effect on feature matching relationship.

[0086] Specifically, this embodiment first uses visual matching scores... Calculate the cross-entropy loss of the visual branch using the corresponding training semantic label y. ,Right now: ; The cross-entropy loss of this visual branch is used to explicitly measure the difference between the optimized training image features and the corresponding training semantic labels, thereby guiding the model to effectively learn the matching relationship between visual features and their corresponding semantic categories.

[0087] Meanwhile, this embodiment uses text matching scores... Calculate the cross-entropy loss of the text branches using the corresponding training semantic labels y. ,Right now: ; The cross-entropy loss of this text branch explicitly quantifies the difference between the optimized text semantic features and the corresponding training semantic labels, enabling the model to learn and accurately capture the semantic mapping relationship between text features and corresponding categories.

[0088] Subsequently, this embodiment further incorporates the cross-entropy loss of the aforementioned visual branches. Cross-entropy loss with text branches Adding them together yields the overall cross-entropy loss function. ,Right now: ; By constructing the overall cross-entropy loss function using the above method, the model can simultaneously optimize the representation of both visual and textual feature modalities during training, enabling them to cluster more accurately near the feature prototypes of their corresponding categories in the common feature space. This improves the model's ability to distinguish between in-distribution and out-of-distribution data, effectively enhancing the generalization, stability, and detection accuracy of the first detection model in vertical domain data scenarios.

[0089] In one possible implementation, refer to Figure 5 , Figure 5 This is the fifth flowchart of the out-of-distribution data detection method provided by the present invention. Step 102 specifically includes steps 501-505: Step 501: Input the image data to be processed into the first detection model to obtain the image feature vector of the image data to be processed.

[0090] In this embodiment, in order to accurately obtain the out-of-distribution detection score of the image data to be processed, thereby achieving effective detection of out-of-distribution data in the vertical domain, this method first inputs the image data to be processed into the first detection model trained and optimized in step 205 to extract the image feature vector corresponding to the image data to be processed.

[0091] Specifically, this embodiment utilizes the visual encoder in the first detection model trained in step 205 to perform step-by-step feature extraction on the image data to be processed. The extracted visual features are then mapped to a unified feature space through a mapping function, thereby obtaining the image feature vector of the image data to be processed. This image feature vector highly condenses and expresses the visual information and semantic features of the original image data in the common feature space, effectively capturing the domain-specific visual semantic information of the image data.

[0092] Step 502: Retrieve multiple positive sample feature vectors with the highest similarity to the image feature vectors from the vector database, and form a similarity group feature vector.

[0093] In this embodiment, in order to more accurately determine whether the image data to be processed belongs to the out-of-distribution data in the vertical domain, this method further utilizes the image feature vector of the image data to be processed extracted in step 501 to retrieve multiple positive sample feature vectors with the highest similarity in the pre-constructed Milvus vector database, so as to form a similar group feature vector for subsequent analysis.

[0094] Specifically, this embodiment uses the image feature vectors of the image data to be processed and the efficient retrieval function provided by the Milvus vector database to perform feature matching using cosine similarity as an indicator. The formula for calculating cosine similarity is: ; In the actual retrieval process, this embodiment uses the aforementioned cosine similarity calculation formula to sequentially compare the similarity between the image feature vector of the image data to be processed and all positive sample image feature vectors stored in the database. The top L positive sample feature vectors are selected in descending order of similarity to form a similarity group feature vector. This similarity group feature vector significantly represents the image data features most similar to the image data to be processed in the feature space, reflecting the close matching relationship between the image data and known business-related data.

[0095] Step 503: Retrieve multiple negative sample feature vectors with the lowest similarity to the image feature vectors from the vector database to form a dissimilar group feature vector.

[0096] In this embodiment, in order to accurately assess the degree of difference between the image data to be processed and the business data, and to improve the accuracy and robustness of out-of-distribution data detection, this method, while constructing similar group feature vectors, further retrieves multiple negative sample feature vectors with the lowest similarity to the image feature vectors of the image data to be processed obtained in step 501 from the Milvus vector database to form dissimilar group feature vectors.

[0097] According to the calculation formula in step 502, this embodiment sequentially calculates the similarity between the image feature vector of the image data to be processed and all negative sample image feature vectors stored in the database, and selects the top 2L negative sample feature vectors in order of increasing similarity to form the dissimilar group feature vector. This dissimilar group feature vector can significantly represent the set of image data features that differ most from the image data to be processed in the feature space, reflecting the typical difference characteristics between the image data to be processed and known business-irrelevant data.

[0098] Step 504: Using the radial basis function kernel function, construct the Gram matrix based on the eigenvectors of similar groups and dissimilar groups, and center the Gram matrix to obtain the centered kernel matrix.

[0099] In this embodiment, in order to more accurately measure the feature relationship between the image data to be processed and the feature vectors of similar groups and dissimilar groups, thereby improving the out-of-distribution detection capability of vertical domain data, this method further constructs a complete Gram matrix based on the feature vectors of similar groups and dissimilar groups obtained in steps 502 and 503 using the radial basis function (RBF) kernel function, and performs centering processing on the Gram matrix to effectively eliminate data offset and redundant information in the feature space.

[0100] In specific implementation, this embodiment first uses a radial basis function kernel function to calculate the feature similarity between the feature vector of the image data to be processed and the feature vectors of similar groups and dissimilar groups. The radial basis function kernel function is defined as follows: ; Using the kernel function described above, this embodiment calculates the pairwise feature similarity between the feature vectors of the image data to be processed, their own feature vectors, similar group feature vectors, and dissimilar group feature vectors, thereby constructing the complete Gram matrix K. : ; The Gram matrix contains fine-grained feature relationships between the features of the image data to be processed and the features of similar and dissimilar groups, and can accurately characterize the relationships between various feature vectors in the feature space.

[0101] To further eliminate the influence of data offset in the feature space and enhance the stability of feature similarity calculation, this embodiment subsequently introduces a centering matrix H to center the Gram matrix. Specifically, the centering matrix is ​​defined as: ; Using the above-described centering matrix, this embodiment processes the Gram matrix to obtain the centering kernel matrix: ; The centralized kernel matrix obtained after the above radial basis function kernel matrix construction and centering process is... This effectively eliminates the influence of redundant information and data distribution bias in the original feature space, making the feature similarity relationship clearer and more reliable. It provides a more solid foundation for the accurate calculation of subsequent out-of-distribution detection scores and effectively improves the model's sensitivity, accuracy and generalization performance for out-of-distribution data detection.

[0102] Step 505: Calculate the out-of-distribution detection score of the image data to be processed based on the kernel matrix.

[0103] In this embodiment, in order to accurately calculate the out-of-distribution detection score of the image data to be processed, thereby achieving accurate identification and exclusion of out-of-distribution data in the vertical domain, this method further calculates the feature relationship between the image data to be processed and the feature vectors of similar and dissimilar groups based on the centered kernel matrix obtained in step 504, thereby obtaining the final out-of-distribution detection score.

[0104] In one possible implementation, refer to Figure 6 , Figure 6 This is the sixth flowchart of the out-of-distribution data detection method provided by the present invention. Step 505 specifically includes steps 601-604: Step 601: Calculate the similarity score between the image data to be processed and the feature vectors of the non-similar groups based on the kernel matrix.

[0105] Step 602: Normalize the similarity scores to obtain normalized similarity scores.

[0106] Step 603: Calculate the mean of the dissimilar groups and the variance of the dissimilar groups based on the normalized similarity scores.

[0107] Step 604: Perform Z-score standardization on the mean and variance of the dissimilar groups to obtain the out-of-distribution detection score of the image data to be processed.

[0108] In this embodiment, to further improve the stability and discrimination accuracy of out-of-distribution detection results, this method, after completing the construction of the centered kernel matrix, continues to perform a series of similarity analysis and statistical normalization operations, namely steps 601 to 604, to calculate the final out-of-distribution detection score. This process ensures that out-of-distribution detection not only relies on the feature distribution of a single sample but also incorporates the statistical information of the entire sample set, thus enhancing the robustness of the detection results.

[0109] First, in step 601, this method calculates the similarity score between the image data to be processed and the feature vectors of similar groups, and the similarity score between the image data and the feature vectors of dissimilar groups, based on the centered kernel matrix obtained in step 504. The similarity score is calculated using a centered kernel function: ; The obtained similarity score reflects the relative proximity of the features of the image to be processed to the two types of samples (similar group and non-similar group) in the feature space, providing the original distribution reference.

[0110] Further, in step 602, the similarity scores are subjected to maximum and minimum normalization.

[0111] Next, in step 603, the similarity score between the image data to be processed and the feature vectors of similar groups is used. Similarity score between feature vectors of dissimilar groups Calculate the mean of the similar groups respectively. Mean of dissimilar groups And further calculate the variance of similar groups. Variance of dissimilar groups The specific calculation formula is as follows: ; ; ; ; By calculating the above statistics, this method can provide a more comprehensive understanding of the relative position of the image to be processed in different categories of data, forming a preliminary basis for judging its classification.

[0112] Finally, in step 604, this method performs Z-score standardization on the mean and variance of dissimilar groups to obtain the out-of-distribution detection score of the image data to be processed. The calculation formula is as follows: ; The Z-score standardization method maps the detection scores of different samples to a unified scale, making the detection results more comparable and capable of threshold discrimination.

[0113] In summary, steps 601 to 604 of this embodiment can significantly improve the stability and accuracy of similarity calculation during out-of-distribution detection. At the same time, the introduction of historical statistical references enables the final out-of-distribution detection score to have good generalization and discriminative power, providing a solid and reliable quantitative basis for subsequent threshold judgment.

[0114] Step 103: Compare the out-of-distribution detection score with the preset threshold to obtain the out-of-distribution detection result.

[0115] In one possible implementation, step 103 specifically includes the following steps: When the out-of-distribution detection score is greater than the preset threshold, the image data to be processed is determined to be in-distribution image data.

[0116] When the out-of-distribution detection score is less than or equal to a preset threshold, the image data to be processed is determined to be out-of-distribution image data.

[0117] In this embodiment, in order to achieve automatic determination of whether the image data to be processed belongs to out-of-distribution data based on the similarity difference between image features and existing samples, after completing the calculation of the out-of-distribution detection score in step 505, this method further performs the discrimination process of out-of-distribution detection results, that is, step 103: compare the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result.

[0118] Specifically, for each image data to be processed, this method first obtains its out-of-distribution detection score. This score has been calculated through steps 601 to 604 based on kernel similarity and Z-score normalization, and can effectively reflect the degree of deviation of the image data from the dissimilar group (i.e., data unrelated to business). The lower the detection score, the closer the image data is to non-business samples in the feature space, and the higher the probability that it belongs to out-of-distribution data.

[0119] Subsequently, this embodiment introduces a preset threshold based on the actual business scenario, and compares the score with the preset threshold to automatically determine the distribution state of the image data. The discrimination rules are as follows: If the out-of-distribution detection score is greater than the preset threshold, then the image data to be processed is determined to be in-distribution image data, i.e. business-related data. If the out-of-distribution detection score is equal to or greater than the preset threshold, the image data to be processed is determined to be out-of-distribution image data, that is, a sample that is irrelevant to the current business scenario and needs to be excluded from the business system in a timely manner.

[0120] This discrimination strategy maintains high recognition accuracy while also offering strong flexibility, allowing for adjustments to preset thresholds to adapt to different business tolerance ranges or detection sensitivity requirements. For example, in security-sensitive scenarios, the threshold can be appropriately lowered to enhance the conservatism of out-of-distribution detection; while in tolerant business scenarios, a higher threshold can be set to avoid false positives.

[0121] In one possible implementation, the method further includes the following steps: When the semantic label belongs to the horizontal domain, the image data to be processed is input into the second detection model to obtain the out-of-distribution detection score of the image data to be processed output by the second detection model; wherein, the second detection model is a zero-shot learning model.

[0122] In this embodiment, to address image data whose semantic labels originate from a horizontal domain rather than a vertical one, this method inputs the image data into a second detection model in this step and determines whether it belongs to out-of-distribution data based on the out-of-distribution detection score output by the second detection model. The second detection model is used because in the horizontal domain, data often has broader distribution characteristics and more uncertain semantic labels. Traditional first detection models, trained with lightweight methods to adapt to specific domains, may suffer from insufficient generalization or high false positive rates when processing such data.

[0123] Specifically, in this embodiment, the image data to be processed, whose semantic labels belong to the horizontal domain, is input into the zero-shot learning model, i.e., the second detection model. Because this zero-shot learning model is pre-trained based on a large amount of general domain data for cross-modal alignment and feature pre-training, it can directly distinguish newly emerging semantic labels or unseen categories without the need for special training on the target dataset.

[0124] During the training of the second detection model, model initialization and preprocessing are performed first. Specifically, data images from the business domain are acquired through cameras or other imaging devices, and preprocessed such as standardization and size normalization are performed on them. At the same time, combined with manual annotation or rule system, the semantic label of each image is determined, thereby obtaining distributed image-semantic data pairs containing K categories.

[0125] Subsequently, the second detection model was initialized using the pre-trained CLIP model, and its parameters were frozen to ensure that the model possessed good basic cross-modal representation capabilities. The model's visual encoder and text encoder are denoted as... and Image features are extracted from image data x using a visual encoder. The semantic label y is combined with the prompt word template and then input into the text encoder to generate text semantic features. Because the CLIP model has good modal alignment capabilities, the two should be highly similar in feature space.

[0126] Next, considering that the semantic features in the traditional CLIP model rely on manually designed prompt words, which can lead to uncontrollability and semantic drift issues in real-world business scenarios, this embodiment generates diverse text semantic features based on a combination of massive prompt word templates and semantic tags. The DB-SCAN clustering algorithm is then used to analyze the distribution of semantic features within the same category, extracting the center of the largest cluster as a stable category semantic representation. This approach maintains the accuracy of category representation while suppressing the interference of outlier semantic features on model training.

[0127] Regarding image features, this embodiment proposes a task-specific image feature adjustment strategy to mitigate interference from irrelevant information in the pre-trained model. First, the visual encoder structure of the CLIP model is modified by introducing a VV attention mechanism, which calculates the self-attention map using only the Value component V. This ensures that the tags retain their semantics while mitigating the impact of anomalous regions on the overall features. Furthermore, the feedforward network module is removed, and the original ViT encoder is completely replaced with the aforementioned structure to improve stability during the inference phase.

[0128] Furthermore, by using attention maps With semantic features Calculate cosine similarity to generate an attention-weighted graph. The final image feature representation is obtained by combining the weighting factor γ. and with After performing element-wise multiplication and summing, the predicted output P is obtained, which is used to measure the degree of matching between the image and the semantic label.

[0129] Finally, to support the out-of-distribution detection process, a feature library of positive and negative samples is constructed. Positive sample set. Includes all adjusted image features and semantic features negative sample set This involves sampling large-scale business-irrelevant images, selecting the images least relevant to the business semantics, extracting their features, and storing them in the database. All of these features are standardized using methods such as max pooling and uniformly stored in the Milvus vector database for subsequent similarity matching and kernel function detection.

[0130] It should be noted that the specific implementation method of inputting the image data to be processed into the second detection model to obtain the out-of-distribution detection score of the image data to be processed output by the second detection model is the same as the specific implementation method of inputting the image data to be processed into the first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model in step 102. You can refer to the specific embodiments of steps 501 to 505 and steps 601 to 604, so they will not be described in detail here.

[0131] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of the off-distribution data detection system provided by the present invention. The system includes: The acquisition module is used to acquire semantic tags for the image data to be processed; The first processing module is used to input the image data to be processed into the first detection model when the semantic label belongs to the vertical domain, and obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; The second processing module is used to compare the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result.

[0132] In one possible implementation, the system further includes a training module for: Obtain multiple image data samples and the corresponding training semantic labels for each image data sample; A preset number of learnable tokens are inserted into multiple Transformer blocks of the visual encoder and text encoder of the contrastive learning multimodal model to obtain a multimodal model structure containing learnable tokens. For any image data sample, the visual encoder extracts features from the image data sample to obtain training image features, and the text encoder extracts features from the corresponding training semantic labels to obtain text semantic features. A loss function is constructed based on all training image features and all text semantic features; Based on the loss function, backpropagation training is performed on the multimodal model structure until the model converges, thus obtaining the first detection model.

[0133] In one possible implementation, the training module is further configured to: Perform group normalization and convolution operations on any training image feature to obtain optimized training image features, and perform layer normalization and linear mapping operations on any text semantic feature to obtain optimized text semantic features. The optimized training image features and optimized text semantic features are mapped to a common feature space; Based on the optimized training image features of all images labeled with the same training semantic label, the mean value of the image features for that training semantic label is calculated, and the visual matching score is calculated based on the optimized training image features and the mean value of the image features. Based on the optimized text semantic features of all training semantic labels, the mean of the semantic text features of the training semantic label is calculated, and the text matching score is calculated based on the optimized text semantic features and the mean of the semantic text features. A loss function is constructed based on visual matching score and text matching score to constrain the matching degree between optimized training image features and optimized text semantic features, thereby improving the discriminative power of in-distribution and out-of-distribution samples in the common feature space.

[0134] In one possible implementation, the training module is further configured to: The cross-entropy loss of the visual branch is calculated based on the visual matching score and the corresponding training semantic label. The cross-entropy loss of the text branches is calculated based on the text matching score and the corresponding training semantic label. The cross-entropy loss function is obtained by adding the cross-entropy loss of the visual branch to the cross-entropy loss of the text branch.

[0135] In one possible implementation, the first processing module is further configured to: The image data to be processed is input into the first detection model to obtain the image feature vector of the image data to be processed; Retrieve the multiple positive sample feature vectors with the highest similarity to the image feature vectors from the vector database, and form a similarity group feature vector; Retrieve multiple negative sample feature vectors with the lowest similarity to the image feature vectors from the vector database to form a dissimilar group feature vector; Using radial basis function kernel function, construct Gram matrix based on eigenvectors of similar groups and dissimilar groups, and then center the Gram matrix to obtain a centered kernel matrix; The out-of-distribution detection score of the image data to be processed is calculated based on the kernel matrix.

[0136] In one possible implementation, the first processing module is further configured to: Calculate the similarity score between the image data to be processed and the feature vectors of non-similar groups based on the kernel matrix; The similarity scores are normalized to obtain normalized similarity scores; Calculate the mean of the dissimilar groups and the variance of the dissimilar groups based on the normalized similarity scores. The Z-score standardization of the mean and variance of the dissimilar groups is performed to obtain the out-of-distribution detection score of the image data to be processed.

[0137] In one possible implementation, the first processing module is further configured to: When the semantic label belongs to the horizontal domain, the image data to be processed is input into the second detection model to obtain the out-of-distribution detection score of the image data to be processed output by the second detection model; wherein, the second detection model is a zero-shot learning model.

[0138] In one possible implementation, the second processing module is further configured to: When the out-of-distribution detection score is greater than the preset threshold, the image data to be processed is determined to be in-distribution image data; When the out-of-distribution detection score is less than or equal to a preset threshold, the image data to be processed is determined to be out-of-distribution image data.

[0139] It should be noted that the off-distribution data detection system provided by the present invention can execute the off-distribution data detection method of any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0140] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device may include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute an out-of-distribution data detection method, which includes: acquiring semantic labels of the image data to be processed; when the semantic labels belong to a vertical domain, inputting the image data to be processed into a first detection model to obtain an out-of-distribution detection score of the image data to be processed output by the first detection model; wherein the first detection model is trained from image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; and comparing the out-of-distribution detection score with a preset threshold to obtain an out-of-distribution detection result.

[0141] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the out-of-distribution data detection method provided in the above embodiments. The method includes: acquiring semantic labels of image data to be processed; when the semantic labels belong to a vertical domain, inputting the image data to be processed into a first detection model to obtain an out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained from image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; and comparing the out-of-distribution detection score with a preset threshold to obtain an out-of-distribution detection result.

[0143] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the out-of-distribution data detection method provided in the above embodiments. The method includes: acquiring semantic labels of image data to be processed; when the semantic labels belong to a vertical domain, inputting the image data to be processed into a first detection model to obtain an out-of-distribution detection score of the image data to be processed output by the first detection model; wherein the first detection model is trained from image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; and comparing the out-of-distribution detection score with a preset threshold to obtain an out-of-distribution detection result.

[0144] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting out-of-distribution data, characterized in that, include: Obtain the semantic labels of the image data to be processed; When the semantic label belongs to a vertical domain, the image data to be processed is input into the first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; The out-of-distribution detection score is compared with a preset threshold to obtain the out-of-distribution detection result.

2. The method for detecting out-of-distribution data according to claim 1, characterized in that, It also includes the training method for the first detection model: Acquire multiple image data samples and training semantic labels corresponding to each image data sample; A preset number of learnable tokens are inserted into multiple Transformer blocks of the visual encoder and text encoder of the contrastive learning multimodal model to obtain a multimodal model structure containing the learnable tokens. For any of the image data samples, the visual encoder extracts features from the image data sample to obtain training image features, and the text encoder extracts features from the corresponding training semantic labels to obtain text semantic features. A loss function is constructed based on all the training image features and all the text semantic features; Based on the loss function, the multimodal model structure is trained by backpropagation until the model converges, thus obtaining the first detection model.

3. The method for detecting out-of-distribution data according to claim 2, characterized in that, The step of constructing a loss function based on all the training image features and all the text semantic features includes: Perform group normalization and convolution operations on any of the training image features to obtain optimized training image features, and perform layer normalization and linear mapping operations on any of the text semantic features to obtain optimized text semantic features. The optimized training image features and the optimized text semantic features are mapped to a common feature space; Based on the optimized training image features of all images labeled with the same training semantic label, the mean value of the image features of that training semantic label is calculated, and the visual matching score is calculated based on the optimized training image features and the mean value of the image features. Based on the optimized text semantic features of all training semantic labels, the mean semantic text feature of the training semantic label is calculated, and the text matching score is calculated based on the optimized text semantic features and the mean semantic text feature. A loss function is constructed based on the visual matching score and the text matching score to constrain the matching degree between the optimized training image features and the optimized text semantic features, thereby improving the discriminative power of in-distribution and out-of-distribution samples in the common feature space.

4. The out-of-distribution data detection method according to claim 3, characterized in that, The step of constructing a loss function based on the visual matching score and the text matching score includes: Based on the visual matching score and the corresponding training semantic label, calculate the cross-entropy loss of the visual branch; Based on the text matching score and the corresponding training semantic label, calculate the cross-entropy loss of the text branch; The cross-entropy loss function is obtained by adding the cross-entropy loss of the visual branch to the cross-entropy loss of the text branch.

5. The method for detecting out-of-distribution data according to claim 1, characterized in that, The step of inputting the image data to be processed into the first detection model to obtain the out-of-distribution detection score of the image data to be processed output by the first detection model includes: The image data to be processed is input into the first detection model to obtain the image feature vector of the image data to be processed; The vector database is used to retrieve multiple positive sample feature vectors that have the highest similarity to the image feature vectors, forming a similarity group feature vector. The vector database is used to retrieve multiple negative sample feature vectors with the lowest similarity to the image feature vectors, forming a dissimilar group feature vector. Using radial basis function kernel function, a Gram matrix is ​​constructed based on the feature vectors of the similar group and the feature vectors of the dissimilar group, and the Gram matrix is ​​centered to obtain a centered kernel matrix; The out-of-distribution detection score of the image data to be processed is calculated based on the kernel matrix.

6. The out-of-distribution data detection method according to claim 5, characterized in that, The calculation of the out-of-distribution detection score of the image data to be processed based on the kernel matrix includes: Calculate the similarity score between the image data to be processed and the feature vectors of the non-similar groups based on the kernel matrix; The similarity scores are normalized to obtain normalized similarity scores; Based on the normalized similarity scores, calculate the mean of the dissimilar groups and the variance of the dissimilar groups; The Z-score is used to standardize the mean and variance of the dissimilar groups to obtain the out-of-distribution detection score of the image data to be processed.

7. The method for detecting out-of-distribution data according to claim 1, characterized in that, Also includes: When the semantic label belongs to the horizontal domain, the image data to be processed is input into the second detection model to obtain the out-of-distribution detection score of the image data to be processed output by the second detection model; wherein, the second detection model is a zero-shot learning model.

8. The method for detecting out-of-distribution data according to claim 1, characterized in that, The step of comparing the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result includes: When the out-of-distribution detection score is greater than the preset threshold, the image data to be processed is determined to be in-distribution image data; When the out-of-distribution detection score is less than or equal to the preset threshold, the image data to be processed is determined to be out-of-distribution image data.

9. An off-distribution data detection system, characterized in that, include: The acquisition module is used to acquire semantic tags for the image data to be processed; The first processing module is used to input the image data to be processed into the first detection model when the semantic label belongs to a vertical domain, and obtain the out-of-distribution detection score of the image data to be processed output by the first detection model; wherein, the first detection model is trained by image data samples, training semantic labels corresponding to the image data samples, and learnable tokens; The second processing module is used to compare the out-of-distribution detection score with a preset threshold to obtain the out-of-distribution detection result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the out-of-distribution data detection method as described in any one of claims 1-8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the out-of-distribution data detection method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the out-of-distribution data detection method as described in any one of claims 1-8.

Images