Out-of-distribution data detection system and method based on image-text bimodal matching

Abstract

The invention discloses an out-of-distribution data detection system and method based on image-text bimodal matching, and belongs to the technical field of out-of-distribution data detection. The current text category feature and the current area visual feature of each tag category to which a to-be-recognized image possibly belongs are aggregated; performing weighted summation on the aggregation result and the current global visual feature to obtain a current image aggregation feature; matching the current image aggregation feature with the storage text mode and the storage visual mode of each label category to obtain matching scores corresponding to the current text mode and the visual mode of each label category, and performing weighted summation on the two matching scores to obtain a bimodal matching score; judging whether the image to be recognized is in-distribution data or out-of-distribution data; according to the method, the fusion information corresponding to the text modality and the visual modality of the data in the distribution is used for carrying out the out-of-distribution detection for the first time, and the out-of-distribution data in the real environment can be effectively detected at low cost.

Description

Technical Field

[0001] The present invention belongs to the technical field of out-of-distribution data detection, and more specifically, relates to an out-of-distribution data detection system and method based on image-text bimodal matching. Background Art

[0002] In the real world, deployed pattern recognition systems often encounter data that deviates from the training label distribution. If the data inside and outside the distribution cannot be distinguished and the out-of-distribution data is mistaken for the category inside the distribution, the reliability of the model will be seriously affected. This problem is called out-of-distribution detection and is very important for the safe deployment of the model. Obviously, models with out-of-distribution detection capabilities are more valuable in practical applications. In recent years, pre-trained large-scale visual language models have demonstrated excellent performance, and transfer learning fine-tuning can be well applied to various downstream fields. However, although visual language models have broad application prospects, the security risks of these models may also be inherited by subsequent fine-tuned models. In order to address these security risks, it is necessary to design out-of-distribution detection methods for such models.

[0003] In order to prevent the model from overconfidently predicting out-of-distribution data as in-distribution data, the existing technology mainly focuses on unimodal data and unimodal models, and only uses the input image data to train the model to achieve the purpose of out-of-distribution detection. There are also studies on bimodal models in the existing technology, but these technologies only use the information of the unimodal (text mode) of the bimodal model.

[0004] In summary, existing out-of-distribution detection systems and methods often only use single-modal information of in-distribution data. Such methods do not make full use of data information, resulting in the inability to effectively and cost-effectively identify out-of-distribution data, and are difficult to adapt to the needs of model deployment in the real world. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides an out-of-distribution data detection system and method based on image-text bimodal matching, with the aim of solving the technical problem that the out-of-distribution detection system and method do not make sufficient use of data information, resulting in the inability to effectively and cost-effectively identify out-of-distribution data.

[0006] To achieve the above object, according to one aspect of the present invention, a system for detecting out-of-distribution data based on bimodal matching of images and texts is provided, comprising:

[0007] A feature extraction module, used to extract current global visual features, current regional visual features of the image to be identified, and current text category features of each label category to which the image to be identified may belong;

[0008] a feature enhancement module, connected to the feature extraction module, for aggregating the current text category feature of each label category to which the image to be identified may belong with the current regional visual feature to obtain a current aggregation result, and then performing a weighted summation of the current aggregation result and the current global visual feature to obtain a current image aggregation feature;

[0009] A calculation matching module, connected to the feature enhancement module, is used to match the current image aggregate features with the stored text modalities of each tag category to obtain the current text modality matching score of each tag category; and is also used to match the current image aggregate features with the stored visual modalities of each tag category to obtain the current visual modality matching score of each tag category;

[0010] A data detection module is connected to the calculation matching module, and is used to perform weighted summation of the current text modality matching score and the current visual modality matching score corresponding to each label category to obtain a bimodal matching score corresponding to each label category; the maximum value of the bimodal matching scores corresponding to all label categories is used to compare with a reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

[0011] In one embodiment, the feature extraction module includes:

[0012] A first visual encoder, used to extract current global visual features and current regional visual features of the image to be recognized;

[0013] The first text encoding unit includes: a class prompt word generator and a first text encoder, wherein the class prompt word generator is used to generate a current text prompt word of each label category to which the image to be identified may belong, and the current text prompt word is used by the first text encoder to extract the current text category features of each label category to which the image to be identified may belong;

[0014] When the stored text modality and the stored visual modality are determined using the in-distribution image data set and its corresponding category label set, the model parameters of the out-of-distribution data detection system based on image-text bimodal matching remain unchanged.

[0015] In one embodiment, when the in-distribution image dataset and its corresponding category label set are used with the first text encoding unit to extract the current text category features of each label category to which the image to be identified may belong:

[0016] The class prompt word generator is also used to use T k,j=M(x,q) generates J intermediate candidate text prompt words for each label category; x is the image of the kth category in the in-distribution image dataset, M is a multimodal large language model, q is the instruction for generating intermediate candidate text prompt words, T k,j The jth intermediate candidate text prompt word of the kth category;

[0017] The class prompt word generator is also used to use Calculate the score of each generated intermediate candidate text prompt word; μ k is the sample global visual feature F corresponding to all sample images of the kth label category V The mean of The current text feature of the i-th class prompt word generated for the k-th label category; the top p intermediate candidate class text prompt words with the highest scores in each label category are selected and input into the first text encoder to obtain the current text category feature of each label category.

[0018] In one of the embodiments, when the stored text modality and the stored visual modality are determined using an in-distribution image dataset and its corresponding category label set:

[0019] The feature enhancement module is also used to utilize Calculate the sample image aggregation features corresponding to each image sample γ is a hyperparameter that controls the weighting of the aggregated features, r is the region number, is the sample region visual feature of the rth region in each image sample, is the sample global visual feature of each image sample, Represented as the text category feature of the jth label category, H is the height of each image sample, and W is the width of each image sample;

[0020] The calculation matching module is also used to utilize Calculate the possible label categories y to which the image sample x may belong k The predicted probability τ is the scaling factor; cos() represents the cosine similarity, and then using the formula The predicted probabilities of each label category are concatenated as the image text matching score s of the image sample x m , m represents the serial number of the image sample x in the image data set within the distribution; using the formula The mean of the image-text matching scores of the image dataset in the distribution in the k-th label category is used as the storage visual modality of the k-th label category n is the total number of images in the kth label category; then use the formula represents the storage visual mode of the image dataset in each category label within the distribution, Concat is the concatenation function, is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

[0021] In one embodiment, the calculation matching module is further used to calculate the stored text modality P of the current image aggregate feature and each tag category. T The similarity of T , and the similarity s T As the current text modality matching score of each tag category; also used to calculate the current image aggregation feature and the stored visual modality P of each tag category V The similarity of V , and the similarity s V as the current visual modality matching score for each label category;

[0022] The data detection module is also used to use the formula DPM(X)=max(s T )-βmin(s V ) calculates the bimodal matching score corresponding to each label category, where β is the visual modality affinity factor, which controls the weight of the visual modality in the bimodal matching.

[0023] In one embodiment, the feature extraction module in the out-of-distribution data detection system includes:

[0024] The second text encoding unit includes: a class prompt word generator, a second text encoder and a first mapping layer, wherein the class prompt word generator is used to generate a current text prompt word of each label category to which the image to be identified may belong, and the second text encoder is used to extract an intermediate text category feature of the current text prompt word, and the current text category feature of each label category to which the image to be identified may belong is obtained by mapping through the first mapping layer;

[0025] The visual encoding unit includes: a second visual encoder, the first mapping layer and the second mapping layer, wherein the second visual encoder is used to extract the intermediate global visual features and the intermediate regional visual features of the image to be recognized, the intermediate global visual features are mapped by the first mapping layer to obtain the current global visual features of the image to be recognized, and the intermediate regional visual features are mapped by the second mapping layer to the current regional visual features corresponding to the image to be recognized;

[0026] The out-of-distribution data detection system also includes: a parameter updating module; when an in-distribution image data set and its corresponding category label set are input into the out-of-distribution data detection system to determine the stored text modality and the stored visual modality, the parameter updating module uses the text modality matching score and the visual modality matching score in the training iteration process and the label to calculate the cross entropy loss, and returns the gradient to update the system parameters.

[0027] In one of the embodiments, the first mapping layer includes: a 1*1 convolution, a layer normalization, an activation function, a 1*1 convolution and a layer normalization connected in sequence;

[0028] The second mapping layer includes a linear layer, a group normalization, an activation function, a linear layer, and a group normalization layer, which are connected in sequence.

[0029] In one embodiment, when the stored text modality and the stored visual modality are determined using the in-distribution image dataset and its corresponding category label set,

[0030] The feature enhancement module is also used to utilize Calculate the sample image aggregation features corresponding to each image sample r is the region number, is the sample region visual feature of the rth region in each image sample after the second mapping layer, L v (F V ) is the sample global visual feature of each image sample after the first mapping layer, It is represented as the text category feature of the kth label category after the first mapping layer, H is the height of each image sample, and W is the width of each image sample; μ k is the sample global visual feature L corresponding to all sample images of the kth label category v (F V )’s mean;

[0031] The calculation matching module is also used to use the formula and Determine the storage visual mode of the image dataset in each category label within the distribution, Concat is a concatenation function, is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

[0032] In one embodiment, the cross entropy loss is expressed as Calculated;

[0033] Indicates the use of p V (y k|x) and label y to calculate the cross entropy loss, By formula Calculated, p V (y k |x) represents the visual modality matching score,

[0034] Indicates the use of p T (y k |x) and label y to calculate the cross entropy loss, By formula Calculated; p T (y k |x) represents the text modality matching score, τ represents the scaling factor.

[0035] In one embodiment, the calculation matching module is used to calculate the current image aggregate feature and the stored text modality P of each tag category. T The similarity of T , and the similarity s T As the current text modality matching score of each tag category; also used to calculate the current image aggregation feature and the stored visual modality P of each tag category V The similarity of V , and the similarity s V as the current visual modality matching score for each label category;

[0036] The data detection module is used to use the formula DPM(X)=max(s T )+βmax(s V The bimodal matching score corresponding to each tag category is calculated, and β is the visual modality affinity factor, which controls the proportion of the visual modality in the bimodal matching.

[0037] According to another aspect of the present invention, a method for detecting out-of-distribution data based on image-text bimodal matching is provided, comprising:

[0038] S1: Input the image to be identified into the out-of-distribution data detection system to obtain the bimodal matching scores corresponding to each label category;

[0039] S2: Compare the maximum value of the bimodal matching scores corresponding to all label categories with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

[0040] In one embodiment, before S1, the method further includes:

[0041] The in-distribution image dataset and its corresponding category label set are input into the out-of-distribution data detection system to obtain the stored text modality and the stored visual modality.

[0042] According to another aspect of the present invention, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method described above are implemented.

[0044] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0045] (1) The present invention provides an out-of-distribution data detection system based on image-text bimodal matching, which aggregates the current text category features of each label category to which the image to be identified may belong with the current regional visual features, and then performs weighted summation of the current aggregation results and the current global visual features to obtain the current image aggregation features; matches the current image aggregation features with the stored text modalities and stored visual modalities of each label category to obtain the current text modality matching scores and the current visual modality matching scores of each label category; performs weighted summation of the current text modality matching scores and the current visual modality matching scores corresponding to each label category to obtain the bimodal matching scores corresponding to each label category, so as to determine whether the image to be identified is in-distribution data or out-of-distribution data; The present invention proposes for the first time to use the fusion information corresponding to the text modality and visual modality of in-distribution data for out-of-distribution detection, which can effectively and cost-effectively detect out-of-distribution data in a real environment.

[0046] (2) This solution provides an out-of-distribution data detection system that does not require training. When the stored text modality and the stored visual modality are determined using an in-distribution image data set and its corresponding category label set, the model parameters of the out-of-distribution data detection system based on image-text bimodal matching remain unchanged. Not only does it utilize bimodal information, but it also does not consume resources for training.

[0047] (3) The class prompt word generator described in this scheme uses a multimodal large model to generate J intermediate candidate class text prompt words for each label category and calculates the score. The top p intermediate candidate class text prompt words with the highest scores are selected and input into the first text encoder to obtain the current text category features of each label category. This can bring more contextual information to the model and enhance the semantic matching ability of images and labels.

[0048] (4) In this scheme, the feature enhancement module uses the cosine similarity of the text category features of each label category and the visual features of the sample area as the aggregation weight, which can highlight the areas in the image that are more relevant to the label category, thereby strengthening the visual information in the image that is more consistent with the label semantics; the calculation matching module compares the image sample x with the possible label categories y. k The predicted probability The image-text matching score concatenated as the image sample x can capture the degree of match between the image and the text in multiple semantic dimensions, rather than being limited to a single category prediction.

[0049] (5) This solution provides an out-of-distribution data detection system that requires lightweight training, and also includes a parameter update module; when the in-distribution image data set and its corresponding category label set are input into the out-of-distribution data detection system to determine the stored text modality and the stored visual modality, the parameter update module uses the text modality matching score and the visual modality matching score in the training iteration process and the label to calculate the cross entropy loss, and returns the gradient to update the system parameters. Considering the user's computing power and cost issues, compared with existing methods, this solution improves the detection accuracy of out-of-distribution data by 5%, and the model has the advantages of short training time, low resource consumption, and high accuracy.

[0050] (6) In this solution, the first mapping layer includes: 1*1 convolution, layer normalization, activation function, 1*1 convolution and layer normalization connected in sequence; the second mapping layer includes a linear layer, group normalization, activation function, linear and group normalization connected in sequence. Such a design can promote cross-channel information fusion of features and enhance the expressiveness of features.

[0051] (7) In this scheme, the feature enhancement module uses the cosine similarity of the text category features of each label category and the visual features of the sample area as the aggregation weight, which can highlight the areas in the image that are more relevant to the label category, thereby strengthening the visual information in the image that is more matched with the label semantics; the calculation matching module uses the formula and Determining the storage visual modality of the in-distribution image dataset in each category label can preserve the visual information of the in-distribution image and solve the ambiguity problem caused by only using text modality matching.

[0052] (8) The cross entropy loss in this scheme is expressed by the formula By calculating, the model can be optimized based on the image matching results and the text matching results at the same time.

[0053] (9) The data detection module in the out-of-distribution data detection system that does not require training in this scheme uses the formula DPM(X)=max(s T )-βmin(s V) calculates the bimodal matching score corresponding to each label category, and only the data detection module in the out-of-distribution data detection system that requires lightweight training uses the formula DPM(X)=max(s T )+βmin(s V ) calculates the bimodal matching score corresponding to each label category; the information of image and text modalities can be used simultaneously to enhance the comprehensiveness of the detection system and make up for the shortcomings of using only a single modality.

[0054] (10) The present invention provides an out-of-distribution data detection method based on bimodal matching of images and texts, which aggregates the current text category features of each label category to which the image to be identified may belong with the current regional visual features, and then performs weighted summation of the current aggregation results and the current global visual features to obtain the current image aggregation features; matches the current image aggregation features with the stored text modalities and stored visual modalities of each label category to obtain the current text modality matching scores and the current visual modality matching scores of each label category; performs weighted summation of the current text modality matching scores and the current visual modality matching scores corresponding to each label category to obtain the bimodal matching scores corresponding to each label category, so as to determine whether the image to be identified is in-distribution data or out-of-distribution data; The present invention proposes for the first time to use the fusion information corresponding to the text modality and visual modality of in-distribution data to perform out-of-distribution detection, which can effectively and cost-effectively detect out-of-distribution data in a real environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic diagram of an out-of-distribution data detection system provided by Example 1 of the present invention;

[0056] Figure 2 is a schematic diagram of an out-of-distribution data detection system provided by Embodiment 2 of the present invention;

[0057] Figure 3 is a schematic diagram of a first mapping layer and a second mapping layer in an out-of-distribution data detection system provided in Embodiment 2 of the present invention;

[0058] Figure 4 is a schematic diagram of an out-of-distribution data detection method provided by Embodiment 3 of the present invention;

[0059] Figure 5 is a schematic diagram of another out-of-distribution data detection method provided by Embodiment 3 of the present invention;

[0060] Figure 6 This is a comparison diagram of the effects of another out-of-distribution data detection method provided by Example 3 of the present invention and the prior art;

[0061] Figure 7 is a schematic diagram of another out-of-distribution data detection method provided by Embodiment 3 of the present invention;

[0062] Figure 8 This is a comparison diagram of the effects of another out-of-distribution data detection method provided by Example 3 of the present invention and the prior art. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] Example 1

[0065] This embodiment provides a distributed data detection system based on image-text bimodal matching. Figure 1 As shown, it includes: a feature extraction module, a feature enhancement module, a calculation matching module and a data detection module.

[0066] The feature extraction module is used to extract the current global visual features, current regional visual features, and current text category features of the image to be identified for each label category to which the image to be identified may belong. For example, the image to be identified may be an image in an autonomous driving recognition scene, and the corresponding out-of-distribution detection refers to automatically detecting abnormal phenomena in the road surface and identifying them as out-of-distribution data. The image to be identified may be an image in an ecological protection scene, and the corresponding out-of-distribution detection refers to the species recognition system identifying the species categories in the training set and detecting new species instances outside the training distribution.

[0067] The feature enhancement module is connected to the feature extraction module and is used to aggregate the current text category features of each label category to which the image to be identified may belong with the current regional visual features to obtain the current aggregation result, and then perform weighted summation of the current aggregation result and the current global visual features to obtain the current image aggregation feature.

[0068] The calculation matching module is connected to the feature enhancement module, and is used to match the current image aggregate features with the stored text modalities of each label category to obtain the current text modality matching score of each label category; it is also used to match the current image aggregate features with the stored visual modalities of each label category to obtain the current visual modality matching score of each label category.

[0069] The data detection module is connected to the calculation matching module, and is used to perform weighted summation of the current text modality matching score and the current visual modality matching score corresponding to each label category to obtain the bimodal matching score corresponding to each label category; the maximum value of the bimodal matching scores corresponding to all label categories is used to compare with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

[0070] As an optional implementation method, the feature extraction module includes: a first visual encoder and a first text encoding unit; the first visual encoder is used to extract the current global visual features and the current regional visual features of the image to be identified; the first text encoding unit is used to extract the text category features of each label category to which each image sample may belong from the category label set corresponding to the in-distribution image data set. Wherein, when the storage text modality and the storage visual modality are determined using the in-distribution image data set and its corresponding category label set, the system parameters of the out-of-distribution data detection system based on image-text bimodal matching remain unchanged; the first visual encoder is also used to extract the sample global visual features and sample regional visual features of each image sample from the in-distribution image data set.

[0071] Specifically, the feature extraction module includes a text encoding unit and a visual encoder, which are used to extract the text features of the category label and the regional visual features and global visual features of the image. The feature enhancement module is used to aggregate the regional visual features of the image with the text features of each category, and weight them with the global visual features of the image to obtain the image features of the aggregated text information. The calculation matching module is used to calculate the similarity between the text modality and the visual modality, and the enhanced features of the matching image with the text modality and the visual modality to obtain the matching score of the text modality and the visual modality. The data detection module calculates the weighted sum of the matching scores of the text modality and the visual modality as the bimodal matching score; and compares the bimodal matching score with a preset threshold. If the score is lower than the threshold, the input is determined to be out-of-distribution data. If the score is higher than the threshold, the input is determined to be in-distribution data.

[0072] Furthermore, in the computational matching module, the text modality is obtained by inputting the category names of each category into the text encoder of the Contrastive Language-Image Pre-training (CLIP) model. The visual modality is obtained by calculating the dot product mean of the image features aggregated from the text information of all images of each category and the text modality.

[0073] As an optional embodiment, when the in-distribution image data set and its corresponding category label set are used with the first text encoding unit to extract the current text category features of each label category to which the image to be identified may belong, the working process of the class prompt word generator is as follows:

[0074] The class prompt word generator is also used to use T k,j =M(x,q) generates J intermediate candidate text prompt words for each label category; x is the image of the kth category in the in-distribution image dataset, M is a multimodal large language model, q is the instruction for generating intermediate candidate text prompt words, T k,j The jth intermediate candidate text prompt word of the kth category;

[0075] The class prompt word generator is also used to use Calculate the score of each generated intermediate candidate text prompt word; μ k is the sample global visual feature F corresponding to all sample images of the kth label category V The mean of The current text feature of the i-th class prompt word generated for the k-th label category; the top p intermediate candidate class text prompt words with the highest scores in each label category are selected and input into the first text encoder to obtain the current text category feature of each label category.

[0076] As an optional embodiment, when the in-distribution image dataset and its corresponding category label set are used to determine the storage text modality and the storage visual modality, the working process of the feature enhancement module and the calculation matching module is described as follows.

[0077] The feature enhancement module is also used to utilize Calculate the sample image aggregation features corresponding to each image sample γ is a hyperparameter that controls the weighting of the aggregated features, r is the region number, is the sample region visual feature of the rth region in each image sample, is the sample global visual feature of each image sample, It is represented as the text category feature of the jth label category, H is the height of each image sample, and W is the width of each image sample.

[0078] Compute matching module, also used to utilize Calculate the possible label categories y to which the image sample x may belong k The predicted probability τ is the scaling factor; cos() represents the cosine similarity, and then using the formula The predicted probabilities of each label category are concatenated as the image text matching score s of the image sample x m , m represents the serial number of image sample x in the image data set within the distribution; using the formula The mean of the image-text matching scores of the in-distribution image dataset in the k-th label category is used as the storage visual modality of the k-th label category n is the total number of images in the kth label category; then use the formula Represents the storage visual mode of the image dataset in each category label within the distribution, Concat is the splicing function, is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

[0079] As an optional implementation manner, after the image to be recognized is input, the working process of the calculation matching module and the data detection module in the out-of-distribution data detection system based on image-text bimodal matching is described as follows.

[0080] The calculation matching module is also used to calculate the current image aggregation features and the stored text modality P of each label category T The similarity of T , and the similarity s T As the current text modality matching score of each label category; it is also used to calculate the current image aggregation feature and the stored visual modality P of each label category V The similarity of V , and the similarity s V As the current visual modality matching score for each label category.

[0081] The data detection module is also used to use the formula DPM(X)=max(s T )-βmin(s V ) calculates the bimodal matching score DPM(X) corresponding to each label category, where β is the visual modality affinity factor, which controls the weight of the visual modality in bimodal matching.

[0082] Example 2

[0083] like Figure 2 As shown, the out-of-distribution data detection system provided in this embodiment includes: a feature extraction module, a feature enhancement module, a calculation matching module, a data detection module and a parameter updating module.

[0084] The feature extraction module is used to extract the current global visual features, current regional visual features, and current text category features of each label category to which the image to be identified may belong. The feature extraction module includes: a second text encoding unit and a visual encoding unit. The second text encoding unit includes: a class prompt word generator, a second text encoder, and a first mapping layer. The class prompt word generator is used to generate the current text prompt words of each label category to which the image to be identified may belong. The second text encoder is then used to extract the intermediate text category features of the current text prompt words, and the current text category features of each label category to which the image to be identified may belong are obtained by mapping through the first mapping layer. The visual encoding unit includes: a second visual encoder, a first mapping layer, and a second mapping layer. The second visual encoder is used to extract the intermediate global visual features and intermediate regional visual features of the image to be identified. The intermediate global visual features are mapped through the first mapping layer to obtain the current global visual features of the image to be identified. The intermediate regional visual features are mapped through the second mapping layer to the current regional visual features corresponding to the image to be identified.

[0085] The feature enhancement module is connected to the feature extraction module and is used to aggregate the current text category features of each label category to which the image to be identified may belong with the current regional visual features to obtain the current aggregation result, and then perform weighted summation of the current aggregation result and the current global visual features to obtain the current image aggregation feature.

[0086] The calculation matching module is connected to the feature enhancement module, and is used to match the current image aggregate features with the stored text modalities of each label category to obtain the current text modality matching score of each label category; it is also used to match the current image aggregate features with the stored visual modalities of each label category to obtain the current visual modality matching score of each label category.

[0087] The data detection module is connected to the calculation matching module, and is used to perform weighted summation of the current text modality matching score and the current visual modality matching score corresponding to each label category to obtain the bimodal matching score corresponding to each label category; the maximum value of the bimodal matching scores corresponding to all label categories is used to compare with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

[0088] A parameter updating module is used to, when an in-distribution image dataset and its corresponding category label set are input into an out-of-distribution data detection system to determine a stored text modality and a stored visual modality, the parameter updating module calculates the cross entropy loss using the text modality matching score and the visual modality matching score in the training iteration process and the label, and returns the gradient to update the system parameters; during this process, the second visual encoder extracts the intermediate global visual features and intermediate regional visual features of each image sample from the in-distribution image dataset, and obtains the sample global visual features and sample regional visual features through the first mapping layer respectively; the second text encoder extracts the intermediate text category features of each label category to which each image sample may belong from the category label set corresponding to the in-distribution image dataset, and obtains the text category features of each label category through the second mapping layer.

[0089] Among them, the feature extraction module: includes the text encoder, visual encoder, several mapping layers, several learnable parameters and the class visual feature mean memory of the CLIP model, which is used to extract the text features of the category label and the regional visual features and global visual features of the image; the feature enhancement module is used to calculate the image features of the text information aggregation and the image features of the image information aggregation; the calculation matching module: is used to calculate the text modality and the visual modality, and respectively calculate the matching scores of the image features of the text information aggregation and the text modality, and the matching scores of the image features of the image information aggregation and the visual modality; the parameter updating module: uses the text modality matching score and the visual modality matching score and the label to calculate the cross entropy loss, and returns the gradient to update the system parameters; the data detection module: calculates the weighted sum of the text modality and the visual modality matching score as the bimodal matching score; compares the bimodal matching score with the preset threshold. If the score is lower than the threshold, the input is determined to be out-of-distribution data. If the score is higher than the threshold, the input is determined to be in-distribution data.

[0090] Furthermore, in the feature extraction module, the visual feature mean memory is updated in a weighted exponential moving average (EMA) manner using the global visual features of the image.

[0091] Furthermore, in the feature enhancement module, the image features aggregated by text information are calculated by performing feature fusion of the regional visual features of the image with the text modalities of each category, and the weighted sum of the results and the global visual features of the image.

[0092] Furthermore, if Figure 3 As shown, the first mapping layer includes: 1*1 convolution, layer normalization, activation function, 1*1 convolution and layer normalization connected in sequence; the second mapping layer includes a linear layer, group normalization, activation function, linear and group normalization connected in sequence.

[0093] Furthermore, in the feature enhancement module, the image features of the image information aggregation are calculated by performing feature fusion of the regional visual features of the image with the global visual features of each category stored in the visual feature mean memory, and weighted summing the results with the global visual features of the image.

[0094] Furthermore, in the calculation matching module, the text modality is obtained by inputting the category names of each category into the text encoder of the CLIP model. The global visual features of each category stored in the visual feature mean memory are used as the visual modality.

[0095] As an optional embodiment, when the in-distribution image dataset and its corresponding category label set are used to determine the storage text mode and the storage visual mode, the feature enhancement module is also used to use Calculate the sample image aggregation features corresponding to each image sample r is the region number, is the sample region visual feature of the rth region in each image sample after the second mapping layer, L v (F V ) is the sample global visual feature of each image sample after the first mapping layer, It is represented as the text category feature of the kth label category after the first mapping layer, H is the height of each image sample, and W is the width of each image sample; μ k is the sample global visual feature L corresponding to the sample images of all k-th label categories v (F V )’s mean.

[0096] The calculation matching module is also used to use the formula and Determine the storage visual modality of the image dataset within the distribution for each category label, is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

[0097] As an optional implementation method, the cross entropy loss is expressed as Calculated; Indicates the use of p V (y k |x) and label y to calculate the cross entropy loss, By formula Calculated, p V (y k |x) represents the visual modality matching score, Indicates the use of p T (y k |x) and label y to calculate the cross entropy loss, By formula Calculated; p T (y k |x) represents the text modality matching score, τ represents the scaling factor.

[0098] As an optional embodiment, after the image to be recognized is input, the working process of the calculation matching module and the data detection module in the out-of-distribution data detection system based on image-text dual-modal matching is as follows. The calculation matching module is used to calculate the current image aggregation feature and the stored text modality P of each label category. T The similarity of T , and the similarity s T As the current text modality matching score of each label category; it is also used to calculate the current image aggregation feature and the stored visual modality P of each label category V The similarity of V , and the similarity s V As the current visual modality matching score of each label category; the data detection module is used to use the formula DPM(X)=max(s T )+βmax(s V ) calculates the bimodal matching score corresponding to each label category, and β is the visual modality affinity factor, which controls the proportion of the visual modality in the bimodal matching.

[0099] Example 3

[0100] like Figure 4 As shown, this embodiment provides an out-of-distribution data detection method based on image-text bimodal matching, including: S1: inputting the image to be identified into the above-mentioned out-of-distribution data detection system to obtain the bimodal matching scores corresponding to each label category; S2: comparing the maximum value of the bimodal matching scores corresponding to all label categories with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

[0101] As an optional implementation method, before S1, it also includes: S0: inputting the in-distribution image dataset and its corresponding category label set into the out-of-distribution data detection system to obtain the storage text modality and the storage visual modality.

[0102] like Figure 5 As shown, the method for out-of-distribution data detection using the out-of-distribution data detection system DPM-F that does not require training includes the following steps.

[0103] First, the camera collects system application field data and performs preprocessing to obtain distributed data including K categories.

[0104] Then, the system is initialized using the parameters of the pre-trained CLIP model and its parameters are frozen. The visual encoder and text encoder of the system are denoted as Φ V and Φ T . Prepare the in-distribution data label set The image data in the distribution is input into the public multimodal large language model to generate J intermediate candidate text prompt words for each label category. The formula is expressed as:

[0105] T k,j =M(x,q),

[0106] Where x is the image of the kth category in the in-distribution image dataset, M is the multimodal large language model, q is the instruction for generating the intermediate candidate text prompt word, and T k,j represents the jth intermediate candidate text prompt word of the kth category. The score of each generated intermediate candidate text prompt word is calculated as follows:

[0107]

[0108] where μ k is the sample global visual feature F corresponding to all sample images of the kth label category V The mean of The current text features of the i-th class prompt word generated for the k-th label category; select the top p intermediate candidate class text prompt words with the highest scores in each label category and input them into the text encoder Φ of the CLIP model T The current text category features of each label category are obtained as the text mode P of each category T , the formula is:

[0109]

[0110] In the formula Represents a given label Text features output by the text encoder.

[0111] Secondly, the image data in the distribution is input into the visual encoder Φ of the CLIP model T , output the regional visual features of the image and the global visual feature F V . Calculate the category text features for each class Regional visual features Cosine similarity of Then, all regional visual features are reweighted according to the similarity as the basis of importance, and the weighted result is combined with the original global image feature F V Weighted to get image features of text information aggregation The formula is:

[0112]

[0113] Among them, γ is a hyperparameter that controls the weighting degree of the aggregated features. The predicted probability can be calculated The formula is: Given the jth data of the kth category, its image-text matching score is s j , the formula is: The mean of the image-text matching scores of the data sampled from the kth category is used as the visual modality of the kth category. The formula is: Calculate the visual modality P of each category V , the formula is:

[0114] Finally, the system is deployed in a real environment and waits for test data to be input. Given an image input x, the system first calculates the image aggregation features Then calculate separately and the text modality of the dataset within the distribution P T and visual model V The cosine similarity of s V and T , the formula is:

[0115] Among them, KL stands for KL divergence, which is used to measure the distance between two probability distributions. The larger the KL divergence value, the greater the difference between the two probability distributions P and Q. The KL divergence calculation formula is expressed as: s V and T The information of the two modalities, vision and text, is emphasized respectively, and the bimodal matching score DPM is calculated using the information of these two modalities. The formula is expressed as: Compare the bimodal matching score DPM with the preset threshold λ. If the score is lower than the threshold, the input is judged to be out-of-distribution data. If the score is higher than the threshold, the input is judged to be in-distribution data. The out-of-distribution data discrimination formula is expressed as:

[0116]

[0117] Among them, 1 means that the test data belongs to the distribution data, and 0 means that it belongs to the distribution data.

[0118] Experimental test. This embodiment adopts the CLIP model with ViT-B / 16 architecture, selects images in the ImageNet dataset as in-distribution data, and selects images in the iNaturalist, SUN, Places365, and Texture datasets as out-of-distribution data. iNaturalist is a real-world dataset containing 8142 fine-grained species. SUN is a scene recognition dataset containing 397 categories and 108,754 images. Places365 consists of images for scene recognition and contains 365 scene categories. Texture is a dataset containing images describing textures. Based on the above network structure and dataset, the DPM-F method disclosed in this embodiment is experimentally compared with the traditional method, as shown in FIG. Figure 6 The method disclosed in this embodiment improves the out-of-distribution detection capability of the system.

[0119] like Figure 7 As shown, the out-of-distribution data detection method using the out-of-distribution data detection system DPM-T that requires lightweight training includes the following steps.

[0120] First, the camera collects system application field data and performs preprocessing to obtain distributed data including K categories.

[0121] Then, the system is initialized using the parameters of the pre-trained CLIP model to freeze its parameters. The visual encoder and text encoder of the system are denoted as Φ V and Φ T . Insert a learnable mapping layer after the visual encoder and text encoder respectively and insert a learnable word vector into the text prompt words in the text encoder. Prepare the in-distribution data label set Input to the text encoder Φ of the CLIP model T , output text features Since the text feature dimension is low, the layer normalization plus 1×1 convolution structure is used for mapping, which has a relatively small amount of calculation and can improve the robustness of the system. After mapping, the text features of each class are recorded as Prompt in the formula represents the text prompt word, into which q learnable word vectors are inserted, as shown in the following formula: Prompt = [V1, V2, V3, ..., V q ].

[0122] That is, prepare the image data within the distribution and input it into the visual encoder Φ of the CLIP model T , calculate the regional visual features of the image and the global visual feature F V For the global image feature F VThe mapping is performed using a layer normalization plus 1×1 convolution structure, and the global image feature after mapping is recorded as L v (F V ). For regional visual features The structure of group normalization plus linear layer is used for mapping, and the image features of the mapped region are recorded as Structure Figure 3 shown.

[0123] Calculate text features for each class Regional visual features Cosine similarity of Then all regional visual features are reweighted according to the similarity, and the weighted result is combined with the original global image feature L v (F V ) weighted to obtain image features aggregated from text information The formula is:

[0124]

[0125] Among them, γ is a hyperparameter that controls the weighting degree of aggregated features. Using the image feature prototype μ k Compute image features that aggregate image information The formula is:

[0126] μ k is the global visual feature L of the k-th data v (F V ), which is continuously updated in the exponential moving average (EMA) mode during the training process. and The predicted probability p can be obtained V (y k |x) and p T (y k ∣x), the formula is expressed as:

[0127]

[0128] Among them, p V (y k |x) and p T (y k |x) and the label to calculate the cross entropy loss to increase the distinguishability of data within and outside the distribution, and use the loss function to return the gradient to update the parameters. The loss function calculation formula is expressed as:

[0129]

[0130] After the system training is completed, the text features L after each category mappingt (F T ) as the text modality P of each category T , the calculation formula is expressed as:

[0131]

[0132] Secondly, use the mean μ of the global visual features of various data calculated in the next step k Calculate the visual modality P of each category V :

[0133]

[0134] Finally, the system is deployed in a real environment and waits for test data to be input. Given an image input x, we first calculate the image features of the text information aggregation Image features aggregated with image information Then calculate separately With each category of text modal P T The cosine similarity of With each category of visual modality P V The cosine similarity of s V and T .

[0135]

[0136] s V and T The information of the two modalities, vision and text, is emphasized respectively, and the bimodal matching score DPM is calculated using the information of these two modalities. The formula is expressed as:

[0137]

[0138] Compare the bimodal matching score DPM with the preset threshold λ. If the score is lower than the threshold, the input is judged as out-of-distribution data. If the score is higher than the threshold, the input is judged as in-distribution data, as shown in the following formula, 1 means it belongs to in-distribution data, and 0 means it belongs to out-of-distribution data. The out-of-distribution data discrimination formula is expressed as:

[0139] Experimental test: This embodiment adopts the CLIP model with ViT-B / 16 architecture, selects images in ImageNet as in-distribution data, and uses images from iNaturalist, SUN, Places365, and Texture as out-of-distribution data. iNaturalist is a real-world dataset containing 8142 fine-grained species. SUN is a scene recognition dataset containing 397 categories and 108754 images. Places365 consists of images for scene recognition and contains 365 scene categories. Texture is a dataset containing images describing textures. The learnable module was trained for 20 epochs using the SGD optimizer on a system equipped with 4 NVIDIARTX4090 (24G) and 2 Intel Xeon Platinum 8249C. The momentum is 0.9, and the cosine learning rate scheduler is used, and the learning rate gradually decays from 0.005 to 0. The weight decay is 0.0005 and the batch size is 256. Based on the above network structure, data set and training method, the DPM-T method disclosed in this embodiment is experimentally compared with the traditional method. Figure 8 The method disclosed in this embodiment improves the out-of-distribution detection capability of the system.

[0140] Example 4

[0141] This embodiment provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0142] Example 5

[0143] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0144] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An out-of-distribution data detection system based on image-text bimodal matching, characterized in that: include: A feature extraction module, used to extract current global visual features, current regional visual features of the image to be identified, and current text category features of each label category to which the image to be identified may belong; a feature enhancement module, connected to the feature extraction module, for aggregating the current text category feature of each label category to which the image to be identified may belong with the current regional visual feature to obtain a current aggregation result, and then performing a weighted summation of the current aggregation result and the current global visual feature to obtain a current image aggregation feature; A calculation matching module, connected to the feature enhancement module, is used to match the current image aggregate features with the stored text modalities of each tag category to obtain the current text modality matching score of each tag category; and is also used to match the current image aggregate features with the stored visual modalities of each tag category to obtain the current visual modality matching score of each tag category; A data detection module, connected to the calculation matching module, for performing weighted summation of the current text modality matching score and the current visual modality matching score corresponding to each tag category to obtain a bimodal matching score corresponding to each tag category; The maximum value of the bimodal matching scores corresponding to all label categories is used to compare with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

2. The out-of-distribution data detection system based on image-text bimodal matching according to claim 1, characterized in that: The feature extraction module comprises: A first visual encoder, used to extract current global visual features and current regional visual features of the image to be recognized; The first text encoding unit includes: a class prompt word generator and a first text encoder, wherein the class prompt word generator is used to generate a current text prompt word of each label category to which the image to be identified may belong, and the current text prompt word is used by the first text encoder to extract the current text category features of each label category to which the image to be identified may belong; When the stored text modality and the stored visual modality are determined using the in-distribution image data set and its corresponding category label set, the model parameters of the out-of-distribution data detection system based on image-text bimodal matching remain unchanged.

3. The out-of-distribution data detection system based on image-text bimodal matching as claimed in claim 2, characterized in that: When the in-distribution image data set and its corresponding category label set are used with the first text encoding unit to extract the current text category features of each label category to which the image to be identified may belong: The class prompt word generator is also used to use T k,j =M(x,q) generates J intermediate candidate text prompt words for each label category; x is the image of the kth category in the in-distribution image dataset, M is a multimodal large language model, q is the instruction for generating intermediate candidate text prompt words, T k,j The jth intermediate candidate text prompt word of the kth category; The class prompt word generator is also used to use Calculate the score of each generated intermediate candidate text prompt word; μ k is the sample global visual feature F corresponding to all sample images of the kth label category V The mean of The current text features of the i-th class prompt word generated for the k-th label category; The first p intermediate candidate class text prompt words with the highest scores in each label category are selected and input into the first text encoder to obtain the current text category features of each label category.

4. The out-of-distribution data detection system based on image-text bimodal matching according to claim 2, characterized in that: When the stored text modality and the stored visual modality are determined using the in-distribution image dataset and its corresponding category label set: The feature enhancement module is also used to utilize Calculate the sample image aggregation features corresponding to each image sample γ is a hyperparameter that controls the weighting of the aggregated features, r is the region number, is the sample region visual feature of the rth region in each image sample, Represented as the text category feature of the jth label category, H is the height of each image sample, and W is the width of each image sample; The calculation matching module is also used to utilize Calculate the possible label categories y to which the image sample x may belong k The predicted probability τ is the scaling factor; cos() represents cosine similarity, and then using the formula The predicted probabilities of each label category are concatenated as the image text matching score s of the image sample x m , m represents the serial number of the image sample x in the image data set within the distribution; using the formula The mean of the image-text matching scores of the image dataset in the distribution in the k-th label category is used as the storage visual modality of the k-th label category n is the total number of images in the kth label category; then use the formula represents the storage visual mode of the image dataset in each category label within the distribution, Concat is the concatenation function, P i V is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

5. The out-of-distribution data detection system based on image-text bimodal matching according to claim 1, characterized in that: The feature extraction module in the out-of-distribution data detection system includes: The second text encoding unit includes: a class prompt word generator, a second text encoder and a first mapping layer, wherein the class prompt word generator is used to generate a current text prompt word of each label category to which the image to be identified may belong, and the second text encoder is used to extract an intermediate text category feature of the current text prompt word, and the current text category feature of each label category to which the image to be identified may belong is obtained by mapping through the first mapping layer; The visual encoding unit includes: a second visual encoder, the first mapping layer and the second mapping layer, wherein the second visual encoder is used to extract the intermediate global visual features and the intermediate regional visual features of the image to be recognized, the intermediate global visual features are mapped by the first mapping layer to obtain the current global visual features of the image to be recognized, and the intermediate regional visual features are mapped by the second mapping layer to the current regional visual features corresponding to the image to be recognized; The out-of-distribution data detection system also includes: a parameter updating module; when an in-distribution image data set and its corresponding category label set are input into the out-of-distribution data detection system to determine the stored text modality and the stored visual modality, the parameter updating module uses the text modality matching score and the visual modality matching score in the training iteration process and the label to calculate the cross entropy loss, and returns the gradient to update the system parameters.

6. The out-of-distribution data detection system based on image-text bimodal matching according to claim 5, characterized in that: The first mapping layer includes: 1*1 convolution, layer normalization, activation function, 1*1 convolution and layer normalization connected in sequence; The second mapping layer includes: a linear layer, a group normalization, an activation function, a linear layer and a group normalization which are connected in sequence.

7. The out-of-distribution data detection system based on image-text bimodal matching according to claim 5, characterized in that: When the stored text modality and the stored visual modality are determined using the in-distribution image dataset and its corresponding category label set, The feature enhancement module is also used to utilize Calculate the sample image aggregation features corresponding to each image sample r is the region number, is the sample region visual feature of the rth region in each image sample after the second mapping layer, L v (F V ) is the sample global visual feature of each image sample after the first mapping layer, It is represented as the text category feature of the kth label category after the first mapping layer, H is the height of each image sample, and W is the width of each image sample; μ k is the sample global visual feature L corresponding to all sample images of the kth label category v (F V )’s mean; The calculation matching module is also used to use the formula and Determine the storage visual mode of the image dataset in each category label within the distribution, Concat is the concatenation function, P i V is the stored visual modality of the i-th category label, i = [1, 2, ...K], K is the total number of category labels.

8. The out-of-distribution data detection system based on image-text bimodal matching according to claim 7, characterized in that: The cross entropy loss is given by the formula Calculated; Indicates the use of p V (y k |x) and label y to calculate the cross entropy loss, By formula Calculated, p V (y k |x) represents the visual modality matching score, Indicates the use of p T (y k |x) and label y to calculate the cross entropy loss, By formula Calculated; p T (y k |x) represents the text modality matching score, τ represents the scaling factor.

9. The out-of-distribution data detection system based on image-text bimodal matching according to claim 2 or 5, characterized in that: The calculation matching module is used to calculate the current image aggregation feature and the stored text modality P of each label category. T The similarity of T , and the similarity s T As the current text modality matching score of each tag category; also used to calculate the current image aggregation feature and the stored visual modality P of each tag category V The similarity of V , and the similarity s V as the current visual modality matching score for each label category; The data detection module is used to use the formula DPM(X)=max(s T )-βmin(s V ) or DPM(X)=max(s T )+βmax(s V ) calculates the bimodal matching score DPM(X) corresponding to each label category, where β is the visual modality affinity factor, which controls the proportion of the visual modality in the bimodal matching.

10. A method for detecting out-of-distribution data based on image-text bimodal matching, characterized in that: include: S1: Inputting the image to be identified into the out-of-distribution data detection system according to any one of claims 1 to 9 to obtain the bimodal matching scores corresponding to each label category; S2: Compare the maximum value of the bimodal matching scores corresponding to all label categories with the reference threshold to determine whether the image to be identified is in-distribution data or out-of-distribution data.

Images