An image classification method and system based on visual language prompt optimization

Abstract

The application relates to the field of artificial intelligence, and more particularly to an image classification method and system based on visual language prompt optimization, which comprises the following steps: acquiring an image training set with image category labels, acquiring image categories according to the image category labels in the image training set; constructing an injection token, injecting the visual encoder and the text encoder of a visual language model based on the injection token to obtain updated visual encoder and text encoder; inputting the image categories, the image training set and a preset text training set into the visual language model for training; inputting an image application set to be processed and a corresponding text application set into the trained visual language model to obtain the image classification result of the image application set. The method is used for improving the generalization ability and robustness of a visual language model in image diversification and low-resource engineering visual classification tasks, and improving the accuracy and reliability of image classification.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to an image classification method and system optimized based on visual language cues. Background Technology

[0002] Currently, visual-language models (such as CLIP, Contrastive Language–Image Pretraining) are cross-modal models based on contrastive learning, used for tasks such as image retrieval, cross-modal image matching, and open-set image classification. However, because visual-language models can only capture general semantic relationships and cannot directly adapt to complex or specialized classification tasks, they rely on prompts for assistance in practical applications. In existing technologies, zero-shot learning methods are an important application mode for prompt assistance in visual-language models. Their core lies in using the text-image relationships learned during the pre-training phase to infer categories not present during training. Prompt learning methods, on the other hand, are a flexible task adaptation technique that explicitly guides the visual-language model to focus on specific task semantics for classification by adding manually designed templates or learnable context vectors to the input text.

[0003] While the aforementioned zero-shot learning methods, cue learning methods, and other existing cueing techniques have enhanced the flexibility of visual-language models, significant limitations remain. Zero-shot learning methods rely heavily on hand-designed text cueing, which is typically generic and fails to capture the complex modalities inherent in specialized domains. This limits the adaptability of visual-language models to complex classification patterns in new categories or image datasets. Cue learning methods, while dynamically optimizing representations for classification tasks, are prone to overfitting training class features under low-data conditions, weakening the generalization ability of visual-language models, especially when there are significant differences in the distribution between the image target domain and the image source domain. Furthermore, existing cue conditioning techniques often operate on single-modal or shallow features, failing to fully exploit the multimodal interaction potential. This results in insufficient semantic alignment during cross-modal task transfer, leading to poor classification performance. Summary of the Invention

[0004] This invention provides an image classification method and system optimized based on visual language cues, which improves the generalization ability and robustness of vision-language models in image diversification and low-resource engineering visual classification tasks, thereby enhancing the accuracy and reliability of image classification.

[0005] According to a first aspect of this application, an image classification method based on visual language cues is provided, the method comprising:

[0006] Obtain an image training set with image category labels, and obtain the image category based on the image category labels in the image training set;

[0007] An injection token is constructed, and the visual encoder and text encoder of the visual language model are injected based on the injection token to obtain the updated visual encoder and text encoder.

[0008] The image category, the image training set, and the preset text training set are input into the visual language model for training. The training process is as follows:

[0009] The image training set is input into the updated visual encoder to obtain a visual feature embedding set; a preset text training set is input into the updated text encoder to obtain a text feature embedding set; an average category text embedding set is obtained according to the image category; a category text embedding set is obtained according to the image category and the visual feature embedding set; feature interaction is performed on the category text embedding set and the visual feature embedding set to obtain a category text embedding set and a visual feature embedding set that have completed the feature interaction; the visual language model is optimized according to the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that has completed the feature interaction, the visual feature embedding set that has completed the feature interaction, and a preset loss function to obtain a trained visual language model;

[0010] The image application set to be processed and the corresponding text application set are input into the trained visual language model to obtain the image classification result of the image application set.

[0011] Understandably, by introducing token injection, feature interaction, image category-based category text embedding sets, and a pre-defined loss function optimization objective, the generalization ability and robustness of the visual language model in low-resource scenarios are effectively improved. This enables the visual language model to autonomously learn domain-related discriminative features and quickly adapt to new tasks with a small amount of labeled data, significantly enhancing cross-domain transfer learning and fine-grained classification performance.

[0012] Optionally, the visual encoder includes a plurality of sequentially connected visual encoding layers, and the injection token includes a visual injection token;

[0013] Based on the injection token, the visual encoder of the visual language model is injected to obtain an updated visual encoder, including:

[0014] Preset visual injection layer number threshold;

[0015] Determine whether the layer number of the visual coding layer exceeds the threshold of the visual injection layer number. If so, then the visual coding layer is used as the visual injection layer.

[0016] The visual injection token is injected into the visual injection layer to obtain the updated visual encoder;

[0017] And / or,

[0018] The text encoder is divided into several sequentially connected text encoding layers, and the injection token includes a text injection token;

[0019] Based on the injection token, the text encoder of the visual language model is injected to obtain an updated text encoder, including:

[0020] Preset text injection layer number threshold;

[0021] Determine whether the layer number of the text encoding layer exceeds the threshold of the text injection layer number; if so, then use the text encoding layer as the text injection layer.

[0022] The text injection token is injected into the text injection layer to obtain the updated text encoder.

[0023] Understandably, by using a hierarchical and differentiated token injection approach, the alignment accuracy and fusion efficiency of cross-modal information are significantly improved. This not only preserves the basic representational capabilities of the underlying general features but also strengthens the logical connections of the high-level semantics, enabling the visual language model to have stronger domain adaptability and fine-grained feature control capabilities. This provides a more discriminative and robust joint feature space for subsequent cross-modal tasks.

[0024] Optionally, the updated visual encoder Represented as:

[0025]

[0026] in, The updated visual encoder represents the first... Layered visual encoding layer, This represents the total number of visual coding layers. , Indicates the first A global representation token for the input image in the layered visual coding layer; Indicates that the input image is in the first... Embedded data of the visual encoding layer; This represents the threshold number of the visual injection layer. Indicates the first Visual injection tokens for the layered visual encoding layer;

[0027] And / or, the updated text encoder Represented as:

[0028]

[0029] in, This represents the first... of the updated text encoder Layered text encoding layer, This indicates the total number of text encoding layers. , Indicates the first In the text encoding layer, the beginning of the input text indicates the token; Indicates the first The end of the input text in the layered text encoding layer represents a token; This indicates that the input text is in the first... Embedded data in the text encoding layer; This indicates the threshold for the text injection layer number. Indicates the first Text injection tokens for the text encoding layer.

[0030] Understandably, high-level token injection into the visual encoder and text encoder preserves the basic representation capabilities of the underlying general features, significantly enhances the visual language model's ability to capture key semantic features and its collaborative expression capabilities across modalities, enables visual and text features to achieve precise alignment at different levels of abstraction, and effectively improves the semantic understanding depth and task adaptability of the visual language model in complex scenarios.

[0031] Optionally, obtaining the average category text embedding set based on the image category includes:

[0032] Obtain the query command corresponding to the image category;

[0033] Retrieve the text description corresponding to the image category based on the query instruction;

[0034] The text description is input into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors;

[0035] The mean value of the initial category text embedding vectors in the initial category text embedding is obtained to obtain the corresponding average category text embedding. All the average category text embeddings form the average category text embedding set.

[0036] Understandably, by generating an average category text embedding set based on image categories, the random bias of individual prompts can be effectively eliminated, the common semantic information of similar samples can be aggregated, and a more universal and stable cross-modal anchoring benchmark can be provided for visual language models. This significantly enhances the visual language model's ability to accurately capture category semantics and the clustering performance of the feature space, and improves the generalization ability of the visual language model in low-resource scenarios.

[0037] Optionally, obtaining the category text embedding set based on the plurality of image categories and the visual feature embedding set includes:

[0038] Obtain the query command corresponding to the image category;

[0039] Retrieve the text description corresponding to the image category based on the query instruction;

[0040] The text description is input into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors;

[0041] Construct a corresponding vector set to be processed from the initial category text embedding vectors at the same position in all the initial category text embeddings;

[0042] The maximum similarity score of each vector set to be processed is obtained based on the vector set to be processed and the visual feature embedding set.

[0043] Based on the maximum similarity score of each vector set to be processed and the preset score threshold, it is determined whether the vector set to be processed is an abnormal vector set. If so, the position of the vector in the abnormal vector set in the initial category text embedding is obtained as the abnormal position. The initial category text embedding vector corresponding to the abnormal position is deleted from each initial category text embedding to obtain the updated initial category text embedding.

[0044] The mean of the initial category text embedding vectors in the updated initial category text embeddings is calculated to obtain the corresponding category text embeddings, and all the category text embeddings form a category text embedding set.

[0045] Understandably, by calculating the maximum similarity score between the initial category text embedding and the visual feature embedding, abnormal initial category text embedding vectors are identified, and interference components that deviate from the core semantics are selectively removed. This effectively filters out noise interference, strengthens the semantic consistency between the initial category text embedding and the visual features, and enables the visual language model to adaptively extract the most discriminative cross-modal association features, significantly improving classification accuracy and cross-domain generalization ability in complex scenarios.

[0046] Optionally, the first Maximum similarity score of a set of vectors to be processed for:

[0047]

[0048] in, For the visual feature embedding set The first in Visual feature embedding, ,and , The total number of visual feature embeddings in the visual feature embedding set. A shared spatial dimension for visual and textual elements. Indicates the first A set of vectors to be processed. Indicates the temperature scaling factor. The total number of image categories, for The vector positions in each row of the multiplication matrix. Indicates request The maximum value of each row vector in the multiplication matrix. This represents the inversion operation on a matrix.

[0049] Understandably, calculating the maximum similarity score between the vector set to be processed and the visual feature embedding set based on the temperature scaling factor not only preserves the spatial distribution characteristics of the visual feature embedding, but also flexibly controls the strictness of feature matching through the temperature parameter. This enables the visual language model to adaptively capture the most discriminative cross-modal association features, effectively enhancing the semantic consistency of visual language representation. It provides a reliable association metric for subsequent feature interaction and optimization, and significantly improves cross-modal alignment accuracy and classification robustness in complex scenarios.

[0050] Optionally, the step of performing feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction includes:

[0051] Based on the preset self-attention unit, text association features between category text embeddings in the category text embedding set are obtained, and visual association features between visual feature embeddings in the visual feature embedding set are obtained based on the self-attention unit.

[0052] Based on a preset cross-attention unit, visual and textual association features between category text embeddings in the category text embedding set and visual feature embeddings in the visual feature embedding set are obtained;

[0053] The text association features, the visual association features, and the visual and text association features are respectively used to perform feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction.

[0054] Understandably, by using text association features, visual association features, and visual text association features to interact with the category text embedding set and the visual feature embedding set, dynamic mutual enhancement of text features and visual features is achieved. This enables the updated category text embedding set to agglomerate more accurate semantic guidance, and the updated visual feature embedding set to obtain more focused task-related representations. This significantly improves the alignment accuracy and fusion efficiency of multimodal information, and provides a more discriminative joint feature representation for subsequent classification decisions.

[0055] Optionally, the loss function Based on cross-entropy loss function Hints at initializing the loss function Suggestions for optimizing the loss function and knowledge distillation loss function The following is the result of the construction:

[0056]

[0057] in, and This is the loss balance coefficient;

[0058] The cross-entropy loss function It is constructed based on the visual feature embedding set and the text feature embedding set;

[0059] The prompt initializes the loss function. It is constructed based on the text feature embedding set and the average category text embedding set;

[0060] The suggestion optimizes the loss function. It is constructed based on the text feature embedding set and the category text embedding set;

[0061] The knowledge distillation loss function It is constructed based on the visual feature embedding set, the text feature embedding set, the category text embedding set that completes the feature interaction, and the visual feature embedding set that completes the feature interaction.

[0062] Understandably, by constructing a loss function that optimizes multiple objectives, a comprehensive performance is achieved that combines high classification accuracy, strong cue adaptation capability, and robust cross-modal alignment, significantly enhancing the generalization ability and engineering application reliability of visual language models in low-resource scenarios.

[0063] Optionally, the cross-entropy loss function Specifically:

[0064]

[0065] in, The total number of categories in the image category. For the first Image categories, For the first Digital labels for each image category; For the first The text feature embeddings for each image category Predict the visual feature embedding set The image in the middle belongs to the first The probability of each image category;

[0066] The prompt initializes the loss function. Specifically:

[0067]

[0068] in, For the first The average category text embedding for each image category, and Text embeddings belonging to the average category;

[0069] The suggestion optimizes the loss function. Specifically:

[0070]

[0071] in, For the first Category text embeddings for each image category, and Belongs to the category of text embedding set;

[0072] The knowledge distillation loss function Specifically:

[0073]

[0074] Among them, based on the first Logarithmic distribution of students for each image category for:

[0075]

[0076] in, For the traversal of the first Image categories, For the first Text feature embeddings for each image category, and Belonging to the text feature embedding set, wherein ;

[0077] Based on the Teacher logarithmic distribution for each image category for:

[0078]

[0079] in, To complete the visual feature embedding set for feature interaction, For the first Category text embedding for image categories to complete feature interactions, For the first Each image category completes the category text embedding for feature interaction, and , Both belong to the category of text embedding sets that complete feature interactions.

[0080] Understandably, by constructing a composite optimization system that includes cross-entropy loss, cue initialization loss, cue optimization loss, and knowledge distillation loss, the visual language model achieves a synergistic improvement in classification accuracy, cue quality, and cross-modal alignment capability.

[0081] According to a second aspect of this application, a visual language cue-based optimized image classification system is provided, the system comprising:

[0082] The acquisition module is used to acquire an image training set with image category labels, acquire image categories based on the image category labels in the image training set, and acquire an application set of images to be processed and a corresponding prompt text set.

[0083] A visual language model, which is pre-trained, is used to classify the image application set to be processed and the corresponding text application set to obtain the image classification result of the image application set.

[0084] An injection module is used to construct an injection token, and inject the visual encoder and text encoder of the visual language model based on the injection token to obtain an updated visual encoder and text encoder.

[0085] The pre-training process of a visual language model is as follows:

[0086] The image training set is input into the updated visual encoder to obtain a visual feature embedding set; a preset text training set is input into the updated text encoder to obtain a text feature embedding set; an average category text embedding set is obtained according to the image category; a category text embedding set is obtained according to the image category and the visual feature embedding set; feature interaction is performed on the category text embedding set and the visual feature embedding set to obtain a category text embedding set and a visual feature embedding set that have completed the feature interaction; the visual language model is optimized according to the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that has completed the feature interaction, the visual feature embedding set that has completed the feature interaction, and a preset loss function to obtain a trained visual language model.

[0087] According to a third aspect of this application, an electronic device is provided, comprising:

[0088] Memory, used to store one or more computer programs;

[0089] A processor, when the one or more computer programs are executed by the processor, implements the image classification method based on visual language cues optimized as described in the first aspect above.

[0090] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image classification method based on visual language cues optimized as described in the first aspect above.

[0091] Based on any of the above aspects, the image classification method, system, electronic device, and storage medium optimized based on visual language cues provided in this application can achieve the following benefits:

[0092] 1. Introducing Injection Tokens to Enhance Cross-Domain Transfer Capabilities: By injecting injection tokens into the visual encoder and text encoder, dynamically mapping them to visual and text spaces, the problem of distribution differences between different professional fields is effectively alleviated. Through a layered injection strategy, injection tokens are embedded only in the deep structure of the visual encoder and text encoder. This preserves the underlying general feature extraction capabilities while giving the visual language model the potential to flexibly adapt to new domains. This allows the visual language model to quickly absorb target domain-specific knowledge, significantly improving cross-domain transfer efficiency.

[0093] 2. Language-assisted evolutionary learning of visual language models based on categorical text embeddings: In this application, the average categorical text embedding set and the categorical text embedding set are generated using a large language model. By iteratively generating diverse categorical cues covering domain knowledge, the limitations of manual templates are overcome, the cost of manual intervention is reduced, and precise alignment of fine-grained semantics and visual features is achieved. Under low-sample conditions, the language assistance in this application effectively suppresses the risk of overfitting, enabling the visual language model to capture more discriminative categorical features and significantly improving classification accuracy and robustness in small-sample scenarios.

[0094] 3. The introduction of self-attention units and cross-attention units improves the accuracy of information fusion: By deploying lightweight self-attention units and cross-attention units in the multimodal interaction stage, self-attention units strengthen the correlation within similar features, while cross-attention units establish fine-grained correspondences between visual and textual features. This feature processing based on hierarchical attention enables the visual language model to focus on key semantic regions and suppress interference from irrelevant information. In complex scenarios such as industrial abrasive particle pattern recognition, it can significantly improve the alignment accuracy of multimodal features. Even under adverse conditions such as changes in illumination and background interference, it can still maintain stable feature extraction capabilities, providing technical support for high-reliability engineering applications. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart of an image classification method based on visual language cues optimized for this embodiment.

[0097] Figure 2 This embodiment provides a flowchart for injecting a token into a visual encoder.

[0098] Figure 3 This embodiment provides a flowchart for injecting tokens into a text encoder.

[0099] Figure 4 This is a flowchart of a visual language model training method provided in this embodiment.

[0100] Figure 5 This is a flowchart for obtaining an average category text embedding set provided in this embodiment.

[0101] Figure 6This is a flowchart for obtaining a set of category text embeddings provided in this embodiment.

[0102] Figure 7 This is a flowchart of a feature interaction provided in this embodiment.

[0103] Figure 8 This is a schematic diagram of the functional modules of an image classification system based on visual language cues optimization provided in this embodiment.

[0104] Figure 9 This embodiment provides a schematic diagram of the electronic device. Detailed Implementation

[0105] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0106] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0107] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0108] This embodiment provides a technical solution that can solve the above problems. The specific implementation of this application will be described in detail below with reference to the accompanying drawings.

[0109] like Figure 1 As shown, this embodiment provides an image classification method optimized based on visual language cues, which can be further divided into the following steps:

[0110] S100: Obtain an image training set with image category labels, and obtain the image category based on the image category labels in the image training set;

[0111] In this embodiment, the image training set contains several training images, each labeled with a corresponding image category. Based on these training images and their corresponding category labels, several image categories can be obtained, providing a data foundation for subsequently obtaining text-based auxiliary prompts.

[0112] S200. Construct an injection token, and inject the visual encoder and text encoder of the visual language model based on the injection token to obtain the updated visual encoder and text encoder.

[0113] Understandably, image classification using visual language models typically involves two input branches: a visual input branch with the input image and a text input branch with the input text. However, input images and input text belong to two different modalities. During the training and learning of visual language models, data from different modalities often face semantic gaps and alignment challenges due to modality heterogeneity. Because visual and textual data differ significantly in representation space, data distribution, and semantic granularity, direct mapping can easily lead to feature misalignment or information loss. Therefore, in this embodiment, an injection token is introduced. This injection token is a modality-independent representation token, which can construct a modality-independent shared latent representation space for visual and textual features, enabling precise alignment of feature data from different modalities within a shared semantic subspace.

[0114] Preferably, a preset mapping function can be used. Inject modality-independent tokens Distribution mapping to visual space and text space .in, To represent the spatial dimension as modality independent, For visual space dimension, For text space dimension, The number of tokens injected.

[0115] Specifically, the visual encoder includes several sequentially connected visual encoding layers, and the injection token includes a visual injection token;

[0116] like Figure 2 As shown, injecting the injection token into the visual encoder of the visual language model to obtain an updated visual encoder may include the following steps:

[0117] S211, Preset visual injection layer number threshold;

[0118] S212. Determine whether the layer number of the visual coding layer exceeds the threshold of the visual injection layer number. If so, then use the visual coding layer as the visual injection layer.

[0119] S213. Inject the visual injection token into the visual injection layer to obtain the updated visual encoder;

[0120] Understandably, in order to avoid disrupting the underlying general representation of the visual encoder, modality-independent injection tokens need to be injected into the higher layers of the visual encoder. In this embodiment, by setting a preset visual injection layer number threshold, the layer number from which the injection token begins to be injected is limited. This ensures that visual encoding layers with layer numbers below the visual injection layer number threshold operate normally according to their original settings, while visual encoding layers with layer numbers exceeding the visual injection layer number threshold need to be injected with visual injection tokens to construct a modality-independent shared latent representation space, thereby obtaining an updated visual encoder.

[0121] Specifically, the updated visual encoder Represented as:

[0122]

[0123] in, The updated visual encoder represents the first... Layered visual encoding layer, This represents the total number of visual coding layers. , Indicates the first The global representation token of the input image in the layered visual coding layer, and ; Indicates that the input image is in the first... Embedded data of the visual encoding layer; This represents the threshold number of the visual injection layer. Indicates the first Visual injection tokens for the visual encoding layer.

[0124] Specifically, the text encoder is divided into several sequentially connected text encoding layers, and the injection token includes a text injection token;

[0125] like Figure 3 As shown, injecting the injection token into the text encoder of the visual language model to obtain an updated text encoder may include the following steps:

[0126] S221, Preset text injection layer number threshold;

[0127] S222. Determine whether the layer number of the text encoding layer exceeds the threshold of the text injection layer number. If so, then use the text encoding layer as the text injection layer.

[0128] S223. Inject the text injection token into the text injection layer to obtain the updated text encoder.

[0129] Understandably, in order to avoid disrupting the underlying general representation of the text encoder, modality-independent injection tokens need to be injected into the higher layers of the text encoder. In this embodiment, by setting a preset text injection layer number threshold, the layer number from which the injection token begins to be injected is limited. This ensures that text encoding layers with layer numbers below the text injection layer number threshold operate normally according to their original settings, while text encoding layers with layer numbers exceeding the text injection layer number threshold need to be injected with text injection tokens to construct a modality-independent shared latent representation space, thereby obtaining the updated text encoder.

[0130] Specifically, the updated text encoder Represented as:

[0131]

[0132] in, This represents the first... of the updated text encoder Layered text encoding layer, This indicates the total number of text encoding layers. , Indicates the first In the text encoding layer, the beginning of the input text indicates the token; Indicates the first The end of the input text in the layered text encoding layer represents the token. ; This indicates that the input text is in the first... Embedded data in the text encoding layer; This indicates the threshold for the text injection layer number. Indicates the first Text injection tokens for the text encoding layer.

[0133] Preferably, the input text can be a text embedding set. With learnable context embeddings The text training set is obtained by splicing together the data. The data is then input into the updated text encoder to obtain the subsequent text feature embedding set.

[0134] Preferably, in this embodiment, the total number of visual coding layers is... The total number of text encoding layers They are equal, that is .

[0135] S300: Input the image category, the image training set, and the preset text training set into the visual language model for training;

[0136] In this embodiment, in addition to using conventional image and text training sets as training data, image categories are introduced and input separately into the large language model for query processing to obtain text-based auxiliary prompts. It is understood that in specialized fields such as industrial defect detection and wear debris analysis, the quality of the prompt content plays a crucial role in determining the generalization ability of the visual language model. Overly specific or biased prompts may cause the visual language model to overfit certain defect patterns, while low-quality prompts that do not conform to the visual features of a specific domain will significantly reduce recognition performance. Therefore, in this embodiment, introducing image category-based auxiliary prompts based on a large language model can solve the problems of semantic sparsity, high manual costs, lack of dynamic optimization, and noise accumulation in existing technologies for prompt optimization in low-resource or specialized domains.

[0137] Specifically, such as Figure 4 As shown, the process of training the visual language model can be divided into the following steps:

[0138] S310. Input the image training set into the updated visual encoder to obtain a visual feature embedding set;

[0139] S320. Input the preset text training set into the updated text encoder to obtain the text feature embedding set;

[0140] S330. Obtain the average category text embedding set according to the image category;

[0141] Specifically, such as Figure 5 As shown, obtaining the average category text embedding set based on the image category may include the following steps:

[0142] S331. Obtain the query instruction corresponding to the image category;

[0143] In this embodiment, standard query instructions are pre-generated. These query instructions can vary based on different image categories, allowing the query instructions to adapt to different image categories while maintaining standardization and avoiding the generation of redundant content. This reduces the cost of manual intervention and improves adaptability to specific domains.

[0144] For example, a certain image training set contains several image categories. CLASS The corresponding adaptive query instruction template It can be " Generate N concise and diverse textual descriptions of the visually discriminative features associated with CLASS as observed in MODALITY images. Each description should highlight different characteristics that help distinguish this class from others. "

[0145] in, MODALITY Used to specify the imaging modality, such as images of defects on the surface of a steel plate.

[0146] S332. Obtain the text description corresponding to the image category according to the query instruction;

[0147] In this embodiment, inputting the query command corresponding to each image category into the large language model can generate a corresponding text description, thereby forming a text description set. , This represents the total number of text descriptions contained in an image category. The total number of image categories, The length of each text description.

[0148] S333. Input the text description into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors;

[0149] Understandably, the initial category text embedding set can provide soft supervision signals to the visual language model, guiding the text feature embedding set to generate semantic representations aligned with the initial category cue text set in the high-dimensional embedding space, thus ensuring the transferability and generalization of the text feature embedding set.

[0150] S334. Calculate the mean of the initial category text embedding vectors in the initial category text embedding to obtain the corresponding average category text embedding. All the average category text embeddings form an average category text embedding set.

[0151] In this embodiment, for the initial category text embedding, there are several initial category text embedding vectors. The average of the several initial category text embedding vectors of an initial category text embedding is calculated to obtain the corresponding average category text embedding. This can effectively smooth the random fluctuations of a single initial category text embedding vector, aggregate the common semantic features of multiple initial category text embedding vectors, and form a more universal and stable category representation benchmark.

[0152] S340. Obtain a category text embedding set based on the image category and the visual feature embedding set;

[0153] Specifically, such as Figure 6 As shown, obtaining a category text embedding set based on several image categories and the visual feature embedding set may include the following steps:

[0154] S341. Obtain the query instruction corresponding to the image category;

[0155] S342. Obtain the text description corresponding to the image category according to the query instruction;

[0156] S343. Input the text description into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors;

[0157] In this embodiment, the steps of obtaining the category text embedding set and obtaining the average category text embedding set are the same as S331, S332 and S333, and will not be repeated here.

[0158] S344. Construct a corresponding vector set to be processed from the initial category text embedding vectors at the same position in all the initial category text embeddings;

[0159] It is understandable that each of the obtained initial category text embeddings contains several initial category text embedding vectors, and the total number of initial category text embedding vectors contained in each initial category text embedding is the same. In this embodiment, each initial category text embedding vector has corresponding position information. Preferably, the position information of each initial category text embedding vector can be marked using a vector index.

[0160] In this embodiment, the set of vectors to be processed is a collection of initial category text embedding vectors at the same position as the initial category text embedding.

[0161] For example, the first The initial category text embedding can be represented as... , , For the first The first initial category text embedding vector is generated from the initial category text embeddings. There are several sets of vectors to be processed, each containing the initial category text embedding vectors at the same position as the initial category text embedding. For example, the first set of vectors to be processed... The first initial category text embedding vector contains the initial category text embedding, and a set of vectors to be processed contains... An initial category text embedding vector is generated, and so on.

[0162] S345. Obtain the maximum similarity score for each vector set to be processed based on the vector set to be processed and the visual feature embedding set.

[0163] Specifically, no. Maximum similarity score of a set of vectors to be processed for:

[0164]

[0165] in, For the visual feature embedding set The first in Visual feature embedding, ,and , The total number of visual feature embeddings in the visual feature embedding set. A shared spatial dimension for visual and textual elements. Indicates the first A set of vectors to be processed. Indicates the temperature scaling factor. The total number of image categories, for The vector positions in each row of the multiplication matrix. Indicates request The maximum value of each row vector in the multiplication matrix. This represents the operation of reversing a matrix.

[0166] For example, for the first set of vectors to be processed ,Will Each initial category text embedding vector in the dataset is multiplied by each visual feature embedding in the visual feature embedding set and the temperature scaling factor to obtain the score matrix of the first vector set to be processed. The maximum similarity score in each row of the score matrix is ​​then calculated; for example, the maximum value in the first row is obtained by finding the maximum similarity score for that row. The other rows are calculated in the same way, resulting in... The maximum similarity score, i.e. and will Add the maximum similarity scores and divide by . The maximum similarity score corresponding to the first set of vectors to be processed is obtained. ;

[0167] The other vector sets to be processed are calculated in the same way to obtain the maximum similarity score for each vector set. The maximum similarity scores of each vector set can form a score set. .

[0168] S346. Determine whether the vector set to be processed is an abnormal vector set based on the maximum similarity score of each vector set to be processed and the preset score threshold. If so, obtain the position of the vector in the abnormal vector set in the initial category text embedding as the abnormal position, delete the initial category text embedding vector corresponding to the abnormal position from each initial category text embedding, and obtain the updated initial category text embedding.

[0169] In this embodiment, the Median Absolute Deviation (MAD) method is used to determine whether the vector set to be processed is an outlier vector set. Specifically, the score set is obtained. median score Specifically:

[0170]

[0171] Based on the median score Obtain the deviation value for each maximum similarity score, and obtain the median deviation of several deviation values. Specifically:

[0172]

[0173] Based on the median score and the median of the deviation Obtain the absolute deviation value for each maximum similarity score, specifically:

[0174]

[0175] in, That is, the maximum similarity score of a set of vectors to be processed corresponds to an absolute deviation value;

[0176] For example, the preset score threshold is The absolute deviation value corresponding to the maximum similarity score of each vector set to be processed is compared with the score threshold. If Then determine the first The set of vectors to be processed is the set of abnormal vectors, and the first set is obtained. The vectors in the set of anomaly vectors are taken as anomaly positions in the initial category text embeddings. The initial category text embedding vectors corresponding to the anomaly positions are deleted from each initial category text embedding to obtain updated initial category text embeddings.

[0177] S347. Calculate the mean of the initial category text embedding vectors in the updated initial category text embeddings to obtain the corresponding category text embeddings. All the category text embeddings form a category text embedding set.

[0178] In this embodiment, for the updated initial category text embedding, there are still several initial category text embedding vectors. The average of the several initial category text embedding vectors of an updated initial category text embedding is calculated to obtain the corresponding category text embedding. This can effectively smooth the random fluctuations of a single initial category text embedding vector, aggregate the common semantic features of multiple initial category text embedding vectors, and form a more universal and stable category representation benchmark.

[0179] S350. Perform feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction.

[0180] In this embodiment, an attention adapter is introduced to facilitate feature interaction between the categorical text embedding set and the visual feature embedding set. This attention adapter is a lightweight adapter that explicitly models the interaction between fine-grained visual feature embeddings and categorical text embeddings through a lightweight, trainable cross-attention structure, achieving dynamic information transfer and semantic alignment across modalities. This adapter can be inserted into the high-level structures of the visual encoder branch and the text encoder branch, enabling complementary fusion of multimodal features while keeping the backbone network parameters frozen.

[0181] The cross-attention adapter module comprises two parts: a self-attention unit and a cross-attention unit. Self-attention captures the correlations within the same modality, while the cross-attention unit guides feature interactions between different modalities. Specifically, for the corresponding text embedding set and visual feature embedding set, the processing layer of the attention adapter calculates attention-weighted fusion features, adaptively emphasizing the most relevant contextual information in both modalities. This selective cross-modal interaction enables the visual language model to learn richer and more accurate contextual cues by effectively aligning semantic cues from the text domain with salient visual patterns.

[0182] Specifically, such as Figure 7 As shown, the step of performing feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction may include the following steps:

[0183] S351. Based on a preset self-attention unit, obtain the text association features between category text embeddings in the category text embedding set, and based on the self-attention unit, obtain the visual association features between each visual feature embedding in the visual feature embedding set;

[0184] S352. Based on a preset cross-attention unit, obtain the visual and textual association features between the category text embeddings in the category text embedding set and the visual feature embeddings in the visual feature embedding set;

[0185] S353. Perform feature interaction on the category text embedding set and the visual feature embedding set respectively using the text association feature, the visual association feature and the visual and text association feature, to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction.

[0186] In this embodiment, when feature interaction is performed based on the visual and text association features, bidirectional interaction between vision and text can be achieved. Specifically, when the visual feature embedding set is optimized by using the visual and text association features to guide the optimization of the visual feature embedding set, the ability of the visual modality to capture professional semantics can be improved; when the visual feature embedding set is optimized by using the visual and text association features to optimize the expressive ability of the visual feature embedding set, the ability of the text modality to perceive visual details can be enhanced.

[0187] S360. Optimize the visual language model based on the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that completes the feature interaction, the visual feature embedding set that completes the feature interaction, and a preset loss function to obtain a trained visual language model.

[0188] Specifically, the loss function Based on cross-entropy loss function Hints at initializing the loss function Suggestions for optimizing the loss function and knowledge distillation loss function The following is the result of the construction:

[0189]

[0190] in, and This is the loss balancing coefficient, used to adjust the contribution ratio of each loss to the final optimization process.

[0191] The cross-entropy loss function It is constructed based on the visual feature embedding set and the text feature embedding set;

[0192] The prompt initializes the loss function. It is constructed based on the text feature embedding set and the average category text embedding set;

[0193] The suggestion optimizes the loss function. It is constructed based on the text feature embedding set and the category text embedding set;

[0194] The knowledge distillation loss function It is constructed based on the visual feature embedding set, the text feature embedding set, the category text embedding set that completes the feature interaction, and the visual feature embedding set that completes the feature interaction.

[0195] In this embodiment, the overall learning objective of the entire visual language model is constructed based on four loss functions, including cross-entropy loss to ensure image classification accuracy, cue initialization loss and cue optimization loss to refine the semantic representation of contextual cues, and knowledge distillation loss to guide semantic consistency from prior knowledge generated from the large language model.

[0196] Specifically, the cross-entropy loss function Specifically:

[0197]

[0198] in, The total number of categories in the image category. For the first Image categories, For the first Digital labels for each image category; For the first The text feature embeddings for each image category Predict the visual feature embedding set The image in the middle belongs to the first The probability of each image category;

[0199] The prompt initializes the loss function. Specifically:

[0200]

[0201] in, For the first The average category text embedding for each image category, and Text embeddings belonging to the average category;

[0202] Understandably, the prompt initialization loss function is constructed based on mean squared error to ensure that the visual language model can utilize generalizable and transferable semantic information during the initialization phase.

[0203] The suggestion optimizes the loss function. Specifically:

[0204]

[0205] in, For the first Category text embeddings for each image category, and Belongs to the category of text embedding set;

[0206] Understandably, the prompting optimization loss function aims to maximize the cosine similarity between the text feature embedding set and the category text embedding set, thereby refining the text feature embedding set and the category text embedding set, and aligning the refined text feature embedding set and the category text embedding set with rich visual semantics.

[0207] The knowledge distillation loss function Specifically:

[0208]

[0209] Among them, based on the first Logarithmic distribution of students for each image category for:

[0210]

[0211] in, For the traversal of the first Image categories, For the first Text feature embeddings for each image category, and Belonging to the text feature embedding set, wherein ;

[0212] Based on the Teacher logarithmic distribution for each image category for:

[0213]

[0214] in, To complete the visual feature embedding set for feature interaction, For the first Category text embedding for image categories to complete feature interactions, For the first Each image category completes the category text embedding for feature interaction, and , Both belong to the category of text embedding sets that complete feature interactions.

[0215] In this embodiment, to achieve consistent knowledge transfer between high-quality category hints and text embedding hints, the visual language model aligns the teacher logits (obtained by combining the visual feature embedding set that completes feature interaction with the category text embedding set that completes feature interaction) with the student logits (obtained by combining the visual feature embedding set and the text feature embedding set). Knowledge distillation is achieved by minimizing the Kullback-Leibler (KL) divergence between the two. Through knowledge distillation, the text feature embedding set can effectively mimic the refined semantic information, thereby improving the hint quality and multimodal alignment performance.

[0216] In this embodiment, the joint optimization of the four losses mentioned above ensures both the classification performance of the visual language model and the high consistency between the text feature embedding set and the category text embedding set that completes the feature interaction in the semantic space. Among them, the knowledge distillation loss plays a key role in guiding the text feature embedding set into the semantic space shaped by the prior of the large language model, effectively preventing the prompt representation of the text feature embedding set from deviating to the suboptimal or even noisy region, and ensuring that the prompt information learned by the visual language model has domain relevance and robustness to multimodal interactions.

[0217] S400. Input the image application set to be processed and the corresponding text application set into the trained visual language model to obtain the image classification result of the image application set.

[0218] In this embodiment, a trained visual language model can be used to classify the image application set to be processed based on the corresponding text application set, thereby obtaining accurate image classification results.

[0219] In practical applications, six publicly available datasets and one self-built dataset are used as inputs to the visual language model for image classification, covering fields such as industrial surface defect detection, medical imaging, remote sensing images, texture classification, and wear analysis based on lubricating oil abrasive particles. Specifically, the FSC-20 and MT-CF datasets in the table are used for industrial defect classification; the MED-3 dataset in the table is used for medical image analysis, including a blood cell image database, HAM10000 dermoscopy lesion analysis, and OCT retinal disease diagnosis; the EuroSAT and RESISC45 datasets in the table are used for remote sensing image classification; the DTD dataset in the table is used for describable texture classification; and the self-built WEAR dataset contains abrasive particle images from industrial rotating machinery lubricating oil. The WEAR dataset contains 788 images distributed across 11 image categories with class imbalance, reflecting the low resource and high complexity characteristics of real industrial fault diagnosis.

[0220] In the experimental design, each dataset is divided into a base image category (Base) and a new image category (Novel). First, the visual language model of this application is trained on the base image category, and then the performance of the visual language model is evaluated on the base image category and the new image category respectively.

[0221] Evaluation metrics include Base Acc, Novel Acc, and their combined mean (HM), to measure the balanced performance of the visual language model in zero-shot or near-zero-shot tests. Specifically, HM is:

[0222] HM = 2 × Base × Novel / (Base + Novel)

[0223] Specifically, as shown in Tables 1, 2, and 3, the classification accuracy of the visual language model in this application compared with other models in terms of basic image categories and new image categories, as well as the accuracy (%) of the mixture mean (HM) of the two. Other model methods included CLIP, CoOp, CoCoOp, KgCoOp, MaPLe, and MMRL. As can be seen from Tables 1, 2, and 3, the classification performance and accuracy of the visual language model used in this application for image classification on various datasets are almost always superior to other models.

[0224] Table 1

[0225]

[0226] Table 2

[0227]

[0228] Table 3

[0229]

[0230] like Figure 8 As shown, this application embodiment also provides an image classification system optimized based on visual language cues. Optionally, the system includes:

[0231] The acquisition module 511, the visual language model 512, and the injection module 513, wherein:

[0232] The acquisition module 511 is used to acquire an image training set with image category labels, acquire image categories based on the image category labels in the image training set, and acquire an application set of images to be processed and a corresponding prompt text set;

[0233] In this embodiment, the acquisition module 511 can be used to perform... Figure 1For a detailed description of the acquisition module 511, please refer to the description of step S100 shown.

[0234] The visual language model 512, which is pre-trained, is used to classify the image application set to be processed and the corresponding text application set to obtain the image classification result of the image application set.

[0235] In this embodiment, the visual language model 512 can be used to execute... Figure 1 For a detailed description of the acquisition module 511, please refer to the description of steps S300 and S400 shown.

[0236] The injection module 513 is used to construct an injection token and inject the visual encoder and text encoder of the visual language model based on the injection token to obtain the updated visual encoder and text encoder.

[0237] In this embodiment, the injection module 513 can be used to perform... Figure 1 For a detailed description of the injection module 513, please refer to the description of step S200 shown.

[0238] The pre-training process of a visual language model is as follows:

[0239] The image training set is input into the updated visual encoder to obtain a visual feature embedding set; a preset text training set is input into the updated text encoder to obtain a text feature embedding set; an average category text embedding set is obtained according to the image category; a category text embedding set is obtained according to the image category and the visual feature embedding set; feature interaction is performed on the category text embedding set and the visual feature embedding set to obtain a category text embedding set and a visual feature embedding set that have completed the feature interaction; the visual language model is optimized according to the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that has completed the feature interaction, the visual feature embedding set that has completed the feature interaction, and a preset loss function to obtain a trained visual language model.

[0240] This application also provides an electronic device, the structure of which is as follows: Figure 9 As shown, the electronic device includes a memory 611, a processor 612, a communication module 613, and an input / output interface 614, etc. Optionally, the memory 611, the processor 612, the communication module 613, and the input / output interface 614 can be connected and communicate with each other through a bus 615.

[0241] The memory 611 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 612; when the one or more computer programs are executed by the processor 612, an image classification method based on visual language cues optimized according to the present application embodiment is implemented.

[0242] Optionally, the electronic device can be connected to a network via communication module 613 to communicate with other devices, such as terminals or servers, to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0243] Optionally, the electronic device can connect to required input / output devices, such as a keyboard or display device, via the input / output interface 614. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 614. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 614 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 611. It is understood that the input / output interface 614 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 614 can be a component of the electronic device or an external device connected to the electronic device when needed.

[0244] Optionally, the memory 611 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0245] Optionally, the computer program stored in the memory 611 can be divided into one or more modules, which are stored in the memory 611 and executed by the processor 612 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0246] Optionally, the processor 612 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 612 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 612 executes the various methods and processes of this embodiment, exemplarily, such as an image classification method optimized based on visual language cues according to an embodiment of this application.

[0247] Optionally, the bus 615 may include a path for transmitting information. Depending on its function, the bus 615 may be divided into an address bus, a data bus, a control bus, etc.

[0248] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above-described method embodiments. Part or all of the computer program can be loaded and / or installed on the memory 611 of an electronic device. When the computer program is executed by the processor 612, one or more steps of a visual language cue-optimized image classification method according to this application embodiment can be performed.

[0249] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0250] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for image classification based on visual language cue optimization, characterized in that, The method includes: Obtain an image training set with image category labels, and obtain the image category based on the image category labels in the image training set; An injection token is constructed, and the visual encoder and text encoder of the visual language model are injected based on the injection token to obtain the updated visual encoder and text encoder. The image category, the image training set, and the preset text training set are input into the visual language model for training. The training process is as follows: The image training set is input into the updated visual encoder to obtain a visual feature embedding set; a preset text training set is input into the updated text encoder to obtain a text feature embedding set; an average category text embedding set is obtained according to the image category; a category text embedding set is obtained according to the image category and the visual feature embedding set; feature interaction is performed on the category text embedding set and the visual feature embedding set to obtain a category text embedding set and a visual feature embedding set that have completed the feature interaction; based on the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that has completed the feature interaction, and the complete... The visual language model is optimized using a set of visual feature embeddings representing the feature interactions and a preset loss function to obtain a trained visual language model. The step of obtaining an average category text embedding set based on the image category includes: obtaining a query instruction corresponding to the image category; obtaining a text description corresponding to the image category based on the query instruction; inputting the text description into the text encoder to obtain initial category text embeddings, wherein the initial category text embeddings contain several initial category text embedding vectors; averaging the initial category text embedding vectors in the initial category text embeddings to obtain the corresponding average category text embeddings; all the average category text embeddings form an average category text embedding set. The image application set to be processed and the corresponding text application set are input into the trained visual language model to obtain the image classification result of the image application set.

2. The method of claim 1, wherein, The visual encoder comprises several sequentially connected visual encoding layers, and the injection token comprises a visual injection token; Based on the injection token, the visual encoder of the visual language model is injected to obtain an updated visual encoder, including: Preset visual injection layer number threshold; Determine whether the layer number of the visual coding layer exceeds the threshold of the visual injection layer number. If so, then the visual coding layer is used as the visual injection layer. The visual injection token is injected into the visual injection layer to obtain the updated visual encoder; And / or, The text encoder is divided into several sequentially connected text encoding layers, and the injection token includes a text injection token; Based on the injection token, the text encoder of the visual language model is injected to obtain an updated text encoder, including: Preset text injection layer number threshold; Determine whether the layer number of the text encoding layer exceeds the threshold of the text injection layer number; if so, then use the text encoding layer as the text injection layer. The text injection token is injected into the text injection layer to obtain the updated text encoder.

3. The method according to claim 2, characterized in that, The updated visual encoder is represented as: wherein, denotes a first layer of visual encoding layers, denotes a total number of visual encoding layers, , denotes a first layer visual encoding layer, denotes a global representation token of the input image in the first layer visual encoding layer; denotes the visual injection layer number threshold, denotes a visual injection token of the first layer visual encoding layer; and / or the updated text encoder is represented as: in, This indicates the first... Layered text encoding layer, This indicates the total number of text encoding layers. , Indicates the first In the text encoding layer, the beginning of the input text indicates the token; Indicates the first The end of the input text in the layered text encoding layer represents a token; This indicates that the input text is in the first... Embedded data in the text encoding layer; This indicates the threshold for the text injection layer number. Indicates the first Text injection tokens for the text encoding layer.

4. The method according to claim 1, characterized in that, The step of obtaining the category text embedding set based on the image category and the visual feature embedding set includes: Obtain the query command corresponding to the image category; Obtain the text description corresponding to the image category according to the query instruction; The text description is input into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors; the initial category text embedding vectors at the same position in all the initial category text embeddings are used to construct the corresponding vector set to be processed; The maximum similarity score of each vector set to be processed is obtained based on the vector set to be processed and the visual feature embedding set; where, the first... Maximum similarity score of a set of vectors to be processed for: in, For the visual feature embedding set The first in Visual feature embedding, ,and , The total number of visual feature embeddings in the visual feature embedding set. A shared spatial dimension for visual and textual elements. Indicates the first A set of vectors to be processed. Indicates the temperature scaling factor. The total number of image categories, for The vector positions in each row of the multiplication matrix. Indicates request The maximum value of each row vector in the multiplication matrix. This indicates the operation of reversing a matrix; Based on the maximum similarity score of each vector set to be processed and the preset score threshold, it is determined whether the vector set to be processed is an abnormal vector set. If so, the position of the vector in the abnormal vector set in the initial category text embedding is obtained as the abnormal position. The initial category text embedding vector corresponding to the abnormal position is deleted from each initial category text embedding to obtain the updated initial category text embedding. The mean of the initial category text embedding vectors in the updated initial category text embeddings is calculated to obtain the corresponding category text embeddings, and all the category text embeddings form a category text embedding set.

5. The method according to claim 1, characterized in that, The step of performing feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction includes: Based on the preset self-attention unit, text association features between category text embeddings in the category text embedding set are obtained, and visual association features between visual feature embeddings in the visual feature embedding set are obtained based on the self-attention unit. Based on a preset cross-attention unit, visual and textual association features between category text embeddings in the category text embedding set and visual feature embeddings in the visual feature embedding set are obtained; The text association features, the visual association features, and the visual and text association features are respectively used to perform feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction.

6. The method according to claim 1, characterized in that, The loss function Based on cross-entropy loss function Hints at initializing the loss function Suggestions for optimizing the loss function and knowledge distillation loss function The following is the result of the construction: in, and This is the loss balance coefficient; The cross-entropy loss function It is constructed based on the visual feature embedding set and the text feature embedding set; The prompt initializes the loss function. It is constructed based on the text feature embedding set and the average category text embedding set; The suggestion optimizes the loss function. It is constructed based on the text feature embedding set and the category text embedding set; The knowledge distillation loss function It is constructed based on the visual feature embedding set, the text feature embedding set, the category text embedding set that completes the feature interaction, and the visual feature embedding set that completes the feature interaction.

7. The method according to claim 6, characterized in that, The cross-entropy loss function Specifically: in, The total number of categories in the image category. For the first Image categories, For the first Digital labels for each image category; For the first The text feature embedding of each image category Predict the visual feature embedding set The image in the middle belongs to the first The probability of each image category; The prompt initializes the loss function. Specifically: in, For the first The average category text embedding for each image category, and Text embeddings belonging to the average category; The suggestion optimizes the loss function. Specifically: in, For the first Category text embeddings for each image category, and Belongs to the category of text embedding set; The knowledge distillation loss function Specifically: Among them, based on the first Logarithmic distribution of students for each image category for: in, For the traversal of the first Image categories, For the first Text feature embeddings for each image category, and Belonging to the text feature embedding set, wherein ; Based on the Teacher logarithmic distribution for each image category for: in, To complete the visual feature embedding set for feature interaction, For the first Category text embedding for image categories to complete feature interactions, For the first Each image category completes the category text embedding for feature interaction, and , Both belong to the category of text embedding sets that complete feature interactions.

8. An image classification system optimized based on visual language cues, characterized in that, The system includes: The acquisition module is used to acquire an image training set with image category labels, acquire image categories based on the image category labels in the image training set, and acquire an application set of images to be processed and a corresponding prompt text set; A visual language model, which is pre-trained, is used to classify the image application set to be processed and the corresponding text application set to obtain the image classification result of the image application set. An injection module is used to construct an injection token, and inject the visual encoder and text encoder of the visual language model based on the injection token to obtain an updated visual encoder and text encoder. The pre-training process of a visual language model is as follows: The image training set is input into the updated visual encoder to obtain a visual feature embedding set; a preset text training set is input into the updated text encoder to obtain a text feature embedding set; an average category text embedding set is obtained according to the image category; a category text embedding set is obtained according to the image category and the visual feature embedding set; feature interaction is performed on the category text embedding set and the visual feature embedding set to obtain a category text embedding set and a visual feature embedding set that have completed the feature interaction; based on the visual feature embedding set, the text feature embedding set, the average category text embedding set, the category text embedding set, the category text embedding set that has completed the feature interaction, and the complete... The visual language model is optimized using the visual feature embedding set of the feature interaction and a preset loss function to obtain a trained visual language model; wherein, obtaining the average category text embedding set according to the image category includes: obtaining the query instruction corresponding to the image category; obtaining the text description corresponding to the image category according to the query instruction; inputting the text description into the text encoder to obtain the initial category text embedding, wherein the initial category text embedding contains several initial category text embedding vectors; averaging the initial category text embedding vectors in the initial category text embedding to obtain the corresponding average category text embedding, and all the average category text embeddings form the average category text embedding set.

9. The system according to claim 8, characterized in that, The visual encoder of the injection module includes several sequentially connected visual encoding layers, and the injection token includes a visual injection token; The injection module is also used for: Preset visual injection layer number threshold; Determine whether the layer number of the visual coding layer exceeds the threshold of the visual injection layer number. If so, then the visual coding layer is used as the visual injection layer. The visual injection token is injected into the visual injection layer to obtain the updated visual encoder; And / or, The text encoder of the injection module is divided into several sequentially connected text encoding layers, and the injection token includes a text injection token; The injection module is also used for: Preset text injection layer number threshold; Determine whether the layer number of the text encoding layer exceeds the threshold of the text injection layer number; if so, then use the text encoding layer as the text injection layer. The text injection token is injected into the text injection layer to obtain the updated text encoder.

10. The system according to claim 8, characterized in that, The visual language model is also used for: Based on the preset self-attention unit, text association features between category text embeddings in the category text embedding set are obtained, and visual association features between visual feature embeddings in the visual feature embedding set are obtained based on the self-attention unit. Based on a preset cross-attention unit, visual and textual association features between category text embeddings in the category text embedding set and visual feature embeddings in the visual feature embedding set are obtained; The text association features, the visual association features, and the visual and text association features are respectively used to perform feature interaction on the category text embedding set and the visual feature embedding set to obtain the category text embedding set and the visual feature embedding set that have completed the feature interaction.

Images