Image classification method, system and equipment based on vision-language interaction fine granularity

Abstract

The invention relates to the field of computer vision, in particular to an image classification method, system and device based on vision-language interaction fine granularity. Comprising the steps of obtaining a to-be-classified image and a prompt instruction, and generating a corresponding text description; extracting visual features of the to-be-classified image, and extracting text features of the text description; performing global average pooling operation on the visual features to obtain global visual embedding, performing average pooling operation on the text features to obtain global semantic embedding, and mapping the global semantic embedding to obtain global semantic priori; acquiring a visual token according to the visual feature; presetting a guide token, and performing splicing processing according to the global semantic priori, the visual token and the guide token to obtain a splicing feature; inputting the splicing features into a relative position sensing Transform model to obtain model output features, wherein a relative position offset table is constructed in the model; and performing image classification according to the model output features to obtain a classification result. According to the method, higher classification accuracy and higher generalization performance are realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more specifically, to a fine-grained image classification method, system, and device based on visual-language interaction. Background Technology

[0002] In computer vision, fine-grained image classification aims to distinguish subclasses within the same semantic category that have subtle differences in appearance, such as different species of birds, car models, or plant varieties. The core challenge of this task lies in the large intra-class differences and small inter-class differences, making traditional recognition methods based on global features ineffective in capturing decisive discriminative cues. Meanwhile, zero-shot learning (aiming to achieve generalized recognition on categories not seen during training) typically accomplishes cross-class inference by aligning visual features with semantic prototypes generated from category names, attribute descriptions, or textual cues.

[0003] In recent years, visual-language pre-trained models have significantly promoted the development of zero-shot classification by mapping images and text to a shared semantic space through contrastive learning.

[0004] However, existing fine-grained zero-shot learning methods still face several technical bottlenecks. First, mainstream methods generally rely on class-level semantics, that is, assigning a uniform text label or attribute vector to each class as a semantic prototype. While this strategy can achieve coarse-grained global alignment, it cannot reflect the morphological, pose, or local structural differences between different instances within the same class, resulting in limited performance in fine-grained scenarios with high intra-class diversity. Second, even with the use of pre-trained Vision Transformer (ViT) or class-level semantic injection, there is still a significant modal gap between visual features and semantic features, leading to limited semantic alignment capabilities.

[0005] Due to the aforementioned shortcomings, the current method has insufficient generalization ability on unseen fine-grained categories, making it difficult to meet the requirements of high accuracy, strong robustness, and interpretability in practical applications. Summary of the Invention

[0006] This invention provides a fine-grained image classification method, system, and device based on vision-language interaction, which improves the accuracy and generalization ability of fine-grained zero-shot image classification.

[0007] According to a first aspect of this application, a fine-grained image classification method based on visual-language interaction is provided, the method comprising: Obtain the image to be classified and the corresponding prompt instruction; generate a text description based on the image to be classified and the corresponding prompt instruction. Extract visual features from the image to be classified, and extract text features from the text description; The visual features are processed by global average pooling to obtain global visual embedding, and the text features are processed by average pooling to obtain global semantic embedding. The global semantic embedding is then mapped to the same dimensional space as the global visual embedding to obtain global semantic prior. Obtain a visual token based on the visual features; A pre-defined guiding token is used to concatenate the global semantic prior, the visual token, and the guiding token to obtain the concatenated feature. The spliced ​​features are input into the relative position-aware Transformer model to obtain the model output features. The relative position-aware Transformer model has a relative position bias table. The relative position bias in the relative position bias table includes relative bias and shared bias. The relative bias comes from the spatial relationship between visual tokens, and the shared bias comes from the fusion relationship between global semantic prior and guiding token and visual token. The classification result is obtained by classifying the image based on the features output by the model.

[0008] Understandably, this application constructs fine-grained semantic representations that correspond one-to-one with the images to be classified by introducing instance-level text descriptions generated from prompts. It then integrates visual tokens, global semantic priors, and the prompt tokens into a unified concatenated feature by combining learnable guiding tokens. Furthermore, it utilizes a relative position-aware Transformer model with an embedded relative position bias table to achieve fine-grained modeling of cross-modal features in the spatial dimension and adaptive modulation in the channel dimension. This application effectively bridges the semantic gap between visual and semantic modalities, significantly enhancing the model's ability to perceive subtle intra-class differences, thereby achieving higher classification accuracy and stronger generalization performance in fine-grained zero-shot image classification tasks.

[0009] Optionally, generating a text description based on the image to be classified and the corresponding prompt instruction includes: inputting the image to be classified and the corresponding prompt instruction into a multimodal large language model to generate a text description.

[0010] Understandably, this approach automatically generates individualized text descriptions that closely match the content of the image to be classified by inputting the image to be classified and its corresponding prompts into a multimodal large language model. This effectively captures fine-grained semantic information at the instance level, breaks through the limitations of traditional class-level semantic labels, provides fine semantic supervision for subsequent cross-modal alignment, and significantly improves the model's ability to identify intra-class differences and its generalization performance to unseen categories in fine-grained image classification tasks.

[0011] Optionally, visual features are extracted from the image to be classified, and textual features are extracted from the text description, including: The image to be classified is input into a visual encoder to extract the visual features; The text description is input into a text encoder to extract the text features.

[0012] Optionally, obtaining a visual token based on the visual features includes: The visual features are divided into a series of visual tokens through an overlapping patch embedding layer, wherein the overlapping patch embedding layer is a learnable two-dimensional convolutional layer in which adjacent convolutional receptive fields have overlapping regions.

[0013] Understandably, this approach, by introducing a learnable overlapping patch embedding layer, divides visual features into visual tokens with local context awareness. The overlapping design between adjacent convolutional receptive fields effectively preserves the spatial continuity and detail information between local regions of the image, alleviating the boundary information loss problem caused by traditional non-overlapping block segmentation. This approach not only enhances the ability of visual tokens to represent fine-grained structures, but also provides richer and more coherent spatial cues for subsequent cross-modal interactions with relative position awareness, thereby significantly improving the model's discriminative performance in fine-grained image classification tasks.

[0014] Optionally, the preset guiding token is concatenated based on the global semantic prior, the visual token, and the guiding token to obtain concatenated features, including: The visual token extracts a global visual context vector through global average pooling; A pre-defined guiding token is used, which is then concatenated with the global semantic prior and the global visual context vector to form a fusion vector. The fusion vector is mapped to the channel modulation vector via a multilayer perceptron. The updated visual token is obtained by updating the visual token using the channel modulation vector; The concatenated feature is obtained by concatenating the global semantic prior, the guiding token, and the updated visual token.

[0015] Understandably, this approach introduces learnable guiding tokens, integrates global semantic priors and global visual context vectors, generates instance-aware channel modulation vectors via a multilayer perceptron, and dynamically reweights the channel responses of the visual tokens, thereby accurately injecting fine-grained instance-level semantic priors into visual features. Finally, the concatenated features are input into the Transformer model to further model the spatial relationship between vision and semantics, achieving global channel-level interaction from semantics to vision, effectively enhancing the robustness and interpretability of cross-modal feature alignment.

[0016] Optionally, for each visual token, the update method is as follows: in, Indicates the firstj A visual token This indicates the total number of visual tokens.

[0017] Understandably, the updated visual token retains the original visual details while embedding global semantic priors, thus making the visual features more discriminative and consistent in the semantic space.

[0018] Optionally, the index value of the relative position offset table is obtained as follows: The relative position-aware Transformer model establishes coordinates for each visual token in the updated visual tokens on a two-dimensional grid and flattens each coordinate into a coordinate vector. Calculate the relative coordinate difference between any two visual tokens based on the coordinate vector; Shift the relative coordinate difference to a positive range; The one-dimensional index value of the visual token is obtained based on the relative coordinate difference offset to the positive interval. : ; The index matrix of the relative position bias table is constructed based on the one-dimensional index value of the visual token, and the index value of the shared bias is extended at the end of the index matrix so that the shared bias and the relative bias are uniformly mapped to the relative position bias table.

[0019] Optionally, the stitched features are input into a relative position-aware Transformer model to obtain the model output features, including: Let Q, K, and V represent the query matrix, key matrix, and value matrix of the concatenation feature M, respectively; The attention weights in the relative position-aware Transformer model are obtained based on the query matrix Q, the key matrix K, and the relative position bias table. The attention weights are defined as follows: in, This represents the relative position offset in the relative position offset table; d represents the feature dimension of the query matrix Q and the key matrix K; The output features are obtained based on the attention weights and value matrix V. Represented as: ; in, , , , , , It is a learnable matrix.

[0020] According to a second aspect of this application, a fine-grained image classification system based on vision-language interaction is provided, the system comprising: The text description generation module is used to acquire the image to be classified and the prompt instruction corresponding to the image to be classified, and to generate a text description based on the image to be classified and the corresponding prompt instruction. The feature extraction module is used to extract visual features from the image to be classified and to extract text features from the text description. The pooling operation module is used to obtain a global visual embedding by performing a global average pooling operation on the visual features, obtain a global semantic embedding by performing a mean pooling operation on the text features, and map the global semantic embedding to the same dimensional space as the global visual embedding to obtain a global semantic prior. A visual token acquisition module is used to acquire a visual token based on the visual features. The splicing feature acquisition module is used to preset the guiding token and perform splicing processing based on the global semantic prior, visual token and guiding token to obtain splicing features; The model processing module is used to input the spliced ​​features into the relative position-aware Transformer model to obtain the model output features. The relative position-aware Transformer model has a relative position bias table. The relative position bias in the relative position bias table includes relative bias and shared bias. The relative bias comes from the spatial relationship between visual tokens, and the shared bias comes from the fusion relationship between global semantic prior, guiding token and visual token. The classification module is used to classify images based on the features output by the model to obtain the classification results.

[0021] According to a third aspect of this application, an electronic device is provided, comprising: Memory, used to store one or more computer programs; A processor, when the one or more computer programs are executed by the processor, implements the fine-grained image classification method based on vision-language interaction described in the first aspect above.

[0022] According to a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the fine-grained image classification method based on vision-language interaction described in the first aspect.

[0023] Based on any of the above aspects, the beneficial effects of the image classification method, system, electronic device, and storage medium based on vision-language interaction provided in the embodiments of this application include: It generates individualized text descriptions that correspond one-to-one with the images to be classified, breaking through the limitations of traditional class-level semantic labels and effectively capturing fine-grained semantic information such as pose, local attributes, and context, providing accurate and rich semantic priors for subsequent cross-modal alignment.

[0024] By constructing a unified shared embedding space, visual features are precisely aligned with instance-level text features, significantly improving the consistency of visual-language representations, effectively alleviating the semantic gap between modalities, and enhancing the model's generalization ability on unseen categories.

[0025] By introducing learnable guiding tokens and fusing global visual context with global semantic priors, instance-aware channel modulation vectors are generated. The channel responses of the visual tokens are dynamically reweighted, enabling refined guidance and deep fusion of visual features by semantic priors.

[0026] By adopting a relative position-aware Transformer architecture with an embedded relative position bias table, the local spatial correspondence between visual regions and semantic concepts is explicitly modeled to obtain a multimodal joint representation that combines spatial awareness and semantic consistency. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is an illustrative application scenario diagram of the fine-grained image classification method based on vision-language interaction provided in this embodiment.

[0029] Figure 2 The flowchart shows the fine-grained image classification method based on vision-language interaction provided in this embodiment.

[0030] Figure 3 This embodiment provides a detailed processing flowchart of a fine-grained image classification method based on visual-language interaction.

[0031] Figure 4 This embodiment provides a schematic diagram of the functional modules of a fine-grained image classification system based on vision-language interaction.

[0032] Figure 5 This embodiment provides a schematic diagram of the electronic device. Detailed Implementation

[0033] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this application. To better illustrate the following embodiments, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product; it is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] An exemplary illustration shows an application scenario diagram of a fine-grained image classification method based on visual-language interaction, provided in an embodiment of this application. For example... Figure 1 As shown, the application scenario includes at least a server 100 and a terminal 200 that can communicate with the server 100.

[0037] Understandably, the server 100 can be an independent electronic device or a cluster of multiple electronic devices; the terminal 200 can be a smartphone terminal, personal computer, tablet computer, vehicle terminal, etc., but is not limited to these.

[0038] In one possible implementation, server 100 and terminal 200 may respectively execute the image classification method based on visual-language interaction provided in the embodiments of this application. Alternatively, the image classification method based on visual-language interaction provided in the embodiments of this application may be partially executed in server 100 and partially executed in terminal 200.

[0039] like Figure 2As shown, this embodiment provides a fine-grained image classification method based on visual-language interaction, which can be further divided into the following steps: S100: Obtain the image to be classified and the prompt instruction corresponding to the image to be classified, and generate a text description based on the image to be classified and the corresponding prompt instruction; Specifically, the image to be classified and the corresponding prompts are input into a multimodal large language model to generate a text description; Among them, large language models refer to artificial intelligence models based on deep learning technology that have massive parameters and training data. Their core capabilities are understanding, generating, and reasoning about human language.

[0040] A multimodal large language model (MLLM) is an artificial intelligence model capable of processing and understanding multiple types of input information (i.e., multimodal) and generating corresponding outputs. It is developed based on traditional large language models and breaks through the limitation of only processing text. The multimodal large language model described in this application can process both image information and text information, and can process both image information and text information simultaneously.

[0041] Specifically, the prompt instruction can be a text instruction, used to guide the multimodal large language model to generate a compliant text description for the image to be classified. The image to be classified and the prompt instruction are input together into the multimodal large language model, which fuses visual and textual information to generate the text description. The text description can be a set of structured and diverse descriptive texts.

[0042] For example, if the image to be classified is an image containing birds, the prompt instruction could be: You are an expert in fine-grained visual descriptions. Given an image of a [domain], generate ten unique and diverse sentences describing its appearance in detail. Each sentence should focus on a different aspect and be presented on a separate line.

[0043] S200, Extract visual features from the image to be classified, and extract text features from the text description; Among them, visual features are Text features are H represents the height, and W represents the width. It represents the dimension of visual features; N represents the number of text descriptions, indicating that there are N text descriptions corresponding to one image to be classified. Dimensions representing text features; Specifically, the image to be classified is input into a visual encoder to extract the visual features, and the text description is input into a text encoder to extract the text features.

[0044] S300. The visual features are processed by global average pooling to obtain a global visual embedding, and the text features are processed by average pooling to obtain a global semantic embedding. The global semantic embedding is then mapped to the same dimensional space as the global visual embedding to obtain a global semantic prior. Specifically, after extracting the visual features, the visual features are further processed through a global average pooling operation to obtain a global visual embedding. After extracting the text features, the text features are then subjected to mean pooling to obtain a global semantic embedding. .

[0045] To unify global visual embedding and global semantic embedding The feature space, embedding global semantics Mapped to global visual embedding The same dimensional space, specifically, allows for the embedding of global semantics. Mapped to global visual embedding via a multilayer perceptron (MLP) Obtaining global semantic priors in the same dimensional space The specific formula is as follows: , in, and Both are learnable matrices and biases.

[0046] S400: Obtain a visual token based on the visual features; Specifically, the visual features After being embedded in an overlapping patch layer, they were divided into a series of visual tokens. ,in, This represents the total number of visual tokens. The overlapping patch embedding layer is a learnable 2D convolutional layer (Conv2d) that converts the visual features into patch-level tokens. The main parameters of this 2D convolutional layer can be set as follows: kernel size 7×7, stride 4, and padding 3. This configuration results in overlapping regions between adjacent convolutional receptive fields, meaning each output patch spatially shares some pixels with surrounding patches, thus achieving overlapping patch embedding.

[0047] S500: A preset guiding token is used to concatenate the global semantic prior, visual token, and guiding token to obtain the concatenated feature. Specifically, step S500 includes the following steps: S510, The visual token extracts the global visual context vector through global average pooling; S520, Preset guidance token, and concatenate the guidance token with the global semantic prior and the global visual context vector to form a fusion vector; Specifically: the bootstrap token is a learnable bootstrap token. fusion vector ,in, Represents the global visual context vector; S530, the fusion vector is mapped to a channel modulation vector via a multilayer perceptron, specifically: in, Represents the channel modulation vector. This represents the Sigmoid activation function. , , and All are learnable parameters. Channel modulation vector. This is used to adaptively reweight visual feature channels, thereby achieving instance-level semantically guided feature fusion.

[0048] S540. Update the visual token using the channel modulation vector to obtain the updated visual token. Specifically, for each visual token Its update method is as follows: , , in, This indicates the number of patches in the vertical direction (height direction). Indicates the number of patches in the horizontal direction (width direction); Updated visual tokens While preserving the original visual details, it embeds global semantic priors, thereby making the visual features more discriminative and consistent in the semantic space.

[0049] S550. The global semantic prior, the guiding token, and the updated visual token are concatenated to obtain the concatenated feature, specifically: Where M represents the splicing feature.

[0050] S600. Input the splicing features into a relative position-aware Transformer model. The relative position-aware Transformer model has a relative position bias table. The relative position bias in the relative position bias table includes relative bias and shared bias. The relative bias comes from the spatial relationship between visual tokens, and the shared bias comes from the fusion relationship between global semantic prior, guiding token and visual token. Specifically, the relative position-aware Transformer module in this application embodiment is based on relative position awareness and is used to introduce learnable spatial relationship modeling in visual-semantic alignment tasks, thereby improving the model's spatial reasoning ability and cross-modal alignment accuracy. By introducing relative position bias, it achieves directional modeling and enhanced correlation between visual and semantic features in the spatial dimension.

[0051] Specifically, the index value of the relative position offset table is obtained in the following way: A610, the relative position-aware Transformer model uses updated visual tokens on a 2D mesh. Each visual token in the array establishes coordinates. and each coordinate Flattening to form coordinate vectors ; A620, Based on coordinate vectors Calculate the relative coordinate difference between any two visual tokens, specifically as follows: A630. Shift the relative coordinate difference to the positive range, specifically as follows: in, This represents the relative offset of the relative coordinate difference in the y-direction. This indicates the relative offset of the relative coordinate difference in the x-direction; A640. Obtain the one-dimensional index value of the visual token based on the relative coordinate difference offset to the positive interval. : ; A650. Construct an index matrix for the relative position bias table based on the one-dimensional index value of the visual token, and extend the index value of the shared bias at the end of the index matrix so that the shared bias and the relative bias are uniformly mapped to the relative position bias table.

[0052] Specifically, inputting the stitched features into a relative position-aware Transformer model to obtain the model output features may include the following steps: B610. Let Q, K, and V represent the query, key, and value matrix of the concatenated features, respectively. Obtain the attention weights in the relative position-aware Transformer model based on Q, K, and the relative position bias table. Specifically, the attention weights are defined as: in, This represents the relative position offset in the relative position offset table; d represents the feature dimensions of Q and K; B620. Based on the attention weights and V, the model output features are obtained, and the model output features are expressed as follows: ; in, , , , , , It is a learnable matrix.

[0053] S700: Based on the model output features, perform image classification to obtain the classification result.

[0054] Specifically, the model outputs features Convert to ;according to Received The data is then mapped to the classification space to obtain the classification results.

[0055] in, The mapping to the classification space is specifically represented as: ; ; in, This is the classification header, where each element represents the score of a classification category; C represents the number of categories. W The weight matrix represents the classification head. The semantic space dimension for classification; Specifically, the total loss function of the Transformer model is calculated based on classification loss, cross-modal alignment loss, and debiasing loss; this loss function, combined with multi-task joint learning, achieves consistency between the visual and semantic feature spaces and improves generalization ability.

[0056] Wherein, the classification loss Used to constrain the model's ability to discriminate between different categories in a shared embedding space; cross-modal alignment loss A contrastive learning approach is employed to minimize the distance between matched visual and semantic features in the embedding space, thereby achieving instance-level semantic alignment; bias reduction loss is used. The mean and variance of the distribution used to match the predicted scores of seen and unseen categories are used to mitigate modality bias.

[0057] Classification loss The definition is as follows: in, Represents a sample image. Represents the set of visible categories; Indicates category label, Represents sample image Predicted category labels.

[0058] Cross-modal alignment loss The definition is as follows: in, This represents the cosine similarity, where T is a learnable temperature parameter used to control the smoothness of the distribution. Indicates the first Global visual embeddings corresponding to each sample; Indicates the first The global semantic prior for each sample; B represents the batch size of the sample batch. This cross-modal alignment loss... The optimization strategy based on contrastive learning ensures consistency between visual and semantic features in a shared embedding space through instance-level matching constraints. Specifically, it utilizes the InfoNCE loss function to calculate the semantic similarity between an image and its corresponding instance, bridging the feature distance between matched samples while separating unmatched samples, thus achieving accurate modality alignment. This process is performed on an instance-by-instance basis, avoiding semantic conflicts between similar samples and effectively improving the consistency and discriminative power of visual and semantic features.

[0059] Dispersion loss The definition is as follows: in, and Let represent the mean and variance of the predicted scores for the observed categories, respectively. and These represent the mean and variance of the predicted scores for unseen categories, respectively. Seen categories refer to those categories the model has encountered during training, while unseen categories are those the model will classify during testing. For example, if the training dataset contains 200 categories, with 150 categories used for training and 50 for testing, then the number of seen categories is 150, and the number of unseen categories is 50.

[0060] The predicted score is calculated using the following formula: in, Represents cosine similarity. Represents sample image Corresponding global semantic prior; γ It's a hyperparameter; It is a collection of unseen classes; For indicator functions, when category Its value is 1 if it belongs to the visible category, and 0 otherwise.

[0061] Total loss function It can be represented as: middle, and The weighting coefficients are used to control the impact of each loss term on the overall optimization objective. By jointly optimizing classification accuracy, modal alignment, and score distribution balance, the optimization module of this invention effectively narrows the gap between visual and semantic modalities, reduces overfitting to seen categories, and improves the model's recognition performance and generalization ability on unseen categories.

[0062] Specifically, the image classification method based on vision-language interaction in this application embodiment undergoes the following specific processing steps: Figure 3 As shown, it takes instance-level semantic generation as the lead, automatically generates individualized semantic descriptions corresponding to the images to be classified, and provides high-precision text semantic supervision; through instance-level cross-modal alignment, it achieves accurate matching of visual features and semantic features in the shared embedding space; then, by using a channel interaction method guided by learnable tokens, semantic priors are injected into the visual channel to achieve global semantic modulation and adaptive fusion; finally, it includes a relative position-aware Transformer module to model the spatial dependency between vision and semantics, thereby obtaining a multimodal representation with spatial awareness and semantic consistency.

[0063] like Figure 4As shown in the embodiments of this application, a fine-grained image classification system based on visual-language interaction is also provided, the system comprising: The text description generation module 811 is used to acquire the image to be classified and the prompt instruction corresponding to the image to be classified, and generate a text description based on the image to be classified and the corresponding prompt instruction. In this embodiment, the text description generation module 811 can be used to perform... Figure 2 For a detailed description of the text description generation module 811, please refer to the description of step S100 shown.

[0064] The feature extraction module 812 is used to extract visual features from the image to be classified and to extract text features from the text description. In this embodiment, the feature extraction module 812 can be used to perform... Figure 2 For a detailed description of the feature extraction module 812, please refer to the description of step S200 shown.

[0065] Pooling operation module 813 is used to obtain global visual embedding by global average pooling operation on the visual features, obtain global semantic embedding by average pooling operation on the text features, and map the global semantic embedding to the same dimensional space as the global visual embedding to obtain global semantic prior. In this embodiment, the pooling operation module 813 can be used to perform... Figure 2 For a detailed description of the pooling operation module 813, please refer to the description of step S300 shown.

[0066] Visual token acquisition module 814 is used to acquire a visual token based on the visual features; In this embodiment, the visual token acquisition module 814 can be used to perform... Figure 2 For a detailed description of the visual token acquisition module 814, please refer to the description of step S400 shown.

[0067] The splicing feature acquisition module 815 is used to preset the guiding token and perform splicing processing based on the global semantic prior, visual token and guiding token to obtain splicing features; In this embodiment, the splicing feature acquisition module 815 can be used to perform... Figure 2 For a detailed description of the splicing feature acquisition module 815 shown in step S500, please refer to the description of step S500.

[0068] The model processing module 816 is used to input the spliced ​​features into the relative position-aware Transformer model to obtain the model output features. The relative position-aware Transformer model has a relative position bias table. The relative position bias in the relative position bias table includes relative bias and shared bias. The relative bias comes from the spatial relationship between visual tokens, and the shared bias comes from the fusion relationship between global semantic prior, guiding token and visual token. In this embodiment, the model processing module 816 can be used to execute... Figure 2 For a detailed description of the model processing module 816, please refer to the description of step S600 shown.

[0069] The classification module 817 is used to classify images based on the features output by the model to obtain classification results.

[0070] In this embodiment, the classification module 817 can be used to perform... Figure 2 For a detailed description of the classification module 817, please refer to the description of step S700 shown.

[0071] This application also provides an electronic device, the structure of which is as follows: Figure 5 As shown, the electronic device includes a memory 911, a processor 912, a communication module 913, and an input / output interface 914, etc. Optionally, the memory 911, the processor 912, the communication module 913, and the input / output interface 914 can be connected and communicate with each other through a bus 915.

[0072] The memory 911 is used to store one or more computer programs and to transfer the code of the computer programs to the processor 912; when the one or more computer programs are executed by the processor 912, a fine-grained image classification method based on visual-language interaction is implemented in this embodiment of the application.

[0073] Optionally, the electronic device can be connected to a network via the communication module 913 to communicate with other devices, such as terminals or servers, to achieve data interaction. The electronic device can be various forms of digital computers, exemplarily such as desktop computers, servers, workbenches, mainframes, or other types of computers. The electronic device can also be various forms of mobile terminals, exemplarily such as smartphones, tablets, wearable devices (such as helmets, glasses, watches, etc.), and other similar mobile terminals.

[0074] Optionally, the electronic device can connect to required input / output devices, such as a keyboard or display device, via the input / output interface 914. The electronic device itself may have a display device, and other display devices can also be connected externally via the input / output interface 914. Optionally, a storage device, such as a hard disk, can also be connected via the input / output interface 914 to store data from the electronic device, read data from the storage device, or store data from the storage device in the memory 911. It is understood that the input / output interface 914 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 914 can be a component of the electronic device or an external device connected to the electronic device when needed.

[0075] Optionally, the memory 911 may be a volatile memory and / or a non-volatile memory. The volatile memory may be a random access memory, etc., and the non-volatile memory may be a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, or a flash memory, etc.

[0076] Optionally, the computer program stored in the memory 911 can be divided into one or more modules, which are stored in the memory 911 and executed by the processor 912 to perform the method provided in this embodiment. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device.

[0077] Optionally, the processor 912 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 912 include, but are not limited to, a central processing unit, a graphics processing unit, a digital signal processor, various special-purpose artificial intelligence computing chips, various processors running machine learning model algorithms, and can also be any suitable controller, microcontroller, processor, etc. The processor 912 executes the various methods and processes of this embodiment, exemplarily, such as a fine-grained image classification method based on vision-language interaction according to an embodiment of this application.

[0078] Optionally, the bus 915 may include a path for transmitting information. Depending on its function, the bus 915 may be classified as an address bus, a data bus, a control bus, etc.

[0079] In an optional implementation, this application embodiment also provides a computer storage medium storing a computer program thereon, which, when executed by a computer, enables the computer to perform the methods described in the above-described method embodiments. Part or all of the computer program may be loaded and / or installed on the memory 911 of an electronic device. When the computer program is executed by the processor 912, it can perform one or more steps of a fine-grained image classification method based on vision-language interaction according to an embodiment of this application.

[0080] Optionally, the computer-readable storage medium may be a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc.

[0081] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A method for image classification based on visual-linguistic interaction fine-grained, characterized in that, The method comprises: acquiring an image to be classified and a prompt instruction corresponding to the image to be classified, and generating a text description according to the image to be classified and the corresponding prompt instruction; extracting visual features in the image to be classified and extracting text features in the text description; obtaining global visual embedding by performing global average pooling operation on the visual features, obtaining global semantic embedding by performing mean pooling operation on the text features, and mapping the global semantic embedding to the same dimension space as the global visual embedding to obtain global semantic prior; acquiring visual tokens according to the visual features; presetting a guide token, and performing splicing processing on the global semantic prior, the visual tokens and the guide token to obtain splicing features; inputting the splicing features into a relative position perception Transformer model to obtain model output features, wherein a relative position bias table is constructed in the relative position perception Transformer model, and the relative position bias in the relative position bias table comprises a relative bias and a shared bias, the relative bias comes from the spatial relationship between the visual tokens, and the shared bias comes from the fusion relationship between the global semantic prior, the guide token and the visual tokens; performing image classification according to the model output features to obtain a classification result.

2. The method of claim 1, wherein, The method comprises:

3. The method of claim 1, wherein, acquiring an image to be classified and a prompt instruction corresponding to the image to be classified, and generating a text description according to the image to be classified and the corresponding prompt instruction; extracting visual features in the image to be classified and extracting text features in the text description, comprising: inputting the image to be classified into a visual encoder to extract the visual features; 4. The method of claim 1, wherein, inputting the text description into a text encoder to extract the text features. acquiring visual tokens according to the visual features, comprising:

5. The method of claim 1, wherein, the visual features are divided into a series of visual tokens through an overlapping patch embedding layer, wherein the overlapping patch embedding layer is a learnable two-dimensional convolution layer, and there is an overlapping area between adjacent convolution receptive fields in the two-dimensional convolution layer. The method comprises: the visual tokens are extracted into global visual context vectors through global average pooling; the guide token is spliced with the global semantic prior and the global visual context vectors to form a fusion vector; the fusion vector is mapped into a channel modulation vector through a multilayer perception machine; the visual tokens are updated by using the channel modulation vector to obtain updated visual tokens; 6. The method of claim 5, wherein, the global semantic prior, the guide token and the updated visual tokens are spliced to obtain splicing features. in, Indicates the first j A visual token This indicates the total number of visual tokens.

7. The method of claim 6, wherein, For each visual token, the updating method is: the index value of the relative position bias table is obtained in the following way: the relative position perception Transformer model establishes a coordinate for each visual token in the updated visual tokens on a two-dimensional grid, and each coordinate is flattened into a coordinate vector; the relative coordinate difference between any two visual tokens is calculated according to the coordinate vector; the relative coordinate difference is offset to the positive number interval; A one-dimensional index value of a visual token is obtained from the relative coordinate difference offset to the positive number interval : ； An index matrix of the relative position bias table is constructed according to a one-dimensional index value of the visual token, and an index value sharing a bias is extended at the end of the index matrix, so that the shared bias is uniformly mapped to the relative position bias table with the relative bias.

8. The method according to any one of claims 1 to 7, characterized in that, The spliced feature is input into a relative position-aware Transformer model to obtain model output features, including: Let Q, K, and V represent the query matrix, key matrix, and value matrix of the spliced feature M, respectively. According to the query matrix Q, the key matrix K, and the relative position bias table, the attention weight in the relative position-aware Transformer model is obtained, and the attention weight is defined as: wherein, represents a relative position bias in the relative position bias table; d represents the feature dimension of the query matrix Q and the key matrix K; An output feature is obtained according to the attention weight and the value matrix V, and the output feature is represented as: is represented as: ； wherein , , , , , is a learnable matrix.

9. A visual-linguistic interaction based fine-grained image classification system, characterized in that, The system comprises: A text description generation module is configured to obtain a to-be-classified image and a prompt instruction corresponding to the to-be-classified image, and generate a text description according to the to-be-classified image and the corresponding prompt instruction. A feature extraction module is configured to extract visual features in the to-be-classified image and extract text features in the text description. A pooling operation module is configured to obtain a global visual embedding by performing a global average pooling operation on the visual features, obtain a global semantic embedding by performing a mean pooling operation on the text features, and map the global semantic embedding to a same dimensional space as the global visual embedding to obtain a global semantic prior. A visual token acquisition module is configured to acquire visual tokens according to the visual features. A spliced feature acquisition module is configured to preset a guide token, and perform splicing processing on the global semantic prior, the visual tokens, and the guide token to obtain spliced features. A model processing module is configured to input the spliced features into a relative position-aware Transformer model to obtain model output features, wherein the relative position-aware Transformer model is constructed with a relative position bias table, and the relative position bias in the relative position bias table includes a relative bias and a shared bias, the relative bias comes from a spatial relationship between visual tokens, and the shared bias comes from a fusion relationship between the global semantic prior, the guide token, and the visual tokens. A classification module is configured to perform image classification according to the model output features to obtain a classification result.

10. An electronic device, comprising: The system comprises: A memory is configured to store one or more computer programs. A processor is configured to implement the image classification method based on visual-linguistic interaction fine granularity according to any one of claims 1-8 when the one or more computer programs are executed by the processor.

Images