An image classification method and device based on cross-modal semantic enhancement

Abstract

The application discloses an image classification method and device based on cross-modal semantic enhancement. First, data remodeling operation, basic convolutional neural network processing, upsampling operation and global average pooling operation are performed on an input image to obtain image features containing significant information. Second, an initialized text prompt is used to construct a multi-task cross-modal source prompt set. Finally, the class probability of a predicted image is constructed according to the image features and the multi-task cross-modal source prompt set. The classification of the input image is completed through the class probability, which can effectively improve the classification accuracy of the few-shot or zero-shot image classification task and make the overall classification effect more ideal.

Description

Technical Field

[0001] This application relates to the field of image classification technology, and more specifically, to an image classification method and apparatus based on cross-modal semantic enhancement. Background Technology

[0002] Image classification aims to enable computers to automatically identify and classify objects or scenes in images. It has wide applications in many practical fields, such as image retrieval, autonomous driving, and medical image analysis. Cross-modal refers to the relationships and interactions between multiple different modalities, typically referring to different data types or representations, such as images, text, audio, and video.

[0003] Current advanced image-text cross-modal classification methods employ fixed text cues to combine images with their text descriptions, while excluding mismatches in the feature space. Through large-scale pre-training, the model can learn different visual concepts and can be easily transferred to any downstream task with the cues.

[0004] Although this method achieves image classification tasks with few or even zero samples through a cross-modal approach involving text and images, the text descriptions are not combined with the features in the images, resulting in unsatisfactory overall classification performance. Therefore, how to better combine text descriptions with the features in the images to improve the overall classification performance has become one of the research directions for those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of existing technologies and provide an image classification method and apparatus based on cross-modal semantic enhancement, which classifies input images by class probability and can effectively improve the classification accuracy of image classification tasks with few or zero samples.

[0006] The objective of this application is achieved through the following technical solution:

[0007] Firstly, this application proposes an image classification method based on cross-modal semantic enhancement, the method comprising:

[0008] The input image is subjected to data reshaping, basic convolutional neural network processing, upsampling, and global average pooling to obtain image features containing significant information.

[0009] Build a multi-task, cross-modal source hint set using initialized text hints;

[0010] Based on image features and a multi-task cross-modal source cue set, the class probabilities of the predicted image are constructed, and the input image is classified using the class probabilities.

[0011] In one possible implementation, the steps of performing data reshaping, basic convolutional neural network processing, upsampling, and global average pooling on the input image to obtain image features containing significant information include:

[0012] The input image is reshaped to obtain an Img image;

[0013] The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features.

[0014] The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by inputting the first feature into the first convolutional layer and the second convolutional layer.

[0015] The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer.

[0016] The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer.

[0017] The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer.

[0018] The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer.

[0019] The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer.

[0020] The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer.

[0021] Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

[0022] In one possible implementation, the step of constructing a multi-task cross-modal source cue set using initialized text prompts includes:

[0023] The text prompts are encoded using a text encoder to obtain text encoding features;

[0024] Image encoding features are obtained by encoding training images in a classification task using an image encoder;

[0025] After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process.

[0026] Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

[0027] In one possible implementation, the step of constructing the class probability of the predicted image based on image features and a multi-task cross-modal source cue set, and then classifying the input image using the class probability, includes:

[0028] The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set;

[0029] The second training cue is obtained by summing the first training cue with the image features;

[0030] After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information.

[0031] The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

[0032] In one possible implementation, the first training cue P s The calculation formula is:

[0033]

[0034] Where P i For multi-task cross-modal source hint sets, Softmax() is the softmax function, α is a hyperparameter, and q task For task-level queries, q ins For instance-level queries, k c For the key set, k i This is a key tip.

[0035] In one possible implementation, the category probability P(y=i|I infer-f The formula for calculating ) is: Where sim() is the cosine similarity calculation, I infer-fLet T be the image feature, τ be the learnable parameter, g represent the number of classes of the predicted sample, and T be the image feature. i Encode information for text.

[0036] Secondly, this application proposes an image classification device based on cross-modal semantic enhancement, the device comprising:

[0037] The image feature extraction module is used to perform data reshaping operations, basic convolutional neural network processing, upsampling operations, and global average pooling operations on the input image to obtain image features containing significant information.

[0038] The cue set building module is used to build multi-task, cross-modal source cue sets using initialized text cue suggestions;

[0039] The classification module is used to construct the class probability of the predicted image based on image features and a multi-task cross-modal source cue set, and to classify the input image using the class probability.

[0040] In one possible implementation, the image feature extraction module is specifically used for:

[0041] The input image is reshaped to obtain an Img image;

[0042] The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features.

[0043] The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by inputting the first feature into the first convolutional layer and the second convolutional layer.

[0044] The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer.

[0045] The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer.

[0046] The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer.

[0047] The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer.

[0048] The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer.

[0049] The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer.

[0050] Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

[0051] In one possible implementation, the prompt set construction module is specifically used for:

[0052] The text prompts are encoded using a text encoder to obtain text encoding features;

[0053] Image encoding features are obtained by encoding training images in a classification task using an image encoder;

[0054] After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process.

[0055] Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

[0056] In one possible implementation, the classification module is specifically used for:

[0057] The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set;

[0058] The second training cue is obtained by summing the first training cue with the image features;

[0059] After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information.

[0060] The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

[0061] The main solution and its various further alternatives described above can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application; furthermore, the (non-conflicting alternatives) can also be freely combined with each other and with other alternatives. Those skilled in the art, after understanding the solution of this application, will realize from the prior art and common general knowledge that there are many combinations, all of which are technical solutions to be protected by this application, and will not be exhaustively listed here.

[0062] This application discloses an image classification method and apparatus based on cross-modal semantic enhancement. First, the input image is subjected to data reshaping, basic convolutional neural network processing, upsampling, and global average pooling to obtain image features containing significant information. Second, an initial text prompt is used to construct a multi-task cross-modal source prompt set. Finally, the class probability of the predicted image is constructed based on the image features and the multi-task cross-modal source prompt set. The classification of the input image is completed through the class probability, which can effectively improve the classification accuracy of image classification tasks with few or zero samples, making the overall classification effect more ideal. Attached Figure Description

[0063] Figure 1 The diagram shows a flowchart of an image classification method based on cross-modal semantic enhancement proposed in an embodiment of this application.

[0064] Figure 2 A schematic diagram illustrating the generation of image features proposed in an embodiment of this application is shown.

[0065] Figure 3 This illustration shows a schematic diagram of the construction of a multi-task cross-modal source cue set proposed in an embodiment of this application.

[0066] Figure 4 A schematic diagram illustrating the classification of input images according to an embodiment of this application is shown. Detailed Implementation

[0067] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0068] Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0069] In existing technologies, cross-modal image-text classification methods use fixed text cues to combine images and their text descriptions, while excluding those that do not match in the feature space. Through large-scale pre-training, the model can learn different visual concepts and can be easily transferred to any downstream task using the cues. However, the text descriptions are not combined with the features in the image, resulting in unsatisfactory overall classification performance.

[0070] To address the aforementioned issues, this application proposes an image classification method and apparatus based on cross-modal semantic enhancement. By classifying input images through category probabilities, it can effectively improve the classification accuracy of image classification tasks with few or zero samples, resulting in a more ideal overall classification effect. The following is a detailed description of this method.

[0071] Please refer to Figure 1 , Figure 1 The diagram illustrates a flowchart of an image classification method based on cross-modal semantic enhancement proposed in this application. The method includes the following steps:

[0072] Step S1: Perform data reshaping, basic convolutional neural network processing, upsampling, and global average pooling operations on the input image to obtain image features containing significant information.

[0073] The process of obtaining image features containing significant information from the input image also includes the following sub-steps;

[0074] The input image is reshaped to obtain an Img image;

[0075] The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features.

[0076] The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by inputting the first feature into the first convolutional layer and the second convolutional layer.

[0077] The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer.

[0078] The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer.

[0079] The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer.

[0080] The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer.

[0081] The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer.

[0082] The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer.

[0083] Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

[0084] First, the input image is processed using RGB three-channel input. A reshaping operation is then performed on the input image to obtain an Img image. The Img image has a size of 224 pixels * 224 pixels. Please refer to... Figure 2 , Figure 2 This diagram illustrates the generation of image features according to an embodiment of this application. After passing an Img image through a ResNet-18 convolutional neural network, four feature maps are obtained: feature R1, feature R2, feature R3, and feature R4. Feature R1 has a dimension of 7*7*512, feature R2 has a dimension of 14*14*256, feature R3 has a dimension of 28*28*128, and feature R4 has a dimension of 56*56*64. After passing the Img image through a Swin-B convolutional neural network, four feature maps are obtained: feature S1, feature S2, feature S3, and feature S4. Feature S1 has a dimension of 14*14*512, feature S2 has a dimension of 14*14*512, feature S3 has a dimension of 28*28*256, and feature S4 has a dimension of 56*56*128.

[0085] Next, we set up the first convolutional layer, Conv_2D. a The second convolutional layer Conv_2D b The third convolutional layer Conv_2D cThese three 2D convolutional layers have the following parameters: First convolutional layer Conv_2D a The convolutional kernel has a dimension of 1*1, a stride of 1, and a padding dimension of 0. The second convolutional layer is Conv_2D. b The convolutional kernel has a dimension of 3*3, a stride of 1, and a padding dimension of 2. The third convolutional layer is Conv_2D. c The convolution kernel has a dimension of 3*3, a stride of 1, and a padding dimension of 1.

[0086] The fifth feature S1 is sequentially input into the second convolutional layer Conv_2D. b and the first convolutional layer Conv_2D a The first generated feature S1 is obtained n (S1 n =Conv_2D a (Conv_2D b (S1))), the first generated feature S1 n After performing upsampling and aligning with the sixth feature S2 dimension, feature fusion is then performed with the sixth feature S2 to obtain the first fused feature FS1. unsample() indicates that the first generated feature S1 is sampled. n Perform an upsampling operation. (This indicates the Concat feature fusion operation).

[0087] The first fusion feature FS1 is input into the third convolutional layer Conv_2D. c The second generated feature S2 is obtained. n (S2 n =Conv_2D c (FS1)4, Conv_2D c ()4 indicates that it is passed through 4 layers of the third convolutional layer Conv_2D c The second generated feature S2) n After performing upsampling and aligning with the seventh feature S3 dimension, feature fusion is then performed with the seventh feature S3 to obtain the second fused feature FS2. This indicates the second generated feature S2 n Perform an upsampling operation. (This indicates the Concat feature fusion operation).

[0088] The second fused feature FS2 is then input into the third convolutional layer Conv_2D. c The third generated feature S3 is obtained. n (S3 n =Conv_2D c (FS2)4), respectively input the eighth feature S4 into the third convolutional layer Conv_2D cThe fourth generated feature S4 is obtained. n (S4 n =Conv_2D c (S4)) and the fourth feature R4 are input into the third convolutional layer Conv_2D c The fifth generated feature R4 is obtained. n (R4 n =Conv_2D c (R4)) Then the fourth generated feature S4 n and the fifth generated feature R4 n The third fused feature is obtained by performing feature fusion.

[0089] The third generated feature S3 n After performing upsampling and aligning with the third fusion feature FS3 dimension, the fourth fusion feature is obtained by feature fusion with the third fusion feature FS3.

[0090] The fourth fusion feature FS4 is input into the third convolutional layer Conv_2D. c The sixth generated feature S5 is obtained. n (S5 n =Conv_2D c (FS4)4), for the sixth generated feature S5 n After performing upsampling and aligning with the third feature R3 dimension, the fifth fused feature is obtained by feature fusion with the third feature R3.

[0091] The fifth fusion feature FS5 is input into the third convolutional layer Conv_2D. c The seventh generated feature S6 is obtained. n (S6 n =Conv_2D c (FS5)4), for the seventh generated feature S6 n After performing upsampling and aligning with the second feature R2 dimension, the sixth fused feature is obtained by feature fusion with the second feature R2.

[0092] The sixth fusion feature FS6 is input into the third convolutional layer Conv_2D. c The eighth generated feature S7 is obtained. n (S7 n =Conv_2D c (FS6)4), the eighth generated feature S7 n After performing upsampling and aligning with the first feature R1 dimension, the seventh fused feature is obtained by feature fusion with the first feature R1.

[0093] The seventh fusion feature FS7 is input into the third convolutional layer Conv_2D. c The ninth generated feature S8 is obtained. n (S8 n =Conv_2D c (FS7)4), for the ninth generated feature S8 n Perform global average pooling to obtain image features (GPs) containing significant information. mean (S8 n ), where GP mean () represents the global average pooling operation.

[0094] Step S2: Construct a multi-task cross-modal source cue set using the initialized text hints.

[0095] The sub-steps for constructing a multi-task cross-modal source cue set include:

[0096] The text prompts are encoded using a text encoder to obtain text encoding features;

[0097] Image encoding features are obtained by encoding training images in a classification task using an image encoder;

[0098] After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process.

[0099] Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

[0100] Please refer to Figure 3 , Figure 3 This diagram illustrates the construction of a multi-task cross-modal source cue set proposed in an embodiment of this application. First, a text cue P of length L is randomly initialized. init :P init =Random L Random L This represents a text prompt of length L that is randomly initialized, and then encoded using a text encoder to obtain the text encoding feature EP. init EP init =Text_Encoder(P init Text_Encoder is a text encoder that uses an image encoder to encode training images in a classification task to obtain image encoding features. The classification task m... ci (ci = 1, 2, ..., N, where N is the total number of tasks), training image MImg mi (mi = 1, 2, ..., N) img , where N img (Total number of training images), image encoding features EImi EI mi =Image_Encoder(MImg) mi ), mi = 1, 2, ..., N img Image_Encoder is an image encoder. It encodes text features EP. init Image coding features EI mi After summing, the cross-modal model M is obtained through a classification training process. ci M ci =Train(EP) init +EI mi ), ci = 1, 2, ..., N, Train represents the classification training process. The cross-modal model M obtained from the training... ci In this process, the text encoding features obtained from training are named source cue SP. ci (ci = 1, 2, ..., N), the source hint SP ci The merging process yields a multi-task cross-modal source cue set P:

[0101] Step S3: Construct the category probability of the predicted image based on image features and multi-task cross-modal source cue set, and complete the classification of the input image through the category probability.

[0102] The sub-steps of step S3 are as follows:

[0103] The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set;

[0104] The second training cue is obtained by summing the first training cue with the image features;

[0105] After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information.

[0106] The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

[0107] Figure 4 This illustration shows a classification diagram of the input image proposed in an embodiment of this application. First, a text prompt P of length L is randomly initialized. init_1 :P init_1 =Random L There are two learnable parameters in the multi-task cross-modal source cue set: the key set Kc and the key cue Ki. The task-level query q... task It is defined as a learnable vector that can be queried for information related to a specific target task. d represents the dimension of the encoding. An instance-level query is defined as a learnable vector that learns target cues by focusing on highly relevant source cues; this vector is essentially P. init_1 , The first training cue Ps is obtained by using the key set and key cues in the multi-task cross-modal source cue set, hyperparameters, task-level queries, and instance-level queries: First training cue P s The calculation formula is:

[0108]

[0109] Where P i For multi-task cross-modal source hint sets, Softmax() is the softmax function, α is a hyperparameter, and q task For task-level queries, q ins For instance-level queries, k c For the key set, k i This is a key point. α is a hyperparameter that takes values ​​between 0 and 1. Indicates q task Find the transpose operation. Indicates q ins Find the transpose operation.

[0110] The second training cue P is obtained by summing the first training cue with the image features. e In the second training prompt P e After adding category information [CLS]j to the end, the text is fed into a text encoder for encoding to obtain text encoded information T. i :T i =T_Encode(P e [CLS] j T_Encode represents the text encoder, j = 1, 2, ..., t, where t is the total number of image categories.

[0111] After obtaining the text encoding information, construct the predicted image class probability P(y=i|I infer-f The calculation formula is as follows: Where sim() is the cosine similarity calculation, I infer-f Let T be the image feature, τ be the learnable parameter, g represent the number of classes of the predicted sample, and T be the image feature. i Encode information for text.

[0112] Therefore, the image classification method based on cross-modal semantic enhancement disclosed in this application classifies the input image by class probability, which can effectively improve the classification accuracy of image classification tasks with few or zero samples, making the overall classification effect more ideal.

[0113] The following provides a possible implementation of an image classification device based on cross-modal semantic enhancement, which performs the various execution steps and corresponding technical effects of the image classification method based on cross-modal semantic enhancement shown in the above embodiments and possible implementations. The device includes:

[0114] The image feature extraction module is used to perform data reshaping operations, basic convolutional neural network processing, upsampling operations, and global average pooling operations on the input image to obtain image features containing significant information.

[0115] The cue set building module is used to build multi-task, cross-modal source cue sets using initialized text cue suggestions;

[0116] The classification module is used to construct the class probability of the predicted image based on image features and a multi-task cross-modal source cue set, and to classify the input image using the class probability.

[0117] In one possible implementation, the image feature extraction module is specifically used for:

[0118] The input image is reshaped to obtain an Img image;

[0119] The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features.

[0120] The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by inputting the first feature into the first convolutional layer and the second convolutional layer.

[0121] The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer.

[0122] The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer.

[0123] The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer.

[0124] The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer.

[0125] The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer.

[0126] The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer.

[0127] Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

[0128] In one possible implementation, the prompt set construction module is specifically used for:

[0129] The text prompts are encoded using a text encoder to obtain text encoding features;

[0130] Image encoding features are obtained by encoding training images in a classification task using an image encoder;

[0131] After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process.

[0132] Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

[0133] In one possible implementation, the classification module is specifically used for:

[0134] The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set;

[0135] The second training cue is obtained by summing the first training cue with the image features;

[0136] After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information.

[0137] The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

[0138] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image classification method based on cross-modal semantic enhancement, characterized in that, The method includes: The input image is subjected to data reshaping, basic convolutional neural network processing, upsampling, and global average pooling to obtain image features containing significant information. Build a multi-task, cross-modal source hint set using initialized text hints; The steps of constructing the class probabilities of the predicted image based on image features and a multi-task cross-modal source cue set, and then classifying the input image using these class probabilities, include: The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set; The second training cue is obtained by summing the first training cue with the image features; After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information. The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

2. The image classification method as described in claim 1, characterized in that, The steps for obtaining image features containing significant information by performing data reshaping operations, basic convolutional neural network processing, upsampling operations, and global average pooling operations on the input image include: The input image is reshaped to obtain an Img image; The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features. The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by sequentially inputting the fifth feature into the set second convolutional layer and the first convolutional layer. The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer. The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer. The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer. The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer. The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer. The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer. Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

3. The image classification method as described in claim 1, characterized in that, The steps for constructing a multi-task, cross-modal source cue set using initialized text hints include: The text prompts are encoded using a text encoder to obtain text encoding features; Image encoding features are obtained by encoding training images in a classification task using an image encoder; After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process. Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

4. The image classification method as described in claim 1, characterized in that, First training tip The calculation formula is: in For multi-task cross-modal source hint set, For the softmax function, For hyperparameters, For task-level queries, For instance-level queries, For the key set, This is a key tip.

5. The image classification method as described in claim 1, characterized in that, Category Probability The calculation formula is: ,in For cosine similarity calculation, For image features, Here, g is a learnable parameter, representing the number of classes in the predicted sample. Encode information for text.

6. An image classification device based on cross-modal semantic enhancement, characterized in that, The device includes: The image feature extraction module is used to perform data reshaping operations, basic convolutional neural network processing, upsampling operations, and global average pooling operations on the input image to obtain image features containing significant information. The cue set building module is used to build a multi-task, cross-modal source cue set using initialized text cue suggestions; The classification module is used to construct the class probability of the predicted image based on image features and a multi-task cross-modal source cue set, and to classify the input image using the class probability. The classification module is specifically used for: The first training cue is obtained by using the key set and key cue, hyperparameters, task-level queries, and instance-level queries in the multi-task cross-modal source cue set; The second training cue is obtained by summing the first training cue with the image features; After adding category information to the second training prompt, the text is fed into the text encoder for encoding to obtain the text encoded information. The predicted image class probability is constructed by using text-encoded information, and the input image is classified using the class probability.

7. The image classification device as described in claim 6, characterized in that, The image feature extraction module is specifically used for: The input image is reshaped to obtain an Img image; The Img image is processed using the basic convolutional neural network ResNet-18 to obtain the corresponding first, second, third, and fourth features, and the Img image is processed using the basic convolutional neural network Swin-B to obtain the fifth, sixth, seventh, and eighth features. The first generated feature is obtained by upsampling the first generated feature, aligning it with the sixth feature dimension, and fusing the features. The first generated feature is obtained by inputting the first feature into the first convolutional layer and the second convolutional layer. The second generated feature is upsampled, aligned with the seventh feature dimension, and fused to obtain the second fused feature. The second generated feature is obtained by inputting the first fused feature into the third convolutional layer. The fourth and fifth generated features are fused to obtain the third fused feature. The fourth generated feature is obtained by inputting the eighth feature into the third convolutional layer, and the fifth generated feature is obtained by inputting the fourth feature into the third convolutional layer. The third generated feature is upsampled, aligned with the dimension of the third fused feature, and fused to obtain the fourth fused feature. The third generated feature is obtained by inputting the second fused feature into the third convolutional layer. The sixth generated feature is upsampled, aligned with the third feature dimension, and fused to obtain the fifth fused feature. The sixth generated feature is obtained by inputting the fourth fused feature into the third convolutional layer. The seventh generated feature is upsampled, aligned with the second feature dimension, and fused to obtain the sixth fused feature. The seventh generated feature is obtained by inputting the fifth fused feature into the third convolutional layer. The seventh fused feature is obtained by upsampling the eighth generated feature, aligning it with the first feature dimension, and fusing the features. The eighth generated feature is obtained by inputting the sixth fused feature into the third convolutional layer. Global average pooling is performed on the ninth generated feature to obtain image features containing significant information. The ninth generated feature is obtained by inputting the seventh fused feature into the third convolutional layer.

8. The image classification device as described in claim 6, characterized in that, The prompt set building module is specifically used for: The text prompts are encoded using a text encoder to obtain text encoding features; Image encoding features are obtained by encoding training images in a classification task using an image encoder; After summing the text-encoded features and image-encoded features, a cross-modal model is obtained through a classification training process. Multiple training text encoding features are extracted from the cross-modal model and merged to obtain a multi-task cross-modal source cue set.

Images