Continuous small sample image classification method based on cross-modal alignment and prototype diversification

Abstract

The present invention discloses a continuous small sample image classification method based on cross-modal alignment and prototype diversification, the method comprising: firstly, adding visual soft prompts to the input image to extract image features, and inputting the image features into a feature adapter to obtain optimized image features, and adding text soft prompts to the category label to extract preliminary category text prototypes; secondly, inputting the optimized image features into a prototype biaser, adjusting the category text prototype, so as to obtain a category text prototype that incorporates visual information; finally, calculating the similarity between the above-mentioned category text prototype and the image features, and predicting the input image as the category with the maximum similarity. The present invention can capture input-specific visual information to perform text prototype correction, which is conducive to improving the classification accuracy of the input image; the implementation method of the present invention is simple and flexible, significantly improves the prediction accuracy of new categories, and can effectively alleviate the catastrophic forgetting phenomenon of the model in old categories.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a continuous small sample image classification method based on cross-modal alignment and prototype diversification. Background Art

[0002] In recent years, deep neural networks with carefully designed architectures have made significant progress in a variety of visual recognition tasks. One of the key factors for its success is the use of pre-collected, static, large-scale labeled datasets for model training. However, in practical applications, new categories usually appear over time, and the training data is relatively small, considering factors such as privacy or annotation costs. For example, in a face recognition system, the system needs to continuously add new users, while only allowing one or a few pictures to be uploaded to identify the newly added face information. In response to the above challenges, researchers proposed the small sample class increase learning problem, which imitates the human learning process and aims to continuously learn new categories from limited labeled samples without excessively forgetting the old category knowledge learned previously.

[0003] Small-sample class augmentation learning usually consists of a base task and multiple consecutive incremental tasks. In the base task, each category has a large number of labeled training samples to build the initial model, while in the incremental tasks, each class has only a few labeled samples to continuously train the model. When learning each task, the model can only be trained using the training data in the current task, and after learning each task, the model needs to be tested on all seen categories. In the process of small-sample class augmentation learning, there are two key challenges that need to be addressed: catastrophic forgetting and model overfitting. Among them, catastrophic forgetting is used to describe the phenomenon that the model often loses the old knowledge that has been learned when adapting to new categories in subsequent incremental tasks; while model overfitting is mainly caused by too little training data in incremental tasks, which in turn exacerbates the occurrence of catastrophic forgetting.

[0004] In response to the above challenges, existing small-sample class augmentation learning methods mainly focus on first learning generalized features from basic tasks, and then optimizing the feature space in incremental tasks using strategies such as weight constraints, knowledge distillation, and feature pre-assignment, thereby alleviating the model overfitting phenomenon and achieving the goal of balancing the model stability-plasticity dilemma. This idea of ​​learning transferable features is similar to the idea of ​​contrastive language and vision pre-training large models, which mainly constructs a network structure with generalization ability through contrastive learning from massive text-image data pairs. Contrasting language and vision pre-training large models have made significant progress in a variety of downstream visual problems such as image classification and image generation. Considering that language and vision pre-training large models usually have massive parameters, how to use language and vision pre-training large models to solve continuous small-sample learning problems in a parameter-efficient manner is a difficult problem that needs to be solved urgently. Summary of the invention

[0005] The purpose of the present invention is to provide a continuous small sample image classification method based on cross-modal alignment and prototype diversification to address the deficiencies of the prior art.

[0006] The objective of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a continuous small sample image classification method based on cross-modal alignment and prototype diversification, comprising the following steps:

[0007] (1) constructing a multimodal alignment network to extract aligned image features and category text prototypes; wherein the multimodal alignment network includes a visual soft prompt, an image encoder, a text soft prompt, a text encoder, and a feature adapter;

[0008] (2) constructing a prototype biaser, inputting the aligned image features obtained in step (1) into the constructed prototype biaser, obtaining a visual bias vector of the category prototype, and using the visual bias vector of the category prototype to correct the category text prototype obtained in step (1) to obtain a corrected category text prototype;

[0009] (3) Calculate the cosine similarity and probability value between the aligned image features obtained in step (1) and the corrected text prototypes of all categories obtained in step (2);

[0010] (4) Use the labeled training data of the current task for iterative training. During the training process, fix the parameters of the image encoder and the text encoder. Calculate the cross entropy loss function based on the probability value obtained in step (3). Take minimizing the cross entropy loss function as the optimization goal, and adjust the parameters in the visual soft prompt, text soft prompt, feature adapter, and prototype bias to obtain the trained visual soft prompt, text soft prompt, feature adapter, and prototype bias.

[0011] (5) Given a test image, use the trained visual soft prompts, text soft prompts, feature adapters and prototype biases as well as the image encoder and text encoder to repeat steps (1) to (3), calculate the probability value between the test image and all category labels, and select the category label corresponding to the maximum probability value as the final category label of the current test image.

[0012] Furthermore, the step (1) includes the following sub-steps:

[0013] (1.1) Use the open source pre-trained comparative language-image pre-trained large model to initialize the image encoder and text encoder;

[0014] (1.2) constructing a visual soft cue vector, concatenating the visual soft cue vector with the input image to obtain a concatenated feature vector, and inputting the concatenated feature vector into the image encoder to obtain the corresponding image features;

[0015] (1.3) Input the image features obtained in step (1.2) into the feature adapter to obtain adapted image features, and then linearly combine the adapted image features with the image features obtained in step (1.2) to obtain aligned image features;

[0016] (1.4) Construct a text soft prompt vector, concatenate the text soft prompt vector with the category label to obtain a concatenated category text vector, input the concatenated category text vector into the text encoder to obtain the corresponding category label feature, also known as the category text prototype.

[0017] Furthermore, the step (1.2) specifically includes: first constructing a visual soft prompt vector ,in Indicates the first visual soft prompt, Indicates the second visual soft prompt, represents the Lth visual soft prompt, and L represents the total number of visual soft prompts; then the input image is evenly divided into M image blocks of fixed size, and the feature representation of the image block is ,in Represents the feature embedding representation of the i-th image block; then obtains the feature vector of the input image based on the feature embedding representation of M image blocks ; Secondly, construct the visual soft prompt vector The feature vector of the input image Perform splicing to obtain the spliced ​​image feature vector ; Finally, the concatenated feature vector Input to the image encoder In the example above, we get the corresponding image features. ;

[0018] The step (1.4) specifically includes: first constructing a text soft prompt vector ,in Indicates the first text soft prompt. Indicates the second text soft prompt. represents the Hth text soft prompt, H represents the total number of text soft prompts; then for each category label, the constructed text soft prompt vector The corresponding category label vector Perform concatenation to obtain the concatenated category text vector ,in Represents the category label vector of the cth category label; finally, the concatenated category text vector Input to the text encoder In the above example, we get the corresponding category label features. , also known as the category text prototype .

[0019] Furthermore, in the step (1.3), the feature adapter includes two fully connected layers and one residual connection layer; the aligned image features are obtained by the following process:

[0020]

[0021]

[0022] in, is the image feature input into the feature adapter; It is the image feature after two fully connected layers; , , , are the parameters of the two fully connected layers in the feature adapter; is the activation function; is the aligned image feature, is the residual coefficient of the residual connection layer.

[0023] Furthermore, the step (2) includes the following sub-steps:

[0024] (2.1) Construct a prototype biaser, input the aligned image features obtained in step (1) into the constructed prototype biaser, and obtain the visual bias vector of the category prototype;

[0025] (2.2) Add the visual bias vector of the category prototype obtained in step (2.1) to the category text prototype obtained in step (1) according to the following formula to obtain the corrected category text prototype:

[0026]

[0027] in, Represents the corrected category text prototype, represents the visual bias vector of the category prototype, is a mixed parameter.

[0028] Furthermore, in the step (2.1), the prototype biaser includes two fully connected layers; the visual bias vector of the category prototype is obtained by the following process:

[0029]

[0030] in, , , , are the parameters of the two fully connected layers in the prototype biaser.

[0031] Furthermore, in step (3), the calculation formula of the probability value is:

[0032]

[0033] in, Represents the input image The corresponding predicted category label, Represents the predicted category label is the probability value of category label c, represents the cosine similarity, represents the exponential function, K represents a total of K category labels, j represents the jth category label, Represents the temperature coefficient.

[0034] Furthermore, in step (4), the calculation formula of the cross entropy loss function is:

[0035]

[0036] in, represents the cross entropy loss function, represents the nth picture, Indicates the nth picture The corresponding predicted category label, Indicates the nth picture The corresponding true category label, Indicates that there are a total of Labeled training data, Represents the predicted category label is the probability value of category label c, and K means there are K category labels in total.

[0037] A second aspect of an embodiment of the present invention provides a continuous small sample image classification device based on cross-modal alignment and prototype diversification, comprising one or more processors and a memory, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned continuous small sample image classification method based on cross-modal alignment and prototype diversification.

[0038] A third aspect of an embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, is used to implement the above-mentioned continuous small sample image classification method based on cross-modal alignment and prototype diversification.

[0039] The beneficial effect of the present invention is that, by adopting a cross-modal alignment method, the present invention can further align the image features of the downstream task with the category text features in the feature space, thereby improving the generalization ability of the contrastive language-image pre-trained large model in the downstream tasks, and on this basis, by adopting a prototype biaser, the category prototype is integrated into the input-specific information, thereby further improving the classification ability; the implementation method of the present invention is simple and flexible, significantly improves the prediction accuracy of new categories, and can effectively alleviate the catastrophic forgetting phenomenon of the model in old categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of a continuous small sample image classification method based on cross-modal alignment and prototype diversification provided by an embodiment of the present invention;

[0041] Figure 2 It is a flowchart of a continuous small sample image classification method based on cross-modal alignment and prototype diversification provided by an embodiment of the present invention;

[0042] Figure 3 is a structural block diagram of a continuous small sample image classification method based on cross-modal alignment and prototype diversification provided by another embodiment of the present invention;

[0043] Figure 4 It is a schematic diagram of the configuration of a continuous small sample task problem provided by an embodiment of the present invention;

[0044] Figure 5 It is a structural schematic diagram of a continuous small sample image classification device based on cross-modal alignment and prototype diversification provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0046] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0047] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0048] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0049] The continuous small sample image classification method based on cross-modal alignment and prototype diversification of the present invention is based on a pre-trained comparative language-image pre-trained large model. By constructing multi-modal prompts and feature adapters, features of images and category labels are aligned. On this basis, a prototype biaser is used to capture input-specific visual information for text prototype correction, thereby improving the classification accuracy of the input image.

[0050] See also Figure 1 The continuous small sample image classification method based on cross-modal alignment and prototype diversification of the present invention specifically includes the following steps:

[0051] (1) Construct a multimodal alignment network to extract aligned image features and category text prototypes; the multimodal alignment network includes visual soft prompts, image encoders, text soft prompts, text encoders, and feature adapters.

[0052] (1.1) Use the open source pre-trained contrastive language-image pre-trained large model to initialize the image encoder and text encoder.

[0053] It should be understood that the network parameters of the image encoder and text encoder in the multimodal alignment network can be initialized based on open source pre-trained contrastive language-image pre-training models, such as CLIP, BLIP, etc. There is no need to use large batches of training data to train the image encoder and text encoder again, saving resources and time.

[0054] (1.2) Construct a visual soft cue vector, concatenate the visual soft cue vector with the input image to obtain a concatenated feature vector, and input the concatenated feature vector into the image encoder to obtain the corresponding image features.

[0055] Specifically, we first construct a visual soft cue vector ,in Indicates the first visual soft prompt, Indicates the second visual soft prompt, represents the Lth visual soft prompt, and L represents the total number of visual soft prompts; then the input image is evenly divided into M image blocks of fixed size, and the feature representation of the image block is ,in Represents the feature embedding representation of the i-th image block; then obtains the feature vector of the input image based on the feature embedding representation of M image blocks ; Secondly, construct the visual soft prompt vector The feature vector of the input image Perform splicing to obtain the spliced ​​image feature vector ; Finally, the concatenated feature vector Input to the image encoder In the example above, we get the corresponding image features. ,like Figure 2 shown.

[0056] It should be noted that when constructing the visual soft prompt vector, each visual soft prompt contained therein needs to be randomly initialized, and then continuously optimized during the training process to obtain the final visual soft prompt vector.

[0057] (1.3) The image features obtained in step (1.2) Input into the feature adapter to obtain the adapted image features, which are then combined with the image features obtained in step (1.2) Perform linear combination to obtain aligned image features ,like Figure 3 shown.

[0058] Furthermore, the feature adapter includes two fully connected layers and one residual connection layer. The aligned image features are obtained through the following process:

[0059]

[0060]

[0061] in, It is the image feature input into the feature adapter, and is also the image feature finally obtained in step (1.2); It is the image feature after two fully connected layers; , , , are the parameters of the two fully connected layers in the feature adapter; is the activation function, usually the ReLU activation function; is the aligned image feature, is the residual coefficient of the residual connection layer.

[0062] (1.4) Construct a text soft prompt vector, concatenate the text soft prompt vector with the category label to obtain a concatenated category text vector, input the concatenated category text vector into the text encoder to obtain the corresponding category label feature, also known as the category text prototype.

[0063] Specifically, first construct a text soft prompt vector ,in Indicates the first text soft prompt. Indicates the second text soft prompt. represents the Hth text soft prompt, H represents the total number of text soft prompts; then for each category label, the constructed text soft prompt vector The corresponding category label vector Perform concatenation to obtain the concatenated category text vector ,in Represents the category label vector of the cth category label; finally, the concatenated category text vector Input to the text encoder In the above example, we get the corresponding category label features. , also known as the category text prototype .

[0064] It should be understood that, similar to the construction of the visual soft prompt vector, when constructing the text soft prompt vector, each text soft prompt contained therein is randomly initialized, and then continuously optimized during the training process to obtain the final text soft prompt vector.

[0065] (2) Construct a prototype biaser, input the aligned image features obtained in step (1) into the constructed prototype biaser, obtain a visual bias vector of the category prototype, and use the visual bias vector of the category prototype to correct the category text prototype obtained in step (1) to obtain a corrected category text prototype, so that the corrected category text prototype incorporates more visual information.

[0066] (2.1) Construct a prototype biaser to align the image features obtained in step (1) Input into the constructed prototype bias device to obtain the visual bias vector of the category prototype.

[0067] Furthermore, the prototype biaser includes two fully connected layers. The visual bias vector of the category prototype is obtained through the following process:

[0068]

[0069] in, represents the visual bias vector of the category prototype, is the aligned image feature obtained in step (1.3), , , , are the parameters of the two fully connected layers in the prototype biaser, It is an activation function, and the ReLU activation function is used in this embodiment.

[0070] (2.2) If Figure 3 As shown, the visual bias vector of the category prototype obtained in step (2.1) is added to the category text prototype obtained in step (1) according to the following formula to obtain the corrected category text prototype:

[0071]

[0072] in, Represents the corrected category text prototype, is the category text prototype obtained in step (1.4), is a mixed parameter.

[0073] (3) Calculate the cosine similarity and probability value between the aligned image features obtained in step (1) and the corrected text prototypes of all categories obtained in step (2).

[0074] Furthermore, the probability value is calculated as follows:

[0075]

[0076] in, Represents the input image The corresponding predicted category label, Represents the predicted category label is the probability value of category label c, represents the cosine similarity, represents the exponential function, K represents a total of K category labels, j represents the jth category label, Represents the temperature coefficient.

[0077] (4) Use the labeled training data of the current task for iterative training. During the training process, fix the parameters of the image encoder and the text encoder. Calculate the cross entropy loss function based on the probability value obtained in step (3). Take minimizing the cross entropy loss function as the optimization goal, and adjust the parameters in the visual soft prompts, text soft prompts, feature adapters, and prototype biasers to obtain trained visual soft prompts, text soft prompts, feature adapters, and prototype biasers.

[0078] Specifically, iterative training is performed using labeled training data of the current task, wherein the labeled training data includes images and their corresponding true category labels. In each round of training, for the nth image, steps (1) to (3) are repeated to obtain the corresponding probability value, and a cross entropy loss function is calculated based on the probability value and its corresponding true category label. The cross entropy loss function is optimized with minimizing the cross entropy loss function, and the parameters of the visual soft prompt, text soft prompt, feature adapter, and prototype bias are adjusted, while the parameters of the image encoder and text encoder are fixed. This is performed until a preset number of training rounds is reached to obtain the trained visual soft prompt, text soft prompt, feature adapter, and prototype bias.

[0079] Furthermore, the calculation formula of the cross entropy loss function is:

[0080]

[0081] in, represents the cross entropy loss function, represents the nth picture, Indicates the nth picture The corresponding predicted category label, Indicates the nth picture The corresponding true category label, Indicates that there are a total of Labeled training data, Represents the predicted category label is the probability value of category label c, and K means there are K category labels in total.

[0082] (5) Given a test image, use the trained visual soft prompts, text soft prompts, feature adapters and prototype biasers as well as the image encoder and text encoder to repeat steps (1) to (3) to calculate the probability value between the test image and all category labels, and select the category label corresponding to the maximum probability value as the final category label of the current test image, such as Figure 4 shown.

[0083] In summary, the present invention can further align the image features of downstream tasks with the category text features in the feature space by adopting a cross-modal alignment method, thereby improving the generalization ability of the contrastive language-image pre-trained large model in downstream tasks. On this basis, by adopting a prototype biaser, the category prototype is integrated into the input-specific information, thereby further improving the classification ability; the implementation method of the present invention is simple and flexible, which significantly improves the prediction accuracy of new categories, and can effectively alleviate the catastrophic forgetting phenomenon of the model in old categories.

[0084] Corresponding to the aforementioned embodiment of the method for continuous small sample image classification based on cross-modal alignment and prototype diversification, the present invention also provides an embodiment of a device for continuous small sample image classification based on cross-modal alignment and prototype diversification.

[0085] See also Figure 5 An embodiment of the present invention provides a continuous small sample image classification device based on cross-modal alignment and prototype diversification, including one or more processors and a memory, and the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the continuous small sample image classification method based on cross-modal alignment and prototype diversification in the above embodiment.

[0086] The embodiment of the continuous small sample image classification device based on cross-modal alignment and prototype diversification of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability where the continuous small sample image classification device based on cross-modal alignment and prototype diversification of the present invention is located. Figure 5In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0087] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0088] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0089] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the continuous small sample image classification method based on cross-modal alignment and prototype diversification in the above embodiment is implemented.

[0090] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0091] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A continuous small sample image classification method based on cross-modal alignment and prototype diversification, characterized in that: The following steps are involved: (1) constructing a multimodal alignment network to extract aligned image features and category text prototypes; wherein the multimodal alignment network includes a visual soft prompt, an image encoder, a text soft prompt, a text encoder, and a feature adapter; The step (1) comprises the following sub-steps: (1.1) Use the open source pre-trained comparative language-image pre-trained large model to initialize the image encoder and text encoder; (1.2) constructing a visual soft cue vector, concatenating the visual soft cue vector with the input image to obtain a concatenated feature vector, and inputting the concatenated feature vector into the image encoder to obtain corresponding image features; (1.3) Input the image features obtained in step (1.2) into the feature adapter to obtain adapted image features, and then linearly combine the adapted image features with the image features obtained in step (1.2) to obtain aligned image features; (1.4) constructing a text soft prompt vector, concatenating the text soft prompt vector with the category label to obtain a concatenated category text vector, and inputting the concatenated category text vector into the text encoder to obtain the corresponding category label feature, also known as the category text prototype; (2) constructing a prototype biaser, inputting the aligned image features obtained in step (1) into the constructed prototype biaser, obtaining a visual bias vector of the category prototype, and using the visual bias vector of the category prototype to correct the category text prototype obtained in step (1) to obtain a corrected category text prototype; (3) Calculate the cosine similarity and probability value between the aligned image features obtained in step (1) and the corrected text prototypes of all categories obtained in step (2); (4) using the labeled training data of the current task for iterative training, fixing the parameters of the image encoder and the text encoder during the training process, calculating the cross entropy loss function based on the probability value obtained in step (3), taking minimizing the cross entropy loss function as the optimization goal, adjusting the parameters in the visual soft prompt, text soft prompt, feature adapter, and prototype biaser to obtain the trained visual soft prompt, text soft prompt, feature adapter, and prototype biaser; (5) Given a test image, use the trained visual soft prompts, text soft prompts, feature adapters and prototype biasers as well as the image encoder and text encoder to repeat steps (1) to (3), calculate the probability value between the test image and all category labels, and select the category label corresponding to the maximum probability value as the final category label of the current test image.

2. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 1, characterized in that: The step (1.2) specifically includes: first constructing a visual soft prompt vector V = {v1, v2, ..., v L }, where v1 represents the first visual soft prompt, v2 represents the second visual soft prompt, and v L represents the Lth visual soft prompt, and L represents the total number of visual soft prompts; then the input image is evenly divided into M image blocks of fixed size, and the feature representation of the image block is where e i represents the feature embedding representation of the i-th image block; then, the feature vector I of the input image is obtained according to the feature embedding representation of the M image blocks. M }; Secondly, concatenate the constructed visual soft prompt vector V with the feature vector I of the input image to obtain the concatenated image feature vector I v =[e1, e2, ..., e M , v1, v2, ...v L ]; Finally, the concatenated feature vector I v Input into the image encoder f to obtain the corresponding image feature f(I v ); The step (1.4) specifically includes: first constructing a text soft prompt vector T = {t1, t2, ..., t H }, where t1 represents the first text soft prompt, t2 represents the second text soft prompt, and t H represents the Hth text soft prompt, H represents the total number of text soft prompts; then for each category label, the constructed text soft prompt vector T is associated with its corresponding category label vector c c Perform splicing to obtain the spliced ​​category text vector p c =[t1, t2, ..., t H , c c ], where c c Represents the category label vector of the cth category label; finally, the concatenated category text vector p c Input into the text encoder g to obtain the corresponding category label feature g(p c ), also known as the category text prototype g(p c ).

3. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 1, characterized in that: In the step (1.3), the feature adapter includes two fully connected layers and one residual connection layer; the aligned image features are obtained by the following process: f′(I v )=ψ(f(I v )W1+b1)W2+b2 f * (I v )=αf′(I v )+(1-α)f(I v ) Among them, f(I v ) is the image feature input to the feature adapter; f′(I v ) is the image feature after two fully connected layers; W1, W2, b1, b2 are the parameters of the two fully connected layers in the feature adapter; ψ is the activation function; f * (I v ) is the aligned image feature, and α∈[0,1] is the residual coefficient of the residual connection layer.

4. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 2, characterized in that: The step (2) comprises the following sub-steps: (2.1) Construct a prototype biaser, input the aligned image features obtained in step (1) into the constructed prototype biaser, and obtain the visual bias vector of the category prototype; (2.2) The visual bias vector of the category prototype obtained in step (2.1) is added to the category text prototype obtained in step (1) according to the following formula to obtain the corrected category text prototype: g′(p c )=βg(p c )+(1-β)Δ Among them, g′(p c ) represents the corrected category text prototype, Δ represents the visual bias vector of the category prototype, and β∈[0, 1] is the mixing parameter.

5. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 4, characterized in that: In the step (2.1), the prototype biaser includes two fully connected layers; the visual bias vector of the category prototype is obtained by the following process: in, are the parameters of the two fully connected layers in the prototype biaser, ψ is the activation function, and f * (I v ) are the aligned image features.

6. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 1, characterized in that: In step (3), the probability value is calculated as follows: in, represents the predicted category label corresponding to the input image I, Represents the predicted category label is the probability value of category label c, sim() represents cosine similarity, exp() represents exponential function, K represents a total of K category labels, j represents the jth category label, τ represents temperature coefficient, and f * (I v ) is the aligned image feature, g ′ (p c ) represents the corrected category text prototype.

7. The continuous small sample image classification method based on cross-modal alignment and prototype diversification according to claim 1, characterized in that: In step (4), the calculation formula of the cross entropy loss function is: Among them, L ce represents the cross entropy loss function, I n represents the nth picture, Indicates the nth image I n The corresponding predicted class label, y n Indicates the nth image I n The corresponding true category label, N t Indicates that there are n in the current task t t Labeled training data, Represents the predicted category label is the probability value of category label c, and K means there are K category labels in total.

8. A continuous small sample image classification device based on cross-modal alignment and prototype diversification, comprising one or more processors and a memory, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the continuous small sample image classification method based on cross-modal alignment and prototype diversification as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that: A program is stored thereon, which, when executed by a processor, is used to implement the continuous small sample image classification method based on cross-modal alignment and prototype diversification as described in any one of claims 1 to 7.

Images