Multi-modal small sample image classification method and system based on multi-scale dynamic feature fusion

Abstract

The invention relates to the technical field of small sample image classification, in particular to a multi-modal small sample image classification method and system based on multi-scale dynamic feature fusion. The method comprises the following steps: acquiring a small sample image data set, and dividing the small sample image data set into a support set and a query set; and inputting the two features into a feature extraction network to extract global features, and inputting the features into a regional proposal network (RPN) to obtain corresponding local features. Extracting semantic information according to the category label of the support set data, and inputting the semantic information into a text encoder to obtain text features; inputting the global feature, the local feature and the text feature of the input image into an adaptive weight network, and performing dynamic feature fusion to effectively integrate multi-modal information; and inputting the fused multi-modal features into a classification model which takes a prototype network as a learning device and takes an MAML algorithm as a meta-learning device for classification prediction, finally training a network model by using a composite loss function, and performing image classification by using the trained classifier.

Description

Technical Field

[0001] The present invention relates to the technical field of small sample image classification, and in particular to a multi-modal small sample image classification method and system for multi-scale dynamic feature fusion. Background Art

[0002] Small sample image classification is an important task that has attracted much attention in the field of computer vision. Its goal is to accurately classify images with little data. With the development of artificial intelligence technology, small sample image classification research has gradually become one of the research hotspots in academia and industry. However, the task faces challenges such as low classification accuracy due to scarce data, overfitting risk, and insufficient model generalization ability. These problems have prompted researchers to explore new learning strategies to improve the classification effect of small sample images.

[0003] The Chinese patent publication number is "CN116863237A", and its name is "An image classification method and system based on reconstruction of category-related features". This method uses feature reconstruction to filter irrelevant features and background features through a feature filtering module, retains category-related key features, and uses these features to reconstruct similar features, thereby improving the problems of visual feature misalignment, irrelevant features, and background interference in small sample classification tasks. However, this method is not comprehensive enough in feature extraction, ignoring possible related detail features and features of other modalities, which affects the subsequent classification accuracy. At the same time, this method only uses a single distance metric module, which makes the model unable to adapt to different needs and limits the generalization ability of the model. Therefore, how to improve the multi-scale feature extraction ability of the small sample image classification feature extraction module and improve the classification accuracy, and how to optimize the subsequent classification module to improve the problem of overfitting and poor model generalization ability caused by small sample number and small intra-class differences, has become a challenge that researchers in this field need to solve urgently. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] The main purpose of the embodiments of the present invention is to propose a multi-modal small sample image classification method and system with multi-scale dynamic feature fusion, aiming to design a classification model that can extract image features in a multi-scale and multi-modal manner, perform adaptive dynamic feature fusion on them, and can continuously adapt to new tasks and improve generalization capabilities.

[0006] (II) Technical solution

[0007] The technical solution of the present invention to solve the above problem is to provide a multi-modal small sample image classification method with multi-scale dynamic feature fusion, comprising the following steps:

[0008] Step 1: Obtain a small sample image dataset: Obtain a public small sample image dataset and divide it into a support set and a query set.

[0009] Step 2, multi-scale feature extraction network: input the support set and query set into the feature extraction network to extract global features; at the same time, input these features into the region proposal network RPN to obtain the corresponding local features; extract semantic information according to the category label of the support set data, and input it into the text encoder to obtain text features.

[0010] Step 3, adaptive weight dynamic feature fusion: The global features, local features and text features of the input image are input into the adaptive weight network for dynamic feature fusion to effectively integrate multimodal information.

[0011] Step 4, prototype adaptive fine-tuning meta-learning classification network model: The fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results.

[0012] Step 5, result prediction: obtain the classification prediction result after the classification network model. Finally, the network model is trained using the composite loss function, and the trained classifier is used for image classification.

[0013] Furthermore, the multi-scale feature extraction network in step 2 includes:

[0014] ResNet12 is used as the feature extraction backbone network, which contains 4 residual blocks. Each residual block has 3 convolutional layers. The first and second residual blocks that extract low-level features and have universality are frozen, and the third and fourth residual blocks are retained to extract more complex global features of the image.

[0015] The local feature extraction branch uses the region proposal network (RPN) to generate multiple candidate regions, and generates 9 candidate frames using 3 scales and 3 aspect ratios. It determines whether each candidate frame contains the target image, outputs the target confidence score, removes the candidate frames with excessive overlap through maximum suppression, retains the high-quality candidate frames as proposal frames, and crops the corresponding regions to obtain the corresponding local features.

[0016] Semantic information is extracted based on category labels, and text encoders are used to generate text features.

[0017] Furthermore, in step 3, the adaptive weight network is composed of a self-attention mechanism, a channel attention mechanism, and a cross-attention mechanism. The global features and the local features are gradually weighted through the self-attention mechanism and the channel attention mechanism to capture the relationship between the features, fuse the global features and the local features, and input them into the cross-attention mechanism together with the text features to enhance multimodal learning. According to the weight generated for each feature by each attention mechanism, the weighted sum is dynamically fused to obtain the feature F. fused .

[0018] Furthermore, in step 4, the fused multimodal features will be input into a classification model using the prototype network as a learner and the MAML algorithm as a meta-learner to obtain a classification prediction result;

[0019] A parameterized mechanism is introduced to optimize the initial class center, so that the originally fixed class center can be fine-tuned according to the characteristics of the current task;

[0020] The optimized prototype network is combined with MAML, combining MAML's ability to quickly adapt to new tasks with the adaptive class center of the prototype network. At the same time, the MAML algorithm optimizes the parameters of the multi-scale feature extraction network.

[0021] Furthermore, in step 5, the classification prediction result is obtained, the network model is trained using a composite loss function, and the trained classifier is used to perform image classification. The composite loss function is calculated by a weighted combination of focal loss, contrast loss and Euclidean distance loss function.

[0022] In order to solve the above technical problems, the present invention further proposes a multi-modal small sample image classification system with multi-scale dynamic feature fusion, which is applied to the small sample image classification method as described above, comprising:

[0023] A data acquisition module is used to acquire a small sample image dataset and divide it into a support set and a query set;

[0024] The multi-scale feature extraction module inputs the divided data set into the feature extraction module to extract its global features, local features and text features;

[0025] Adaptive weight dynamic feature fusion module, inputs the global features, local features and text features of the input image into the adaptive weight network, performs dynamic feature fusion, and effectively integrates multimodal information

[0026] In the prototype adaptive fine-tuning meta-learning classification module, the fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results.

[0027] The training module uses a composite loss function to train the network model and uses the trained classifier to classify images.

[0028] (III) Beneficial effects

[0029] Compared with the prior art, the present invention provides a multi-modal small sample image classification method and system with multi-scale dynamic feature fusion, which has the following beneficial effects:

[0030] 1. The present invention proposes a network model for multi-scale and multi-modal feature extraction, which can efficiently capture the global features of an image and focus on local detail features such as texture and shape. At the same time, the model introduces text features to complement image features, thereby optimizing classification results and significantly improving classification accuracy.

[0031] 2. The present invention proposes an adaptive dynamic feature fusion mechanism, which can automatically adjust the weights of each modal feature according to the characteristics of the input image, weaken the problems of irrelevant features and information asymmetry, improve the flexibility of feature expression, and thus enhance the generalization ability of the model.

[0032] 3. The present invention proposes a meta-learning network model with prototype adaptive fine-tuning, introduces a parameter mechanism, fine-tunes the class center, better balances the feature representation, and combines it with the MAML algorithm, which can quickly adapt to new tasks and reduce the risk of overfitting under small samples. At the same time, the present invention proposes a composite loss function that integrates focal loss, contrast loss, and Euclidean distance loss to form a synergistic effect, solve the problems of small sample size and small intra-class differences, and improve the overall performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 This is a flow chart of the multi-modal small sample image classification method with multi-scale dynamic feature fusion according to the present invention;

[0035] Figure 2 A network block diagram of the multi-modal small sample image classification method with multi-scale dynamic feature fusion according to the present invention;

[0036] Figure 3 A structural diagram of the multi-scale feature extraction backbone network of the present invention;

[0037] Figure 4 This is a diagram of the regional proposal network RPN network structure in the local feature extraction of the present invention;

[0038] Figure 5 This is a diagram of the adaptive weight dynamic feature fusion network structure of the present invention;

[0039] Figure 6 A network structure diagram of various attention mechanisms in the dynamic feature fusion module of the present invention;

[0040] Figure 7A meta-learning classification network diagram for the prototype adaptive fine-tuning described in the present invention;

[0041] Figure 8 A schematic diagram showing the comparison between the corresponding evaluation indicators of the method proposed in the present invention and the indicators of the prior art;

[0042] Fig. 9 This is a structural block diagram of the multi-modal small sample image classification system with multi-scale dynamic feature fusion described in the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Embodiment 1:

[0045] like Figure 1 As shown, a multi-modal small sample image classification method based on multi-scale dynamic feature fusion proposed in one embodiment of the present invention specifically includes the following steps:

[0046] Step 1, obtain a small sample image dataset: obtain a public small sample image dataset and divide it into a support set and a query set;

[0047] It should be noted that the small sample image datasets use two datasets, MinilmageNet and CUB-200-2011. The MinilmageNet image content is complex and can well verify the multi-scale feature extraction and dynamic feature fusion capabilities. CUB-200-2011 contains rich attribute labels and is particularly suitable for multimodal tasks.

[0048] Step 2, multi-scale feature extraction network: input the divided support set and query set into the feature extraction network to extract global features; at the same time, input these features into the region proposal network RPN to generate candidate regions, select high-confidence proposal boxes from the RPN, and use these proposal boxes to crop the feature map to obtain the corresponding local features; extract semantic information according to the category label of the support set data, and input it into the text encoder to obtain text features;

[0049] It should be noted that if Figure 2 As shown in the figure, the divided support set and query set are input into the network diagram of the multi-modal small sample image classification method with multi-scale dynamic feature fusion, as shown in Figure 3As shown, ResNet12 is selected as the feature extraction backbone network to avoid the gradient vanishing problem in deep networks. ResNet12 consists of four residual blocks, and the residual blocks have jump connections to avoid the gradient vanishing problem in deep networks. Each residual block has three convolutional layers, and the convolutional layers all perform 3*3 convolution operations. Each convolutional layer is followed by a batch normalization layer (BN), a ReLU activation function, and a pooling layer. After the last residual block, global average pooling and a fully connected layer are used to globally summarize the features. In addition, the first and second residual blocks that extract low-level features and are universal are frozen, and the third and fourth residual blocks are retained to extract more complex global features of the image. The frozen residual blocks 1 and 2 are represented by gray and bells. Since the content in each residual block is the same, only the composition of residual block 1 and residual block 4 is shown in detail;

[0050] It should be further explained that local feature extraction uses the region proposal network (RPN), such as Figure 4 As shown in the figure. Although RPN is widely used in target detection tasks, it can also significantly improve the classification accuracy in small sample image classification. First, the extracted features are input into RPN. RPN, as a binary classification branch, extracts local features through 3x3 convolution layers in turn, and uses ReLU activation function for nonlinear transformation. Then, RPN integrates the output using 1x1 convolution to unify the dimension of the convolution layer output and the dimension of the candidate box prediction. Subsequently, the output of the shared convolution layer is passed to the region proposal layer to generate candidate regions. The common number of anchor boxes is 9, covering 3 scales and 3 aspect ratios. RPN determines whether each candidate box contains the target object and outputs the corresponding target confidence score. Through the maximum suppression algorithm, RPN removes candidate boxes with excessive overlap and only retains high-quality candidate boxes as proposal boxes. Finally, RPN crops the area corresponding to the retained proposal box to obtain the corresponding local features. Through this series of fine steps, RPN effectively extracts local features related to the target object, further improving the performance of small sample image classification;

[0051] It is important to further explain that the category names or identifier labels in the support set cannot be used directly in the model, so semantic information must be extracted from them. After extraction, this information is input into the text encoder to capture deeper text features. Through this process, the model can better understand the semantic relationship between categories, thereby improving classification performance.

[0052] Step 3, adaptive weight dynamic feature fusion: input the global features, local features and text features of the input image into the adaptive weight network to perform dynamic feature fusion to effectively integrate multimodal information;

[0053] It should be noted that if Figure 5 As shown, the adaptive weight network dynamic feature fusion consists of three attention mechanisms: self-attention mechanism, cross-attention mechanism and channel attention mechanism;

[0054] In the feature extraction module, the global features and local features extracted first will be gradually fused through the self-attention mechanism and the channel attention mechanism. This process helps to capture the subtle relationship between global features and local features. The fused global features and local features are input into the cross-attention mechanism together with the text features. This mechanism can enhance multimodal learning, dynamically adjust the weight ratio for different task types, and finally realize dynamic feature fusion through weighted summation to obtain multimodal fusion features;

[0055] It needs to be further explained that if Figure 6 As shown in the figure, global features and local features are usually represented as high-dimensional vectors. After entering the self-attention mechanism, they must first be linearly transformed to obtain the query Q, key K, and value V. Next, the similarity between the query and the key is calculated to obtain the attention weight, and these weights are weighted summed to generate a new feature Z. Subsequently, feature Z is input into the channel attention mechanism to highlight important channels and suppress unimportant channels. In this process, the obtained features and text features will be input into the cross-attention module together. The structure of cross-attention is similar to that of self-attention, but it inputs two different types of features and performs cross-processing when calculating the attention weights. The rest of the structure is consistent with the self-attention mechanism, and finally the fused multimodal features are output. Through this series of steps, the model can more effectively integrate information from different sources and improve the accuracy of classification and recognition;

[0056] By performing weighted fusion after each attention mechanism, we can dynamically adjust the features at each step. This approach not only builds a richer feature representation, but also ensures that the final fused features have higher information content and expressiveness.

[0057] Step 4: Meta-learning classification network model with prototype adaptive fine-tuning: The fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results;

[0058] It should be noted that if Figure 7As shown in the figure, by optimizing the original prototype network, an adjustable parameter is added to fine-tune the fixed class center, thereby enhancing the stability of the class center. In the case of a small number of samples, the model can flexibly adjust the position of the class center to more accurately capture the category characteristics, thereby achieving better classification results. This optimization not only improves the robustness of the model, but also significantly enhances its adaptability in small sample tasks. At the same time, combined with the MAML algorithm in meta-learning, the class center of the original prototype network and the optimized prototype adaptive fine-tuning network are further optimized, so that the classification model can quickly adapt to new small sample tasks.

[0059] Step 5, result prediction: obtain the classification prediction result after passing the classification network model. Finally, use the composite loss function to train the network model and use the trained classifier to classify the image;

[0060] It should be noted that the loss function adopts a composite loss function, including focal loss, contrast loss and Euclidean distance loss function;

[0061] The focal loss L(p t ) is an improved cross entropy loss, which is used to solve the problem of uneven number of positive and negative samples and imbalanced categories. Its calculation formula is:

[0062] L(p t )=-α t (1-pt) γ log(p t )

[0063] Among them, p t is the model’s predicted probability for the correct category, α t is the balancing factor, γ is the regulating factor, controlling the intensity of the focus loss;

[0064] The contrast loss L(y,D) brings similar sample pairs closer together and pushes different class sample pairs further apart. It is used to learn more discriminative features in small sample classification. Its calculation formula is:

[0065]

[0066] Where y represents the label of the sample pair, D is the Euclidean distance of the sample pair, and m is the boundary distance between the sample pairs;

[0067] Euclidean distance loss It is used to measure the difference between the predicted value and the true value and minimize the difference so that the model learning can be represented more accurately. The calculation formula is:

[0068]

[0069] Where y is the true value, is the predicted value of the model, ‖*‖ represents the Euclidean distance, that is, the squared difference between the two;

[0070] The total loss L is composed of the losses of the above three parts, which can be calculated by weighted combination. The specific calculation formula of the total loss L is as follows:

[0071]

[0072] Among them, α, β, and γ are weight hyperparameters of the loss function, representing the proportion coefficients of different loss functions in the total loss, which are used to balance the three losses;

[0073] It should be further explained that the network model is trained using the above composite loss function, and 150 epochs are set. The specific time is determined according to the performance of the validation set. The learning rate during training is set to 0.001 to ensure that the small sample task model does not skip the optimal point. In order to optimize the network model parameters, the Adam adaptive learning rate optimizer is selected to achieve better results without fine-tuning the parameters. At the same time, the weight decay is set to 1e-4 to prevent overfitting. When the composite loss function remains in a relatively stable state for several consecutive epochs, it means that the model has converged and the training is nearly completed. Use the trained classifier for image classification;

[0074] The corresponding evaluation indicators of the method proposed in this invention are compared with the indicators of the prior art, such as Figure 8 As shown in the figure, it can be seen that the accuracy of the method proposed by the present invention is significantly higher than that of the prior art, which illustrates the effectiveness of the method.

[0075] Embodiment 2:

[0076] like Fig. 9 As shown, an embodiment of the present invention proposes a structural block diagram of a multi-modal small sample image classification system with multi-scale dynamic feature fusion, the system comprising:

[0077] The data acquisition module is used to obtain a small sample image dataset and divide it into a support set and a query set. The datasets used are MinilmageNet and CUB-200-2011.

[0078] The multi-scale feature extraction module inputs the divided data set into the feature extraction module to extract its global features, local features and text features;

[0079] Adaptive weight dynamic feature fusion module, inputs the global features, local features and text features of the input image into the adaptive weight network, performs dynamic feature fusion, and effectively integrates multimodal information

[0080] In the prototype adaptive fine-tuning meta-learning classification module, the fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results.

[0081] The training module uses a composite loss function to train the network model and uses the trained classifier to classify images.

[0082] The system is an intelligent system for small sample image classification, which includes five core modules and is designed for processing data scarcity and classification requirements of multimodal tasks. MiniImageNet and CUB-200-2011 data sets are selected, and comprehensive features are extracted from three aspects: global, local and text, respectively, through the multi-scale feature extraction module to adapt to diverse tasks and improve classification accuracy. The dynamic feature fusion module can adaptively allocate attention weights according to the task type to achieve effective fusion of features. The fused multimodal features are input into the adaptive fine-tuning prototype network optimized by MAML to more accurately determine the class center of each type of task, quickly adapt to new tasks, and achieve accurate classification. The training module uses a composite loss function to optimize the network training, and finally classifies it through the trained model. The system of the present invention is based on the model in Example 1, and the specific steps are consistent, so it is not repeated here.

[0083] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-modal small sample image classification method based on multi-scale dynamic feature fusion, characterized in that: The following steps are involved: Step 1, obtain a small sample image dataset: obtain a public small sample image dataset and divide it into a support set and a query set; Step 2, multi-scale feature extraction network: input the support set and query set into the feature extraction network to extract global features; at the same time, input these features into the region proposal network RPN to obtain the corresponding local features; Extract semantic information based on the category labels of the support set data and input it into the text encoder to obtain text features; Step 3, adaptive weight dynamic feature fusion: input the global features, local features and text features of the input image into the adaptive weight network to perform dynamic feature fusion to effectively integrate multimodal information; Step 4: Meta-learning classification network model with prototype adaptive fine-tuning: The fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results; Step 5, result prediction: obtain the classification prediction result through the classification network model; finally, use the composite loss function to train the network model, and use the trained classifier to classify the image.

2. The multi-modal small sample image classification method with multi-scale dynamic feature fusion according to claim 1 is characterized in that: In the step 1, a public small sample image dataset is obtained and divided into a support set and a query set in an N-way K-shot format.

3. The multi-modal small sample image classification method with multi-scale dynamic feature fusion according to claim 1 is characterized in that: The step 2 includes: The support set and query set are input into the feature extraction network to extract global features. The feature extraction network uses ResNet12 as the backbone network, which contains 4 residual blocks. Each residual block has 3 convolutional layers. The first and second residual blocks that extract low-level features and have universality are frozen, and the third and fourth residual blocks are retained to extract more complex global features of the image. These features are input into the region proposal network (RPN) to generate multiple candidate regions. Nine candidate boxes are generated using three scales and three aspect ratios. It is determined whether each candidate box contains the target image and the target confidence score is output. The candidate boxes with too large overlaps are removed through maximum suppression, and high-quality candidate boxes are retained as proposal boxes. The corresponding areas are cropped to obtain the corresponding local features. Semantic information is extracted according to the category labels of the support set data and input into the text encoder to obtain text features.

4. The multi-modal small sample image classification method with multi-scale dynamic feature fusion according to claim 1 is characterized in that: The step 3 includes: The global features, local features and text features of the input image are input into the adaptive weight network for dynamic feature fusion to effectively integrate multimodal information. The adaptive weight network consists of self-attention mechanism, channel attention mechanism, and cross-attention mechanism. Global features and local features are gradually weighted through self-attention mechanism and channel attention mechanism to capture the relationship between features, fuse global features and local features, and input them into the cross-attention mechanism together with text features to enhance multimodal learning. According to the weight generated for each feature by each attention mechanism, the weighted sum is dynamically fused to obtain feature F. fused : The specific expression is as follows: F fused =w global ·F global +w local ·F local +w test ·F test Among them, F * Represents the corresponding features, global represents global features, local represents local features, test represents text features, and w is the weight calculated by the attention mechanism.

5. The multi-modal small sample image classification method with multi-scale dynamic feature fusion according to claim 1 is characterized in that: The step 4 includes: The fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results; A parameterized mechanism is introduced to optimize the initial class center, so that the originally fixed class center can be fine-tuned according to the characteristics of the current task; The prototype center c of the traditional prototype network i is through all training samples x j The average eigenvector of is obtained, and its calculation formula is as follows: Where f(*) is the feature parameter after the fusion of the adaptive weighted dynamic features of the previous level, and N is the sample of category i; Add parameter α i The calculation formula of the prototype network after adaptive dynamic fine-tuning is as follows: The prototype network is placed in the MAML framework for training, combining MAML's ability to quickly adapt to new tasks with the prototype network's adaptive class center. At the same time, the MAML algorithm optimizes the parameters of the multi-scale feature extraction network.

6. The multi-modal small sample image classification method with multi-scale dynamic feature fusion according to claim 1 is characterized in that: The step 5 includes: The loss function adopts a composite weighted loss function, including focal loss, contrast loss and Euclidean distance loss function; The focal loss L(p t ) is an improved cross entropy loss used to solve the category imbalance problem that may be encountered. Its calculation formula is: L(p t )=-a t (1-p t ) γ log(p t ) Among them, p t is the model’s predicted probability for the correct category, α t is the balancing factor, γ is the regulating factor, controlling the intensity of the focus loss; The contrast loss L(y,D) brings similar sample pairs closer together and pushes different class sample pairs further apart. It is used to learn more discriminative features in small sample classification. Its calculation formula is: Where y represents the label of the sample pair, D is the Euclidean distance of the sample pair, and m is the boundary distance between the sample pairs; Euclidean distance loss It is used to measure the difference between the predicted value and the true value and minimize the difference so that the model learning can be represented more accurately. The calculation formula is: Where y is the true value, is the predicted value of the model, ‖*‖ represents the Euclidean distance, that is, the squared difference between the two; The total loss L is composed of the losses of the above three parts, which can be calculated by weighted combination. The specific calculation formula of the total loss L is as follows: Among them, α, β, and γ are the weight hyperparameters of the loss function, representing the proportion coefficients of different loss functions in the total loss, which are used to balance the three losses.

7. A multi-modal small sample image classification system with multi-scale dynamic feature fusion, characterized in that: include: A data acquisition module is used to acquire a small sample image dataset and divide it into a support set and a query set; The multi-scale feature extraction module inputs the divided data set into the feature extraction module to extract its global features, local features and text features; Adaptive weight dynamic feature fusion module: the global features, local features and text features of the input image are input into the adaptive weight network to perform dynamic feature fusion to effectively integrate multimodal information; Prototype adaptive fine-tuning meta-learning classification module, the fused multimodal features will be input into the classification model with the prototype network as the learner and the MAML algorithm as the meta-learner to obtain the classification prediction results; The training module uses a composite loss function to train the network model and uses the trained classifier to classify images; A multimodal small sample image classification method applied to the multi-scale dynamic feature fusion as described in any one of claims 1 to 6.

Images