Semantic enhancement few-sample learning method based on large language model guidance

Abstract

The invention discloses a semantic enhancement few-sample learning method based on large language model guidance. The method comprises the following steps: acquiring an image data set for training a deep neural network model and corresponding semantic features; the image data set is used for pre-training a pre-defined deep neural network model A, the optimization target is cross entropy optimization loss and local-global image text comparison loss, and semantic features needed by the local-global image text comparison loss are obtained through a text encoder of pre-training CLIP; performing meta-learning training on a pre-trained deep neural network model A and a progressive visual semantic aggregation framework B by using the image data set, wherein the optimization target is cross entropy loss; and outputting a prediction result of given query and support image classification by using the trained deep neural network model A + B. The method is suitable for tasks such as image recognition in a few-sample scene, a more representative classification prototype can be generated, and the generalization performance of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a semantically enhanced few-sample learning method guided by a large language model. Background Art

[0002] Deep learning has revolutionized many fields, especially computer vision and natural language processing. These learning systems are inherently data-intensive and require a large amount of labeled data for model training. In some real-world applications, data acquisition and annotation may be very expensive or infeasible. Therefore, few-shot learning has gained great attention as a promising solution to learn from scarce data and thus alleviate the challenges associated with data scarcity. Most few-shot learning methods face significant challenges due to sample scarcity, especially when only one sample per category is available. To alleviate this problem, some methods introduce semantic information to help construct category concepts. However, these semantic integration methods usually only use high-level information to enrich category representations, while ignoring the local visual context structure that is crucial for effective generalization. Although humans are able to recognize new categories by combining abstract attributes and detailed category descriptions with visual signals, it can be challenging for machine vision systems to extract low-level descriptions of categories directly from visual samples because images often fail to comprehensively capture all the attributes and features that define a category. In addition, limitations of visual representations, such as illumination variations, angles, and occlusions, may hinder the model's ability to fully describe a category based only on the images at hand. Summary of the invention

[0003] Given the success in natural language processing where semantic descriptions or attributes can be generated by large language models, this paper proposes a new paradigm to learn semantically-aware visual patterns from semantic descriptions to enrich category representations for few-shot learning.

[0004] The present invention adopts a filtering strategy to evaluate the similarity between the generated semantic description and the corresponding category name, and discard entities with low similarity. In addition, in order to solve the misalignment problem between visual samples and semantic descriptions, the present invention proposes a progressive visual semantic aggregation framework, which uses category names and category-related entities to generate semantically-aware visual patterns and gradually enrich visual prototypes. The framework comprehensively reveals the potential visual features of the category through a semantically-guided visual pattern extraction module, and integrates the semantically-aware visual patterns into the visual prototype through a prototype calibration module, so that a classifier with significant discrimination can be obtained under limited visual samples.

[0005] To achieve the above object, the technical solution of the present invention is a semantically enhanced few-sample learning method based on a large language model, comprising the following steps:

[0006] S1, obtain the image dataset and corresponding semantic feature description for training the deep neural network model;

[0007] S2. Pre-training a pre-defined deep neural network model A using the image dataset, wherein the optimization target is the cross entropy optimization loss and the local-global image-text contrast loss, and the semantic features required by the local-global image-text contrast loss are obtained by pre-training the text encoder of CLIP;

[0008] S3, using the image dataset to perform meta-learning training on the pre-trained deep neural network model A and the progressive visual semantic aggregation framework B, where the optimization target is the cross entropy loss;

[0009] S4. Use the trained deep neural network model A+B to output the prediction results of the given query and supporting image classification.

[0010] Furthermore, step S2 specifically includes:

[0011] S21, the deep neural network model A adopts ResNet12 network;

[0012] S22, the image description feature cap i Under the prompt "Describe this photo.", i The entity semantic feature entity is input into the large model CogVLM in sequence and extracted using the pre-trained CLIP text encoder. i* , at the prompt “Given a category named {name i},Please list Nrelevant entities.", and change the class name to name i The large model GPT4o is input in sequence to obtain the initial entity, and then the initial entity is screened based on the entity selector Seleter and the pre-trained CLIP text encoder is used to extract features to obtain the final entity semantic features;

[0013] S23, when pre-training the deep neural network A, a batch processing method is adopted. First, a batch of image samples is randomly sampled from the image dataset and the image description semantic features. The batch size N bs given in advance;

[0014] S24. Batch pre The image in is input into the deep neural network A to get the corresponding prediction output Corresponding local image features Corresponding global image features z i ,l i ,g i They are the output prediction vector result of the model, the local features of the model, and the global features of the model, with sizes of 1*C and h respectively. l *w l *hdim,1*hdim,h l ,w l is the length and width of the local feature, and hdim is the dimension of the hidden layer.

[0015] S25. Use all predictions in a batch to form a cross entropy loss function L cls , using local features and image description semantic features to form the local image text contrast loss L local , and use global features and image description semantic features to form the global image-text contrast loss L global ;

[0016] S26, based on the cross entropy loss, the local image text contrast loss and the global image text contrast loss, the total loss function is obtained as follows:

[0017] L=L cls +λ*L global +η*L local

[0018] Among them, λ and η are pre-set weight parameters used to balance the contribution of the three loss functions;

[0019] S27. According to the obtained total loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm to train the deep neural network model A;

[0020] S28. Repeat steps S23 to S27 until the model converges.

[0021] Furthermore, in step S25, the cross entropy loss function is: where y i is the true label of the image, z i Output the prediction result for the model. The local image text contrast loss is:

[0022]

[0023] Where Π is the indicator function and τ represents the temperature parameter. The global image-text contrast loss is:

[0024]

[0025] Furthermore, the operation of using local features and image description semantic features to form a local image text contrast loss in step S25 is: In the formula is the local feature l i The kth local feature representation is, and cos is the cosine similarity.

[0026] Furthermore, the global image text contrast loss operation using the global features and the image description semantic features in step S25 is: golbal (x i ,cap i )=cos(g i ,cap i ).

[0027] Furthermore, in step S21, the deep neural network model A adopts a ResNet12 network. ResNet12 is composed of four ResBlocks. Each ResBlock structure is composed of three convolutional layers with a convolution kernel of 3*3. Batch normalization and ReLU activation functions are added to each convolutional layer. In addition, within each ResBlock, the input is added to the original input after passing through all convolutional layers and activation functions. A global average pooling layer GAP is added after the last ResBlock.

[0028] Furthermore, in step S24, the local image feature L is the output without GAP, and the global image feature G is the output after GAP.

[0029] Furthermore, in step S22, the entity selector Selecter operates as follows: calculating the similarity between the class name and each entity of the class, retaining the first M entities, M∈{5,…,15}. The similarity calculation formula is: sim(entity ij ,name i )=cos(θ T (entity ij ),θ T (name i )), where θ T A text encoder for pre-training CLIP.

[0030] Furthermore, step S3 specifically includes:

[0031] S31, loading the pre-trained model A weights, appending the progressive visual semantic aggregation framework B to the architecture of model A, and updating the gradients of A and B simultaneously during training. The progressive visual semantic aggregation framework B consists of a semantically guided visual pattern extraction module C and a prototype calibration module D;

[0032] S32. When meta-learning and training the deep neural network A+B, a batch processing method is adopted. First, a batch of image samples are randomly sampled from the image dataset and entity semantic features.

[0033]

[0034] Among them, the batch size N bs Pre-given, entity i ∈{entity 1 ,entity 2 ,…,entity C}, name i is the class name of the i-th sample;

[0035] S33. In Batch meta The query set and support set are constructed based on the following construction rules: select N categories, K samples in each category as the support set, and Q samples in each category as the query set;

[0036] S34, input the support set and the query set into model A at the same time, and obtain the output Z of model A after GAP A And the output Z of each ResBlock block1 , Z block2 , Z block3 , Z block4 ;

[0037] S35, Z A , Z block1 , Z block2 , Z block3 , Z block4 And the entity semantic feature S corresponding to each sample entity , class name semantic feature S name Input to module C in framework B results in output C 1out , C 2out , C 3out , C 4out . 1out , C 2out , C 3out , C 4out The concatenation is performed along the last dimension and then input into the feature fusioner, whose output is then added to the classification prototype proto and overwrites the original proto;

[0038] S36, proto, entity semantic features S entity , class name semantic feature S name Input into the D module in framework B to obtain the final classification prototype proto, and according to the formula: Get the probability distribution of each query sample P = {p 1 ,p 2 ,…,p Q*N}, where f(q i ) is the query sample q i feature embedding, proto i is the classification prototype of category i;

[0039] S37. Use the probability distribution of all query samples in a batch to form a cross entropy loss function L CE ;

[0040] S38. According to the obtained cross entropy loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm to train the deep neural network model A+B;

[0041] S39. Repeat steps S32 to S38 until the model converges.

[0042] Furthermore, the semantically guided visual pattern extraction module C in step S31 is composed of a number of CBlocks, and the input of CBlock is the entity semantic feature S entity and visual feature Z from ResBlock block , or two sets of visual features Z from different ResBlocks of model A blocki , Z blockj The calculation process is as follows: first, the input semantic features are transformed through a linear layer, then the cosine similarity between the semantic features and the image features is calculated, and these similarities are used as weights to weight the image features. Finally, the original semantic features are fused with the weighted image features. According to the ResBlock in model A, 1, 2, 3, and 4 CBlocks are assigned to the 1st, 2nd, 3rd, and 4th ResBlocks in turn, denoted as C 1 , C 2 , C 3 , C 4 ,These 4 sub-modules constitute the semantically guided visual pattern extraction module C.

[0043] Furthermore, the prototype calibration module D in step S31 is composed of two semantic alignment modules DBlock, and the structure of DBlock is linear layer, ReLu activation function, Dropout layer, linear layer, with an input dimension of hdim*3, an output dimension of hdim, and a hidden layer dimension of 4096. These two DBlocks are denoted as D 1 , D 2 .

[0044] Furthermore, the class name semantic feature S in step S35 name By the formula: Get it.

[0045] Further, in step S35, the specific process of processing the input in framework B is as follows: for the support set sample features corresponding to the features And the entity semantic feature S corresponding to each support set sample entity ,Will S entity Input to C 1 In the output C 1out . 1out , S entity Input to C 2 In, C 2 The input of the first CBlock in is S entity , and get the output C 2out , the input of the second Cblock is C 1out , C 2out , the output covers the original C 1out . 1out , C 2out , S entity Input to C 3 In, C 3 The input of the first CBlock in is S entity , the output is C 3out , the input of the second Cblock is C 1out , C 3out , the output covers the original C 1out , the input of the third Cblock is C 2out , C 3out , the output covers the original C 2out , C 4 The processing process can be deduced by analogy.

[0046] Furthermore, in step S35, the feature fusion structure is a linear layer, a ReLu activation function, a Dropout layer, and a linear layer in sequence, and its input dimension is hdim*3, the output dimension is hdim, and the hidden layer dimension is 4096.

[0047] Further, the classification prototype proto in step S35 is composed of Z A Corresponding support set sample features The support set samples of each category are summed and divided by the total number of support set samples K of each category.

[0048] Further, in step S36, the prototype calibration module D receives the classification prototype proto, the entity semantic feature S entity , class name semantic feature S name As input, proto and S entity , S name Fusion is obtained by splicing along the last dimension entity 、fusion name , fusion entity 、fusion name Input to D 1 , D 2 Get the output D 1out , D 2out , D 1out , D 2out Add and sum the support set samples of each category and divide by the total number of support set samples K of each category to get the output fusion all , fusion all Add to proto to get the final classification prototype proto, the formula is: proto = λ 1 *proto+λ 2 *fusion all , where λ 1 , 2 It is a pre-set weight parameter used to compare the contribution between semantic features and visual features.

[0049] Furthermore, in step S37, the cross entropy classification loss function L CE for: where y i is the true label, p i Predict the results for the model.

[0050] Further, in step S4, a given query image x is calculated i The method for classifying and predicting the results of the given support set image Support is as follows: i Input the trained deep neural network A+B with Support to get the predicted output result p i , the predicted label is:

[0051] y pred = argmax(p i )

[0052] That is y pred is the prediction result of the image.

[0053] Compared with the existing few-sample learning methods, the semantically enhanced few-sample learning method based on the large language model of the present invention has the following beneficial effects:

[0054] Firstly, the present invention designs a multimodal pre-training method, so that the pre-training model can make full use of semantic information, thereby achieving higher accuracy in classification tasks;

[0055] Secondly, we design a progressive visual semantic aggregation framework that uses category names and category-related entities to generate semantically-aware visual patterns and gradually enrich visual prototypes. This framework consists of a semantically-guided visual pattern extraction module and a prototype calibration module. The former is responsible for extracting semantically-related multi-scale visual patterns, while the latter integrates these patterns to improve the representativeness of visual prototypes.

[0056] In addition, the present invention also designs a filtering strategy to evaluate the similarity between the generated semantic description and the corresponding category name, and discard entities with low similarity to solve the hallucination problem that may occur when a large language model generates semantic entities.

[0057] In summary, the semantically enhanced few-sample learning method based on large language model guidance of the present invention has great application value in scenarios such as computer vision recognition and small-sample scene training. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 The figure is a flow chart of the semantically enhanced few-sample learning method guided by a large language model according to the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0060] On the contrary, the present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention as defined by the claims. Further, in order to make the public have a better understanding of the present invention, some specific details are described in detail in the detailed description of the present invention below. Those skilled in the art can fully understand the present invention without the description of these details.

[0061] refer to Figure 1 In a preferred embodiment of the present invention, a semantically enhanced few-sample learning method based on a large language model comprises the following steps:

[0062] First, an image dataset and corresponding semantic features for training a deep neural network model are obtained. The image dataset is {(x1 ,y 1 ),(x 2 ,y 2 ),…,(x N ,y N )}, where N is the total number of images in the dataset, x i and i Respectively represent the i-th image and its corresponding class label, y i ∈{1,…,C}, C represents the total number of categories, and each category contains multiple images. The semantic features are divided into two parts: image description semantic features and entity semantic features. The entity semantic features are {(y 1 ,entity 1 ),(y 2 ,entity 2 ),…,(y C ,entity C ), the image description feature is {(x 1 ,cap 1 ),(x 2 ,cap 2 ),…,(x N ,cap N )},x i and cap i Respectively represent the i-th image and its corresponding image description features. The image description feature cap i Under the prompt "Describe this photo.", i The entity semantic feature entity is input into the large model CogVLM in sequence and extracted using the pre-trained CLIP text encoder. i , at the prompt “Given a category named {name i},Please list N relevant entities.", and change the class name to i The initial entities are input into the large model GPT4o in sequence, and then the initial entities are screened based on the entity selector Seleter and the feature extraction is performed using the pre-trained CLIP text encoder to obtain the final entity semantic features. The entity selector Selecter operation is: calculate the similarity between the class name and each entity of the class, and retain the first M entities, M∈{5,…,15}. The similarity calculation formula is: sim(entity ij ,name i )=cos(θ T (entity ij ),θ T(name i )), where θ T A text encoder for pre-training CLIP.

[0063] Secondly, the pre-defined deep neural network model A is pre-trained using the image dataset, and the optimization objectives are the cross entropy optimization loss and the local-global image text contrast loss. The deep neural network model A adopts the ResNet12 network. ResNet12 consists of four ResBlocks. Each ResBlock structure consists of three convolutional layers with a convolution kernel of 3*3. Batch normalization and ReLU activation functions are appended to each convolutional layer. In addition, within each ResBlock, the input will be added to the original input after passing through all convolutional layers and activation functions. A global average pooling layer GAP is appended after the last ResBlock. Specifically, the following steps are included:

[0064] In the first step, batch processing is used when pre-training the deep neural network A. First, a batch of image samples is randomly sampled from the image dataset and the image description semantic features. The batch size N bs Given in advance.

[0065] The second step is to batch pre The image in is input into the deep neural network A to get the corresponding prediction output Corresponding local image features Corresponding global image features z i ,l i ,g i They are the output prediction vector result of the model, the local features of the model, and the global features of the model, with sizes of 1*C and h respectively. l *w l *hdim,1*hdim,h l ,w l is the length and width of the local feature, and hdim is the dimension of the hidden layer. Wherein, the local image feature L is the output without GAP, and the global image feature G is the output after GAP.

[0066] The third step is to use all the predictions in a batch to form a cross entropy loss function L cls , using local features and image description semantic features to form the local image text contrast loss L local , and use global features and image description semantic features to form the global image-text contrast loss L global Specifically, the cross entropy loss function is: where yi is the true label of the image, z i Output the prediction result for the model. The local image text contrast loss is:

[0067]

[0068] Where Π is the indicator function, τ represents the temperature parameter, and the global image-text contrast loss is:

[0069]

[0070] In the above formula, sim global (x i ,cap i )=cos(g i ,cap i ), where is the local feature l i The kth local feature representation of , cos is the cosine similarity. Based on the cross entropy loss, local image text contrast loss and global image text contrast loss, the total loss function is:

[0071] L=L cls +λ*L global +η*L local

[0072] Among them, λ and η are pre-set weight parameters used to balance the contribution of the three loss functions.

[0073] The fourth step is to train the deep neural network model A using the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm according to the obtained total loss function.

[0074] Step 5: Repeat steps 1 to 4 until the model converges.

[0075] Then, the pre-trained model A weights are loaded, and the progressive visual semantic aggregation framework B is appended to the architecture of model A. Meta-learning training is performed on the A+B model, and the gradients of A and B are updated simultaneously during training. The progressive visual semantic aggregation framework B consists of a semantically guided visual pattern extraction module C and a prototype calibration module D. The visual pattern extraction module C consists of several CBlocks, and the input of CBlock is the entity semantic feature S entity and visual feature Z from ResBlock block, the calculation process is as follows: first, the input semantic features are transformed through a linear layer, then the cosine similarity between the semantic features and the image features is calculated, and these similarities are used as weights to weight the image features. Finally, the original semantic features are fused with the weighted image features. According to the ResBlock in model A, 1, 2, 3, and 4 CBlocks are assigned to the 1st, 2nd, 3rd, and 4th ResBlocks in turn, denoted as C 1 , C 2 , C 3 , C 4 Finally, the semantic-guided visual pattern extraction module C is formed. The prototype calibration module D consists of two semantic alignment modules DBlock. The structure of DBlock is linear layer, ReLu activation function, Dropout layer, linear layer, with input dimension hdim*3, output dimension hdim, and hidden layer dimension 4096. These two DBlocks are denoted as D 1 , D 2 The specific steps include:

[0076] In the first step, batch processing is used when meta-learning to train the deep neural network A+B. First, a batch of image samples is randomly sampled from the image dataset and entity semantic features.

[0077]

[0078] Among them, the batch size N bs Pre-given, entity i ∈{entity 1 ,entity 2 ,…,entity C}, name i is the class name of the i-th sample. meta The query set and support set are constructed based on the following construction rules: select N categories, K samples in each category as the support set, and Q samples in each category as the query set;

[0079] The second step is to input the support set and query set into model A at the same time to obtain the output Z of model A after GAP. A And the output Z of each ResBlock block1 , Z block2 , Z block3 , Z block4 .

[0080] The third step is to A , Z block1 , Z block2 , Z block3 , Zblock4 And the entity semantic feature S corresponding to each sample entity Input to module C in framework B results in output C 1out , C 2out , C 3out , C 4out The specific operation is: for the support set sample features corresponding to the features And the entity semantic feature S corresponding to each support set sample entity , class name semantic feature S name ,Will S entity Input to C 1 In the output C 1out . 1out , S entity Input to C 2 In, C 2 The input of the first CBlock in is S entity , and get the output C 2out , the input of the second Cblock is C 1out , C 2out , the output covers the original C 1out . 1out , C 2out , S entity Input to C 3 In, C 3 The input of the first CBlock in is S entity , the output is C 3out , the input of the second Cblock is C 1out , C 3out , the output covers the original C 1out , the input of the third Cblock is C 2out , C 3out , the output covers the original C 2out , C 4 The processing of C 1out , C 2out , C 3out , C 4outThe concatenation is performed along the last dimension and then input into the feature fusion Fusioner, whose output is then added to the classification prototype proto and overwrites the original proto. The structure of the feature fusion Fusioner is a linear layer, a ReLu activation function, a Dropout layer, and a linear layer. Its input dimension is hdim*3, its output dimension is hdim, and its hidden layer dimension is 4096. The classification prototype proto consists of Z A Corresponding support set sample features The support set samples of each category are summed and divided by the total number of support set samples K of each category.

[0081] The fourth step is to convert proto and entity semantic features S entity , class name semantic feature S name The D module in the framework B is input to obtain the final classification prototype proto. The specific operation of the prototype calibration module D is as follows: accept the classification prototype proto, entity semantic feature S entity , class name semantic feature S name As input, proto and S entity , S name Fusion is obtained by splicing along the last dimension entity 、fusion name , fusion entity 、fusion name Input to D 1 , D 2 Get the output D 1out , D 2out , D 1out , D 2out Add and sum the support set samples of each category and divide by the total number of support set samples K of each category to get the output fusion all , fusion all Add to proto to get the final classification prototype proto, the formula is: proto = λ 1 *proto+λ 2 *fusion all , where λ 1 , 2 is a pre-set weight parameter used to determine the contribution between semantic features and visual features. name By the formula: Get it.

[0082] Step 5, and according to the formula: Get the probability distribution of each query sample P = {p 1 ,p 2 ,…,pQ*N}, where f(q i ) is the query sample q i feature embedding, proto i is the classification prototype of category i.

[0083] Step 6: Use the probability distribution of all query samples in a batch to form a cross entropy loss function where y i is the true label, p i Predict the results for the model.

[0084] In the seventh step, according to the obtained cross entropy loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm to train the deep neural network model A+B.

[0085] Step 8. Repeat steps 1 to 7 until the model converges.

[0086] Finally, given a query image x, i The method for classifying and predicting the results of the given support set image Support is as follows: i Input the trained deep neural network A+B with Support to get the predicted output result p i , the predicted label is:

[0087] y pred = argmax(p i )

[0088] That is y pred is the prediction result of the image.

[0089] Through the above technical solutions, the embodiment of the present invention is a semantically enhanced few-shot learning method guided by a large language model. The present invention formulates a strategy to generate and filter class-related entities, thereby making full use of the powerful expert knowledge in the large language model to build a comprehensive class concept. In addition, by utilizing a progressive semantic visual aggregation module, the present invention is able to generate more representative classification prototypes, which helps to improve classification accuracy, especially in few-shot learning tasks.

[0090] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A semantically enhanced few-shot learning method based on a large language model, characterized in that: The following steps are involved: S1, obtain the image dataset and corresponding semantic features for training the deep neural network model; S2. Pre-training a pre-defined deep neural network model A using the image dataset, wherein the optimization target is the cross entropy optimization loss and the local-global image-text contrast loss, and the semantic features required by the local-global image-text contrast loss are obtained by pre-training the text encoder of CLIP; S3, using the image dataset to perform meta-learning training on the pre-trained deep neural network model A and the progressive visual semantic aggregation framework B, where the optimization target is the cross entropy loss; S4. Use the trained deep neural network model A and the progressive visual semantic aggregation framework B to output the prediction results of the given query and supporting image classification.

2. The semantically enhanced few-sample learning method based on large language model guidance as claimed in claim 1, characterized in that: In step S1, the image data set is {(x1,y1),(x2,y2),…,(x N ,y N )}, where N is the total number of images in the dataset, x i represents the i-th image, y i Indicates the class label corresponding to the i-th image, y i ∈{1,…,C}, C represents the total number of categories, each category contains multiple images; the semantic features include image description semantic features and entity semantic features, the entity semantic features are {(y1,entity1),(y2,entity2),…,(y C ,entity C ), entity i Obtained through the large model GPT4o and pre-trained CLIP; the image description features are {(x1,cap1),(x2,cap2),…,(x N ,cap N )}, cap i represents the image description feature corresponding to the i-th image, cap i Obtained through the large model CogVLM and pre-trained CLIP.

3. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 2, characterized in that: Step S2 specifically includes: S21, the deep neural network model A adopts ResNet12 network; S22, the image description feature cap i At the prompt "Describe this photo.", copy the image to i The entity semantic feature entity is obtained by sequentially inputting the large model CogVLM and extracting features using the pre-trained CLIP text encoder; i* , at the prompt "Given a category named{name i },Please list N relevantentities.", and change the class name to name i The large model GPT4o is input in sequence to obtain the initial entity, and then the initial entity is screened based on the entity selector Seleter and the pre-trained CLIP text encoder is used to extract features to obtain the final entity semantic features; S23. Randomly sample a batch of image samples from the image dataset and image description semantic features The batch size N bs given in advance; S24. Batch pre The image in is input into the deep neural network A to get the corresponding prediction output Corresponding local image features Corresponding global image features z i ,l i ,g i They are the output prediction vector results of the deep neural network A; S25. Use all the predicted outputs and features in a batch to form a cross entropy loss function L cls , using local features and image description semantic features to form the local image text contrast loss L local , and use global features and image description semantic features to form the global image-text contrast loss L global ; S26, based on the cross entropy loss, the local image text contrast loss and the global image text contrast loss, the total loss function is obtained as follows: L=L cis +λ*L global +η*L local Among them, λ and η are pre-set weight parameters used to balance the contribution of the three loss functions; S27. According to the obtained total loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm to train the deep neural network model A; S28. Repeat steps S23 to S27 until the model converges.

4. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 3, characterized in that: In step S25, the cross entropy loss function is: where y i is the true label of the image, z i Output prediction results for the model; The local image text contrast loss is: Where Π is the indicator function and τ represents the temperature parameter; In the formula is the local feature l i The kth local feature representation of , cos is the cosine similarity; The global image-text contrast loss is: sim global (x i ,cap i )=cos(g i ,cap i )。 5. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 3, characterized in that: In step S22, the entity selector Selecter operates as follows: calculating the similarity between the class name and each entity of the class, retaining the first M entities, M∈{5,...,15}; The similarity calculation formula is: sim(entity ij ,name i )=cos(θ T (entity ij ), θ T (name i )), where θ T A text encoder for pre-training CLIP.

6. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 1, characterized in that: Step S3 specifically includes: S31, loading the pre-trained weights of the deep neural network model A, appending the progressive visual semantic aggregation framework B to the architecture of the deep neural network model A, and updating the gradients of A and B simultaneously during training; wherein the progressive visual semantic aggregation framework B is composed of a semantically guided visual pattern extraction module C and a prototype calibration module D; S32. When meta-learning the deep neural network model A and the progressive visual semantic aggregation framework B, a batch processing method is adopted. First, a batch of image samples are randomly sampled from the image dataset and entity semantic features. Among them, the batch size N bs Pre-given, entity i ∈{entity1,entity2,...,entity C }, name i is the class name of the i-th sample; S33. In Batch meta The query set and support set are constructed based on the following construction rules: select N categories, K samples in each category as the support set, and Q samples in each category as the query set; S34, input the support set and the query set into the deep neural network model A at the same time, and obtain the output Z of the deep neural network model A after GAP A And the output Z of each ResBlock block1 , Z block2 , Z block3 , Z block4 ; S35, Z A , Z block1 , Z block2 , Z block3 , Z block4 And the entity semantic feature S corresponding to each sample entity , class name semantic feature S name Input to module C in framework B results in output C 1out , C 2out , C 3out , C 4out ; C 1out , C 2out , C 3out , C 4out The concatenation is performed along the last dimension and then input into the feature fusioner, whose output is then added to the classification prototype proto and overwrites the original proto; S36, proto, entity semantic features S entity , class name semantic feature S name The D module input into the progressive visual semantic aggregation framework B obtains the final classification prototype proto, and according to the formula: Get the probability distribution of each query sample P = {p1, p2, ..., p Q*N }, where f(q i ) is the query sample q i feature embedding, proto i is the classification prototype of category i; S37. Use the probability distribution of all query samples in a batch to form a cross entropy loss function L CE ; S38. According to the obtained cross entropy loss function, use the Adam optimizer or the stochastic gradient descent optimizer with momentum and the back propagation algorithm to train the deep neural network model A and the progressive visual semantic aggregation framework B; S39. Repeat steps S32 to S38 until the model converges.

7. The semantically enhanced few-sample learning method based on large language model guidance according to claim 6, characterized in that: The semantically guided visual pattern extraction module C in step S31 is composed of several CBlocks, and the input of Cblock is the entity semantic feature S entity and visual feature Z from ResBlock block , the calculation process is as follows: first, the input semantic features are transformed through a linear layer, then the cosine similarity between the semantic features and the image features is calculated, and these similarities are used as weights to weight the image features; finally, the original semantic features are fused with the weighted image features; according to the ResBlock in model A, 1, 2, 3, and 4 CBlocks are assigned to the 1st, 2nd, 3rd, and 4th ResBlocks in turn, denoted as C1, C2, C3, and C4, and finally a semantically guided visual pattern extraction module C is formed; The prototype calibration module D is composed of two semantic alignment modules DBlock. The structure of DBlock is linear layer, ReLu activation function, Dropout layer, linear layer in sequence. Its input dimension is hdim*3 and its output dimension is hdim. The two DBlocks are denoted as D1 and D2.

8. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 7, characterized in that In step S35, the class name semantic feature S name By the formula: get; The specific process of processing input in the progressive visual semantic aggregation framework B is as follows: for the support set sample features corresponding to the features And the entity semantic feature S corresponding to each support set sample entity , class name semantic feature S name ,Will S entity Input into C1 and get output C 1out ; C 1out , S entity Input into C2. The input of the first CBlock in C2 is S entity , and get the output C 2out , the input of the second Cblock is C 1out , C 2out , the output covers the original C 1out ; C 1out , C 2out , S entity Input into C3. The input of the first CBlock in C3 is S entity , the output is C 3out , the input of the second Cblock is C 1out , C 3out , the output covers the original C 1out , the input of the third Cblock is C 2out , C 3out , the output covers the original C 2out ; The processing of C4 is similar.

9. The semantic enhancement few-sample learning method based on large language model guidance as claimed in claim 6, characterized in that In step S35, the feature fusion structure is a linear layer, a ReLu activation function, a Dropout layer, and a linear layer in sequence, with an input dimension of hdim*3 and an output dimension of hdim; The classification prototype proto is composed of Z A Corresponding support set sample features The support set samples of each category are summed and divided by the total number of support set samples K of each category.

10. The semantic enhancement few-sample learning method based on large language model guidance according to claim 6, characterized in that: In step S36, the prototype calibration module D receives the classification prototype proto, entity semantic features S entity , class name semantic feature S name As input, proto and S entity , S name Fusion is obtained by splicing along the last dimension entity 、fusion name , fusion entity 、fusion name Input to D1 and D2 in sequence to get output D 1out , D 2out , D 1out , D 2out Add and sum the support set samples of each category and divide by the total number of support set samples K of each category to get the output fusion all , fusion all Add it to proto to get the final classification prototype proto, the formula is: proto = λ1*proto+λ2*fusion all , where λ1 and λ2 are pre-set weight parameters for the contribution between semantic features and visual features; In step S37, the cross entropy classification loss function L CE for:

11. The semantic enhancement few-sample learning method based on large language model guidance according to any one of claims 1 to 10, characterized in that: In step S4, a given query image x is calculated i The method for classifying and predicting the results of the given support set image Support is as follows: i Input the trained deep neural network A+B with Support to get the predicted output result p i , the predicted label is: y pred =argmax(p i ) y pred is the prediction result of the image.

Images