Small sample image classification method and device based on semantic adaptive fusion mechanism

Abstract

The application discloses a small sample image classification method and device based on a semantic adaptive fusion mechanism, and relates to the field of image recognition classification.The technical scheme is as follows: the application realizes high-level multi-modal feature extraction of each class of a target through target visual feature prototype extraction and semantic feature extraction; semantic feature and visual feature dimension adaptation is proposed, solving the fusion between different dimension modalities; through semantic feature and visual feature adaptive convex combination, effective fusion between different dimension modalities is realized, and a multi-modal fusion feature prototype with enhanced representation is obtained; a weight-based support set query set sample splicing is proposed, which can splice features according to the importance of sample distribution.Through a relationship scoring network, a relationship score between samples is obtained, and the one with the highest score is the same class.The method proposed by the application is used for image classification under small sample conditions, fully utilizes multi-modal information, increases target representation information, and improves target classification accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition classification, more particularly, it relates to a small sample image classification method and device based on a semantic adaptive fusion mechanism. BACKGROUND

[0002] Deep learning has been widely applied in image recognition, speech recognition and chess playing. However, deep learning has a great dependence on sample data volume, and most deep learning algorithms need a large amount of data for training, and converge slowly, which hinders the further application of deep learning in many scenarios where it is difficult to obtain a large amount of samples.

[0003] For the target recognition problem under the condition of small samples, the existing technology proposes data enhancement, initialization and metric learning methods. Data enhancement mainly uses a certain method to expand the sample data volume to solve the problem of overfitting, including using an adversarial generative network to generate samples and fantasy features, but these generated samples and features have high similarity, which is difficult to effectively improve the target classification effect. The optimization-based method is based on the idea of meta-learning, aiming to learn a set of meta-classifiers, which can achieve good classification performance on new tasks through parameter fine-tuning, but the trained model can only be pre-trained and migrated on fixed tasks. The method based on graph neural network takes a single sample as a node and the similarity between samples as an edge, and finally infers the similarity between the to-be-recognized sample and all support samples by iteratively calculating the connection matrix of the graph model through the neural network model, but the model training process consumes a large amount of memory space, and the computational complexity increases sharply with the increase of the number of samples.

[0004] However, the traditional classification has poor target classification accuracy under the condition of small samples. Specifically, there are the following deficiencies: first, the difficulty of target feature representation caused by the scarcity of samples; second, the fixed nearest neighbor classifier and linear classifier hinder performance optimization; third, the introduction of auxiliary information is too direct and lacks targeted introduction of different sample characteristics. SUMMARY

[0005] The present application provides a small sample image classification method and device based on a semantic adaptive fusion mechanism to solve the problem of poor target classification accuracy under the condition of small samples. The classification method proposed by the present application obtains the semantic features of the image through the textual description information of the image for the image classification under the condition of small samples, introduces the semantic features into the small sample image classification, combines the visual features of the image, increases the representation information of the image, and improves the classification accuracy of the image.

[0006] The above technical purposes of the present application are achieved by the following technical solutions:

[0007] In a first aspect of the present application, a small sample image classification method based on a semantic adaptive fusion mechanism is provided, and the method comprises the following steps:

[0008] An image dataset is obtained, and a support set and a query set with at least one category are obtained according to the image dataset;

[0009] Text description information of images of each category in the support set is obtained;

[0010] Visual feature prototypes of each category in the support set are extracted, and first semantic features of the text description information are extracted;

[0011] The dimensions of the first semantic features are matched to obtain second semantic features with the same dimensions as the visual feature prototypes;

[0012] Fusion weights are calculated according to the distribution of the first semantic features of the text description information of the images of each category in the support set, the visual feature prototypes and the second semantic features are fused according to the fusion weights, and fusion feature prototypes are obtained;

[0013] Visual features of images to be tested in the query set are extracted, a splicing weight calculation network is constructed to calculate the visual features of the query set and the fusion feature prototypes corresponding to each category in the support set, and a first weight factor corresponding to the query set and a second weight factor corresponding to the support set are outputted, the images to be tested and the fusion feature prototypes of all categories in the support set are spliced along the channel dimension of the visual features according to the first weight factor and the second weight factor to obtain spliced features, and the spliced features corresponding to each category in the support set are spliced along the length direction to obtain total spliced features;

[0014] The total spliced features are scored by using a pre-trained relationship scoring network, and the classification result of the query set is determined according to the scoring result.

[0015] In an embodiment, the support set and the query set with at least one category are obtained according to the image dataset, specifically, N images of categories are randomly selected from the image dataset, and K images and T images are randomly selected from the images of each category in the N categories as the support set and the query set.

[0016] In an embodiment, the visual feature prototypes of each category in the support set are extracted, specifically including:

[0017] A visual feature prototype extraction network is constructed, wherein the visual feature prototype extraction network is composed of a visual feature extraction network and a feature weight calculation network, the visual feature extraction network is a convolutional neural network, and the feature weight calculation network is composed of multiple convolutional layers and a fully connected layer;

[0018] The visual feature prototypes of each category in the support set are extracted based on the pre-trained visual feature prototype extraction network.

[0019] The first semantic feature of the text description information is extracted, specifically by using a word vector learning algorithm to extract the first semantic feature of the text description information.

[0020] In an embodiment, the visual feature prototype of each category of the support set is extracted based on a pre-trained visual feature prototype extraction network, including:

[0021] The target visual feature of each image of each category of the support set is extracted according to the visual feature extraction network, the target visual feature corresponding to each image of each category is spliced along the channel dimension direction of the target visual feature, and the spliced result is sent to the feature weight calculation network to calculate the weight of each image;

[0022] The target visual feature of each image is weighted according to the weight of each image, and the visual feature prototype of each category of the support set is obtained.

[0023] In an embodiment, the dimensions of the first semantic feature are matched, specifically including:

[0024] A feature dimension matching network is constructed, wherein the feature dimension matching network is obtained by sequentially connecting a fully connected layer and a deconvolution layer;

[0025] The first semantic feature is sent to the fully connected layer for depth matching to obtain a first semantic sub-feature which is depth-matched with the visual feature prototype;

[0026] The first semantic sub-feature is sent to the deconvolution layer for length matching to obtain a second semantic feature which is both depth-matched and length-matched with the visual feature prototype.

[0027] In an embodiment, the calculation formula of the fusion weight according to the distribution of the first semantic feature of the support set is where λ represents the fusion weight, represents the first semantic feature, h represents the fully connected network, n represents the category index of each category of the support set, and S represents the support set.

[0028] In an embodiment, the calculation formula of the fusion feature prototype is where λ represents the fusion weight, represents the second semantic feature, represents the visual feature prototype, n represents the category index of each category of the support set, and S represents the support set.

[0029] In an embodiment, the splicing weight calculation network is constructed to calculate the second semantic feature and the visual feature, specifically: the visual feature of the query set and the fusion feature prototype of the support set are respectively sent into the splicing weight calculation network, and the first weight factor corresponding to the query set and the second weight factor corresponding to the support set are output; wherein the visual feature of each image to be tested in the query set is extracted by using a pre-trained convolutional neural network; wherein the splicing weight calculation network is composed of a first convolutional module, a first maximum value pooling 2*2, a second convolutional module, a second maximum value pooling 2*2 and a full connection layer connected in sequence.

[0030] In an embodiment, the fusion feature prototype of the image to be tested and the support set of all categories are spliced along the channel dimension of the visual feature according to the first weight factor and the second weight factor to obtain splicing features, specifically including:

[0031] determining a first proportion of the first weight factor in the sum of the first weight factor and the second weight factor, and a second proportion of the second weight factor in the sum of the first weight factor and the second weight factor;

[0032] splicing the fusion feature prototype and the visual feature along the channel direction according to the first proportion and the second proportion respectively to obtain the splicing features of the support set of all categories corresponding to the query set.

[0033] In a second aspect of the present application, a small sample image classification device based on a semantic adaptive fusion mechanism is provided, comprising:

[0034] a first data module for obtaining an image data set, and obtaining a support set and a query set with at least one category according to the image data set;

[0035] a second data module for obtaining the textual description information of the images of each category of the support set;

[0036] a feature extraction module for extracting the visual feature prototype of each category of the support set, and extracting the first semantic feature of the textual description information;

[0037] a dimension matching module for matching the dimension of the first semantic feature to obtain a second semantic feature with the same dimension as the visual feature prototype;

[0038] a feature fusion module for calculating a fusion weight according to the distribution of the first semantic feature of the textual description information of the images of each category of the support set, and fusing the visual feature prototype and the second semantic feature according to the fusion weight to obtain a fusion feature prototype;

[0039] The feature splicing module is used for extracting visual features of the to-be-tested image in the query set, constructing a splicing weight calculation network to calculate the visual features of the query set and the fusion feature prototype corresponding to each category of the support set, outputting a first weight factor corresponding to the query set and a second weight factor corresponding to the support set, splicing the to-be-tested image and the fusion feature prototype of all categories of the support set along the channel dimension of the visual features according to the first weight factor and the second weight factor to obtain spliced features, and splicing the spliced features corresponding to each category of the support set along the length direction to obtain total spliced features.

[0040] The scoring and classification module is used for scoring the total spliced features by using a pre-trained relationship scoring network, and determining the classification result of the query set according to the scoring result.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. The present application realizes high-level multi-modal feature extraction of each category of the image through visual feature prototype extraction and first semantic feature extraction; the first semantic feature and the visual feature prototype are dimensionally adapted to solve the problem of difficulty in fusing different dimensional features; the fusion feature prototype with enhanced representation is obtained through self-adaptive convex combination of the second semantic feature and the visual feature prototype, the problem of weak feature representation caused by direct feature fusion is solved, and self-adaptive fusion enhancement between features is realized; the support set query set sample splicing based on weight is proposed, the features can be spliced according to the importance of sample distribution, the influence of irrelevant features on the subsequent relationship scoring network is weakened, and finally the relationship score between the samples of the image data set is calculated through the relationship scoring network, and the one with the highest score is the same category. In summary, the classification method proposed in the present application is aimed at image classification under small sample conditions, multi-modal information of the image is used to increase the representation information of the image, and the classification accuracy of the image is improved.

[0043] 2. The present application is based on the visual feature extraction network and the feature weight calculation network of the convolutional neural network, the weight of each image is calculated through the feature weight calculation network, the target visual features of each image are weighted according to the weight of each image, and the features with obvious features are allocated with larger weights, so that the advantage features are retained and the visual feature representation is enriched.

[0044] 3. The present application designs a modal dimension matching method based on a full connection layer and a deconvolution layer, the depth matching of the second semantic feature and the visual feature prototype is realized through the full connection layer calculation, the length-width matching of the second semantic feature and the visual feature prototype is realized through the deconvolution layer, and the problem of different feature sizes that cannot be fused is solved.

[0045] 4、The application is based on the influence of the distribution of the first semantic feature corresponding to the literal description information on the fusion weight, and the fusion weight of each feature is given through the full connection network, so that the adaptive fusion enhancement between the first semantic features of the literal description information is realized, and the representation features of the image are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings described herein are used to provide further understanding of the embodiments of the present application, form a part of the present application, and do not constitute a limitation of the embodiments of the present application. In the drawings:

[0047] Figure 1 A flowchart of a small sample image classification method based on a semantic adaptive fusion mechanism provided by the embodiments of the present application is shown in the figure.

[0048] Figure 2 A structural block diagram of a visual feature prototype extraction network provided by the embodiments of the present application is shown in the figure.

[0049] Figure 3 A first semantic feature and visual feature prototype dimension adaptation flowchart provided by the embodiments of the present application is shown in the figure.

[0050] Figure 4 A structural diagram of a splicing weight calculation network provided by the embodiments of the present application is shown in the figure.

[0051] Figure 5 A structural block diagram of a small sample image classification device based on a semantic adaptive fusion mechanism provided by the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the present application clearer and more apparent, the present application will be further described in detail below with reference to the embodiments and the drawings, and the illustrative embodiments of the present application and the description thereof are only used to explain the present application, and cannot be regarded as a limitation of the present application.

[0053] It should be understood that the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features limited by "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0054] As described in the background, the traditional classification has poor target classification accuracy under small sample conditions. Specifically, there are the following deficiencies: first, the difficulty of target feature representation caused by the small sample; second, the fixed nearest neighbor classifier and linear classifier hinder performance optimization; third, the introduction of auxiliary information is too direct and lacks targeted introduction of the characteristics of different samples. Therefore, the embodiment of the present application proposes a small sample image classification method based on a semantic adaptive fusion mechanism. The method is applied to a terminal device and runs a small sample image classification device based on a semantic adaptive fusion mechanism on the terminal device. As shown in the classification device of Figure 5 The device obtains an image data set and obtains a support set and a query set with at least one category according to the image data set; obtains the text description information of the images of each category of the support set; extracts the visual feature prototype of each category of the support set, and extracts the first semantic feature of the text description information; match the dimension of the first semantic feature to obtain the second semantic feature with the same dimension as the visual feature prototype; calculate the fusion weight according to the distribution of the first semantic feature of the text description information of the images of each category of the support set, fuse the visual feature prototype and the second semantic feature according to the fusion weight, and obtain the fusion feature prototype; extract the visual feature of the image to be tested in the query set, construct a splicing weight calculation network to calculate the visual feature of the query set and the fusion feature prototype corresponding to each category of the support set, output the first weight factor corresponding to the query set and the second weight factor corresponding to the support set, and splice the fusion feature prototype of all categories of the support set along the channel dimension of the visual feature according to the first weight factor and the second weight factor. The splicing feature is obtained, and the splicing feature corresponding to each category of the support set is spliced along the length direction to obtain the total splicing feature; score the total splicing feature using a pre-trained relationship scoring network, and determine the classification result of the query set according to the scoring result. Based on the above working principle, the classification method provided by the embodiment can obtain the semantic feature of the image through the text description information of the image for the image under the small sample condition, introduce the semantic feature into the small sample image classification, combine the semantic feature with the visual feature of the image, increase the representation information of the image, and thus improve the classification accuracy of the image.

[0055] For example, a personal computer, a tablet computer, a computer, etc. can be used as a terminal device. The terminal device can also be referred to as a user equipment. The terminal device and the cloud server are connected through a wireless communication mode. The wireless communication mode includes but is not limited to Bluetooth, WIFI, ZigBee, GPRS, 3G, 4G, 5G, WiTax, etc.

[0056] The cloud server described above, as a distribution center of information, is used to receive, process and store image information; a user sends an information acquisition instruction to the cloud server through the terminal device, and the cloud server sends a classification result of image classification to the terminal device after receiving the information acquisition instruction. The terminal device receives the classification result of the image for the user to view.

[0057] In combination with the above implementation environment, refer to Figure 1 , Figure 1 A flowchart of a small sample image classification method based on a semantic adaptive fusion mechanism provided by the embodiment of the application is provided. The method provided by the embodiment is specifically as follows. The method comprises the following steps:

[0058] In S110, an image data set is obtained, and a support set and a query set with at least one category are obtained according to the image data set.

[0059] Specifically, the image data set refers to a data collection generated by an image acquisition device, including images, graphics, photos, images, graphics, photos converted from video data streams, etc., which are collectively referred to as images. The image acquisition device can be a camera, or a camera, or a CCD image array, or a CTOS image array, etc. for converting optical images into digital data devices, or devices, or components, or instruments. Further, the support set and the query set with at least one category are obtained according to the image data set, specifically: N images of each category are randomly selected from the image data set, and K images and T images are randomly selected from the images of each category in the N categories as the support set and the query set. For example, first, N images of each category are randomly selected from the training data set, and K images of each category are randomly selected from the training set as the support set, and T training images are randomly selected from the training set of N categories to form the query set. In this embodiment, the number of categories N of the training images in the support set is 10, the number of training images K of each category is 10, and the number of training images T in the query set is 5. It can be understood that the category is different kinds of images.

[0060] In S120, the text description information of each category of images in the support set is obtained.

[0061] Specifically, the text description information refers to the target text description information obtained through the Wikipedia entry. The target semantic information extraction converts the text description of the image into a one-dimensional semantic feature vector. Specifically, a one-dimensional semantic feature vector can be obtained by training a corpus for object description such as Wikipedia.

[0062] In S130, the visual feature prototype of each category of images in the support set is extracted, and the first semantic feature of the text description information is extracted.

[0063] In this embodiment, first, the feature tensors of multiple images are extracted by the convolutional neural network, and the feature tensors of the multiple images are used to form a target representation of more effective visual feature prototype, wherein the feature tensors reflect the visual features of image contours and colors, and the feature prototype can more completely represent the visual features of the image, thereby solving the problem of incomplete representation of the visual feature prototype.

[0064] The first semantic feature of the textual description information is extracted by using a word vector learning algorithm, and the semantic information of the image is obtained by using the word vector learning algorithm (Glove method). The first semantic feature corresponding to each category of image in the support set is obtained (d S is the depth of the first semantic feature, and R represents the tensor space). The semantic feature related to the image is extracted, which is a multi-modal representation in addition to the visual feature for image classification, which can enrich the feature information of the image and help improve the classification effect of the image. Correspondingly, in addition to the word vector learning algorithm, the semantic feature of the textual description information can also be extracted by using the one-hot encoding or TF-IDF method. The specific extraction method is a routine technical means for those skilled in the art, and therefore, this place is not described in detail.

[0065] S140, matching the dimension of the first semantic feature to obtain a second semantic feature with the same dimension as the visual feature prototype.

[0066] In this embodiment, the first semantic feature extracted based on the word vector learning algorithm is a two-dimensional vector feature, which cannot be directly combined with the three-dimensional visual feature prototype. Therefore, to solve the problem of how to match the dimensions of the two, this embodiment provides a feature fusion method, which is specifically: constructing a feature dimension matching network, wherein the feature dimension matching network is obtained by sequentially connecting a fully connected layer and a deconvolution layer; the first semantic feature is sent to the fully connected layer for depth matching to obtain a first semantic sub-feature that is depth-matched with the visual feature prototype; and the first semantic sub-feature is sent to the deconvolution layer for length matching to obtain a second semantic feature that is matched in depth and length with the visual feature prototype.

[0067] As shown in Figure 3 , the embodiment provides a feature dimension adaptation method based on a fully connected layer and a deconvolution layer (semantic feature dimension: → visual feature tensor dimension: ), wherein d f is the depth of the visual feature, n f is the length and width of the visual feature

[0068] First, the channel dimension matching of the first semantic feature, i.e., the depth matching, sends the first semantic feature to the fully connected layer fc1 to obtain the first semantic sub-feature

[0069] Then is the spatial dimension matching, i.e. length matching, sending the first semantic sub-feature into a deconvolution layer deconv to obtain the second semantic feature which is finally and visually feature dimension matched In summary, the embodiment described above realizes the size matching of the length and depth of the semantic feature and the visual feature through the full connection layer and the deconvolution layer, and solves the problem that the features cannot be fused due to different sizes.

[0070] S150, calculating the fusion weight according to the distribution of the first semantic feature of the image of each category in the support set, and fusing the visual feature prototype and the second semantic feature according to the fusion weight to obtain the fusion feature prototype.

[0071] In this embodiment, the fusion weight is generated for the fusion of the second semantic feature and the visual feature prototype according to the data distribution of the first semantic feature, the second semantic feature and the visual feature prototype are adaptively fused according to the fusion weight to obtain the fusion feature prototype of the enhanced image representation feature, so as to improve the accuracy of image classification.

[0072] S160, extracting the visual feature of the image to be tested in the query set, constructing a splicing weight calculation network to calculate the visual feature of the query set and the fusion feature prototype corresponding to each category of the support set, outputting the first weight factor corresponding to the query set and the second weight factor corresponding to the support set, splicing the fusion feature prototype of the image to be tested and all categories of the support set along the channel dimension of the visual feature according to the first weight factor and the second weight factor to obtain spliced features, and then splicing the spliced features corresponding to each category of the support set along the length direction to obtain the total spliced feature.

[0073] Specifically, the visual feature of the image to be tested in the query set is extracted by using a pre-trained convolutional neural network (CNN neural network), and in addition, the extraction of the image feature can also be realized by RNN, deep learning model, etc., which is well known to those skilled in the art and will not be described here. In one embodiment, as shown in Figure 4 The splicing weight calculation network is composed of a first convolution module, a first maximum value pooling 2*2, a second convolution module, a second maximum value pooling 2*2, and a full connection layer connected in sequence.

[0074] S170, scoring the total spliced feature by using a pre-trained relationship scoring network, and determining the classification result of the query set according to the scoring result.

[0075] ​In this embodiment, the relationship scoring network χ calculates the score of the relationship between each image in the query set and each target class in the support set, and the higher the score is, the greater the probability that the query set image and the support set target class are of the same class. For the tth query set image, Fconcat t is input into the relationship scoring network to obtain the relationship score S t t t 1 N The item with the highest score is determined as the class, and the one-hot vector r is output. The training loss. The entire loss is defined as: The model χ and the parameters of the network such as the fully connected layer h are continuously updated based on the training loss through back propagation. The above process is repeated until the parameters of each network or module converge. The to-be-tested image is input into the trained relationship scoring network to obtain the relationship scores of each class of images of the to-be-classified image and the existing sample image, and finally the class label with the highest relationship with the to-be-classified image is output, which is the classification result of the to-be-tested image. It should be understood that the construction and training of the relationship scoring network are prior art, and will not be described in detail here.

[0076] In one embodiment, please refer to the structural block diagram of the visual feature prototype extraction network shown in Figure 2 , which is used to extract the visual feature prototype of each class of the support set, and specifically includes:

[0077] The visual feature prototype extraction network is constructed, wherein the visual feature prototype extraction network is composed of a visual feature extraction network and a feature weight calculation network, the visual feature extraction network is a convolutional neural network, the feature weight calculation network is composed of multiple convolutional layers and a fully connected layer; and the visual feature prototype of each class of the support set is extracted based on the pre-trained visual feature prototype extraction network.

[0078] In this embodiment, first, the length and width of the image are adjusted to 224x224 to obtain the support set image and the query set x and y represent the pixel position indexes of the image, and the visual features of the support set image and the query set image are calculated respectively, as follows: wherein, is the visual feature extraction network, n∈[1,N] is the class index of all classes of the support set, k∈[1,K] is the image index of each class of the support set target, is the visual feature obtained from the nth class of the support set, t∈[1,T] is the index of the image in the query set, and f t Q is the visual feature obtained from the tth image in the query set. n​​​​f is the length of the visual feature, f is the depth of the visual feature.

[0079] In one embodiment, the visual feature prototype of each class of the support set is extracted based on the pre-trained visual feature prototype extraction network, including:

[0080] According to the visual feature extraction network, the target visual feature of each image of each class of the support set is extracted, the target visual feature corresponding to each image of each class is spliced along the channel dimension direction of the target visual feature, and the splicing result is sent to the feature weight calculation network to calculate the weight of each image; the target visual feature of each image is weighted according to the weight of each image, and the visual feature prototype of each class of the support set is obtained.

[0081] In this embodiment, for the visual features obtained from the support set images, all visual features of the images of each class are spliced along the channel direction, and the splicing result is sent to the feature weight calculation network φ to calculate the weight of each image in the calculation of the target prototype feature, as follows: wherein, is the weight corresponding to the kth image of the nth class. Concat is an operation of splicing features along the channel direction.

[0082] The visual features of the K images are weighted according to the support set feature weight to obtain the visual feature prototype of each class of the support set that is,

[0083] In one embodiment, the calculation formula of the fusion weight according to the distribution of the first semantic feature of the support set is wherein, λ represents the fusion weight, represents the first semantic feature, h represents the full connection network, n represents the class index of each class of the support set, and S represents the support set.

[0084] In a further embodiment, the calculation formula of the fusion feature prototype is wherein, λ represents the fusion weight, represents the second semantic feature, represents the visual feature prototype, n represents the class index of each class of the support set, and S represents the support set.

[0085] In one embodiment, the first weight factor and the second weight factor are used to splice the to-be-tested image and the fusion feature prototype of all classes of the support set along the channel dimension of the visual feature to obtain spliced features, specifically including:

[0086] determining a first proportion of the first weight factor in a total of the first weight factor and the second weight factor, and a second proportion of the second weight factor in the total of the first weight factor and the second weight factor;

[0087] According to the first proportion and the second proportion, the fusion feature prototype and the visual feature are spliced along the channel direction to obtain spliced features of all categories of the support set corresponding to the query set.

[0088] Specifically, the first proportion and the second proportion are well-known knowledge, and thus are not explained in detail. According to the first proportion and the second proportion, the fusion feature prototype and the visual feature are spliced along the length direction, and the spliced feature calculation formula is specifically: represents the second proportion, represents the first proportion, represents the target visual feature of the tth image of the query set. The fusion feature prototypes of all categories of the support set corresponding to the query set are spliced along the length of the feature tensor of the image to obtain total spliced features

[0089] Based on the same inventive concept, the embodiment of the present application also provides a small sample image classification device based on a semantic adaptive fusion mechanism. The classification device uses multi-modal information of an image to increase representation information of the image, and thus improves classification accuracy of the image. Please refer to Figure 5 , Figure 5 The embodiment of the present application also provides a structure block diagram of a small sample image classification device based on a semantic adaptive fusion mechanism, which includes:

[0090] The first data module 510 is configured to obtain an image data set, and obtain a support set and a query set with at least one category according to the image data set.

[0091] The second data module 520 is configured to obtain textual description information of images of each category of the support set.

[0092] The feature extraction module 530 is configured to extract a visual feature prototype of each category of the support set, and extract a first semantic feature of the textual description information.

[0093] The dimension matching module 540 is configured to match a dimension of the first semantic feature to obtain a second semantic feature with the same dimension as the visual feature prototype.

[0094] The feature fusion module 550 is configured to calculate a fusion weight according to a distribution of the first semantic feature of the textual description information of each category of the image of the support set, and fuse the visual feature prototype and the second semantic feature according to the fusion weight to obtain a fusion feature prototype.

[0095] The feature splicing module 560 is configured to extract visual features of the to-be-tested image in the query set, construct a splicing weight calculation network to calculate the visual features of the query set and the fusion feature prototype corresponding to each category of the support set, output a first weight factor corresponding to the query set and a second weight factor corresponding to the support set, splice the to-be-tested image and the fusion feature prototype of all categories of the support set along the channel dimension of the visual features according to the first weight factor and the second weight factor to obtain spliced features, and then splice the spliced features corresponding to each category of the support set along the length direction to obtain total spliced features.

[0096] The scoring and classification module 570 is configured to score the total spliced features by using a pre-trained relationship scoring network, and determine the classification result of the query set according to the scoring result.

[0097] It can be seen that the classification device provided in the above embodiments realizes high-level multi-modal feature extraction of each category of images through visual feature prototype extraction and first semantic feature extraction; solves the problem of difficulty in fusing features of different dimensions by dimensionally adapting the first semantic features and the visual feature prototypes; solves the problem of weak feature representation ability caused by direct feature fusion by adaptively combining the second semantic features and the visual feature prototypes, realizes adaptive fusion and enhancement of features, proposes weight-based support set and query set sample splicing, splices features according to the importance of sample distribution, weakens the influence of irrelevant features on the subsequent relationship scoring network, and finally calculates the relationship scores between samples in the image data set by the relationship scoring network, and the one with the highest score is the same category.

[0098] Embodiments of the present application also disclose a terminal device. The terminal device includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the small sample image classification method based on the semantic adaptive fusion mechanism.

[0099] The terminal device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the terminal device is configured to provide computing and control capabilities. The memory of the terminal device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the terminal device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, a carrier network, near field communication (NFC) or other technologies. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen. The input device of the terminal device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the housing of the terminal device, or an external keyboard, touchpad or mouse, etc.

[0100] Those skilled in the art can understand that the above-mentioned embodiments show the structure of the terminal device, and only the structure of part of the technical solution of the present disclosure, and do not constitute a limitation on the electronic device to which the technical solution of the present application is applied. The specific terminal device can include more or fewer components than those described in the above-mentioned embodiments, or combine certain components, or have a different arrangement of components.

[0101] The computer readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the small sample image classification method based on the semantic adaptive fusion mechanism are implemented.

[0102] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A small sample image classification method based on a semantic adaptive fusion mechanism, characterized in that, The method comprises: obtaining an image data set, and obtaining a support set and a query set with at least one category according to the image data set; obtaining the text description information of each category of images in the support set; extracting the visual feature prototype of each category of images in the support set, and extracting the first semantic feature of the text description information; matching the dimensions of the first semantic feature to obtain the second semantic feature with the same dimensions as the visual feature prototype; The fusion weight is calculated according to the distribution of the first semantic feature of the text description information of the image of each category of the support set, and the visual feature prototype and the second semantic feature are fused according to the fusion weight to obtain a fusion feature prototype; wherein the calculation formula of the fusion weight calculated according to the distribution of the first semantic feature of the support set is wherein, the fusion weight is represented by f, the first semantic feature is represented by h, h represents a full connection network, n represents the category index of each category of the support set, and S represents the support set. The visual features of the images to be tested in the query set are extracted, a splicing weight calculation network is constructed to calculate the visual features of the query set and the fusion feature prototypes corresponding to each category of the support set, output the first weight factor corresponding to the query set and the second weight factor corresponding to the support set, splice the fusion feature prototypes of the images to be tested and all categories of the support set along the channel dimension of the visual features according to the first weight factor and the second weight factor, obtain spliced features, and then splice the spliced features corresponding to each category of the support set along the length direction to obtain total spliced features; wherein the calculation formula of the fusion feature prototype is wherein, indicates the fusion weight, indicates the second semantic feature, indicates the visual feature prototype, n indicates the category index of each category of the support set, and S indicates the support set. scoring the total splicing feature by using a pre-trained relationship scoring network, and determining the classification result of the query set according to the scoring result.

2. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 1, characterized in that, The support set and the query set with at least one category are obtained from the image data set, specifically: randomly selecting N category images from the image data set, and randomly selecting K images and T images from each of the N categories as the support set and the query set.

3. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 1, characterized in that, The visual feature prototype of each category of images in the support set is extracted, specifically including: constructing a visual feature prototype extraction network, wherein the visual feature prototype extraction network is composed of a visual feature extraction network and a feature weight calculation network, the visual feature extraction network is a convolutional neural network, and the feature weight calculation network is composed of multiple convolutional layers and a fully connected layer; extracting the visual feature prototype of each category of images in the support set based on the pre-trained visual feature prototype extraction network; extracting the first semantic feature of the text description information, specifically: using a word vector learning algorithm to extract the first semantic feature of the text description information.

4. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 3, characterized in that, The visual feature prototype of each category of images in the support set is extracted based on the pre-trained visual feature prototype extraction network, including: extracting the target visual feature of each image of each category of images in the support set according to the visual feature extraction network, splicing the target visual feature corresponding to each image of each category along the channel dimension direction of the target visual feature, and sending the splicing result to the feature weight calculation network to calculate the weight of each image; weighting the target visual feature of each image according to the weight of each image to obtain the visual feature prototype of each category of images in the support set.

5. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 1, characterized in that, The dimensions of the first semantic feature are matched, specifically including: constructing a feature dimension matching network, wherein the feature dimension matching network is obtained by sequentially connecting a fully connected layer and an inverse convolution layer; sending the first semantic feature to the fully connected layer for depth matching to obtain the first semantic sub-feature which is depth-matched with the visual feature prototype; sending the first semantic sub-feature to the inverse convolution layer for length matching to obtain the second semantic feature which is both depth-matched and length-matched with the visual feature prototype.

6. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 1, characterized in that, constructing a splicing weight calculation network to calculate the second semantic feature and the visual feature, specifically: sending the visual feature of the query set and the fusion feature prototype of the support set into the splicing weight calculation network respectively to output the first weight factor corresponding to the query set and the second weight factor corresponding to the support set; wherein a pre-trained convolutional neural network is used to extract the visual feature of each image to be tested in the query set; wherein the splicing weight calculation network is composed of a first convolutional module, a first maximum value pooling 2*2, a second convolutional module, a second maximum value pooling 2*2, and a fully connected layer connected in sequence.

7. The small sample image classification method based on semantic adaptive fusion mechanism according to claim 1, characterized in that, The fusion feature prototype of each class of the support set and the to-be-tested image are spliced along the channel dimension of the visual feature according to the first weight factor and the second weight factor to obtain spliced features, and the splicing specifically includes: A first proportion of the first weight factor in the sum of the first weight factor and the second weight factor is determined, and a second proportion of the second weight factor in the sum of the first weight factor and the second weight factor is determined; The fusion feature prototype and the visual feature are spliced along the channel direction according to the first proportion and the second proportion respectively to obtain spliced features of all classes of the support set corresponding to the query set.

8. A small sample image classification device based on a semantic adaptive fusion mechanism, characterized in that, The device uses the small sample image classification method based on the semantic adaptive fusion mechanism according to any one of claims 1 to 7, and the device comprises: A first data module is configured to obtain an image data set, and obtain a support set and a query set with at least one class according to the image data set; A second data module is configured to obtain text description information of images of each class of the support set; A feature extraction module is configured to extract a visual feature prototype of each class of the support set, and extract a first semantic feature of the text description information; A dimension matching module is configured to match the dimension of the first semantic feature to obtain a second semantic feature with the same dimension as the visual feature prototype; A feature fusion module is configured to calculate a fusion weight according to the distribution of the first semantic feature of the text description information of the images of each class of the support set, fuse the visual feature prototype and the second semantic feature according to the fusion weight, and obtain a fusion feature prototype; A feature splicing module is configured to extract a visual feature of a to-be-tested image in the query set, construct a splicing weight calculation network to calculate the visual feature of the query set and the fusion feature prototype corresponding to each class of the support set, output a first weight factor corresponding to the query set and a second weight factor corresponding to the support set, splice the fusion feature prototype of each class of the support set and the to-be-tested image along the channel dimension of the visual feature according to the first weight factor and the second weight factor to obtain spliced features, and splice the spliced features of each class of the support set along the length direction to obtain total spliced features; A scoring and classification module is configured to score the total spliced features by using a pre-trained relationship scoring network, and determine a classification result of the query set according to the scoring result.

Images