A semantic soft-label image recognition method and device with limited and unbalanced data

Abstract

The present invention discloses a semantic soft label image recognition method and device with limited and unbalanced data, the method comprising: constructing a semantic soft label image recognition model; pre-training a self-supervised network on a large-scale text data set to obtain a word embedding module; using the word embedding module to generate a corresponding soft label for each category in the training data set; inputting the training data set into a feature extractor to obtain a feature vector, using the corresponding soft label to guide the training, and obtaining a trained semantic soft label image recognition model; inputting a test data set into the trained semantic soft label image recognition model for testing, and obtaining an image recognition result. The method trains a word embedding module on a large-scale text data set, and uses it to generate a corresponding soft label containing rich semantic information for each class of the data set, so as to help train an image recognition model with strong generalization performance under limited and unbalanced data conditions, thereby improving recognition performance.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition, and in particular relates to a method and device for semantic soft label image recognition with limited and unbalanced data. Background Art

[0002] In recent years, the field of deep learning image processing has made rapid progress. The existing image recognition model has achieved an accuracy of 88.3% in the ImageNet image recognition dataset, which contains 1,000 categories of natural images such as airplanes, cars, birds, cats, etc. The image recognition task is essentially a classification task, that is, classifying the input image into the real category to which it belongs, which usually requires a well-performing image feature extractor and feature classifier.

[0003] However, image recognition based on deep learning relies on a large amount of training data, and applies statistical principles to learn the data feature distribution of the corresponding real category. In practical applications, such as face recognition and medical image analysis, it is difficult to obtain images (such as images of rare diseases), and the labeling cost is also high (professionally trained doctors need to spend time labeling), and it is usually difficult to have a large amount of training data for deep learning models to learn. When the amount of training data is limited, the deep learning model will have serious overfitting on the training data set, that is, as the classification accuracy on the training data set increases, the classification accuracy on the test data set decreases. In the case of unbalanced training data, the deep learning model tends to predict the image as the class with more appearances due to the uneven distribution of data between different classes in the training set, which will lead to model bias problems. In both cases, the performance of existing deep learning models is not ideal, which limits the further development of deep learning in these fields.

[0004] In order to solve the above problems, the scientific community has proposed corresponding methods for limited data and imbalanced data. For the case of limited data, the scientific community has proposed two main methods: 1. Transfer the knowledge learned about the training data from an additional training data set with sufficient data related to the current training data set to the currently trained deep learning model. This method usually uses the additional training data set to train a well-performing feature extractor, then fine-tune or fix the parameters of the feature extractor on the current task, and train a new feature classifier; 2. Perform data augmentation methods such as rotation, cropping, and translation transformation on the training images to generate new training images, which in turn increases the amount of training data. The newer methods include Cutout, Randomerasing, and Gridmask. Cutout replaces a random area of ​​the original image with a value of 0 to generate a new training image; Randomerasing replaces a random area on the original image with a random value to generate a new training image; Gridmask replaces multiple different random areas on the original image with a value of 0 or a random value to generate a new training image. In the case of data imbalance, the scientific community has proposed a series of methods for class balance in the training process, such as increasing the importance of training samples of less common classes, or performing more repeated sampling of less common classes. Recent methods focus on the design of loss functions for deep learning training, mainly considering the prediction challenges at the instance level and the data distribution at the class level. The challenge of instance-level prediction is represented by CBFocal, which targets the marginal diminishing effect of the insignificant improvement in model performance when the amount of training data increases. Considering that different images may contain similar information, that is, information overlap, this method proposes an effective number of samples, that is, the number of samples required to provide sufficient information in each class, and balances the classes through the effective number of samples. The data distribution at the class level is represented by LDAMloss, which, through the design of the loss function, makes the distribution boundary of the less common class larger than the boundary of other classes in the feature space, thereby improving the generalization performance on the less common sample class.

[0005] However, there are also certain defects. In the case of limited data: the knowledge transfer method requires an additional training data set with a large number of annotated images related to the current training data. However, in tasks such as medical image analysis or pest monitoring, such an additional training data set with a large number of annotated images is almost impossible to obtain due to the difficulty of collection and annotation; the data augmentation method generates new images by some transformation operations on the original training images. However, compared with the original images, it is difficult to effectively introduce effective high-level semantic information (such as some highly generalized knowledge about the category of the current image) in these transformed images, thereby limiting the ability to further improve the classification performance of the model; the rebalancing method in the case of data imbalance is actually a trade-off between the classification performance of the more common classes and the less common classes, that is, sacrificing the classification performance of the more common classes to improve the classification performance of the less common classes, which usually leads to underfitting of the classification performance of the model on the more common classes (there is still room for further improvement); when the training data in the less common classes is relatively small, this artificial rebalancing method may also lead to overfitting of the classification performance of the model on the less common classes. Summary of the invention

[0006] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a method and device for semantic soft label image recognition with limited and unbalanced data. The method generates soft labels by training a word embedding model obtained through a large amount of natural language materials, which is not affected by the image training data set. When the training data is limited, it can help train an image recognition model with stronger generalization performance and improve the image recognition accuracy.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] On the one hand, the present invention provides a method for semantic soft label image recognition with limited and unbalanced data, comprising the following steps:

[0009] Constructing a semantic soft label image recognition model; the semantic soft label image recognition model includes a feature extractor and a word embedding module;

[0010] Pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module;

[0011] Using the word embedding module, generate corresponding soft labels for each category in the training data set;

[0012] Input the training data set into the feature extractor to obtain feature vectors, use the corresponding soft labels to guide training, and obtain a trained semantic soft label image recognition model;

[0013] The test data set is input into the trained semantic soft-label image recognition model for testing to obtain the image recognition results.

[0014] As a preferred technical solution, the word embedding module training steps are:

[0015] Process the training text in the large-scale text data set, remove garbled characters and symbols, and generate a vocabulary of V words in total;

[0016] Determine each word in the vocabulary as an N-dimensional learnable word vector;

[0017] Use the N-dimensional learnable word vectors of V words in the vocabulary to train the self-supervised network to obtain a trained self-supervised network;

[0018] The feature encoder part of the trained self-supervised network is retained as the word embedding module.

[0019] As a preferred technical solution, the self-supervised network includes Word2Vec, GloVe, Fasttext, Kazumachar and Bert networks.

[0020] As a preferred technical solution, the generation of the corresponding soft label is specifically as follows:

[0021] Input the training data set into the word embedding module to obtain the soft label corresponding to each category, expressed as:

[0022] w c ∈R D

[0023] Among them, w c represents the soft label of the cth class in the training dataset, R D represents a D-dimensional vector on the real number space R;

[0024] The training data set is represented as:

[0025] DA={(x i ,y i ),i=1,…,N}

[0026] Among them, x i represents the i-th training image, y i represents the true unique hot label corresponding to the i-th training image, and N represents the total number of training images in the training dataset.

[0027] As a preferred technical solution, the obtaining of the feature vector is specifically as follows:

[0028] Use feature extractor to extract training images x i The eigenvector of is expressed as:

[0029] f i =F(x i )

[0030] where f i Represents the feature extractor for the input training image x i The D-dimensional feature vector output.

[0031] As a preferred technical solution, the trained semantic soft label image recognition model is specifically:

[0032] For each training image x i , the cosine distance loss function is used to calculate the f output by the feature extractor i The cosine similarity between the corresponding soft labels is used to obtain the cross entropy loss function, which is:

[0033]

[0034] Among them, s(f i ,w i ) represents the feature vector f of the i-th training image i The soft label w corresponding to the i-th training image i The cosine similarity between them, C represents the total number of categories in the training data set, s(f i ,w j ) is the feature vector f of the i-th training image i The soft label w corresponding to the jth category in the training dataset j The cosine similarity between , τ is the temperature hyperparameter;

[0035] The feature extractor is trained using the cross entropy loss function and its parameters are updated by stochastic gradient descent.

[0036] Get the trained semantic soft label image recognition model.

[0037] As a preferred technical solution, the feature extractor adopts ResNet, VGG, DenseNet or ViT network structure.

[0038] As a preferred technical solution, the generation of the corresponding soft label is specifically as follows:

[0039] The deep learning network trained with a large amount of image data is used to extract features for each type of image in the training data set, and the class center is calculated as a soft label.

[0040] Another aspect of the present invention provides a semantic soft label image recognition system with limited and unbalanced data, which is applied to the above-mentioned semantic soft label image recognition method with limited and unbalanced data, including a model construction module, a word embedding training module, a soft label generation module, a model training module and an image recognition module;

[0041] The model building module is used to build a semantic soft label image recognition model, and the model includes a feature extractor and a word embedding module;

[0042] The word embedding training module is used to pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module;

[0043] The soft label generation module generates a corresponding soft label for each category in the training data set using the word embedding module;

[0044] The model training module inputs the training data set into the feature extractor to obtain feature vectors, uses corresponding soft labels to guide training, and obtains a trained semantic soft label image recognition model;

[0045] The image recognition module is used to input the test data set into the trained semantic soft label image recognition model for testing to obtain the image recognition result.

[0046] The present invention also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned data-limited and unbalanced semantic soft label image recognition method.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] 1. This method uses a self-supervised network trained on a large-scale text dataset to obtain a word embedding module, and introduces the prior knowledge of the relationship between categories into the model training process in the form of soft labels, which helps to train a model with stronger generalization performance when the data is limited or unbalanced;

[0049] 2. This method directly calculates the similarity between the extracted feature vector and the soft label to obtain the classification probability of the corresponding category. It can directly guide the image feature extractor to learn to extract visual features at the feature level, and at the same time learn to extract category-specific features to distinguish between semantically similar categories, thereby improving image recognition performance;

[0050] 3. The image recognition method proposed in this method obtains a word embedding model and generates soft labels through training with a large amount of natural language materials. It is not affected by the image training data set and is widely applicable to image recognition tasks in situations where data is limited or unbalanced. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 It is a flow chart of a method for semantic soft label image recognition with limited and unbalanced data in an embodiment of the present invention;

[0053] Figure 2 It is a structural diagram of a semantic soft label image recognition system with limited and unbalanced data in an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention;

[0055] Figure 4 (a) is the cosine similarity between two soft labels of similar categories on the CIFAR-100 dataset according to an embodiment of the present invention;

[0056] Figure 4 (b) is an example of two categories of soft labels with relatively high cosine similarity according to an embodiment of the present invention;

[0057] Figure 5 The figure is a comparison chart of the performance of the embodiment of the present invention and the benchmark method on a limited data set. DETAILED DESCRIPTION

[0058] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0059] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0060] Example

[0061] The main goal of training a deep learning image recognition model is to let the model learn the knowledge of the categories of training images during the training process, that is, some representative features or discriminative features of each class, so that it can accurately distinguish various new images. This ability is called the generalization performance of the model. However, for training data with limited data or imbalanced data categories, it is very challenging for the model to fully learn the corresponding knowledge from such a small number of samples. In the case of limited data or imbalanced data categories, migrating or embedding prior knowledge of categories into the model can help the model effectively learn category knowledge. The word embedding vector of each category, that is, the vector representation of the word in natural language embedded in a mathematically quantized embedding space, is considered as a soft label of the category, so the relationship between categories is naturally embedded in the process of training deep learning image recognition models as part of the category knowledge.

[0062] like Figure 1 As shown, this embodiment provides a semantic soft label image recognition method with limited and unbalanced data, comprising the following steps:

[0063] S1. Build a semantic soft label image recognition model, including feature extractor and word embedding module;

[0064] S2. Pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module;

[0065] S3, using the word embedding module, generate corresponding soft labels for each category in the training data set;

[0066] S4, input the training data set into the feature extractor to obtain feature vectors, use the corresponding soft labels to guide training, and obtain a trained semantic soft label image recognition model;

[0067] S5. Input the test data set into the trained semantic soft label image recognition model for testing to obtain image recognition results.

[0068] More specifically, in step S2, the training steps of the word embedding module are:

[0069] Process the training text in the large-scale text data set, remove garbled characters and symbols, and generate a vocabulary of V words in total;

[0070] Determine each word in the vocabulary as an N-dimensional learnable word vector;

[0071] Use the N-dimensional learnable word vectors of V words in the vocabulary to train the self-supervised network to obtain a trained self-supervised network;

[0072] The feature encoder part of the trained self-supervised network is retained as the word embedding module.

[0073] The word embedding model uses a self-supervised network (that is, it does not need to manually annotate the training data, but directly uses the training data itself as the supervision information for training) on ​​a large-scale text dataset (such as the large amount of text crawled from the Internet by Google using crawler technology). The training task is based on the context vector of a word, which is connected and then predicted by a neural network to predict the current word vector, or based on the vector of the current word, it is predicted by a neural network to predict the vector of the context word centered on it. When a self-supervised network is trained, its feature encoder part (that is, the training network removes the network part related to the task, such as the neural network used for prediction mentioned above) is used as a word embedding model, that is, for each input word, the word embedding model outputs the corresponding semantic expression in the embedding feature space, that is, the expression of the vector. Since the word embedding model is trained on training materials with a number of sentences at the level of millions or even billions, the embedding feature vector of each word has the potential semantic relationship between words. Specifically, two words that are closely related in semantics (such as boy and man) are often more similar in the feature embedding space.

[0074] More specifically, the self-supervised networks used to obtain the word embedding module in the embodiments of the present invention include Word2Vec, GloVe, Fasttext, Kazumachar, and Bert networks;

[0075] Here are two ways to get the word embedding module:

[0076] One is to use the Word2Vec network to obtain the word embedding module. The training steps are:

[0077] Set a window size, sample V words and their context in the window in the training text as training data, find them in the vocabulary and convert them into corresponding vector representations;

[0078] Denote the word as w(t), and the context as w(t+i) and w(ti);

[0079] There are two training modes for Word2Vec networks:

[0080] In the CBOW mode, the vectors of the context words w(t+i) and w(ti) are summed and then a neural network is used to predict the middle word w(t);

[0081] In Skip-gram mode, a neural network is used to predict the context words w(t+i) and w(ti) through the vector of the current word w(t);

[0082] Optimize the N-dimensional vector representation of V words in the vocabulary through any training mode; after the training is completed, discard the neural network and retain the N-dimensional vector representation of V words in the vocabulary to obtain the word embedding module.

[0083] The second is to use the GloVe network to obtain the word embedding module. The training steps are as follows:

[0084] According to the training text, set a window size, build a V×V co-occurrence matrix for V words, and use X i,j Indicates the number of times the i-th word and the j-th word of each element appear in the same window in the training text. The window moves on the training text, and the co-occurrence matrix is ​​updated every time it moves.

[0085] The loss function of the GloVe training network is:

[0086]

[0087] Indicates that the dot product of two word vectors is used to reflect the predicted similarity, and the frequency of two words appearing together is used to reflect the actual similarity; where f(X i,j ) represents an artificially set weight matrix, which is used to eliminate the influence of some words that appear in large quantities, such as the, a, etc.; by continuously training the square error loss function, the predicted similarity can be continuously approached to the actual similarity. When the predicted similarity and the actual similarity are very close, the expected word vector is obtained;

[0088] Finally, the N-dimensional expected word vector representation of V words is retained to obtain the word embedding module.

[0089] More specifically, in step S3, the training data set is input into the trained word embedding module to generate soft labels corresponding to each category, which are expressed as:

[0090] w c ∈R D

[0091] Among them, w c represents the soft label of the cth class in the training dataset, R D represents a D-dimensional vector on the real number space R;

[0092] The training data set is represented as:

[0093] DA={(x i ,y i ),i=1,…,N}

[0094] Among them, x i represents the i-th training image, y i represents the true one-hot label corresponding to the i-th training image, that is, a label vector consisting of only 0 and 1 and only 1 at the corresponding category sequence position; N represents the total number of training images in the training dataset.

[0095] More specifically, in step S4, a feature extractor is used to extract the training image x i The eigenvector of is expressed as:

[0096] f i =F(x i )

[0097] where f i Represents the feature extractor for the input training image x i D-dimensional feature vector output;

[0098] For each training image x i , the cosine distance loss function is used to calculate the f output by the feature extractor i The cosine similarity between the corresponding soft labels is used to obtain the cross entropy loss function, which is:

[0099]

[0100] Among them, s(f i ,w i ) represents the feature vector f of the i-th training image i The soft label w corresponding to the i-th training image i The cosine similarity between them, C represents the total number of categories in the training data set, s(f i ,w j ) is the feature vector f of the i-th training image i The soft label w corresponding to the jth category in the training dataset j The cosine similarity between , τ is the temperature hyperparameter;

[0101] In addition to using cosine similarity to calculate similarity, this method can also use other similarity calculation methods, which will not be repeated here.

[0102] If the feature extractor output is close to the corresponding soft label, it means that the feature extractor can extract visual features with inter-class relations contained in the soft label vector; therefore, using soft labels to guide the training of the feature extractor can help the feature extractor learn prior knowledge of inter-class relations even when the training data is limited or the classes are unbalanced;

[0103] Therefore, the feature extractor is trained using the cross entropy loss function, and the parameters of the feature extractor are updated by stochastic gradient descent;

[0104] For each training image, the soft labels of all classes are involved in the calculation of the cosine distance loss function. The feature extractor will be trained to minimize the distance between each feature vector and the corresponding soft label, while maximizing the distance between the soft labels and feature vectors of other classes. In this way, for semantically similar categories, the feature extractor can be trained to extract more discriminative features, otherwise similar soft labels will lead to larger loss function values. Finally, the trained semantic soft label image recognition model can better distinguish different categories, that is, find the category whose corresponding soft label vector is closest to the feature extractor output vector.

[0105] Figure 4 (a) shows the cosine similarity between soft labels of two similar categories on the CIFAR-100 dataset. Each point represents the cosine similarity of the soft labels of the two categories at the corresponding coordinates. The brighter the point, the higher the similarity. Figure 4 (b) is an example image of a category where the two soft labels have a high similarity.

[0106] In this embodiment, there are multiple network frameworks that can be used by the feature extractor, and network structures such as ResNet, VGG, DenseNet or ViT can be combined with the method of the present invention.

[0107] The key point of this method is to use a large amount of natural language materials in the field of natural language processing to train a word embedding module, and use it to generate a corresponding soft label containing rich semantic information for each class in the data set; based on the generated soft labels, an effective image recognition method is proposed, that is, the semantic information in the soft labels is used to help train an image recognition model with better generalization performance, which is not affected by the image training data set and can achieve good image recognition performance under limited and unbalanced data conditions.

[0108] Figure 5 It is a performance diagram of the benchmark method tested on a limited data set and the performance effect diagram of the benchmark method combined with this method; a limited number of training images (50, 100, 200) are sampled for each category on the CIFAR10, CIFAR100, and mini-Imagenet datasets for training, and the commonly used data augmentation methods Cutout, Randomerasing, and Gridmask are combined with this method to test on multiple limited data classification tasks, and the accuracy of the test results has been further improved.

[0109] At the same time, in addition to generating soft labels through the above-mentioned trained word embedding module, the present invention can also use a deep learning network trained with a large amount of image data to extract features of each category of images in the current data set and calculate the class center as a soft label.

[0110] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0111] Based on the same idea as the semantic soft label image recognition method with limited and unbalanced data in the above-mentioned embodiment, the present invention also provides a semantic soft label image recognition system with limited and unbalanced data, which can be used to execute the above-mentioned semantic soft label image recognition method with limited and unbalanced data. For ease of explanation, the structural diagram of an embodiment of a semantic soft label image recognition system with limited and unbalanced data only shows the parts related to the embodiment of the present invention. Those skilled in the art can understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0112] like Figure 2 As shown, another embodiment of the present invention provides a semantic soft label image recognition system with limited and unbalanced data, comprising the following modules:

[0113] The model building module is used to build a semantic soft label image recognition model, which includes a feature extractor and a word embedding module;

[0114] The word embedding training module is used to pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module;

[0115] The soft label generation module uses the word embedding module to generate corresponding soft labels for each category in the training data set;

[0116] The model training module inputs the training data set into the feature extractor to obtain feature vectors, uses the corresponding soft labels to guide training, and obtains a trained semantic soft label image recognition model;

[0117] The image recognition module is used to input the test data set into the trained semantic soft-label image recognition model for testing and obtain image recognition results.

[0118] It should be noted that a semantic soft label image recognition system with limited and unbalanced data of the present invention corresponds one-to-one to a semantic soft label image recognition method with limited and unbalanced data of the present invention. The technical features and beneficial effects described in the embodiment of the above-mentioned semantic soft label image recognition method with limited and unbalanced data are applicable to the embodiment of a semantic soft label image recognition system with limited and unbalanced data. For specific contents, please refer to the description in the embodiment of the method of the present invention, which will not be repeated here. This is hereby declared.

[0119] In addition, in the implementation of a semantic soft label image recognition system with limited and unbalanced data in the above-mentioned embodiment, the logical division of each program module is only an example. In actual applications, the above-mentioned functions can be assigned to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the semantic soft label image recognition system with limited and unbalanced data is divided into different program modules to complete all or part of the functions described above.

[0120] like Figure 3 As shown, in one embodiment, a computer-readable storage medium is provided, in which a program is stored in a memory, and when the program is executed by a processor, the above-mentioned method for semantic soft label image recognition with limited and unbalanced data is implemented, specifically:

[0121] Constructing a semantic soft label image recognition model; the semantic soft label image recognition model includes a feature extractor and a word embedding module;

[0122] Pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module;

[0123] Using the word embedding module, generate corresponding soft labels for each category in the training data set;

[0124] Input the training data set into the feature extractor to obtain feature vectors, use the corresponding soft labels to guide training, and obtain a trained semantic soft label image recognition model;

[0125] The test data set is input into the trained semantic soft-label image recognition model for testing to obtain the image recognition results.

[0126] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0127] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A semantic soft-label image recognition method with limited and unbalanced data, characterized in that: The steps include: Constructing a semantic soft label image recognition model; the semantic soft label image recognition model includes a feature extractor and a word embedding module; Pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module; The training steps of the word embedding module are: Process the training text in the large-scale text data set, remove garbled characters and symbols, and generate a vocabulary of V words in total; Determine each word in the vocabulary as an N-dimensional learnable word vector; Use the N-dimensional learnable word vectors of V words in the vocabulary to train the self-supervised network to obtain a trained self-supervised network; Keep the feature encoder part of the trained self-supervised network as the word embedding module; Using the word embedding module, generate corresponding soft labels for each category in the training dataset; The generating of the corresponding soft label is specifically as follows: Input the training data set into the word embedding module to obtain the soft label corresponding to each category, expressed as: In c ∈R D Among them, w c represents the soft label of the cth class in the training dataset, R D represents a D-dimensional vector on the real number space R; The training data set is represented as: DA={(x i ,y i ),i=1,…,N} Among them, x i represents the i-th training image, y i represents the true unique hot label corresponding to the i-th training image, and N represents the total number of training images in the training dataset; Input the training data set into the feature extractor to obtain the feature vector, use the corresponding soft label to guide the training, and obtain the trained semantic soft label image recognition model; The test data set is input into the trained semantic soft-label image recognition model for testing to obtain the image recognition results.

2. The method for semantic soft label image recognition with limited and unbalanced data according to claim 1, characterized in that: The self-supervised networks include Word2Vec, GloVe, Fasttext, Kazumachar and Bert networks.

3. The method for semantic soft label image recognition with limited and unbalanced data according to claim 1, characterized in that: The obtaining of the feature vector is specifically as follows: Use feature extractor to extract training images x i The eigenvector of is expressed as: f i =F(x i ) where f i Represents the feature extractor for the input training image x i The D-dimensional feature vector output.

4. The method for semantic soft label image recognition with limited and unbalanced data according to claim 3, characterized in that: The trained semantic soft label image recognition model is specifically: For each training image x i , the cosine distance loss function is used to calculate the feature extractor output f i The cosine similarity between the corresponding soft labels is used to obtain the cross entropy loss function, which is: Among them, s(f i ,w i ) represents the feature vector f of the i-th training image i The soft label w corresponding to the i-th training image i The cosine similarity between them, C represents the total number of categories in the training data set, s(f i ,w j ) is the feature vector f of the i-th training image i The soft label w corresponding to the jth category in the training dataset j The cosine similarity between , τ is the temperature hyperparameter; The feature extractor is trained using the cross entropy loss function and its parameters are updated by stochastic gradient descent. Get the trained semantic soft label image recognition model.

5. The method for semantic soft label image recognition with limited and unbalanced data according to claim 4, characterized in that: The feature extractor adopts ResNet, VGG, DenseNet or ViT network structure.

6. The method for semantic soft label image recognition with limited and unbalanced data according to claim 1, characterized in that: The generating of the corresponding soft label is specifically as follows: The deep learning network trained with a large amount of image data is used to extract features for each type of image in the training data set, and the class center is calculated as a soft label.

7. A semantic soft-label image recognition system with limited and unbalanced data, characterized in that: A semantic soft-label image recognition method with limited and unbalanced data applied to any one of claims 1-6, comprising a model building module, a word embedding training module, a soft-label generation module, a model training module and an image recognition module; The model building module is used to build a semantic soft label image recognition model, and the model includes a feature extractor and a word embedding module; The word embedding training module is used to pre-train the self-supervised network on a large-scale text dataset to obtain a word embedding module; The training steps of the word embedding module are: Process the training text in the large-scale text data set, remove garbled characters and symbols, and generate a vocabulary of V words in total; Determine each word in the vocabulary as an N-dimensional learnable word vector; Use the N-dimensional learnable word vectors of V words in the vocabulary to train the self-supervised network to obtain a trained self-supervised network; Keep the feature encoder part of the trained self-supervised network as the word embedding module; The soft label generation module generates a corresponding soft label for each category in the training data set using the word embedding module; The generating of the corresponding soft label is specifically as follows: Input the training data set into the word embedding module to obtain the soft label corresponding to each category, expressed as: In c ∈R D Among them, w c represents the soft label of the cth class in the training dataset, R D represents a D-dimensional vector on the real number space R; The training data set is represented as: DA={(x i ,y i ),i=1,…,N} Among them, x i represents the i-th training image, y i represents the true unique hot label corresponding to the i-th training image, and N represents the total number of training images in the training dataset; The model training module inputs the training data set into the feature extractor to obtain feature vectors, uses corresponding soft labels to guide training, and obtains a trained semantic soft label image recognition model; The image recognition module is used to input the test data set into the trained semantic soft label image recognition model for testing to obtain the image recognition result.

8. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the method for semantic soft label image recognition with limited and unbalanced data as described in any one of claims 1 to 6 is implemented.

Images