A feature enhancement method and system based on text-guided mining

Abstract

The present invention provides a feature enhancement method and system based on text-guided mining. The method extracts multimodal features from received sample images to generate multimodal features, and classifies the multimodal features based on a classifier, and then determines a text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated after the classifier, and then determines the target text feature matched by each image based on the prediction score corresponding to the image feature generated after the classifier and the text feature set, and after obtaining the image shallow features corresponding to each sample image, feature enhancement is performed on the feature vector to be enhanced based on the matching degree of the image shallow features and the target text features, and target enhanced features are generated, and the features of the detail area in the image are enhanced through the attention mechanism guided by the text, so that the difference between the images is more obvious, and the classification effect of subsequent image classification is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a feature enhancement method and system based on text-guided mining. Background Art

[0002] At present, in the fine-grained classification process of natural images, the ability of feature extraction has been improved by optimizing the network structure and designing attention modules. For example, GLPM uses deep features and global information to explore key local information from shallow features, and EFM-Net fuses multi-scale features from different layer outputs of ConvNeXt to determine the attention area. Wei et al. designed a causal multi-head attention module based on causal inference theory to obtain the attention area. Although these methods design attention mechanisms to obtain representative features, they ignore the impact of unbalanced datasets on the model. In order to obtain attention features and solve the long tail problem, P2Net first uses multiple prototypes to separate and aggregate samples. ABC-Norm proposes an adversarial loss to improve the learning ability of the model. However, unlike the diversity of natural images, the limited information provided by single perspective images makes it difficult to fully describe these classes, especially the tail classes.

[0003] The successful application of visual-language (VL) models such as CLIP in various image recognition tasks has demonstrated their powerful feature representation capabilities. In natural image classification, some models have achieved impressive performance by fusing textual features with deep visual features in different ways such as addition, concatenation, and multi-head attention mechanisms. However, most works aim to solve general classification problems and fail to effectively explore the potential of shallow features in fine-grained classification tasks. For example, fine-grained ship classification (FGSC) in remote sensing (RS) images faces similar challenges to fine-grained classification of natural images. FGSC also suffers from the problems of low inter-class variance and large intra-class variance. In addition, RS images are captured from a top-down perspective, which has limited diversity of visual information compared to natural images. In addition, due to the finer granularity of the warship subclass, the long-tail distribution problem of the data is more prominent. These factors make FGSC in RS images a challenging task.

[0004] Therefore, how to enhance image features to improve inter-class differences and further improve image classification effects is of great significance. Summary of the invention

[0005] In order to solve the problem in the prior art that the difference between image classes is low, resulting in poor classification effect, the present invention proposes a feature enhancement method and system based on text-guided mining.

[0006] In a first aspect, the present invention provides a feature enhancement method based on text-guided mining, comprising:

[0007] Extracting multimodal features from the received sample image to generate multimodal features, and classifying the multimodal features based on a classifier, wherein the multimodal features include image features and text features;

[0008] Determine the text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier;

[0009] Determine the target text feature for each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set;

[0010] The shallow features of the image corresponding to each sample image are obtained, and feature enhancement is performed on the feature vector to be enhanced which is determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

[0011] Preferably, performing multimodal feature extraction on the received sample image to generate multimodal features, and classifying the multimodal features based on a classifier includes:

[0012] Inputting the received sample image into a feature extraction model to generate multimodal features, wherein the multimodal features include image features and text features;

[0013] The image features and text features corresponding to the sample image are classified by the same classifier.

[0014] Preferably, determining the text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier includes:

[0015] Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score;

[0016] Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category;

[0017] The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

[0018] Preferably, the calculation formula of the difference is as follows:

[0019]

[0020] Among them, g j represents the difference of the jth category, b is a hyperparameter, Represents its text feature t j The score when the prediction is correct after passing the classifier, It represents the score when the prediction is wrong, and N represents the number of categories in the data set.

[0021] Preferably, determining the target text feature for each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set includes:

[0022] Constructing a multi-head attention mechanism based on the prediction score corresponding to the image feature and the text feature set to generate an initial prediction score;

[0023] The prediction score corresponding to the image feature and the initial prediction score are added according to a preset ratio and then multiplied by the mean of each type of text feature set to generate a target text feature that matches each image in the text feature set;

[0024] Based on the prediction scores corresponding to the image features and the initial prediction scores, the average loss is calculated and back-propagated to optimize the parameters of the classifier.

[0025] Preferably, the determination of the target text feature of each image match is as shown in the following formula, including:

[0026] T p =σ(a*P1+(1-a)*P2)*T m

[0027] Among them, T p represents the target text features, T m Represents the mean of each type of text feature set, P1 represents the prediction score corresponding to the image feature, P2 represents the initial prediction score generated by constructing a multi-head attention mechanism with P1 and the text feature set, a represents a hyperparameter, and σ represents the standard deviation function.

[0028] Preferably, the calculation of the average loss function to optimize the classifier parameters is as shown below:

[0029]

[0030] Among them, L fine represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(p1 i ,y i) represents the cross loss value of the i-th text, CE(p2 i ,y i ) represents the cross loss value of the i-th image.

[0031] Preferably, the step of performing feature enhancement on the feature vector to be enhanced determined based on the matching degree between the shallow features of the image and the target text features to generate target enhanced features includes:

[0032] Calculating the matching degree between each shallow feature vector of the image and the target text feature in the shallow feature of the image;

[0033] The shallow feature vector of the image having a matching degree higher than a preset matching degree is determined as the feature vector to be enhanced;

[0034] The feature vector to be enhanced is subjected to maximum and minimum normalization processing, and the processed feature vector to be enhanced is combined with the shallow feature vector of the image to generate a target enhancement feature.

[0035] In a second aspect, the present invention provides a feature enhancement system based on text-guided mining, comprising:

[0036] A classification module, configured to extract multimodal features from the received sample image, generate multimodal features, and classify the multimodal features based on a classifier, wherein the multimodal features include image features and text features;

[0037] A first determination module is used to determine a text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated after the classifier;

[0038] A second determination module is used to determine the target text feature of each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set;

[0039] The generation module is used to obtain the shallow features of the image corresponding to each sample image, and perform feature enhancement on the feature vector to be enhanced determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

[0040] Preferably, the classification module is further used for:

[0041] Inputting the received sample image into a feature extraction model to generate multimodal features, wherein the multimodal features include image features and text features;

[0042] The image features and text features corresponding to the sample image are classified by the same classifier.

[0043] Preferably, the first determining module is further used to:

[0044] Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score;

[0045] Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category;

[0046] The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

[0047] Preferably, the calculation formula of the difference in the first determination module is as follows:

[0048]

[0049] Among them, g j represents the difference of the jth category, b is a hyperparameter, Represents its text feature t j The score when the prediction is correct after passing the classifier, It represents the score when the prediction is wrong, and N represents the number of categories in the data set.

[0050] Preferably, the second determining module is further used to:

[0051] Constructing a multi-head attention mechanism based on the prediction score corresponding to the image feature and the text feature set to generate an initial prediction score;

[0052] The prediction score corresponding to the image feature and the initial prediction score are added according to a preset ratio and then multiplied by the mean of each type of text feature set to generate a target text feature that matches each image in the text feature set;

[0053] Based on the prediction scores corresponding to the image features and the initial prediction scores, the average loss is calculated and back-propagated to optimize the parameters of the classifier.

[0054] Preferably, the determination of the target text feature of each image match in the second determination module is as shown in the following formula, including:

[0055] T p =σ(a*P1+(1-a)*P2)*T m

[0056] Among them, T p represents the target text features, T mRepresents the mean of each type of text feature set, P1 represents the prediction score corresponding to the image feature, P2 represents the initial prediction score generated by constructing a multi-head attention mechanism with P1 and the text feature set, a represents a hyperparameter, and σ represents the standard deviation function.

[0057] Preferably, the second determination module calculates the average loss function to optimize the classifier parameters as shown in the following formula:

[0058]

[0059] Among them, L fine represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(p1 i ,y i ) represents the cross loss value of the i-th text, CE(p2 i ,y i ) represents the cross loss value of the i-th image.

[0060] Preferably, the generating module is further used for:

[0061] Calculating the matching degree between each shallow feature vector of the image and the target text feature in the shallow feature of the image;

[0062] The shallow feature vector of the image having a matching degree higher than a preset matching degree is determined as the feature vector to be enhanced;

[0063] The feature vector to be enhanced is subjected to maximum and minimum normalization processing, and the processed feature vector to be enhanced is combined with the shallow feature vector of the image to generate a target enhancement feature.

[0064] In another aspect, the present application further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0065] The memory is used to store one or more programs;

[0066] When the one or more programs are executed by the at least one processor, a feature enhancement method based on text-guided mining as described above is implemented.

[0067] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the feature enhancement method based on text-guided mining as described above is implemented.

[0068] Compared with the prior art, the present invention has the following beneficial effects:

[0069] The present invention provides a feature enhancement method and system based on text-guided mining. The method extracts multimodal features from received sample images to generate multimodal features, and classifies the multimodal features based on a classifier, and then determines a text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated after the classifier, and then determines the target text feature matched by each image based on the prediction score corresponding to the image feature generated after the classifier and the text feature set, and after obtaining the image shallow features corresponding to each sample image, the feature vector to be enhanced is determined based on the matching degree of the image shallow features and the target text features to generate target enhanced features, and an attention mechanism guided by text is implemented to enhance the features of the detail area in the image, so that the difference between the images is more obvious, and the classification effect of subsequent image classification is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A flowchart of the feature enhancement method based on text-guided mining of the present invention;

[0071] Figure 2 A network flow diagram of the feature enhancement method based on text-guided mining of the present invention;

[0072] Figure 3 Schematic diagram of loss function comparison of the feature enhancement method based on text-guided mining of the present invention

[0073] Figure 4 It is a schematic diagram of a heat map before and after feature enhancement of the feature enhancement method based on text-guided mining of the present invention;

[0074] Figure 5 It is a structural schematic diagram of the feature enhancement system based on text-guided mining of the present invention;

[0075] Figure 6 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0076] The present invention proposes a feature enhancement method and system based on text-guided mining. The method uses text to explore the attention area of ​​an image and enhances the area to obtain highly discriminative features, which solves the problem of high target similarity and long-tail data distribution interfering with model performance in image classification and improves the accuracy of image classification.

[0077] In order to better understand the present invention, the content of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0078] Embodiment 1:

[0079] A feature enhancement method based on text-guided mining, such as Figure 1 As shown, including:

[0080] Step S1: extract multimodal features from the received sample image to generate multimodal features, and classify the multimodal features based on a classifier, wherein the multimodal features include image features and text features.

[0081] Step S2: Determine the text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier.

[0082] Step S3: determining the target text feature for each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set.

[0083] Step S4: Obtain the shallow features of the image corresponding to each sample image, and perform feature enhancement on the feature vector to be enhanced determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

[0084] In this embodiment, in step S1, the process of extracting multimodal features from the received sample image, generating multimodal features, and classifying the multimodal features based on a classifier includes:

[0085] Inputting the received sample image into a feature extraction model to generate multimodal features, wherein the multimodal features include image features and text features;

[0086] The image features and text features corresponding to the sample image are classified by the same classifier.

[0087] Specifically, the image features of the sample image are extracted by the visual encoder in the feature extraction model, and the text features of the sample image are extracted by the text encoder in the feature extraction model. The same classifier is used to classify the image features and text features, and then the same type of image features and text features are divided into this category to ensure the consistency of the feature attributes of the two modalities in the same feature space.

[0088] In addition, after using the same classifier to classify the image features and text features corresponding to the sample image, the average loss function can be calculated to balance the impact of the loss values ​​of each category on the overall loss. The calculation formula is as follows:

[0089]

[0090] Among them, L prerepresents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(pl i ,y i ) represents the cross loss value of the i-th text, CE(p xi ,y i ) represents the cross loss value of the i-th image.

[0091] In this embodiment, in step S2, based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier, the text feature set corresponding to each category is determined, including:

[0092] Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score;

[0093] Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category;

[0094] The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

[0095] Specifically, representative text features of each category are screened and stored for subsequent use. All text features of each category are screened, and the number of screened features depends on the number of images contained in the category. For example, the head category corresponds to 1 text feature, the tail category corresponds to 6 text features, and the middle category corresponds to 4 text features. The selection of text features depends on the difference between the scores of the multimodal features predicted as the correct category and the scores predicted as the wrong category after the classifier. The larger the difference, the better. Then, the text features screened out of each category are constructed into a text feature set, and the mean T of the text features screened out of each category is calculated. m .

[0096] In one embodiment, the difference between the correct prediction score and the incorrect prediction score of the text feature corresponding to each category is calculated as follows:

[0097]

[0098] Among them, g j represents the difference of the jth category, b is a hyperparameter, Represents its text feature t j The score when the prediction is correct after passing the classifier, where 1 represents the first category and is the category number, that is, Specifically expressed as the score of the jth text being predicted as the first category, It represents the score when the j-th text is predicted incorrectly (that is, when the j-th text is predicted as the w-th category instead of the first category), and N represents the number of categories contained in the dataset.

[0099] In this embodiment, in step S3, the process of determining the target text feature of each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set includes:

[0100] Constructing a multi-head attention mechanism based on the prediction score corresponding to the image feature and the text feature set to generate an initial prediction score;

[0101] The prediction score corresponding to the image feature and the initial prediction score are added according to a preset ratio and then multiplied by the mean of each type of text feature set to generate a target text feature that matches each image in the text feature set;

[0102] Based on the prediction scores corresponding to the image features and the initial prediction scores, the average loss is calculated and back-propagated to optimize the parameters of the classifier.

[0103] In one embodiment, the target text feature of each image match is determined as follows:

[0104] T p =σ(a*P1+(1-a)*P2)*T m

[0105] Among them, T p represents the target text features, T m Represents the mean of each type of text feature set, P1 represents the prediction score corresponding to the image feature, P2 represents the initial prediction score generated by constructing a multi-head attention mechanism with P1 and the text feature set, a represents a hyperparameter, and σ represents the standard deviation function.

[0106] In one embodiment, the average loss function is calculated to optimize the classifier parameters as shown in the following formula:

[0107]

[0108] Among them, L fine represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(p1 i ,y i ) represents the cross loss value of the i-th image after passing through the classification head, CE(p2 i ,y i) represents the cross loss value of the i-th image after the text block interacts with the text, p1 i represents the prediction score of the i-th image after two linear layers, p2 i Represents the prediction score obtained by constructing the multi-head attention mechanism with the i-th text feature set.

[0109] In this embodiment, in step S4, the feature vector to be enhanced determined based on the matching degree of the shallow features of the image and the target text features is enhanced to generate the target enhanced features, including:

[0110] Calculating the matching degree between each shallow feature vector of the image and the target text feature in the shallow feature of the image;

[0111] The shallow feature vector of the image having a matching degree higher than a preset matching degree is determined as the feature vector to be enhanced;

[0112] The feature vector to be enhanced is subjected to maximum and minimum normalization processing, and the processed feature vector to be enhanced is combined with the shallow feature vector of the image to generate a target enhancement feature.

[0113] Specifically, the similarity is calculated by using the vector of each position point of the target text feature and the shallow feature of the image, which can reflect how much information of this type is contained in the position point. The selected similar vectors are filtered, and the elements greater than the mean of the similar vectors are retained, and the elements less than the mean of the similar vectors are set to zero. Then, the maximum and minimum factors are obtained, and the factors are multiplied by the shallow features of the image to obtain the target enhanced features. The target enhanced features can then be input into the subsequent model network for prediction, further improving the accuracy of image classification.

[0114] Embodiment 2:

[0115] See also Figure 2 , Figure 2 The network flow diagram of the feature enhancement method based on text-guided mining of the present invention is as follows: Figure 2 The feature enhancements of the present invention are described in detail.

[0116] Step 1: In the pre-training stage (i.e., Pre-training), a shared classifier is used to classify the corresponding modal features extracted by the dual encoders. The text is generated by large language models such as Chat GPT 3.5. The specific process is as follows:

[0117] Step 11: Convert the image to x i ∈X and the text l corresponding to the image i∈L is input into their respective encoders (visual encoder VE (i.e. Visual Encoder) and text encoder TE (i.e. Textual Encoder)), and classified using the same classifier FC, so that the same type of two-modal features are divided into this category, ensuring the consistency of the two-modal feature attributes in the same feature space.

[0118]

[0119] Among them, B is the batch size, P X Represents the prediction score of the image, P L represents the prediction score of the text, fi represents the features of the i-th image, VE(xi) represents the process of the i-th image xi passing through the image encoder, ti represents the features of the text corresponding to the i-th image, LE(li) represents the process of the i-th text li passing through the text encoder, and p xi represents the prediction score of image i, p li represents the prediction score of text i, F represents multiple image feature sets, T represents multiple text feature sets, FC(fi) represents the process of image feature i passing through the classifier, and FC(ti) represents the process of text feature i passing through the classifier.

[0120] Step 12: Considering that the loss value of the tail class samples is usually large, but due to the small sample size, the impact on the overall loss is small, so the average loss is set to balance the impact of the loss value of each category on the overall loss. The process is as follows Figure 1 L pre As shown, the calculation formula is as follows:

[0121]

[0122] Among them, L pre represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(pl i ,y i ) represents the cross loss value of the i-th text, CE(p xi ,y i ) represents the cross loss value of the i-th image. Loss per Batch represents the loss of each sample, Loss per Modality represents the loss of each modality, Loss per Class represents the loss of each category, Loss perSample represents the loss of each sample, avg represents the average operation, Class1 represents the first class, and ClassM represents the Mth class.

[0123] See also Figure 3 , Figure 3 A schematic diagram showing the loss function comparison of the feature enhancement method based on text-guided mining of the present invention.

[0124] Comparing the three loss functions, the average loss (L pre ), cross entropy loss (CE), and weighted CE. The highest accuracy was obtained using the average loss function. Focusing on the categories with accuracy differences exceeding 0.1, the average loss demonstrated the effectiveness of minority class performance improvement by improving the accuracy of 2 out of 3 tail / middle class categories compared to CE. Meanwhile, for the head class, only one category showed a significant difference with both the average loss and weighted CE. The average loss function achieved higher accuracy, indicating minimal impact on the performance of the head class.

[0125] Among them, average loss(ours) represents the average loss function, crossEntropy loss represents the cross loss function, weighted crossentropy loss represents the weighted cross loss, tail class represents the tail class, head class represents the head class, and middle class represents the middle class.

[0126] Step 2: After pre-training, the fine-tuning phase begins. In the fine-tuning phase, high-quality text features are first selected for each category. The text processing process is as follows: Figure 1 The specific steps are as follows:

[0127] Step 21: Use the Representative Feature Collection module to filter out high-quality text for each category of text. The following is a detailed explanation of the quantity and quality of the text assigned to each category.

[0128] (1) A hierarchical strategy is used to deal with the long-tail problem: the tail class (i.e., Tail class) (containing ≤ 20 images) is assigned 6 sentences; the head class (i.e., Head class) (containing ≥ 100 images) is assigned 1 sentence; the middle class (i.e., Middle class) (containing 20 to 100 images) is assigned 1 sentence.

[0129] (2) High-quality text features: By using a pre-trained classifier to predict the prediction score of the text features extracted from TE, if the prediction score of the correct category is significantly higher than the prediction score of any wrong category, the text feature is considered to be of high quality. An example is provided below to describe this process in detail. Let the first category be the intermediate category, whose text set L contains a total of K texts, that is, |L| = K. We need to select 4 high-quality sentences from them. For each sentence l j ∈L 1 , input it into the TE and classifier trained in the pre-training phase to obtain the text feature t j and prediction scores The fraction of correct categories (i.e. ) and the maximum score of the error category (i.e. ) is calculated as follows:

[0130]

[0131] Represents its text feature t j The score when the prediction is correct after passing the classifier, Indicates the score when the prediction is wrong, and the difference between the two scores is g j , b is a hyperparameter, and N represents the number of categories in the dataset. We divide the text in L according to g j Sort and select the texts corresponding to the top 4 highest scores for the next calculation.

[0132] The text feature processing process for other classes is the same as above. In this way, the text feature set corresponding to all classes is recorded as T f , and then calculate the text mean of each category to form a set T m .

[0133] Among them, Data Augmentation represents the data enhancement process, Layer0 represents the 0th layer, and other layer1, layer2, and layer3 represent the corresponding number of layers. Attentionpool represents the attention pooling layer, Textblock represents the text block, L_fine represents the fine-tuning loss function, TGS (Text-Guided Selection Module) represents the text selection module, TVD (Textual Vector Database) represents the text vector database, JFE (joint feature enhancement) represents the joint feature enhancement module, Shared params represents the shared parameters, Extractive Textual Features represents the extracted text features, TVD (Textual Vector Database) represents the text vector database, RFC (Representative Feature Collection) represents the representative feature selection module, Textual Prototype represents the text prototype, Optimal Textual Features represents the optimal text features, 1-st class (middle class) has L textual features, which means that the first class (middle class) has L text features, C represents the number of channels, N represents the total number of categories, b represents the hyperparameter, Logits represents the score, Gaps represents the gap, and Sort represents selection.

[0134] Step 3: The image processing process is as follows Figure 1 As shown in the Image Process process in Figure 1, the process includes two propagations. The first propagation aims to select appropriate text for each image. The specific steps are as follows:

[0135] Step 31: First propagation (i.e. 1 st Backpropagation) selects appropriate text for each image, which is actually a process of predicting the class to which the image belongs. We use the "Text Block" module to predict the class for each image.

[0136] First, the image encoder is used to extract the features of the image, and then it is sent to the "Text Block" module for prediction. The calculation process of this module is as follows: Figure 1 The calculation formula is as follows:

[0137] Q=Linear(LayerNorm(F 3 )),V=T f,

[0138] K=Linear(LayerNorm(T f )),

[0139] P2= <F 3 ,G> / τ,P1=MLP(F 3 ),

[0140] Among them, F 3 represents the image features, T f is the text feature constructed in step 21, and the two are subjected to a multi-head attention mechanism to obtain a score P2, where F 3 Another prediction score P1 is obtained through two linear layers. Q represents the query vector obtained by the linear layer for the image feature F3. K represents the key vector obtained by the linear layer for the text feature. V represents the value vector, i.e., the text feature. G represents the attention vector. K T represents the transpose of the value vector, represents the adjustment factor, and τ represents the temperature factor, i.e., the hyperparameter.

[0141] Step 32: Use the predicted results to select the appropriate text. The calculation formula for the selection process is as follows:

[0142]

[0143] Where a is a hyperparameter, T m is the text prototype set obtained in step 21, Represents the prediction score of image feature F3.

[0144] Step 33: P1 and P2 simultaneously calculate the loss function to perform back propagation, giving penalties to misjudged categories for better prediction. The loss function is still the average loss:

[0145]

[0146] The meanings of the relevant parameters are the same as L pre .

[0147] Among them, Concat means concatenation, and a means a hyperparameter.

[0148] Step 4: The second forward propagation aims to use the enhanced features for further classification. The enhanced module is the Joint Feature Enhancement module. The process is as follows: Figure 1 The specific steps of the process are as follows:

[0149] Step 41: For sample i, its shallow feature f 2 ∈F2 Each position point and its matching text feature Find the similarity. If the similarity is high, the point is retained, otherwise it is set to zero, and then the maximum and minimum normalization is performed to obtain the final x f . f Combined with the original features, the enhanced features are obtained. The calculation process of the whole process is as follows:

[0150]

[0151] a∈H,b∈W

[0152] Among them, H and W refer to the characteristics f 2 Height and width, Represents the feature point at the (a, b) position, x m represents the image-text similarity graph, The vector representing the i-th position point of the image feature f2, F 2 Represents the image features output by the second layer of the image encoder, i.e. layer2.

[0153]

[0154] in, Represents the enhanced features.

[0155] The enhanced features are sent to the next network for subsequent prediction and loss calculation, where the loss is still L fine .

[0156] Among them, Negative point represents negative corresponding points, positive point represents positive corresponding points, filtering represents filtering, and min-max normalization represents minimum and maximum normalization.

[0157] See also Figure 4 , Figure 4 Schematic diagram of the heat map before and after feature enhancement of the feature enhancement method based on text-guided mining of the present invention.

[0158] Use EigenCAM to activate the feature map. 2 The features and The features after the joint feature mining module. (a) The chimney position of some car carriers is different from that of other ships. It is located in the middle of the ship to facilitate vehicle loading and unloading. Enhanced features The chimney can be accurately identified, but the focus of the unenhanced feature is biased. (b) A medical ship is distinguished from other ships by the red cross symbol. After enhancement, The red cross can be focused accurately, and the unenhanced F 2 The region of interest is loose and inaccurate, where input represents the input.

[0159] The feature enhancement method based on text-guided mining provided by the present invention can enhance the features of detail areas in remote sensing ship images during ship image classification and recognition through a text-guided attention mechanism, making the slight differences between ships of different models more obvious, and further improving the accuracy of subsequent image classification; and the present invention adopts a representative feature set module to mine more useful information from long-tail data, reduce the negative impact of uneven data distribution on model training, and improve the performance of the model on long-tail categories.

[0160] Embodiment 3:

[0161] The present invention based on the same inventive concept also provides a feature enhancement system based on text-guided mining, such as Figure 5 As shown, including:

[0162] A classification module, configured to extract multimodal features from the received sample image, generate multimodal features, and classify the multimodal features based on a classifier, wherein the multimodal features include image features and text features;

[0163] A first determination module is used to determine a text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated after the classifier;

[0164] A second determination module is used to determine the target text feature of each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set;

[0165] The generation module is used to obtain the shallow features of the image corresponding to each sample image, and perform feature enhancement on the feature vector to be enhanced determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

[0166] Preferably, the classification module is further used for:

[0167] Inputting the received sample image into a feature extraction model to generate multimodal features, wherein the multimodal features include image features and text features;

[0168] The image features and text features corresponding to the sample image are classified by the same classifier.

[0169] Preferably, the first determining module is further used to:

[0170] Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score;

[0171] Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category;

[0172] The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

[0173] Preferably, the calculation formula of the difference in the first determination module is as follows:

[0174]

[0175] Among them, g j represents the difference of the jth category, b is a hyperparameter, Represents its text feature t j The score when the prediction is correct after passing the classifier, It represents the score when the prediction is wrong, and N represents the number of categories in the data set.

[0176] Preferably, the second determining module is further used to:

[0177] Constructing a multi-head attention mechanism based on the prediction score corresponding to the image feature and the text feature set to generate an initial prediction score;

[0178] The prediction score corresponding to the image feature and the initial prediction score are added according to a preset ratio and then multiplied by the mean of each type of text feature set to generate a target text feature that matches each image in the text feature set;

[0179] Based on the prediction scores corresponding to the image features and the initial prediction scores, the average loss is calculated and back-propagated to optimize the parameters of the classifier.

[0180] Preferably, the determination of the target text feature of each image match in the second determination module is as shown in the following formula, including:

[0181] T p =σ(a*P1+(1-a)*P2)*T m

[0182] Among them, T p represents the target text features, T mRepresents the mean of each type of text feature set, P1 represents the prediction score corresponding to the image feature, P2 represents the initial prediction score generated by constructing a multi-head attention mechanism with P1 and the text feature set, a represents a hyperparameter, and σ represents the standard deviation function.

[0183] Preferably, the second determination module calculates the average loss function to optimize the classifier parameters as shown in the following formula:

[0184]

[0185] Among them, L fine represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(p1 i ,y i ) represents the cross loss value of the i-th text, CE(p2 i ,y i ) represents the cross loss value of the i-th image.

[0186] Preferably, the generating module is further used for:

[0187] Calculating the matching degree between each shallow feature vector of the image and the target text feature in the shallow feature of the image;

[0188] The shallow feature vector of the image having a matching degree higher than a preset matching degree is determined as the feature vector to be enhanced;

[0189] The feature vector to be enhanced is subjected to maximum and minimum normalization processing, and the processed feature vector to be enhanced is combined with the shallow feature vector of the image to generate a target enhancement feature.

[0190] Embodiment 4:

[0191] like Figure 6 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0192] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of a feature enhancement method based on text-guided mining in the above-mentioned embodiment.

[0193] Embodiment 5:

[0194] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a feature enhancement method based on text-guided mining in the above embodiment.

[0195] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0196] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0197] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0198] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0199] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A feature enhancement method based on text-guided mining, characterized in that: include: Extracting multimodal features from the received sample image to generate multimodal features, and classifying the multimodal features based on a classifier, wherein the multimodal features include image features and text features; Determine the text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier; Determine the target text feature for each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set; The shallow features of the image corresponding to each sample image are obtained, and feature enhancement is performed on the feature vector to be enhanced which is determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

2. The method according to claim 1, characterized in that Extracting multimodal features from the received sample image to generate multimodal features, and classifying the multimodal features based on a classifier, including: Inputting the received sample image into a feature extraction model to generate multimodal features, wherein the multimodal features include image features and text features; The image features and text features corresponding to the sample image are classified by the same classifier.

3. The method according to claim 1, characterized in that The method of determining the text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated by the classifier includes: Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score; Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category; The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

4. The method according to claim 3, characterized in that The calculation formula of the difference is as follows: Among them, g j represents the difference of the jth category, b is a hyperparameter, Represents its text feature t j The score when the prediction is correct after passing the classifier, It represents the score when the prediction is wrong, and N represents the number of categories in the data set.

5. The method according to claim 1, characterized in that The step of determining the target text feature for each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set includes: Constructing a multi-head attention mechanism based on the prediction score corresponding to the image feature and the text feature set to generate an initial prediction score; The prediction score corresponding to the image feature and the initial prediction score are added according to a preset ratio and then multiplied by the mean of each type of text feature set to generate a target text feature that matches each image in the text feature set; Based on the prediction scores corresponding to the image features and the initial prediction scores, the average loss is calculated and back-propagated to optimize the parameters of the classifier.

6. The method according to claim 5, characterized in that The determination of the target text features of each image match is shown in the following formula, including: T p =σ(a*P1+(1-a)*P2)*T m Among them, T p represents the target text features, T m Represents the mean of each type of text feature set, P1 represents the prediction score corresponding to the image feature, P2 represents the initial prediction score generated by constructing a multi-head attention mechanism with P1 and the text feature set, a represents a hyperparameter, and σ represents the standard deviation function.

7. The method according to claim 5, characterized in that The calculation of the average loss function to optimize the classifier parameters is shown in the following formula: Among them, L fine represents the average loss function, M represents the number of categories contained in the class, class_num[k] represents the number of samples of the kth class in the class, and y i represents the label of the i-th sample, CE(p1 i ,y i ) represents the cross loss value of the i-th text, CE(p2 i ,y i ) represents the cross loss value of the i-th image.

8. The method according to claim 1, characterized in that The step of performing feature enhancement on the feature vector to be enhanced determined based on the matching degree between the shallow features of the image and the target text features to generate target enhanced features includes: Calculating the matching degree between each shallow feature vector of the image and the target text feature in the shallow feature of the image; The shallow feature vector of the image having a matching degree higher than a preset matching degree is determined as the feature vector to be enhanced; The feature vector to be enhanced is subjected to maximum and minimum normalization processing, and the processed feature vector to be enhanced is combined with the shallow feature vector of the image to generate a target enhancement feature.

9. A feature enhancement system based on text-guided mining, characterized in that: include: A classification module, configured to extract multimodal features from the received sample image, generate multimodal features, and classify the multimodal features based on a classifier, wherein the multimodal features include image features and text features; A first determination module is used to determine a text feature set corresponding to each category based on the classification result and the prediction score of the text feature corresponding to each category generated after the classifier; A second determination module is used to determine the target text feature of each image match based on the prediction score corresponding to the image feature generated by the classifier and the text feature set; The generation module is used to obtain the shallow features of the image corresponding to each sample image, and perform feature enhancement on the feature vector to be enhanced determined based on the matching degree of the shallow features of the image and the target text features to generate target enhanced features.

10. The system according to claim 9, characterized in that The first determining module is further used to: Determine the text features corresponding to each category generated by the classifier based on the classification result, and determine the prediction score of the text features corresponding to each category based on the text features corresponding to each category and the sample labels corresponding to each category, wherein the prediction score includes a correct prediction score and an incorrect prediction score; Calculate the difference between the correct prediction score and the wrong prediction score of the text feature corresponding to each category; The text features whose difference is greater than a preset difference condition are determined as a text feature set corresponding to each category.

Images