A specific picture recognition method based on multi-modal

Abstract

The application discloses a specific picture recognition method based on multi-modal, relates to the field of specific picture recognition, and comprises two parts of feature information extraction of pictures and texts and a specific picture matching strategy of FH-LANet, adopts an image matching idea to extract features of pictures through a multi-modal feature extraction network FH-LANet model, selects a specific picture as a query library picture, the query feature is not a single modal picture feature, but a text feature, similarity calculation is carried out by using the features of the input picture and the query feature, threshold judgment is then carried out, the matched picture and filtered text are matched again, noise data is filtered out, the picture that does not match the query picture is secondarily matched with the query text, and finally, the recognition result of the picture is obtained, and the specific picture in the mass pictures can be accurately found out.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of certain picture recognition, in particular to a certain picture recognition method based on multi-modal. BACKGROUND

[0002] With the development of the Internet, more and more pictures are generated on the Internet, and different types of picture data are mixed together, which is inconvenient for use and viewing, and it is necessary to classify and recognize the pictures to find the desired pictures in the massive data.

[0003] The traditional manual searching method is time-consuming and high in cost, and the existing certain picture recognition method is mainly single modal, that is, only the picture features are used, the recognition result is relatively single, and it is mainly concentrated on a certain picture with high characteristic, and the solution is to classify the pictures, each type of picture is taken as a classification category, when a new category needs to be recognized, data needs to be collected, and then the model is trained and iterated.

[0004] The existing certain picture recognition method recognizes each type of certain picture, which is relatively single, and the effect is poor for the large differences in certain picture categories, and many pictures cannot be recognized, if you want to recognize it, you must do fine classification, which is a large amount of work, when a new certain category is encountered, data must be collected, the model must be trained, and then the model is updated, the process is complex and the cost is high. SUMMARY

[0005] The technical problem to be solved by the present application is to more accurately recognize certain pictures and meet the requirements of rapid application and low cost, and to provide a certain picture recognition method based on multi-modal, which can accurately find certain pictures in massive pictures.

[0006] The present application adopts the following technical solutions to solve the above technical problems:

[0007] A certain picture recognition method based on multi-modal includes two parts:

[0008] (I) extracting the feature information of the picture and the text;

[0009] (ii) certain picture matching strategy;

[0010] Specifically, the following steps are included:

[0011] Step 1, extracting features from the picture through a multi-modal feature extraction network FH-LANet model; wherein the multi-modal feature extraction network FH-LANet model includes an image encoder and a text encoder, and specifically as follows:

[0012] Step 1.1, collect picture-text pairs, and the text is a description of the image content;

[0013] Step 1.2, convert the picture and text into a sequence by image blocking and word vectorization of the text;

[0014] Step 1.3, select an image encoder and a text encoder to forward propagate the image and the text to extract features, and use a contrastive learning loss function to backward propagate to train the multi-modal image-text model;

[0015] Step 2, calculate the similarity using the features of the input picture and the query features, then perform threshold judgment, match the filtered pictures and filtered texts, filter out noise data, and perform secondary matching with the query text for pictures that do not match the query picture, to obtain the recognition result of the picture, wherein the cosine similarity is calculated as follows:

[0016]

[0017] Where A and B represent two N-dimensional vectors.

[0018] As a further preferred scheme of the specific picture recognition method based on multi-modal, in step 2, steps 2.1 to 2.7 are performed to obtain the specific picture recognition result:

[0019] Step 2.1, pre-process the query picture and the filtered picture, crop to the required size of the model, and then normalize;

[0020] Step 2.2, extract features from the query text and the filtered text, fuse the query picture features and the query text features as the final query features, and fuse the filtered picture features and the corresponding text features as the filtered features;

[0021] Step 2.3, pre-process the input picture, crop to the required size of the model, and then normalize;

[0022] Step 2.4, extract features from the input picture, and then calculate the cosine similarity with the query features;

[0023] Step 2.5, threshold judgment, calculate the cosine similarity with the filtered features if greater than the threshold, and calculate the cosine similarity with the query text if less than the threshold;

[0024] Step 2.6, calculate the cosine similarity with the filtered text again after calculating the similarity with the filtered picture if less than the threshold;

[0025] Step 2.7, finally obtain the recognition result.

[0026] As a further preferred scheme of the specific picture recognition method based on multi-modal, in step 1.3, an image encoder is used to extract features from the picture;

[0027] The image encoder is a picture feature extractor of the FH-LANet, and is composed of an encoder module containing a multi-head attention mechanism, and the encoder module is composed of a self-attention layer and a feedforward neural network;

[0028] The self-attention layer is obtained by multiplying the same input using different parameter matrices, and the output values are named Q, K and V, Q can be understood as attention weight, K attention weighting, and V can be understood as a word vector, then the dot product of Q and K is Q*K T , the vector similarity is calculated, the probability is converted into a probability distribution, the probability distribution and the value of V are weighted and summed, the self-attention mechanism can pay attention to the correlation between different positions of the picture, the feedforward neural network layer is composed of two fully connected layers, a normalization layer and a random inactivation layer, and is used to reduce overfitting, accelerate convergence and add nonlinearity.

[0029] As a further preferred scheme of the specific picture recognition method based on multi-modal in the application, in step 1.3, the text encoder is a text feature extractor of the FH-LANet, and is composed of an encoder module containing a multi-head attention mechanism, and the basic structure is the same as that of the picture encoding, but the number of the encoder modules is different, and the type of the input is different; the text encoder is composed of a multi-head attention mechanism module.

[0030] As a further preferred scheme of the specific picture recognition method based on multi-modal in the application, in step 1.3, the contrast learning loss function is specifically as follows:

[0031]

[0032] Wherein xi, yi represents the features of the image and the text, N is the size of the batch processing, alpha is the temperature coefficient, and T is the matrix transpose.

[0033] Compared with the prior art, the application has the following technical effects:

[0034] The application extracts features of pictures through the FH-LANet model independently researched by the application, and the recognition method is not the idea of picture classification, but the idea of image matching, a specific picture is selected as a query library picture, the query feature is not a single modal picture feature, but a text feature is fused, similarity calculation is carried out by using the features of the input picture and the query feature, then threshold judgment is carried out, the matched and filtered pictures, filtered text are matched again, noise data is filtered out, the picture without matching with the query picture is matched with the query text for the second time, and finally the recognition result of the picture is obtained, which can accurately find a specific picture in a large number of pictures; the method of fusing picture features and text features is proposed to recognize, the FH-LANet is a model based on a deep neural network independently researched by the application, which is composed of an image encoder and a text encoder, the picture encoder extracts features of pictures, the text encoder extracts features of text, the model is optimized through pairing of images and text, so that the model can obtain more internal semantic information of pictures. BRIEF DESCRIPTION OF DRAWINGS

[0035] The accompanying drawings are included to provide a further understanding of the application, and constitute a part of the specification, illustrate the application together with the embodiments of the application, and do not constitute a limitation on the application.

[0036] Figure 1 is the overall flowchart of the specific picture recognition method based on the multi-modal of the application;

[0037] Figure 2 is the structure diagram of the multi-modal feature extraction network FH-LANet model of the application. DETAILED DESCRIPTION

[0038] The technical solutions of the application will be further described in detail below with reference to the drawings:

[0039] The application will be further supplemented by combining the drawings and specific embodiments, and it should be understood that the specific embodiments described here are only used to explain the application, and do not limit the application. Those skilled in the art should understand that various substitutions and changes can be made to the application according to the basic technical knowledge and means in the art without departing from the technical idea of the application, and all should be included in the protection scope of the application.

[0040] In order to more accurately identify a specific picture and meet the requirements of rapid application and low cost, we have created a specific picture recognition method based on multi-modal, and innovatively proposed a picture feature fusion text feature method to identify. FH-LANet is a model based on deep neural network independently developed by Fenghuo, which is composed of an image encoder and a text encoder. The image encoder extracts features from the picture, and the text encoder extracts features from the text. By pairing images and texts, the model can obtain more internal semantic information of the picture. Because there are a lot of noise data in the massive data, Fenghuo has also independently developed a matching strategy to reduce the influence of noise data on the control

[0041] FH-LANet is a specific picture recognition model based on multi-modal independently developed by Fenghuo. It can deeply understand the semantic information inside the picture by pairing images and texts. In order to reduce the interference of noise data in massive data, Fenghuo has also independently developed a matching strategy.

[0042] A specific picture recognition method based on multi-modal contains two parts:

[0043] (1) Extracting feature information of pictures and texts;

[0044] (2) Specific picture matching strategy;

[0045] Specifically, the following steps are included:

[0046] Step 1, extract features from the picture by multi-modal feature extraction network FH-LANet model; wherein the multi-modal feature extraction network FH-LANet model contains an image encoder and a text encoder, specifically as follows:

[0047] Step 1.1, collect image-text pairs of picture and text pairs, and the text is a description of the image content;

[0048] Step 1.2, convert the picture and text into a sequence by image blocking and text word vectorization;

[0049] Step 1.3, select the image encoder and the text encoder to forward propagate the image and the text to extract features, and use the contrast learning loss function to back propagate to train the multi-modal image-text model;

[0050] Step 2, calculate the similarity between the features of the input picture and the query features, then perform threshold judgment, match the filtered pictures and filtered texts, filter out noise data, and perform secondary matching with the query text, and obtain the recognition result of the picture.

[0051] Wherein, the cosine similarity is calculated as follows:

[0052]

[0053] wherein A, B are two N-dimensional vectors.

[0054] In step 2, steps 2.1 to 2.7 are performed to obtain the specific picture recognition result:

[0055] Step 2.1, pre-process the query picture and the filtered picture, crop to the size required by the model, and then normalize the processing;

[0056] Step 2.2, extract features from the query text and the filtered text, fuse the query picture features and the query text features as the final query features, and fuse the filtered picture features and the corresponding text features as the filtered features;

[0057] Step 2.3, pre-process the input picture, crop to the size required by the model, and then normalize the processing;

[0058] Step 2.4, extract features from the input picture, and then calculate the cosine similarity with the query features;

[0059] Step 2.5, threshold judgment, greater than the threshold value and the filtered features are calculated for cosine similarity, less than the threshold value and the query text are calculated for cosine similarity;

[0060] Step 2.6, after calculating the cosine similarity with the filtered picture, less than the threshold value is calculated for cosine similarity with the filtered text;

[0061] Step 2.7, finally obtain the recognition result.

[0062] The specific implementation is as follows; the overall steps of specific picture recognition:

[0063] The specific picture recognition steps are as shown in Figure 1 , which are mainly divided into two parts: FH-LANet feature information extraction of pictures and texts and specific picture matching strategy.

[0064] As shown in Figure 1 , we first extract features from the picture by FH-LANet model, the recognition method is not a picture classification idea, but an image matching idea, select a specific picture as a query library picture, the query feature is not a single modal picture feature, but a fusion of text features, use the input picture features and the query features to calculate the similarity, then perform threshold judgment, match the filtered picture and the filtered text, filter out noise data, and match the query picture with the query text for the second time, and finally obtain the picture recognition result.

[0065] Multimodal feature extraction network: FH-LANet

[0066] The FH-LANet mainly includes two parts: an image encoder and a text encoder, and the network structure is as shown in Figure 2

[0067] Figure 2 Structure diagram of the multimodal feature extraction network FH-LANet;

[0068] The image encoder is a picture feature extractor of the FH-LANet, which is composed of an encoder module containing a multi-head attention mechanism, and the module is mainly composed of a self-attention layer and a feedforward neural network.

[0069] The self-attention layer is obtained by multiplying the same input with different parameter matrices to obtain the output values named Q, K and V. Q can be understood as attention weight, K attention weighting, and V can be understood as a word vector. Then, the values of Q and K are point multiplied, i.e. Q*K T , the vector similarity is calculated, the probability distribution is converted by using probability, and the weighted sum of the probability distribution and the value of V is calculated. The self-attention mechanism can pay attention to the correlation between different positions of the picture. The feedforward neural network layer is composed of two fully connected layers, a normalization layer and a random inactivation layer, which is used to reduce overfitting, speed up convergence and add nonlinearity.

[0070] The text encoder is a text feature extractor of the FH-LANet, which is also composed of an encoder module containing a multi-head attention mechanism. The basic structure is the same as that of the picture encoding, and the difference lies in that the number of encoder modules is different, and the type of input is different. The text encoder uses a multi-head attention mechanism module, which has a great improvement in parallel computing compared with the traditional RNN network, and the correlation between words in the text is very strong.

[0071] Contrastive learning loss function: first, a large number of paired images and text pairs are collected, and then the FH-LANet model independently developed by the author is used to extract features from the images and texts, and the model is optimized by using contrastive learning, i.e. the similarity between the paired pictures and texts is maximized, and the similarity between the unpaired image-text pairs is smaller. The contrastive learning loss is as shown in

[0072]

[0073] wherein xi and yi represent the features of the images and texts, N is the size of the batch processing, a is the temperature coefficient, and T is the matrix transpose.

[0074] ​Matching strategy: the simple method of matching strategy is picture and control matching, which is correct in recognition, but we do not use this method, but redesign a new method, which can significantly improve the accuracy. First, in the selection of specific pictures, for each category of specific picture selection multiple pictures as query picture, as long as the picture and the specific category of picture has one of the similarity over the threshold is considered similar. Because of the mass of pictures, there are a lot of noise data, so the design of the filter link, by filtering pictures, filtering text filtering, also using the query text to further improve the recall rate.

[0075] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and unless defined as such, should not be interpreted in an idealized or overly formal sense.

[0076] The above examples are only to illustrate the technical idea of the present application, and cannot limit the protection scope of the present application. Any modification made according to the technical idea of the present application on the basis of technical solutions falls within the protection scope of the present application. The embodiments of the present application are described in detail above, but the present application is not limited to the above-mentioned embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the purpose of the present application.

Claims

1. A method for specific image recognition based on multimodality, characterized in that: It contains two parts: (I) Extraction of feature information from images and text; (II) Specific image matching strategies; Specifically, it includes the following steps: Step 1: Extract features from the image using the FH-LANet multimodal feature extraction network model. The FH-LANet model comprises an image encoder and a text encoder, as detailed below: Step 1.1: Collect image-text pairs, where the text is a description of the image content; Step 1.2: Convert images and text into serialized data through image segmentation and text word vectorization; Step 1.3: Select an image encoder and a text encoder to perform forward propagation to extract features from the image and text, and use the contrastive learning loss function for backpropagation to train a multimodal image-text model. Step 2: Calculate cosine similarity using the features of the input image and the query features, then perform a threshold judgment. Matching images and filtered text are then matched again to remove noise data. Images that do not match the query image are matched a second time with the query text to obtain the image recognition result. The specific calculation method for cosine similarity is as follows: Here, A and B represent two N-dimensional vectors.

2. The method for specific image recognition based on multimodality according to claim 1, characterized in that: In step 2, steps 2.1 to 2.7 are executed to obtain the recognition results for a specific image: Step 2.1: Perform preprocessing on the query images and filtered images, cropping them to the size required by the model, and then normalizing them; Step 2.2: Extract features from query text and filter text, fuse query image features and query text features to obtain the final query features, and fuse filter image features and corresponding text features to obtain the filter features. Step 2.3: Preprocess the input image by cropping it to the size required by the model, and then normalize it. Step 2.4: Extract features from the input image and then calculate the cosine similarity with the query features; Step 2.5: Perform threshold judgment. For features greater than the threshold, calculate cosine similarity with the filtered features. For features less than the threshold, calculate cosine similarity with the query text. Step 2.6: After calculating the similarity with the filtered images, calculate the cosine similarity between the images with a similarity less than the threshold and the filtered text. Step 2.7, finally the recognition result is obtained.

3. The method for specific image recognition based on multimodality according to claim 1, characterized in that: In step 1.3, an image encoder is used to extract features from the image; The image encoder is an image feature extractor of FH-LANet, which consists of an encoder module containing a multi-head attention mechanism. The encoder module consists of a self-attention layer and a feedforward neural network. A self-attention layer multiplies different parameter matrices with the same input to obtain output values ​​named Q, K, and V. Q can be understood as attention weights, K as attention weighting, and V as word vectors. Then, the values ​​of Q and K are multiplied by a dot product, i.e., Q*K. T The algorithm calculates vector similarity by converting probability into probability distribution, and then weights and sums the probability distribution and the value of V. The self-attention mechanism can focus on the correlation between different positions in the image. The feedforward neural network layer consists of two fully connected layers, a normalization layer, and a random deactivation layer, which are used to reduce overfitting, speed up convergence, and add nonlinearity.

4. The method for specific image recognition based on multimodality according to claim 1, characterized in that: In step 1.3, the text encoder is an FH-LANet text feature extractor, which consists of encoder modules with a multi-head attention mechanism. Its basic structure is the same as that of image encoding, except that the number of encoder modules and the type of input are different. The text encoder is composed of modules with a multi-head attention mechanism.

5. The method for specific image recognition based on multimodality according to claim 1, characterized in that: In step 1.3, the contrastive learning loss function is as follows: Where xi and yi represent the features of the image and text, N is the batch size, α is the temperature coefficient, and T is the matrix transpose.

Images