A method for matching images and texts, a large multimodal model for matching images and texts, and a large model training method

Abstract

The present invention discloses a method for matching images and texts, a large multimodal model for matching images and texts, and a large model training method. The matching method includes feature extraction and feature determination. The feature extraction includes: S1, encoding an input image to obtain an initial image feature; S2, encoding an input text to obtain an initial text feature; S3, adding a different position code to the initial image feature and the initial text feature to obtain a secondary image feature and a secondary text feature; S4, fusing the secondary image feature and the secondary text feature obtained in step S3 to obtain a fused text feature I. T And fusion image features T I ; Feature determination includes: S5, integrating text feature I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated, and the discriminant linear layer outputs 0 to 1 to indicate the degree of matching. The present invention can simultaneously extract and deeply fuse visual and language features.

Description

Technical Field

[0001] The present invention relates to the field of computer vision processing, and in particular to a picture-text matching method, a picture-text matching multimodal large model and a large model training method. Background Art

[0002] Currently, the field of multimodal processing of vision and language generally faces the challenge of feature extraction and fusion. Traditional methods mostly use separate visual and language feature extractors, which not only increases the complexity of the model, but also makes it difficult to effectively capture the deep correlation between cross-modalities. In addition, the feature extraction and encoding processes are often performed independently, resulting in information loss during the transmission process, which limits the depth and accuracy of multimodal understanding. Therefore, there is an urgent need for an innovative architecture that can efficiently integrate visual and language features and achieve tighter cross-modal fusion to improve the processing effect of multimodal tasks. Summary of the invention

[0003] The present invention mainly aims to solve the technical problems existing in the prior art, such as high model complexity, difficulty in effectively capturing deep cross-modal correlations, and easy loss of information during transmission, and provides a picture-text matching method that efficiently integrates visual and language features and achieves tighter cross-modal fusion, a picture-text matching multimodal large model, and a large model training method.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a method for image-text matching, based on a large multimodal model of image-text matching, including feature extraction and feature determination, wherein the feature extraction includes the following steps:

[0005] S1, encode the input image to obtain the initial image features;

[0006] S2, encode the input text to obtain initial text features;

[0007] S3, adding a position code to the initial image feature and the initial text feature respectively, the two position codes are different from each other, and the secondary image feature and the secondary text feature are obtained; the position code uses absolute position code, which can be obtained by sine and cosine position code or learnable position code;

[0008] S4: Fusing the secondary image features and secondary text features obtained in step S3 to obtain fused text features I T And fusion image features T I ;

[0009] The feature determination includes:

[0010] S5. Fusion text feature I T And fusion image features T IThe features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. The discriminant linear layer outputs 0 to 1 to indicate the degree of matching, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text.

[0011] Preferably, the step S4 is specifically as follows: inputting the secondary text features and the secondary image features into a multimodal feature fusion encoder composed of several layers of encoding modules, the output of the encoding module of the previous layer is used as the input of the encoding module of the next layer, and the processing process of the k-th layer encoding module is as follows:

[0012] S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector;

[0013] S42, the first query vector passes through a random drop layer, and then performs feature normalization, and then compares it with the original text feature I k Add them together to get the intermediate text features;

[0014] S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector;

[0015] S44, the second query vector passes through a random dropout layer, then performs feature normalization, and then compares it with the original image feature T k Add together to get the intermediate image features;

[0016] S45, the intermediate text features and the intermediate image features are respectively passed through a linear layer to obtain the encoded text features I k+1 and image features T k+1 ;

[0017] The image features and text features input to the first layer encoding module are the secondary image features and secondary text features obtained in step S3; the text features output by the last encoding layer are the fused text features I T , the image feature output by the last encoding layer is the fused image feature T I .

[0018] Preferably, the step S1 is specifically as follows:

[0019] S11, scaling the input image to 224 pixels × 224 pixels, and then dividing the scaled image into image blocks of 16 pixels × 16 pixels, to obtain 196 image blocks in total;

[0020] S12, stretching each 16 pixel × 16 pixel image block into a 16 × 16 × 3 = 768-dimensional feature according to the RGB value, and obtaining a total of 196 768-dimensional features;

[0021] S13, map the 196 768-dimensional features into a 196×256-dimensional image feature T through a linear layer; this is equivalent to multiplying a 196×768 matrix by a 768×256 matrix to obtain a 196×256 image feature.

[0022] Preferably, the step S5 is specifically as follows:

[0023] S51, fusion image feature T I And fusion text features I T After feature pooling, they are transformed into feature vectors with a dimension of 768;

[0024] S52, scaling the two feature vectors to 256 dimensions through a linear layer respectively;

[0025] S53, concatenate the two 256-dimensional features into a 512-dimensional feature;

[0026] S54, feature discrimination is performed on the 512-dimensional features, and a linear layer is used to output a value from 0 to 1 to indicate the degree of matching, where 0 represents that the image and text do not match at all, and 1 represents that the image and text match completely.

[0027] A multimodal large model for image-text matching, running the image-text matching method as described above, the large model includes a feature extractor and a feature discriminator, and the feature extractor includes:

[0028] Image encoder: After scaling the image to a specified size, it is divided into several 16×16 pixel image blocks. Then, each image block is stretched into a 16×16×3 dimension feature according to the RGB value. All features are mapped through a linear layer to obtain the initial image features.

[0029] Text encoder: Encode the input text to obtain the initial text features;

[0030] Position coding adding module: The initial image features and initial text features are respectively added with two different position codings to obtain secondary image features of 196×768 dimensions and secondary text features of 196×768 dimensions;

[0031] Multimodal feature fusion encoder: fuses the secondary image features and the secondary text features to obtain the fused image feature I T And fusion image features T I ;

[0032] The feature discriminator will fuse the image features I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. A linear layer is used to output a value from 0 to 1 to indicate the degree of match, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text.

[0033] Preferably, the multimodal feature fusion encoder includes several layers of encoding modules, the output of the encoding module of the previous layer is used as the input of the encoding module of the next layer, each encoding module includes a multi-head attention layer, a random drop layer, a normalization layer, a linear layer and an addition layer, and the processing process of the k-th layer encoding module is as follows:

[0034] S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector;

[0035] S42, the first query vector passes through a random discard layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine with the original text feature I k Add them together to get the intermediate text features;

[0036] S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector;

[0037] S44, the second query vector passes through a random drop layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine it with the original image feature T k Add together to get the intermediate image features;

[0038] S45, the intermediate text features and the intermediate image features are respectively passed through the linear layer to obtain the encoded text features I k+1 and image features T k+1 .

[0039] A method for training a multimodal large model for image-text matching is used to train the multimodal large model for image-text matching as described above. First, the training data set is completed, and the matching degree is calculated for each completed image-text pair, and the matching degree is stored as a label of the image-text pair; then the image-text pair is input into the image-text matching large model for training, and the loss function is a cross entropy loss, and the training process is completed by back propagation;

[0040] The completion process is as follows: if a certain training data in the training data set is pure text, a picture is randomly selected from the picture-text pair or pure picture training data to form a picture-text pair with the text; if a certain training data in the training data set is pure picture, a sentence of text is randomly selected from the picture-text pair or pure text training data to form a picture-text pair with the picture;

[0041] The matching degree calculation process is as follows:

[0042] Use the object recognition model to extract image descriptions from images;

[0043] Use a natural language processing model to extract verbs and nouns from image descriptions and image-text pairs, respectively;

[0044] The matching degree is calculated using the following formula:

[0045] ;

[0046] ;

[0047] Where N 交 is the number of intersections between the image description and the text of the image-text pair, N 并 is the union number of nouns between the image description and the text of the image-text pair, V 交 is the number of intersections between the image description and the verbs in the image-text pair text, V 并 is the union number of verbs between the image description and the text of the image-text pair; k is the coefficient, and R is the matching degree of the image-text pair.

[0048] Preferably, the k is 0.1-0.3.

[0049] The substantial effects brought by the present invention are: 1. Only one visual language feature extractor is needed to extract visual and language features simultaneously, without the need to use two feature extractors separately for extraction; 2. Feature extraction and feature encoding are integrated together to deeply integrate visual and language features. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flow chart of a picture-text matching method of the present invention. DETAILED DESCRIPTION

[0051] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0052] Embodiment: A method for image-text matching, based on a large multimodal model for image-text matching, such as Figure 1 As shown, the matching method includes feature extraction and feature determination, and the feature extraction includes the following steps:

[0053] S1, encode the input image to obtain the initial image features;

[0054] S2, encode the input text to obtain initial text features;

[0055] S3, adding a position code to the initial image feature and the initial text feature respectively, the two position codes are different from each other, and the secondary image feature and the secondary text feature are obtained; the position code uses absolute position code, which can be obtained by sine and cosine position code or learnable position code;

[0056] S4: Fusing the secondary image features and secondary text features obtained in step S3 to obtain fused text features I T And fusion image features T I ;

[0057] The feature determination includes:

[0058] S5. Fusion text feature I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. The discriminant linear layer outputs 0 to 1 to indicate the degree of matching, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text.

[0059] The step S4 is specifically as follows: inputting the secondary text features and the secondary image features into a multimodal feature fusion encoder composed of several layers of encoding modules, and the output of the encoding module of the previous layer is used as the input of the encoding module of the next layer. The processing process of the encoding module of the kth layer is as follows:

[0060] S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector;

[0061] S42, the first query vector passes through a random drop layer, and then performs feature normalization, and then compares it with the original text feature I k Add them together to get the intermediate text features;

[0062] S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector;

[0063] S44, the second query vector passes through a random dropout layer, then performs feature normalization, and then compares it with the original image feature T kAdd together to get the intermediate image features;

[0064] S45, the intermediate text features and the intermediate image features are respectively passed through a linear layer to obtain the encoded text features I k+1 and image features T k+1 ;

[0065] The image features and text features input to the first layer encoding module are the secondary image features and secondary text features obtained in step S3; the text features output by the last encoding layer are the fused text features I T , the image feature output by the last encoding layer is the fused image feature T I .

[0066] The step S1 is specifically as follows:

[0067] S11, scaling the input image to 224 pixels × 224 pixels, and then dividing the scaled image into image blocks of 16 pixels × 16 pixels, to obtain 196 image blocks in total;

[0068] S12, stretching each 16 pixel × 16 pixel image block into a 16 × 16 × 3 = 768-dimensional feature according to the RGB value, and obtaining a total of 196 768-dimensional features;

[0069] S13, map the 196 768-dimensional features into a 196×256-dimensional image feature T through a linear layer; this is equivalent to multiplying a 196×768 matrix by a 768×256 matrix to obtain a 196×256 image feature.

[0070] The step S5 is specifically as follows:

[0071] S51, fusion image feature T I And fusion text features I T After feature pooling, they are transformed into feature vectors with a dimension of 768;

[0072] S52, scaling the two feature vectors to 256 dimensions through a linear layer respectively;

[0073] S53, concatenate the two 256-dimensional features into a 512-dimensional feature;

[0074] S54, feature discrimination is performed on the 512-dimensional features, and a linear layer is used to output a value from 0 to 1 to indicate the degree of matching, where 0 represents that the image and text do not match at all, and 1 represents that the image and text match completely.

[0075] A multimodal large model for image-text matching, running the image-text matching method as described above, the large model includes a feature extractor and a feature discriminator, and the feature extractor includes:

[0076] Image encoder: After scaling the image to a specified size, it is divided into several 16×16 pixel image blocks. Then, each image block is stretched into a 16×16×3 dimension feature according to the RGB value. All features are mapped through a linear layer to obtain the initial image features.

[0077] Text encoder: Encode the input text to obtain the initial text features;

[0078] Position coding adding module: The initial image features and initial text features are respectively added with two different position codings to obtain secondary image features of 196×768 dimensions and secondary text features of 196×768 dimensions;

[0079] Multimodal feature fusion encoder: fuses the secondary image features and the secondary text features to obtain the fused image feature I T And fusion image features T I ;

[0080] The feature discriminator will fuse the image features I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. A linear layer is used to output a value from 0 to 1 to indicate the degree of match, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text.

[0081] The multimodal feature fusion encoder includes several layers of encoding modules. The output of the encoding module of the previous layer is used as the input of the encoding module of the next layer. Each encoding module includes a multi-head attention layer, a random drop layer, a normalization layer, a linear layer and an addition layer. The processing process of the k-th layer encoding module is as follows:

[0082] S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector;

[0083] S42, the first query vector passes through a random discard layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine with the original text feature I k Add them together to get the intermediate text features;

[0084] S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector;

[0085] S44, the second query vector passes through a random drop layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine it with the original image feature T k Add together to get the intermediate image features;

[0086] S45, the intermediate text features and the intermediate image features are respectively passed through the linear layer to obtain the encoded text features I k+1 and image features T k+1 .

[0087] A method for training a multimodal large model for image-text matching is used to train the multimodal large model for image-text matching as described above. First, the training data set is completed, and the matching degree is calculated for each completed image-text pair, and the matching degree is stored as a label of the image-text pair; then the image-text pair is input into the image-text matching large model for training, and the loss function is a cross entropy loss, and the training process is completed by back propagation;

[0088] The completion process is as follows: if a certain training data in the training data set is pure text, a picture is randomly selected from the picture-text pair or pure picture training data to form a picture-text pair with the text; if a certain training data in the training data set is pure picture, a sentence of text is randomly selected from the picture-text pair or pure text training data to form a picture-text pair with the picture;

[0089] The matching degree calculation process is as follows:

[0090] Use the object recognition model to extract image descriptions from images;

[0091] Use a natural language processing model to extract verbs and nouns from image descriptions and image-text pairs, respectively;

[0092] The matching degree is calculated using the following formula:

[0093] ;

[0094] ;

[0095] Where N 交 is the number of intersections between the image description and the text of the image-text pair, N 并 is the union number of nouns between the image description and the text of the image-text pair, V 交 is the number of intersections between the image description and the verbs in the image-text pair text, V 并 is the union number of verbs between the image description and the text of the image-text pair; k is the coefficient, and R is the matching degree of the image-text pair.

[0096] The k is 0.1-0.3.

[0097] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0098] Although this article uses more terms such as image features, text features, and linear layers, it does not exclude the possibility of using other terms. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A method for matching images and texts, characterized in that: Based on the multimodal large model of image-text matching, including feature extraction and feature determination, the feature extraction includes the following steps: S1, encode the input image to obtain the initial image features; S2, encode the input text to obtain initial text features; S3, adding a position code to the initial image feature and the initial text feature respectively, the two position codes are different from each other, and obtaining a secondary image feature and a secondary text feature; S4: Fusing the secondary image features and secondary text features obtained in step S3 to obtain fused text features I T And fusion image features T I ; The feature determination includes: S5. Fusion text feature I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. The discriminant linear layer outputs 0 to 1 to indicate the degree of matching, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text. The step S4 is specifically as follows: inputting the secondary text features and the secondary image features into a multimodal feature fusion encoder composed of several layers of encoding modules, and the output of the encoding module of the previous layer is used as the input of the encoding module of the next layer. The processing process of the encoding module of the kth layer is as follows: S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector; S42, the first query vector passes through a random dropout layer, then performs feature normalization, and then compares it with the original text feature I k Add them together to get the intermediate text features; S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector; S44, the second query vector passes through a random dropout layer, then performs feature normalization, and then compares it with the original image feature T k Add together to get the intermediate image features; S45, the intermediate text features and the intermediate image features are respectively passed through a linear layer to obtain the encoded text features I k+1 and image features T k+1 ; The image features and text features input to the first layer encoding module are the secondary image features and secondary text features obtained in step S3; the text features output by the last encoding layer are the fused text features I T , the image feature output by the last encoding layer is the fused image feature T I .

2. A method for image-text matching according to claim 1, characterized in that: The step S1 is specifically as follows: S11, scaling the input image to 224 pixels × 224 pixels, and then dividing the scaled image into image blocks of 16 pixels × 16 pixels, to obtain 196 image blocks in total; S12, stretching each 16 pixel × 16 pixel image block into a 16 × 16 × 3 = 768-dimensional feature according to the RGB value, and obtaining a total of 196 768-dimensional features; S13, map the 196 768-dimensional features into a 196×256-dimensional image feature T through a linear layer.

3. The image-text matching method according to claim 1, characterized in that: The step S5 is specifically as follows: S51, fusion image feature T I And fusion text features I T After feature pooling, they are transformed into feature vectors with a dimension of 768; S52, scaling the two feature vectors to 256 dimensions through a linear layer respectively; S53, concatenate the two 256-dimensional features into a 512-dimensional feature; S54, feature discrimination is performed on the 512-dimensional features, and a linear layer is used to output a value from 0 to 1 to indicate the degree of matching, where 0 represents that the image and text do not match at all, and 1 represents that the image and text match completely.

4. A multimodal large model for image-text matching, running the image-text matching method as claimed in claim 1, characterized in that: It includes a feature extractor and a feature discriminator, wherein the feature extractor includes: Image encoder: After scaling the image to a specified size, it is divided into several 16×16 pixel image blocks. Then, each image block is stretched into a 16×16×3 dimension feature according to the RGB value. All features are mapped through a linear layer to obtain the initial image features. Text encoder: Encode the input text to obtain the initial text features; Position coding adding module: The initial image features and initial text features are respectively added with two different position codings to obtain secondary image features of 196×768 dimensions and secondary text features of 196×768 dimensions; Multimodal feature fusion encoder: fuses the secondary image features and the secondary text features to obtain the fused image feature I T And fusion image features T I ; The feature discriminator will fuse the image features I T And fusion image features T I The features are pooled separately, then scaled and spliced, and the spliced ​​features are discriminated. A linear layer is used to output a value from 0 to 1 to indicate the degree of match, where 0 represents a complete mismatch between the image and the text, and 1 represents a complete match between the image and the text.

5. The multimodal large model of image-text matching according to claim 4, characterized in that: The multimodal feature fusion encoder includes several layers of encoding modules. The output of the encoding module of the previous layer is used as the input of the encoding module of the next layer. Each encoding module includes a multi-head attention layer, a random drop layer, a normalization layer, a linear layer and an addition layer. The processing process of the k-th layer encoding module is as follows: S41, text feature I k As the query vector, the image feature T k As the key vector and value vector, the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the first query vector; S42, the first query vector passes through a random discard layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine with the original text feature I k Add them together to get the intermediate text features; S43, the image feature T k As the query vector, the intermediate text features are used as the key vector and value vector, and the query vector, key vector and value vector are input into the multi-head attention layer for encoding to obtain the second query vector; S44, the second query vector passes through a random drop layer, then passes through a normalization layer to perform feature normalization, and then passes through an addition layer to combine it with the original image feature T k Add together to get the intermediate image features; S45, the intermediate text features and the intermediate image features are respectively passed through the linear layer to obtain the encoded text features I k+1 and image features T k+1 .

6. A method for training a large multimodal model for image-text matching, used for training the large multimodal model for image-text matching as claimed in claim 5, characterized in that: First, the training data set is completed, and the matching degree of each completed image-text pair is calculated, and the matching degree is stored as the label of the image-text pair; then the image-text pair is input into the image-text matching model for training, and the loss function is the cross entropy loss, and the training process is completed through back propagation; The completion process is as follows: if a certain training data in the training data set is pure text, a picture is randomly selected from the picture-text pair or pure picture training data to form a picture-text pair with the text; if a certain training data in the training data set is pure picture, a sentence of text is randomly selected from the picture-text pair or pure text training data to form a picture-text pair with the picture; The matching degree calculation process is as follows: Use the object recognition model to extract image descriptions from images; Use a natural language processing model to extract verbs and nouns from image descriptions and image-text pairs, respectively; The matching degree is calculated using the following formula: ； ； Where N 交 is the number of intersections between the image description and the text of the image-text pair, N 并 is the union number of nouns between the image description and the text of the image-text pair, V 交 is the number of intersections between the image description and the verbs in the text, V 并 is the union number of verbs between the image description and the text of the image-text pair; k is the coefficient, and R is the matching degree of the image-text pair.

7. The method for training a multimodal large model for image-text matching according to claim 6, characterized in that: The k is 0.1-0.3.

Images