A cross-modal visual and text information matching method and device

Abstract

The present invention discloses a cross-modal visual and text information matching method and device, the method comprising the following steps: using a mature target recognition system to detect blocks of important object scenes in a picture, the total number of blocks being X; inputting the visual vector information of these blocks into a visual encoder, thereby constructing the relationship and comprehensive information between these blocks, and obtaining the visual vector information h containing context. x , x∈X; for text information of length Y, the pre-trained text encoder model is used for encoding to obtain the context embedding vector h of each word y , y∈Y; for all h x and h y Perform matching calculation on g to obtain X×Y scores; take out the maximum scores of each column and each row on the x-axis and y-axis respectively, and perform nonlinear transformation on the scores; calculate the final score s as the matching degree, and the final score s is the average of all g'.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for cross-modal visual and text information matching. Background Art

[0002] Cross-modal data matching plays a core role in many commercial applications. For example, video retrieval technology requires the system to match the query text with all video key frame information in order to find relevant video frames. Another example is in video automatic generation technology, it is necessary to automatically find relevant materials in the video library based on the text script, and then the video generation software will combine these materials into the final product. For another example, in a smart office scenario, when the text content of the PPT is completed, it is necessary to use cross-modal matching technology to automatically match key text such as titles from the gallery. Therefore, improving the cross-modal visual-text matching algorithm can improve the performance of the above and other related applications from the bottom up and generate greater commercial value. Summary of the invention

[0003] The present invention mainly provides a cross-modal visual and text information matching method and device, which has high matching accuracy.

[0004] The present invention aims to solve the above technical problems mainly through the following technical solutions: a cross-modal visual and text information matching method, comprising the following steps:

[0005] S1, detect the blocks of important object scenes in the picture through the target recognition system, the total number of blocks is X;

[0006] S2. Input the visual vector information of these blocks into the TRANSFORMER model, build the relationship and comprehensive information between these blocks, and obtain the visual vector information h containing the context x , x∈X;

[0007] S3. For text information of length Y, use the pre-trained text TRANSFORMER encoder model to encode and obtain the context embedding vector h of each word y , y∈Y;

[0008] S4. Use dot product (vector dot product) for all h x and h y Perform matching calculations to obtain X×Y scores, which form an X×Y matrix;

[0009] S5. For the x-axis (row) and y-axis (column) of the matrix, the maximum score of each column and each row is taken out respectively, and a total of X+Y scores are obtained. These scores are represented as g, and the taken scores are nonlinearly transformed by g'=logReLU(g);

[0010] S6. Calculate the final score s, which is the matching degree. The final score s is the sum of all g ′ The average of .

[0011] Preferably, the cross-modal visual and text information matching method further includes a training process, and the training process is as follows:

[0012] Collection of training data: First, collect paired pictures and text information;

[0013] Loss function and training: Given the above model and data, mini-batch is used for training. The training loss function is the cross entropy loss function. The manually annotated text is regarded as the correct answer, and the other text information in the same mini-batch is regarded as the wrong answer. The cross entropy loss function is:

[0014]

[0015] Where A is the number of texts in a mini-batch, g is the calculated score, and e is a natural constant.

[0016] A cross-modal visual and text information matching device, comprising:

[0017] Visual TRANSFORMER encoder: This module can be divided into two parts. The first step is to detect the blocks of important object scenes in the picture through a mature object recognition system. The total number of blocks is X. The visual vector information of these blocks is input into the TRANSFORMER model to build the relationship and comprehensive information between these blocks and obtain the visual vector information h containing the context. x , x∈X;

[0018] Text TRANSFORMER encoder: For text information of length Y, the pre-trained text TRANSFORMER encoder model is used to encode and obtain the context embedding vector h of each word. y , y∈Y;

[0019] Matching scoring model: given h x and h y After that, the matching degree is calculated through the following process:

[0020] First, use the dot product for all h x and h yA matching calculation is performed on , and X×Y scores are obtained. Then, the maximum scores of each column and each row are taken out for the x-axis and y-axis respectively. These scores are g, and they are nonlinearly transformed by g'=logReLU(g). The final score s is the average of all g'.

[0021] Preferably, the model training of the device is performed as follows:

[0022] Collection of training data: First, collect paired pictures and text information;

[0023] Loss function and training: Given the above model and data, mini-batch is used for training. The training loss function is the cross entropy loss function. The manually annotated text is regarded as the correct answer, and the other text information in the same mini-batch is regarded as the wrong answer. The cross entropy loss function is:

[0024]

[0025] Where A is the number of texts in a mini-batch, g is the calculated score, and e is a natural constant.

[0026] The substantial effect brought about by the present invention is that it has high accuracy and provides a basis for subsequent applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of a cross-modal matching neural network model architecture of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings.

[0029] Embodiment: This solution proposes a novel cross-modal information matching technology based on neural networks. It calculates efficient matching scores by performing fine-grained matching between the image detail area and each text. The overall architecture is as follows: Figure 1 As shown, including:

[0030] 1. Visual TRANSFORMER encoder: This module can be divided into two parts. The first step is to detect the blocks of important object scenes in the image through mature object recognition systems (such as FasterRCNN, Yolov, etc.). Assuming there are X blocks in total, the visual vector information of these blocks will be input into TRANSFORMER to build the relationship and comprehensive information between these blocks and obtain the visual vector information h containing context. x , x∈X.

[0031] 2. Text TRANSFORMER Encoder: For text information of length Y, we use a pre-trained text TRANSFORMER encoder (such as BERT, ROBERTA) and other models to encode and obtain the context embedding vector h of each word y , y∈Y.

[0032] 3. Matching scoring model: given h x and h y Afterwards, the algorithm model of this solution calculates the matching degree through the following process:

[0033] First, we use the dot product for all h x and h y We perform matching calculations on , and get X×Y scores, and then we take the maximum score of each column and each row for the x-axis and y-axis respectively. The maximum score of each column represents the text relevance that a visual block can best match, and the score of each row represents the visual relevance that a text can match. Assuming these scores are g, we let them be nonlinearly transformed by g'=logReLU(g). The final score s is the average of all g'.

[0034] The above model is trained by SGD. Specifically, the training is divided into two parts:

[0035] Training data collection: First, you need to collect paired pictures and text information. Data collection can be obtained by crawling pictures and related text titles on the Internet. Alternatively, you can also use manual annotation to annotate a picture library by looking at the picture and writing a description, so that each picture has corresponding text description information.

[0036] Loss function and training: Given the above model and data, we use mini-batch for training. The training loss function is the conventional cross entropy loss function. The manually annotated text is regarded as the correct answer, and the other text information in the same mini-batch is regarded as the wrong answer.

[0037] After the model training is completed, it is used to perform matching calculations on any image and text input. The larger the score, the higher the degree of matching.

[0038] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

[0039] Although the terms encoder, loss function, etc. are used more frequently in this article, the possibility of using other terms is not excluded. These terms are used only to more conveniently describe and explain the essence of the present invention; interpreting them as any additional restrictions is contrary to the spirit of the present invention.

Claims

1. A cross-modal visual and text information matching method, characterized in that: The following steps are involved: S1, detect the blocks of important object scenes in the picture through the target recognition system, the total number of blocks is X; S2. Input the visual vector information of these blocks into the TRANSFORMER model, build the relationship and comprehensive information between these blocks, and obtain the visual vector information h containing the context x , x∈X; S3. For text information of length Y, the text TRANSFORMER encoder model is used to encode and obtain the context embedding vector h of each word y , y∈Y; S4. Use dot product for all h x and h y Perform matching calculations to obtain X×Y scores, which form an X×Y matrix; S5. For the x-axis and y-axis of the matrix, the maximum score of each column and each row is taken out respectively, and a total of X+Y scores are obtained. These scores are represented as g, and the taken scores are nonlinearly transformed by g'=logReLU(g); S6. Calculate the final score s, which is the matching degree. The final score s is the average of all g's. The cross-modal visual and text information matching method also includes a training process, which is as follows: Collection of training data: First, collect paired pictures and text information; Loss function and training: Given the above model and data, mini-batch is used for training. The training loss function is the cross entropy loss function. The manually annotated text is regarded as the correct answer, and the other text information in the same mini-batch is regarded as the wrong answer. The cross entropy loss function is: Where A is the number of texts in a mini-batch, g is the calculated score, and e is a natural constant.

2. A cross-modal visual and text information matching device, characterized in that: include: Visual TRANSFORMER encoder: This module can be divided into two parts. The first step is to detect the blocks of important object scenes in the picture through a mature object recognition system. The total number of blocks is X. The visual vector information of these blocks is input into the TRANSFORMER model to build the relationship and comprehensive information between these blocks and obtain the visual vector information h containing the context. x , x∈X; Text TRANSFORMER encoder: For text information of length Y, the pre-trained text TRANSFORMER encoder model is used to encode and obtain the context embedding vector h of each word. y , y∈Y; Matching scoring model: given h x and h y After that, the matching degree is calculated through the following process: First, use the dot product for all h x and h y Perform matching calculations on , and obtain X×Y scores. Then, for the x-axis and y-axis, take out the maximum scores of each column and each row, respectively. These scores are g, and they are nonlinearly transformed by g'=logReLU(g). The final score s is the average of all g'; The model training of the whole device is carried out as follows: Collection of training data: First, collect paired pictures and text information; Loss function and training: Given the above model and data, mini-batch is used for training. The training loss function is the cross entropy loss function. The manually annotated text is regarded as the correct answer, and the other text information in the same mini-batch is regarded as the wrong answer. The cross entropy loss function is: Where A is the number of texts in a mini-batch, g is the calculated score, and e is a natural constant.

Images