Cross-language text-image retrieval method based on multilingual multimodal representation learning

Abstract

The present invention discloses a cross-language text image retrieval method based on multi-language multi-modal representation learning, comprising: 1. obtaining English text images and their corresponding image descriptions and forming a multi-modal data set; 2. constructing an image synthesis model for generating multi-language text images; 3. synthesizing multi-language natural scene text images; 4. constructing a neural network for multi-language natural scene text image representation learning; 5. performing a pre-training task on the network: using the visual information of the text area in the text image as a clue, cross-language recovery of the masked text in the description sentence; 6. performing matching training between the cross-language description sentence and the scene text image on the network to obtain a trained multi-language multi-modal representation model; 7. using the representation model to represent any input multi-language text image, and performing cross-language cross-modal retrieval of the text image. The present invention can provide support for cross-language cross-modal retrieval of natural scene text images.

Description

Technical Field

[0001] The present invention relates to information processing related issues of multilingual text images, and in particular to a cross-language text image retrieval method for natural scene text images. Background Art

[0002] In a multilingual environment, text images contain information in different modalities and languages. Current studies have proposed to jointly model the information of the text modality and the visual modality in natural scene text images through pre-training, and map them to a common semantic space. However, most of the current pre-training models for natural scene text images are for images containing English text and cannot be used in a multilingual environment. It is extremely important to enable the model to have the ability to understand multimodal and multilingual information at the same time and to solve the semantic gap problem caused by data of different modalities and languages ​​in natural scene text images, which is extremely important for intelligent applications for text images in a multilingual environment, such as cross-language and cross-modal retrieval of text images. Summary of the invention

[0003] The present invention aims to address the deficiencies of the above-mentioned prior art and proposes a multilingual and multimodal representation learning method for natural scene text images, in order to synthesize a large amount of graphic data, use a series of pre-training tasks to train the representation model, and let the model learn how to map multilingual and multimodal information into a unified space, thereby eliminating the semantic gap and providing support for cross-language and cross-modal retrieval of natural scene text images.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The present invention provides a cross-language text image retrieval method based on multi-language multi-modal representation learning, which comprises the following steps:

[0006] Step 1: Get English text images and their corresponding description sentences constitute a multimodal English data set ,in, represents the i-th English text image, express The corresponding description statement is: express The number of Chinese and English text images;

[0007] Step 2: Build an image synthesis model for generating multilingual text images , including: English text detection and recognition module, stroke erasure module, text translation module, stroke synthesis module;

[0008] The English text detection and recognition module includes: an English text detector and an English text recognizer;

[0009] The erasing module comprises: The stroke erasing downsampler consists of convolution operations, A stroke erasing upsampler consisting of deconvolution operations;

[0010] The stroke synthesis module includes: a text renderer, an image fusion device, and an image harmonizer;

[0011] Step 3: According to the target language other than English ,Will Input Synthesis Model and generate a set of realistic text images in the target language and its corresponding target language description sentence set ,in, represents the kth target language, Indicates the number of target languages. Representation and The background is the same and contains the kth target language Realistic text images, express The corresponding k-th target language Descriptive sentences;

[0012] Step 3.1: The English text detection and recognition module Processing is performed to obtain the cropped image of the English text area and English character recognition result sequence ;

[0013] Step 3.2: The stroke erasing module Processed to get erased Cropped image of text area after stroke ;

[0014] Step 3.3: The text translation module and Processed, we get A collection of translation results of description sentences in the target language as well as A collection of character translation results in the target language ;in, express In the kth target language The translation results on express In the kth target language The translation results on

[0015] Step 3.4: The stroke synthesis module , and Processing, synthesis Photorealistic text images in the target language ,…, ,…, ;in, express Photorealistic text images in the k-th target language;

[0016] Step 3.5: and its corresponding After combining the elements in one by one, we get a multimodal target language data set and Merge into a multimodal and multilingual data set ,in, express The Multimodal samples, Indicates Text images in multimodal samples, express The corresponding description statement is: express The number of multimodal samples in ;

[0017] Step 4: Build a neural network for learning multilingual natural scene text image representations , including: multilingual image and text detection and recognition module, multilingual visual information encoding module, multilingual text information encoding module, multimodal Transformer fusion module and linear classifier;

[0018] Step 5: Based on , using the visual information of the text region in the text image as a clue, we can restore the masked text in the description sentence across languages, thus Perform pre-training to obtain a pre-trained network ;

[0019] Step 6: Based on ,right Perform matching training between cross-language description sentences and scene text images to obtain a trained cross-language text image retrieval model ;

[0020] Step 7: Obtain a multilingual text image set And its query statement ,in, express The i-th image in express The number of images in ;

[0021] Using the trained cross-language text-image retrieval model Calculation query statement and The matching degree of each image in Zhongyu The best matching text image.

[0022] The cross-language text image retrieval method based on multi-language multi-modal representation learning described in the present invention is also characterized in that step 3.1 includes the following steps:

[0023] Step 3.1.1: The English text detector uses an adaptive Bezier curve network to Perform text detection and get Position coordinates of Chinese and English text areas , thus based on right Cut and get Chinese and English text area images ;

[0024] Step 3.1.2: The English text recognizer uses a recognition network based on visual language modeling to Perform text recognition and obtain Chinese and English character recognition result sequence .

[0025] Further, the step 3.2 includes the following steps:

[0026] Step 3.2.1: The downsampler of the stroke erasing module uses The convolution operation Process and obtain Corresponding character image feature map ;

[0027] Step 3.2.2: The stroke eraser upsampler uses The deconvolution operation is Process to erase , thus generating .

[0028] Further, the step 3.4 includes the following steps:

[0029] Step 3.4.1: Based on The text renderer uses the Pango rendering library to Processing to obtain The text area image sequence for the background[ ,…, ,…, ],in, Indicates For background, including The text area image of the Chinese character;

[0030] Step 3.4.2: Based on , the image fuser will Fusion to Position coordinates of Chinese and English text areas At the corresponding position, we get The background is the same and contains the kth target language Text image ;

[0031] Step 3.4.3: The image and harmonizer are Processing to obtain The background is the same and contains the kth target language Photorealistic text images .

[0032] Further, the step 5 comprises the following steps:

[0033] Step 5.1: Use token mask instead The description of each multimodal sample in the description sentence contains part of the visual scene description or the text description in the text image to construct the current pre-training sample set ,in, express Middle pre-training sample text images, express The description statement after the corresponding mask part text, express The masked text in ; express The mth character to be masked in ; M represents the total number of characters in the masked text;

[0034] Step 5.2: Multimodal samples enter In the multilingual image and text detection and recognition module Process and obtain The text area position coordinates in , crop the image in the text area And recognition results ;

[0035] Step 5.3: The multilingual visual information encoding module uses the VIT network to and Processing to obtain text image Visual encoding and crop image to text area Visual encoding , and according to and The parts in The positions in and The various parts are respectively The corresponding position code in and ;

[0036] Will and , After splicing, we get the text image Visual encoding ;

[0037] Will and , After splicing, we get the cropped image of the text area Visual encoding ,in, An encoding representing the type of visual modality;

[0038] Step 5.4: The multilingual text information encoding module uses the mT5 encoder to and Process and get the description statement Text encoding And recognition results Text encoding , and according to and The parts in and The relative position in and The parts in and The corresponding position code in and ;

[0039] Will and , After splicing, we get the description statement Text encoding ;

[0040] Will and , After splicing, we get the recognition result Text encoding ,in, The code for the text modality type;

[0041] Step 5.5: and , and Input into the multimodal Transformer fusion module for processing and output The predicted feature vector corresponding to the masked text in ,in, express The corresponding predicted feature vector;

[0042] Will Input into the linear classifier for processing to obtain the prediction result probability vector sequence of the corresponding mask position ,in, Indicates The predicted result probability vector of each position;

[0043] Step 5.6: and The converted one-hot vectors are then compared to construct a loss function and used to Perform back propagation to update the parameters of each part of the model to obtain the pre-trained network .

[0044] Further, the step 6 comprises the following steps:

[0045] Step 6.1: Random Selection Select some multimodal samples and change some of the visual scene descriptions or text descriptions in the text images in their description sentences, so that the text images do not match the description sentences, thereby obtaining a negative sample set, and the remaining multimodal samples are used as a positive sample set;

[0046] The training set consists of a positive sample set and a negative sample set. ,in, express Middle Text images in multimodal samples, express Middle A description statement of a multimodal sample, express and Matching tags, if they match, then = , otherwise, let = ;

[0047] Step 6.2: Multimodal samples enter In the The multilingual image and text detection and recognition module, the multilingual visual information encoding module, and the multilingual text information encoding module are used for processing to obtain the text image. Visual encoding , crop the image in the text area Visual encoding , description statement Text encoding , recognition results Text encoding ;

[0048] Step 6.3: and , and The initialized cls code is input into the multimodal Transformer fusion module for processing, and the vector corresponding to the position of the cls code is output. ;

[0049] Step 6.4: Input a linear classifier for classification to predict and Is it a match? The prediction result vector is recorded as ;

[0050] Step 6.5: and The one-hot vectors converted from each other are compared to construct a loss function for Perform back propagation to update the parameters of each part of the network and obtain the trained cross-language text image retrieval model .

[0051] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the cross-language text image retrieval method, and the processor is configured to execute the program stored in the memory.

[0052] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the cross-language text image retrieval method when the computer program is executed by a processor.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. The present invention trains a representation network of multilingual natural scene text images by synthesizing massive data. Compared with the general multimodal representation network pre-training method, it can better train the representation network without spending a lot of effort to annotate massive real data. Compared with the existing methods, the method of the present invention can obtain more diverse and rich training data, can better simulate real data sets, and has a larger scale of available training data, which improves the effect of multilingual multimodal representation learning, thereby improving the accuracy of cross-language text image retrieval.

[0055] 2. The present invention specifically designs a series of pre-training tasks to enable the model to learn how to map multilingual and multimodal information into a unified space. Compared with general multimodal pre-training tasks, the model can better possess the ability to align text within the image with text outside the image across languages ​​and modalities, thereby improving the model's text image retrieval performance in a multilingual environment.

[0056] 3. The present invention takes into account the multilingual situation of text content in images and can realize the mapping of text information in different languages ​​in images to a common semantic space. Compared with general multimodal representation learning methods, it can be better applied to multilingual environments and realize cross-language and cross-modal retrieval of natural scene text images. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of the use of the cross-language text image retrieval method based on multi-language multi-modal representation learning of the present invention;

[0058] Figure 2 It is a network structure diagram of the cross-language text image retrieval method based on multi-language multi-modal representation learning of the present invention. DETAILED DESCRIPTION

[0059] In this embodiment, Figure 1 As shown, a cross-language text image retrieval method based on multi-language and multi-modal representation learning is to train a representation learning neural network on a synthetic multi-language text image, map the input natural scene text image and the information of different languages ​​and modalities in the text into a common semantic space, and form an embedded representation for cross-language and cross-modal text image retrieval. Specifically, the following steps are included:

[0060] Step 1: Get English text images and their corresponding description sentences constitute a multimodal English data set ,in, represents the i-th English text image, express The corresponding description statement is: express The number of Chinese and English text images. The acquired English text images and their description sentences are used to construct a multilingual text image description generation dataset.

[0061] Step 2: Build an image synthesis model for generating multilingual text images , including: English text detection and recognition module (used to detect and recognize character sequences in text images), stroke erasure module (used to erase character strokes in English text images), text translation module (used to translate character sequences in English text images into other languages), stroke synthesis module (used to synthesize strokes of character sequences in a specified language);

[0062] English text detection and recognition module, including: 1 English text detector and 1 English text recognizer. The text detector uses a pre-trained adaptive Bezier curve network, and the text recognizer uses a recognition network based on visual language modeling.

[0063] Erase module, including: The stroke erasing downsampler consists of convolution operations, The stroke erasing upsampler consists of a deconvolution operation. The convolution operation and the deconvolution operation use lateral connections to capture more features;

[0064] The stroke synthesis module includes: a text renderer, an image fusion device, and an image harmonizer. The text renderer is composed of the Pango rendering library.

[0065] Step 3: According to the target language other than English ,Will Input Synthesis Model and generate a set of realistic text images in the target language and its corresponding target language description sentence set ,in, represents the kth target language, Indicates the number of target languages. Representation and The background is the same and contains the kth target language Realistic text images, express The corresponding k-th target language The training data obtained by the method in step 3 is more diverse and rich, which can better simulate the real data set and the available training data scale is larger, thus improving the effect of multi-language multi-modal representation learning.

[0066] Step 3.1: English text detection and recognition module Processing is performed to obtain the cropped image of the English text area and English character recognition result sequence ;

[0067] Step 3.1.1: English text detector uses adaptive Bezier curve network to Perform text detection and get Position coordinates of Chinese and English text areas , thus based on right Cut and get Chinese and English text area images ;

[0068] Step 3.1.2: English text recognizer uses a recognition network based on visual language modeling to Perform text recognition and obtain Chinese and English character recognition result sequence .

[0069] Step 3.2: Stroke erasing module Processed to get erased Cropped image of text area after stroke ;

[0070] Step 3.2.1: Downsampler usage of stroke erasing module The convolution operation Process and obtain Corresponding character image feature map ;

[0071] Step 3.2.2: Upsampler usage for stroke erasing The deconvolution operation is Process to erase , thus generating .

[0072] Step 3.3: Text translation module and Processed, we get A collection of translation results of description sentences in the target language as well as A collection of character translation results in the target language ;in, express In the kth target language The translation results on express In the kth target language The translation results on .

[0073] Step 3.4: Stroke synthesis module , and Processing, synthesis Photorealistic text images in the target language ,…, ,…, ;in, express Photorealistic text images in the k-th target language;

[0074] Step 3.4.1: Based on The text renderer uses the Pango rendering library to Processing to obtain The text area image sequence for the background[ ,…, ,…, ],in, Indicates For background, including The text area image of the Chinese character.

[0075] Step 3.4.2: Based on , the image fuser will Fusion to Position coordinates of Chinese and English text areas At the corresponding position, we get The background is the same and contains the kth target language Text image ;

[0076] Step 3.4.3: Image and Harmonizer Pairs Processing to obtain The background is the same and contains the kth target language Photorealistic text images .

[0077] Step 3.5: and its corresponding After combining the elements in one by one, we get the multimodal target language data set and Merge into a multimodal and multilingual data set ,in express The Multimodal samples, Indicates Text images in multimodal samples, express The corresponding description statement is: express The number of multimodal samples in . In the text image and description sentence, the language categories include English and other target languages.

[0078] Step 4: Figure 2 As shown in the figure, a neural network is constructed for learning multilingual natural scene text image representations , including: multilingual image and text detection and recognition module (used to detect and recognize character sequences in multilingual text images), multilingual visual information encoding module (used to encode visual information in text images), multilingual text information encoding module (used to encode multilingual text information), multimodal Transformer fusion module (used to map information of different languages ​​and modalities into a common semantic space) and linear classifier (used to perform classification tasks).

[0079] Step 5: Based on the dataset and Network , perform the pre-training task: using the visual information of the text area in the text image as a clue, restore the masked text in the description sentence across languages, and thus obtain the pre-trained network .

[0080] Step 5.1: Use token mask instead The description of each multimodal sample in the description sentence contains part of the visual scene description or the text description in the text image to construct the current pre-training sample set ,in, express Middle pre-training sample text images, express The description statement after the corresponding mask part text, express The masked text in ; express The mth character to be masked in ; M represents the total number of characters in the masked text;

[0081] Step 5.2: Multimodal samples enter In the multilingual image and text detection and recognition module Process and obtain The text area position coordinates in , crop the image in the text area And recognition results ;

[0082] Step 5.3: The multilingual visual information encoding module uses the VIT network to and Processing to obtain text image Visual encoding and crop image to text area Visual encoding , and according to and The parts in The positions in and The various parts are respectively The corresponding position code in and ;

[0083] Will and , After splicing, we get the text image Visual encoding ;

[0084] Will and , After splicing, we get the cropped image of the text area Visual encoding ,in, An encoding representing the type of visual modality;

[0085] Step 5.4: The multilingual text information encoding module uses the mT5 encoder to and Process and get the description statement Text encoding And recognition results Text encoding , and according to and The parts in and The relative position in and The parts in and The corresponding position code in and ;

[0086] Will and , After splicing, we get the description statement Text encoding ;

[0087] Will and , After splicing, we get the recognition result Text encoding ,in, The code for the text modality type;

[0088] Step 5.5: and , and The input is processed in the multimodal Transformer fusion module, the information of different languages ​​and modalities in the text image and description sentence is mapped to a common semantic space, the dependency relationship between different languages ​​and modalities is modeled, and the output is The predicted feature vector corresponding to the masked text in ,in, express The corresponding predicted feature vector;

[0089] Will Input into the linear classifier for processing to obtain the prediction result probability vector sequence of the corresponding mask position ,in, Indicates The predicted result probability vector of each position;

[0090] Step 5.6: and The converted one-hot vectors are then compared to construct the loss function. Perform back propagation to update the parameters of each part of the model to obtain the pre-trained network This pre-training method enables the model to better align the text inside the image with the text outside the image across languages ​​and modalities, thereby improving the model's text image retrieval performance in a multilingual environment.

[0091] Step 6: Based on the dataset , for the network Perform matching training between cross-language description sentences and scene text images to obtain a trained cross-language text image retrieval model ;

[0092] Step 6.1: Random Selection We select some multimodal samples and change some of the visual scene descriptions or text descriptions in the text images in the description sentences, so that the text images do not match the description sentences, thus obtaining a negative sample set, and the remaining multimodal samples are used as positive sample sets. In the actual construction process, the ratio of positive samples to negative samples can be set to 50%;

[0093] The training set consists of a positive sample set and a negative sample set. ,in, express Middle Text images in multimodal samples, express Middle A description statement of a multimodal sample, express and Matching tags, if they match, then Recorded as ,otherwise, Recorded as .

[0094] Step 6.2: Multimodal samples Enter the neural network In, by The multilingual image and text detection and recognition module, the multilingual visual information encoding module, and the multilingual text information encoding module are used to obtain the text image. Visual encoding , crop the image in the text area Visual encoding , description statement Text encoding , recognition results Text encoding .

[0095] Step 6.3: and , and The initialized cls code is input into the multimodal Transformer fusion module for processing, and the vector corresponding to the position of the cls code is output. . The information contains the characteristics of whether the input text image matches the description sentence;

[0096] Step 6.4: Input a linear classifier for classification to predict and Is it a match? The prediction result vector is recorded as ;

[0097] Step 6.5: and The one-hot vectors converted from each other are compared to construct a loss function to train the neural network Perform back propagation to obtain the parameters of each part of the network and obtain the trained cross-language text image retrieval model .

[0098] Step 7: Input Multilingual Text Image Set And its query statement ,in, express The i-th image in express The number of images in .

[0099] Using the trained cross-language text-image retrieval model Calculation query statement and The matching degree of each image in the image is obtained to obtain a multilingual text image set Zhongyu The best matching text image. Specifically, for i , each query-image pair The class token is input into the above-trained representation network. The representation learning network maps the input information of different modalities and languages ​​to a unified space and outputs a new representation vector sequence. Input into the linear classifier, and the linear classifier outputs the predicted value of the matching degree . Based on the matching prediction value ,right Sort them and then select the text image with the greatest matching degree and return it as the retrieval result.

[0100] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0101] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A cross-language text image retrieval method based on multilingual multimodal representation learning, characterized in that: The following steps are involved: Step 1: Get English text images and their corresponding description sentences constitute a multimodal English data set ,in, represents the i-th English text image, express The corresponding description statement is: express The number of Chinese and English text images; Step 2: Build an image synthesis model for generating multilingual text images , including: English text detection and recognition module, stroke erasure module, text translation module, stroke synthesis module; The English text detection and recognition module includes: an English text detector and an English text recognizer; The erasing module comprises: The stroke erasing downsampler consists of convolution operations, A stroke erasing upsampler consisting of deconvolution operations; The stroke synthesis module includes: a text renderer, an image fusion device, and an image harmonizer; Step 3: According to the target language other than English ,Will Input Synthesis Model and generate a set of realistic text images in the target language and its corresponding target language description sentence set ,in, represents the kth target language, Indicates the number of target languages. Representation and The background is the same and contains the kth target language Realistic text images, express The corresponding k-th target language Descriptive sentences; Step 3.1: The English text detection and recognition module Processing is performed to obtain the cropped image of the English text area and English character recognition result sequence ; Step 3.2: The stroke erasing module Processed to get erased Cropped image of text area after stroke ; Step 3.3: The text translation module and Processed, we get A collection of translation results of description sentences in the target language as well as A collection of character translation results in the target language ;in, express In the kth target language The translation results on express In the kth target language The translation results on Step 3.4: The stroke synthesis module , and Processing, synthesis Photorealistic text images in the target language ,…, ,…, ;in, express Photorealistic text images in the k-th target language; Step 3.5: and its corresponding After combining the elements in one by one, we get a multimodal target language data set and Merge into a multimodal and multilingual data set ,in, express The Multimodal samples, Indicates Text images in multimodal samples, express The corresponding description statement is: express The number of multimodal samples in ; Step 4: Build a neural network for learning multilingual natural scene text image representations , including: multilingual image and text detection and recognition module, multilingual visual information encoding module, multilingual text information encoding module, multimodal Transformer fusion module and linear classifier; Step 5: Based on , using the visual information of the text region in the text image as a clue, we can restore the masked text in the description sentence across languages, thus Perform pre-training to obtain a pre-trained network ; Step 6: Based on ,right Perform matching training between cross-language description sentences and scene text images to obtain a trained cross-language text image retrieval model ; Step 7: Obtain a multilingual text image set And its query statement ,in, express The i-th image in express The number of images in ; Using the trained cross-language text-image retrieval model Calculation query statement and The matching degree of each image in Zhongyu The best matching text image.

2. The cross-language text image retrieval method based on multilingual multimodal representation learning according to claim 1, characterized in that: The step 3.1 comprises the following steps: Step 3.1.1: The English text detector uses an adaptive Bezier curve network to Perform text detection and get Position coordinates of Chinese and English text areas , thus based on right Cut and get Chinese and English text area images ; Step 3.1.2: The English text recognizer uses a recognition network based on visual language modeling to Perform text recognition and obtain Chinese and English character recognition result sequence .

3. The cross-language text image retrieval method based on multi-language multi-modal representation learning according to claim 2, characterized in that: The step 3.2 comprises the following steps: Step 3.2.1: The downsampler of the stroke erasing module uses The convolution operation Process and obtain Corresponding character image feature map ; Step 3.2.2: The stroke eraser upsampler uses The deconvolution operation is Process to erase , thus generating .

4. The cross-language text image retrieval method based on multi-language multi-modal representation learning according to claim 3, characterized in that: The step 3.4 comprises the following steps: Step 3.4.1: Based on The text renderer uses the Pango rendering library to Processing to obtain The text area image sequence for the background[ ,…, ,…, ],in, Indicates For background, including The text area image of the Chinese character; Step 3.4.2: Based on , the image fuser will Fusion to Position coordinates of Chinese and English text areas At the corresponding position, we get The background is the same and contains the kth target language Text image ; Step 3.4.3: The image and harmonizer are Processing to obtain The background is the same and contains the kth target language Photorealistic text images .

5. The cross-language text image retrieval method based on multilingual multimodal representation learning according to claim 4, characterized in that: The step 5 comprises the following steps: Step 5.1: Use token mask instead The description of each multimodal sample in the description sentence contains part of the visual scene description or the text description in the text image to construct the current pre-training sample set ,in, express Middle pre-training sample text images, express The description statement after the corresponding mask part text, express The masked text in ; express The mth character to be masked in ; M represents the total number of characters in the masked text; Step 5.2: Multimodal samples enter In the multilingual image and text detection and recognition module Process and obtain The text area position coordinates in , crop the image in the text area and recognition results ; Step 5.3: The multilingual visual information encoding module uses the VIT network to and Processing to obtain text image Visual encoding and crop image to text area Visual encoding , and according to and The parts in The positions in and The various parts are respectively The corresponding position code in and ; Will and , After splicing, we get the text image Visual encoding ; Will and , After splicing, we get the cropped image of the text area Visual encoding ,in, An encoding that represents the type of visual modality; Step 5.4: The multilingual text information encoding module uses the mT5 encoder to and Process and get the description statement Text encoding and recognition results Text encoding , and according to and The parts in and The relative position in and The parts in and The corresponding position code in and ; Will and , After splicing, we get the description statement Text encoding ; Will and , After splicing, we get the recognition result Text encoding ,in, The code for the text modality type; Step 5.5: and , and Input into the multimodal Transformer fusion module for processing and output The predicted feature vector corresponding to the masked text in ,in, express The corresponding predicted feature vector; Will Input into the linear classifier for processing to obtain the prediction result probability vector sequence of the corresponding mask position ,in, Indicates The predicted result probability vector of each position; Step 5.6: and The converted one-hot vectors are then compared to construct a loss function and used to Perform back propagation to update the parameters of each part of the model to obtain the pre-trained network .

6. The cross-language text image retrieval method based on multi-language multi-modal representation learning according to claim 5, characterized in that: The step 6 comprises the following steps: Step 6.1: Random Selection Select some multimodal samples and change some of the visual scene descriptions or text descriptions in the text images in their description sentences, so that the text images do not match the description sentences, thereby obtaining a negative sample set, and the remaining multimodal samples are used as a positive sample set; The training set consists of a positive sample set and a negative sample set. ,in, express Middle Text images in multimodal samples, express Middle A description statement of a multimodal sample, express and Matching tags, if they match, then = , otherwise, let = ; Step 6.2: Multimodal samples enter In the The multilingual image and text detection and recognition module, the multilingual visual information encoding module, and the multilingual text information encoding module are used for processing to obtain the text image. Visual encoding , crop the image in the text area Visual encoding , description statement Text encoding , recognition results Text encoding ; Step 6.3: and , and The initialized cls code is input into the multimodal Transformer fusion module for processing, and the vector corresponding to the position of the cls code is output. ; Step 6.4: Input a linear classifier for classification to predict and Is it a match? The prediction result vector is recorded as ; Step 6.5: and The one-hot vectors converted from each other are compared to construct a loss function for Perform back propagation to update the parameters of each part of the network and obtain the trained cross-language text image retrieval model .

7. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the cross-language text image retrieval method according to any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-language text image retrieval method according to any one of claims 1 to 6 are executed.

Images