Multi-language text image-oriented cross-language description generation method

Abstract

The invention discloses a multi-language text image-oriented cross-language description generation method. The method comprises the following steps of: 1, acquiring a multi-language text image and labeling description statements; 2, constructing a cross-language description generation network oriented to multi-language text images; 3, constructing a data set of a multi-modal text error correction task, and pre-training part of modules in the description generation network; 4, generating a data set based on multilingual text image description to train all modules of the network; and 5, generating a description statement of a specified language for any input multilingual text image by using the trained cross-lingual description generation network. According to the method, the input multi-language natural scene text image can be deeply understood in a multi-language scene, and the description statement of the specified language of the multi-language text image is output.

Description

Technical Field

[0001] The present invention relates to information processing related issues of multilingual natural scene text images, and in particular to a cross-language description generation method for multilingual text images. Background Art

[0002] Most current research on text image description generation focuses on generating English image descriptions based on English text images. In the context of the rapid development of cross-border trade and transnational tourism, it is necessary to study text image description generation in a multilingual environment. Text image description generation methods rely on upstream models, and incorrect text recognition results will have an adverse effect on subsequent modeling steps. For natural scene text images, there is a correlation between text areas at different locations in the image, and there is also a correlation between visual targets and text content in the image. Existing text recognition models do not fully utilize this correlation when recognizing text. Incorrect text recognition results affect the generation of high-quality image descriptions. Summary of the invention

[0003] In order to address the deficiencies of the above-mentioned prior art, the present invention proposes a cross-language description generation method for multilingual text images, in order to correct the results of text recognition based on the correlation between information from multiple perspectives in the text image, and make the correction results used for description generation, thereby generating higher quality multilingual text image descriptions.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The invention provides a method for generating cross-language descriptions of multilingual text images, comprising the following steps:

[0006] Step 1: Obtain multilingual text images and annotate description sentences to obtain a training set ,in, Represents multilingual text images, express The corresponding image description, and , express The characters, express The number of characters in ; express The corresponding language, express The corresponding linear representation of structured knowledge, and , express The characters, express The number of characters in ; express The number of multilingual text images in ;

[0007] Step 2: Build a cross-language description generation network for multilingual text images , including: multilingual image and text detection and recognition module, multilingual text information encoding module, multilingual visual information encoding module, multimodal Transformer module, knowledge extraction module, language embedding module, Transformer decoding module; and Each sample input in Processed in to obtain the decoded character probability sequence , and used to construct the total loss function ;

[0008] Step 3: Construct a dataset for the multimodal text error correction task , and used for Pre-training a multi-language text information encoding module, a multi-language visual information encoding module, and a multi-modal Transformer module to obtain a pre-trained language text information encoding module, a pre-trained multi-language visual information encoding module, and a pre-trained multi-modal Transformer module;

[0009] Step 4: Based on the total loss function , through the back propagation algorithm to generate a multilingual text image cross-language description network The multilingual image and text detection and recognition module, the pre-trained language text information encoding module, the pre-trained multilingual visual information encoding module, the pre-trained multimodal Transformer module, the knowledge extraction module, the language embedding module, and the Transformer decoding module are trained to update the network parameters, thereby obtaining a trained multilingual text image cross-language description generation model;

[0010] Step 5: Use the trained multilingual text image cross-language description generation model to generate the text image in the specified language category. Input any multilingual text image Make predictions to get the specified language category Down Description statement.

[0011] The cross-language description generation method for multilingual text images described in the present invention is also characterized in that the step 2 includes the following steps:

[0012] Step 2.1: The multilingual image and text detection and recognition module first detects The text area and output the position coordinates , thus based on right Crop to get the text area image , and identify The character sequence in the text recognition result is obtained ;

[0013] Step 2.2: The multilingual text information encoding module uses the encoder of the mT5 model Process and output the text feature representation matrix ,in, express Middle The text representation vector of the position, for The number of semantic vectors in ;

[0014] Step 2.3: The multilingual visual information encoding module uses the Vit model to Extract visual features and get The visual feature representation matrix ,in, express Middle The visual representation vector of the image region features, for The number of image regions in

[0015] Step 2.4: The multilingual visual information encoding module uses the Vit model to Extract visual features and get representation The visual feature representation matrix ,in, express The visual representation vector of the mth image region in , for The number of image regions in

[0016] Step 2.5: , , The multimodal Transformer modules are inputted respectively, and then go through multiple stacked multi-head attention mechanisms, feedforward operations and residual connection processing, modality fusion and semantic enhancement, and then the corresponding outputs are Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding ;

[0017] Step 2.6: , , Input the knowledge extraction module to make predictions and obtain the character probability sequence of the knowledge representation sequence ,in, represents the first The probability vector of characters, K represents the number of characters in the knowledge representation sequence;

[0018] Step 2.7: Converted into a one-hot vector sequence and Compare to construct the loss function ;

[0019] Step 2.8: The knowledge extraction module to convert and Select the character with the largest probability value at each position in the prediction result of the sequence represented by the knowledge of the characters at all positions ,in, express The characters, express The number of characters in ;

[0020] The knowledge extraction module embeds the knowledge representation sequence to obtain the embedding representation of the knowledge representation sequence ;

[0021] Step 2.9: The language embedding module Embedding representation, we get The representation vector ;

[0022] Step 2.10: , , , , Input the Transformer decoding module to perform stacked multi-head attention operations to obtain the decoded character probability sequence ;in, express Middle The character probability vector of the position, express The number of characters in ;

[0023] Step 2.11: Converted into a one-hot vector sequence and Compare to construct the loss function ;

[0024] Step 2.12: Calculate the total loss function ,in, express The corresponding weight.

[0025] Furthermore, the step 3 comprises the following steps:

[0026] Step 3.1: Construct a dataset for the multimodal text error correction task ,in, express The corresponding identification tag containing the wrong character, express The corresponding correct identification label;

[0027] Step 3.2: The data is sequentially input into the multilingual text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module for processing. The corresponding semantically enhanced feature vector sequence ,in, express The corresponding feature vectors, for The number of characters in ;

[0028] Step 3.3: Send it to the character classifier for prediction and output the probability of the character category corresponding to each position ;in, Indicates the output probability vector of positions;

[0029] Step 3.4: Convert to one-hot vector , and Compare to construct the loss function , which is used to perform back-propagation on the language text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module to update the module parameters, thereby obtaining the pre-trained language text information encoding module, the pre-trained multilingual visual information encoding module, and the pre-trained multimodal Transformer module.

[0030] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the cross-language description generation method, and the processor is configured to execute the program stored in the memory.

[0031] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the cross-language description generation method when the computer program is executed by a processor.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The cross-language description generation network of multilingual text images of the present invention can understand multilingual text images, and realize the generation of description sentences in specified languages ​​for multilingual natural scene text images. It is not limited to generating description sentences in English or Chinese, and can be better applied to multilingual environments. The present invention uses a multilingual image and text detection and recognition module and a multilingual text information encoding module to process information in different languages, and maps information in different languages ​​and modes to a common space through a multimodal Transformer module, thereby overcoming the semantic gap between multilingualism and multimodality. The present invention uses a language embedding module to obtain the representation vector of the language and input it into the decoder, thereby achieving the goal of guiding the decoder to generate a description sentence in the specified language.

[0034] 2. The present invention corrects the recognition results of text images based on the correlation between different perspective information in multilingual text images, and uses the correction results for description generation, which weakens the impact of text misrecognition on description sentence generation, thereby generating higher quality multilingual text image description sentences. The present invention uses multimodal text error correction tasks to pre-train multiple parts of the cross-language description generation network, overcoming the problem of text recognition error accumulation in the prior art, thereby improving the accuracy of description generation of multilingual text images.

[0035] 3. The network framework designed by the present invention integrates the information extraction task and description generation task of multilingual text images, so that the two tasks can complement each other. The inherent relationship between the two tasks enables the two tasks to promote each other, overcoming the shortcomings of the prior art that the relationship between different modal information in the image is not fully mined when generating description sentences, thereby generating higher quality text image description sentences. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of the method for generating cross-language description of multilingual text images of the present invention;

[0037] Figure 2 It is a network structure diagram of the method for generating cross-language description of multilingual text images according to the present invention. DETAILED DESCRIPTION

[0038] In this embodiment, Figure 1 As shown, a structured information extraction method for multilingual natural scene text images includes the following steps:

[0039] Step 1: Obtain multilingual text images and annotate description sentences to obtain a training set ,in, Represents multilingual text images, express The corresponding image description, and , express The characters, express The number of characters in . express The corresponding language, express The corresponding linear representation of structured knowledge, and , express The characters, express The number of characters in . express The number of multilingual text images in . Samples , during training, and As the input of the network, and As a supervision signal for the network.

[0040] Step 2: If Figure 2 As shown in Figure 2, a cross-language description generation network is constructed for multilingual text images. , including: multilingual image and text detection and recognition module (for detecting and recognizing multilingual text in images), multilingual text information encoding module (for building embedded representations of multilingual text modal data), multilingual visual information encoding module (for building embedded representations of visual modal data), multimodal Transformer module (for mapping multilingual multimodal data to a common space), knowledge extraction module (for extracting structured knowledge representations in multilingual text images), language embedding module (for building embedded representations for language categories), Transformer decoding module (for decoding character sequences of image description sentences); and Each sample input in Processed in to obtain the decoded character probability sequence .

[0041] Step 2.1: The multilingual image and text detection and recognition module first detects The text area and output the position coordinates , thus based on right Crop to get the text area image , and identify The character sequence in the text recognition result is obtained ;

[0042] Step 2.2: The multilingual text information encoding module uses the encoder of the mT5 model Process and output the text feature representation matrix ,in, express Middle The text representation vector of the position, for The number of semantic vectors in ;

[0043] Step 2.3: The multilingual visual information encoding module uses the Vit model to Extract visual features and get The visual feature representation matrix ,in, express Middle The visual representation vector of the image region features, for The number of image regions in .

[0044] Step 2.4: The multilingual visual information encoding module uses the Vit model to Extract visual features and get representation The visual feature representation matrix ,in, express Middle The visual representation vector of the image region, for The number of image regions in

[0045] Step 2.5: , , They are input into the multimodal Transformer module respectively, and then go through multiple stacked multi-head attention mechanisms, feedforward operations and residual connection processing, and then perform modal fusion and semantic enhancement. The outputs of the corresponding positions of the multimodal Transformer module are represented respectively. Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding ; The multimodal Transformer module maps information in different languages ​​and modalities into a common space, overcoming the semantic gap between multilingualism and multimodality.

[0046] Step 2.6: , , Input into the knowledge extraction module for prediction to obtain the character probability sequence of the knowledge representation sequence ,in, represents the first The probability vector of characters, K represents the number of characters in the knowledge representation sequence;

[0047] Step 2.7: Converted into a one-hot vector sequence and Compare to construct the loss function ; It is used to train the network through auxiliary tasks: structured knowledge extraction tasks, giving the network the ability to extract structured knowledge and allowing the network to use structured knowledge to improve the quality of description generation.

[0048] Step 2.8: Knowledge extraction module To convert, The character with the largest probability is selected at each position, and the knowledge of the characters at all positions represents the prediction result of the sequence ,in, express The characters, express The knowledge extraction module embeds the knowledge representation sequence to obtain the embedding representation of the knowledge representation sequence. ;

[0049] Step 2.9: Language Embedding Module Embedding representation, we get The representation vector ; The language category of the description sentence specified by the user is contained in it, so that the network can easily receive language information to generate description sentences in the corresponding language.

[0050] Step 2.10: , , , , Input the Transformer decoding module to perform stacked multi-head attention operations to obtain the decoded character probability sequence ;in, express Middle The character probability vector of the position, express The number of characters in .

[0051] Step 2.11: Converted into a one-hot vector sequence and Compare to construct the loss function ; It is used to train the network on the main task: cross-lingual description generation task of text images.

[0052] Step 2.12: Calculate the total loss function ,in, express Corresponding weights; during training, you can try different The best value is determined through experimental results. Get the value.

[0053] Step 3: Construct a dataset for the multimodal text error correction task ,right The multi-language text information encoding module, multi-language visual information encoding module, and multi-modal Transformer module are pre-trained; when performing training, the multi-modal text error correction task is first used to pre-train part of the network structure to give the network text error correction capabilities and alleviate the problem of text recognition error accumulation. After pre-training is completed, other training tasks are performed.

[0054] Step 3.1: Construct a dataset for the text error correction task ,in, express The corresponding identification tag containing the wrong character, express The corresponding correct identification label;

[0055] Step 3.2: The data is sequentially input into the multilingual text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module for processing. The corresponding semantically enhanced feature vector sequence ,in, express The corresponding feature vectors, for The number of characters in ;

[0056] Step 3.3: Send it to the character classifier for prediction and output the probability of the character category corresponding to each position ;in, Indicates the output Probability vector of positions.

[0057] Step 3.4: Convert to one-hot vector ,and Compare to construct the loss function ; and used to perform back propagation on the language text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module to update the module parameters, thereby obtaining the language text information encoding module after preliminary training, the multilingual visual information encoding module after preliminary training, and the multimodal Transformer module after preliminary training.

[0058] Step 4: Based on the total loss function , through the back propagation algorithm to generate a multilingual text image cross-language description network The multilingual image and text detection and recognition module, the language text information encoding module after preliminary training, the multilingual visual information encoding module after preliminary training, the pre-trained multimodal Transformer module after preliminary training, the knowledge extraction module, the language embedding module, and the Transformer decoding module are trained to update the network parameters, thereby obtaining the trained multilingual text image cross-language description generation model. Step 4 is performed through the total loss function Two training tasks are performed simultaneously: knowledge extraction task and description generation task to update the parameters in the network. The two tasks can promote each other and thus improve the effect of description generation.

[0059] Step 5: Use the trained multilingual text image cross-language description generation model to generate the text image in the specified language category. Input any multilingual text image Make predictions to get the specified language category Down Specifically, the input multilingual text image is , the specified language category is , model pair and Process and predict each token in the description sentence one by one to get a probability sequence , take the token with the largest probability value at each position in the probability sequence, form a description statement, and return it to the user.

[0060] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0061] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A cross-language description generation method for multilingual text images, characterized in that: The following steps are involved: Step 1: Obtain multilingual text images and annotate description sentences to obtain a training set ,in, Represents multilingual text images, express The corresponding image description, and , express The characters, express The number of characters in ; express The corresponding language, express The corresponding linear representation of structured knowledge, and , express The characters, express The number of characters in ; express The number of multilingual text images in ; Step 2: Build a cross-language description generation network for multilingual text images , including: multilingual image and text detection and recognition module, multilingual text information encoding module, multilingual visual information encoding module, multimodal Transformer module, knowledge extraction module, language embedding module, Transformer decoding module; and Each sample input in Processed in to obtain the decoded character probability sequence , and used to construct the total loss function ; Step 3: Construct a dataset for the multimodal text error correction task , and used for Pre-training a multi-language text information encoding module, a multi-language visual information encoding module, and a multi-modal Transformer module to obtain a pre-trained language text information encoding module, a pre-trained multi-language visual information encoding module, and a pre-trained multi-modal Transformer module; Step 4: Based on the total loss function , through the back propagation algorithm to generate a multilingual text image cross-language description network The multilingual image and text detection and recognition module, the pre-trained language text information encoding module, the pre-trained multilingual visual information encoding module, the pre-trained multimodal Transformer module, the knowledge extraction module, the language embedding module, and the Transformer decoding module are trained to update the network parameters, thereby obtaining a trained multilingual text image cross-language description generation model; Step 5: Use the trained multilingual text image cross-language description generation model to generate the text image in the specified language category. Input any multilingual text image Make predictions to get the specified language category Down Description statement.

2. The method for generating cross-language descriptions for multilingual text images according to claim 1, characterized in that: The step 2 comprises the following steps: Step 2.1: The multilingual image and text detection and recognition module first detects The text area and output the position coordinates , thus based on right Crop to get the text area image , and identify The character sequence in the text recognition result is obtained ; Step 2.2: The multilingual text information encoding module uses the encoder of the mT5 model Process and output the text feature representation matrix ,in, express Middle The text representation vector of the position, for The number of semantic vectors in ; Step 2.3: The multilingual visual information encoding module uses the Vit model to Extract visual features and get The visual feature representation matrix ,in, express Middle The visual representation vector of the image region features, for The number of image regions in Step 2.4: The multilingual visual information encoding module uses the Vit model to Extract visual features and get representation The visual feature representation matrix ,in, express The visual representation vector of the mth image region in , for The number of image regions in Step 2.5: , , The multimodal Transformer modules are inputted respectively, and then go through multiple stacked multi-head attention mechanisms, feedforward operations and residual connection processing, modality fusion and semantic enhancement, and the corresponding outputs are Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding , Corresponding semantically enhanced feature encoding ; Step 2.6: , , Input the knowledge extraction module to make predictions and obtain the character probability sequence of the knowledge representation sequence ,in, represents the first The probability vector of characters, K represents the number of characters in the knowledge representation sequence; Step 2.7: Converted into a one-hot vector sequence and Compare to construct the loss function ; Step 2.8: The knowledge extraction module to convert and Select the character with the largest probability value at each position in the prediction result of the sequence represented by the knowledge of the characters at all positions ,in, express The characters, express The number of characters in ; The knowledge extraction module embeds the knowledge representation sequence to obtain the embedding representation of the knowledge representation sequence ; Step 2.9: The language embedding module Embedding representation, we get The representation vector ; Step 2.10: , , , , Input the Transformer decoding module to perform stacked multi-head attention operations to obtain the decoded character probability sequence ;in, express Middle The character probability vector of the position, express The number of characters in ; Step 2.11: Converted into a one-hot vector sequence and Compare to construct the loss function ; Step 2.12: Calculate the total loss function ,in, express The corresponding weight.

3. The method for generating cross-language descriptions for multilingual text images according to claim 2, characterized in that: The step 3 comprises the following steps: Step 3.1: Construct a dataset for the multimodal text correction task ,in, express The corresponding identification tag containing the wrong character, express The corresponding correct identification label; Step 3.2: The data is sequentially input into the multilingual text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module for processing. The corresponding semantically enhanced feature vector sequence ,in, express The corresponding feature vectors, for The number of characters in ; Step 3.3: Send it to the character classifier for prediction and output the probability of the character category corresponding to each position ;in, Indicates the output probability vector of positions; Step 3.4: Convert to one-hot vector , and Compare to construct the loss function , which is used to perform back-propagation on the language text information encoding module, the multilingual visual information encoding module, and the multimodal Transformer module to update the module parameters, thereby obtaining the pre-trained language text information encoding module, the pre-trained multilingual visual information encoding module, and the pre-trained multimodal Transformer module.

4. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the cross-language description generation method described in any one of claims 1 to 3, and the processor is configured to execute the program stored in the memory.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the cross-language description generation method according to any one of claims 1 to 3 are executed.

Images