Structured information extraction method for multilingual natural scene text image

Abstract

The invention discloses a structured information extraction method oriented to a multilingual natural scene text image. The method comprises the following steps: 1, constructing a data set oriented to multilingual natural scene text image information extraction; 2, constructing a natural scene text image-oriented multi-language multi-modal information extraction network; 3, pre-training text branches of the multi-modal information extraction network on the multi-language text information extraction data set; 4, training a natural scene text image-oriented multi-language multi-modal information extraction network; and 5, performing information extraction on any input multilingual text image by utilizing the trained multimodal information extraction network to obtain structured knowledge representation of vision and language information in the text image. According to the method, information extraction can be performed on the input multilingual natural scene text image in a multilingual scene, information of different languages and different modes in the text image is deeply understood, and structured knowledge representation of the text image is output.

Description

Technical Field

[0001] The present invention relates to information processing related issues of multilingual natural scene text images, and in particular to a structured information extraction method for multilingual natural scene text images. Background Art

[0002] Existing research on information extraction from text images is mainly oriented towards document images, and the information extraction of document images is realized based on the layout information and text semantic information in document images. Compared with document images, the background of natural scene text images is more complex, containing specific visual targets and meaningful visual scene information. At the same time, the text content and the form of text presentation in natural scene text images are more flexible and diverse, which brings challenges to the research on structured knowledge extraction. For natural scene text images, the number of characters in the text area contained therein is usually small. The context of the text type in natural scene text images provides relatively weak support for information extraction. Summary of the invention

[0003] In order to address the deficiencies of the above-mentioned prior art, the present invention proposes a structured information extraction method for multilingual natural scene text images, in order to simultaneously understand three different modalities of information corresponding to the visual targets in the text images, the text contents in the text images, and the descriptive sentences of the text images in a multilingual environment, thereby realizing the efficient extraction of structured knowledge.

[0004] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme:

[0005] The structured information extraction method for multilingual natural scene text images of the present invention is characterized in that it comprises the following steps:

[0006] Step 1: Obtain a multilingual text information extraction dataset ,in, represents the i-th multilingual text, A language for representing structured knowledge, express The corresponding structured knowledge, express The number of multilingual texts in

[0007] Obtain annotated multilingual natural scene text image sets ,in, represents the jth multilingual natural scene text image, The language in which the structured knowledge is annotated. express Structured knowledge annotation of express The number of multilingual natural scene text images in ;

[0008] Step 2: Build a structured information extraction network for multilingual natural scene text images , including: a text branch, a visual branch, an image descriptor, and a multimodal information extractor;

[0009] The text branch includes: a multilingual text information encoding module, a Transformer module, and a text information extraction module;

[0010] The visual branch includes: a multilingual image and text detection and recognition module, a multilingual visual information encoding module, a multilingual text information encoding module, and a pre-trained multimodal Transformer module;

[0011] The multimodal information extractor comprises: a multimodal information fusion module and a decoding module;

[0012] Step 3: Input to structured information extraction network Pre-train the text branch in to obtain the pre-trained text branch;

[0013] Step 4: Input structured information extraction network Train in and obtain the trained structured information extraction model;

[0014] Step 5: Use the trained structured information extraction model to extract any input multilingual text image Information extraction is performed to obtain a predicted structured knowledge representation, which is output as the information extraction result.

[0015] The structured information extraction method for multilingual natural scene text images described in the present invention is also characterized in that step 3 includes the following steps:

[0016] Step 3.1: Input the multilingual text information into the encoding module and use the mT5 encoder to Process and obtain The embedding representation matrix for each position in ,in, express Middle The embedding representation vector of the position, express The number of characters in represents the embedding dimension;

[0017] Step 3.2: Input into the Transformer layer, and after multiple stacked multi-head attention mechanisms, feedforward operations and residual connections, we get The semantic feature matrix ,in, express Middle Semantic features of each position;

[0018] Step 3.3: Input the text information extractor for prediction to obtain linearized knowledge representation ,in, express The corresponding linearized knowledge representation characters, Indicates the number of characters in the linearized knowledge representation;

[0019] Step 3.4: Transformation into linear knowledge representation based on tree-structured rules , and then with the predicted linear knowledge representation A comparison is performed to construct a loss function, which is used to backpropagate the text branch to update the network parameters in the text branch, thereby obtaining a pre-trained text branch.

[0020] Further, the step 4 comprises the following steps:

[0021] Step 4.1: Transfer the weights of each node of the text branch network trained in step 3 to the nodes at the corresponding positions of the multilingual multimodal information extraction network;

[0022] Step 4.2: Input the multilingual image and text detection and recognition module in the visual branch to perform text detection and recognition, and obtain the text area position coordinates respectively , crop the image in the text area And recognition results ;

[0023] Step 4.3: and Input the multilingual visual information encoding module for processing to obtain the visual embedding representation of the entire text image and visual embedding representation of cropped images of text regions ,in, express The A representation vector, express The A representation vector, express The number of vectors in , express The number of vectors in ;

[0024] Step 4.4: Based on and The parts in The position in and The parts in The corresponding position in and ;

[0025] Will and After splicing at the corresponding positions, the overall visual feature encoding of the text image is obtained. ;

[0026] Will and After splicing at the corresponding positions, the visual feature encoding of the cropped image of the text area is obtained. ;

[0027] Step 4.5: Input the multilingual text information into the encoding module for processing to obtain text embedding representation ,in, express The represents a vector, Represents the number of embedded vectors;

[0028] based on The parts in The relative position in is calculated The position codes of the parts in ;

[0029] Will and After splicing, we get the text feature encoding ;

[0030] Step 4.6: , , Input the pre-trained multimodal Transformer layer for processing and output multilingual text images Corresponding semantically enhanced visual feature encoding , crop the image in the text area Corresponding semantically enhanced visual feature encoding , recognition results Corresponding semantically enhanced text feature encoding ;

[0031] Step 4.7: Input the image description module for processing to obtain Image description sentence ;

[0032] Step 4.8: The pre-trained text is input into the branch and processed by the multilingual text information encoding module and the Transformer layer in turn, and then output Corresponding semantically enhanced text feature encoding ,in, express Middle The encoding vector, Indicates the number of vectors;

[0033] Step 4.9: , , and Input the multimodal information extractor to extract information of multilingual text images, and obtain the prediction result of linearized knowledge representation of text images ;

[0034] Step 4.10: Transformed into the i-th linearized knowledge representation through tree-based rules , and the linearized knowledge representation of the i-th prediction Compare to construct a loss function for structured information extraction network Back propagation is performed to update the network parameters to obtain the trained structured information extraction model.

[0035] Further, the step 4.9 includes the following steps:

[0036] Step 4.9.1: The multimodal information fusion module , , and Fusion is performed to obtain the feature encoding after the i-th fusion ,in, express No. The fused feature vectors, Indicates the number of fused feature vectors;

[0037] Step 4.9.2: Input into the decoding module for prediction, and obtain the linearized knowledge representation of the i-th prediction ,in, express The corresponding linearized knowledge representation characters; Indicates the total number of characters;

[0038] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the structured information extraction method, and the processor is configured to execute the program stored in the memory.

[0039] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the structured information extraction method when the computer program is executed by a processor.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention can realize information extraction of multilingual natural scene text images. When the image contains complex visual scenes, the structured information extraction network of multilingual natural scene text images can be used to simultaneously understand the multilingual text information and visual scene information in the text image, thereby overcoming the deficiency of the prior art that only uses the text information and text layout information in the image for information extraction, and can be better applied to multilingual and multimodal environments.

[0042] 2. The present invention transfers the knowledge learned by the information extraction model on the massive text modal information extraction dataset to the multilingual multimodal information extraction model, and alleviates the problem of insufficient sample size of the multilingual text image information extraction dataset through the use of external knowledge. Through the transfer of knowledge, the effect of multilingual text image structured knowledge extraction is improved.

[0043] 3. The network framework designed by the present invention integrates information from different sources, different modalities, and different granularities in multilingual text images, so that various information can complement each other and support the model's deep understanding of multilingual natural scene text images from different angles, thereby achieving better structured information extraction performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flowchart of the use of the structured information extraction method for multilingual natural scene text images of the present invention;

[0045] Figure 2 It is a network structure diagram of the structured information extraction method for multilingual natural scene text images of the present invention. DETAILED DESCRIPTION

[0046] In this embodiment, Figure 1 As shown, a structured information extraction method for multilingual natural scene text images includes the following steps:

[0047] Step 1: Obtain a multilingual text information extraction dataset ,in, represents the i-th multilingual text, A language for representing structured knowledge, express The corresponding structured knowledge, express The number of multilingual texts in Samples , during training and For input, as a supervisory signal during training.

[0048] Obtain annotated multilingual natural scene text image sets ,in, represents the jth multilingual natural scene text image, The language in which the structured knowledge is annotated. express Structured knowledge annotation of express The number of multilingual natural scene text images in . Samples , during training and For input, as a supervisory signal during training.

[0049] Step 2: Build a structured information extraction network for multilingual natural scene text images , including: a text branch (for knowledge extraction from text modality input), a visual branch (for building features for multimodal information extraction from visual modality input), an image descriptor (for building description sentences for input multilingual text images), and a multimodal information extractor (for information extraction based on multimodal features);

[0050] Text branch, including: 1 multilingual text information encoding module, 1 Transformer module, and 1 text information extraction module;

[0051] The visual branch includes: 1 multilingual image and text detection and recognition module, 1 multilingual visual information encoding module, 1 multilingual text information encoding module, and 1 pre-trained multimodal Transformer module;

[0052] Multimodal information extractor, including: 1 multimodal information fusion module and 1 decoding module.

[0053] Step 3: Input to structured information extraction network The text branch in is pre-trained to obtain the pre-trained text branch. The pre-training aims to solve the problem of insufficient training samples for multilingual natural scene text image information extraction, so that the model can learn external knowledge from massive text data, thereby improving the performance of the multilingual text image structured knowledge extraction network.

[0054] Step 3.1: Input the multilingual text information into the encoding module and use the mT5 encoder to Process and obtain The embedding representation matrix for each position in ,in, express Middle The embedding representation vector of the position, express The number of characters in represents the embedding dimension;

[0055] Step 3.2: Input into the Transformer layer, and after multiple stacked multi-head attention mechanisms, feedforward operations and residual connections, we get The semantic feature matrix ,in, express Middle Semantic features of each position;

[0056] Step 3.3: Input the text information extractor for prediction and obtain linearized knowledge representation ,in, express The corresponding linearized knowledge representation characters, Indicates the number of characters in the linearized knowledge representation;

[0057] Step 3.4: Transformation into linear knowledge representation based on tree-structured rules , and then with the predicted linear knowledge representation A comparison is performed to construct a loss function, which is used to backpropagate the text branch to update the network parameters in the text branch, thereby obtaining a pre-trained text branch.

[0058] Step 4: Input structured information extraction network Train in and obtain the trained structured information extraction model;

[0059] Step 4.1: Use the weights of each node of the text branch network trained in step 3 to migrate to the nodes at the corresponding positions of the multilingual multimodal information extraction network. This is used to migrate the knowledge learned by the information extraction model on the massive text modal information extraction dataset to the multilingual multimodal information extraction model.

[0060] Step 4.2: Input the multilingual image and text detection and recognition module in the visual branch to perform text detection and recognition, and obtain the text area position coordinates respectively , crop the image in the text area And recognition results ;

[0061] Step 4.3: and Input the multilingual visual information encoding module for processing to obtain the overall visual embedding representation of the text image and visual embedding representation of cropped images of text regions ,in, express The A representation vector, express The A representation vector, express The number of vectors in , express The number of vectors in .

[0062] Step 4.4: Based on and The parts in The position in and The parts in The corresponding position in and ;

[0063] Will and After splicing at the corresponding positions, the overall visual feature encoding of the text image is obtained. ;

[0064] Will and After splicing at the corresponding positions, the visual feature encoding of the cropped image of the text area is obtained. .

[0065] Step 4.5: Input the multilingual text information into the encoding module for processing to obtain the text embedding representation ,in, express The represents a vector, Represents the number of embedded vectors;

[0066] based on The parts in The relative position in is calculated The position codes of the parts in ;

[0067] Will and After splicing, we get the text feature encoding ;

[0068] Step 4.6: , , Input the pre-trained multimodal Transformer layer for processing and output multilingual text images Corresponding semantically enhanced visual feature encoding , crop the image in the text area Corresponding semantically enhanced visual feature encoding , recognition results Corresponding semantically enhanced text feature encoding .

[0069] Step 4.7: Input the image description module for processing and obtain Image description sentence ;

[0070] Step 4.8: After inputting the pre-trained text branch, it is processed by the multilingual text information encoding module and the Transformer layer in turn, and then output Corresponding semantically enhanced text feature encoding ,in, express Middle The encoding vector, Indicates the number of vectors;

[0071] Step 4.9: , , and Input the multimodal information extractor to extract information of multilingual text images and obtain the prediction results of linearized knowledge representation of text images ;

[0072] Step 4.9.1: Multimodal information fusion module , , and Fusion is performed to obtain the feature encoding after the i-th fusion ,in, express No. The fused feature vectors, Indicates the number of fused feature vectors. By integrating information from different sources, different modalities, and different granularities in multilingual text images, various information can complement each other and support the model's deep understanding of multilingual natural scene text images from different angles, thereby achieving better structured information extraction performance;

[0073] Step 4.9.2: Input the prediction into the decoding module to obtain the linearized knowledge representation of the i-th prediction ,in, express The corresponding linearized knowledge representation characters; Indicates the total number of characters;

[0074] Step 4.10: Transformed into the i-th linearized knowledge representation through tree-based rules , and the linearized knowledge representation of the i-th prediction Compare to construct a loss function for structured information extraction network Perform back propagation to update the network parameters to obtain the trained structured information extraction model;

[0075] Step 5: Use the trained structured information extraction model to extract any input multilingual text image Information extraction is performed to obtain the predicted structured knowledge representation and output it as the information extraction result. Specifically, the input multilingual text image is denoted as , right Process and predict each character in the linearized knowledge representation one by one to obtain a character sequence , using tree-based rules to transform character sequences It is converted into a structured knowledge representation and returned to the user as an information extraction result.

[0076] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0077] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.

Claims

1. A method for extracting structured information from multilingual natural scene text images, characterized in that: The following steps are involved: Step 1: Obtain a multilingual text information extraction dataset ,in, represents the i-th multilingual text, A language for representing structured knowledge, express The corresponding structured knowledge, express The number of multilingual texts in Obtain annotated multilingual natural scene text image sets ,in, represents the jth multilingual natural scene text image, The language in which the structured knowledge is annotated. express Structured knowledge annotation of express The number of multilingual natural scene text images in ; Step 2: Build a structured information extraction network for multilingual natural scene text images , including: a text branch, a visual branch, an image descriptor, and a multimodal information extractor; The text branch includes: a multilingual text information encoding module, a Transformer module, and a text information extraction module; The visual branch includes: a multilingual image and text detection and recognition module, a multilingual visual information encoding module, a multilingual text information encoding module, and a pre-trained multimodal Transformer module; The multimodal information extractor comprises: a multimodal information fusion module and a decoding module; Step 3: Input to structured information extraction network Pre-train the text branch in to obtain the pre-trained text branch; Step 4: Input structured information extraction network Train in and obtain the trained structured information extraction model; Step 5: Use the trained structured information extraction model to extract any input multilingual text image Information extraction is performed to obtain a predicted structured knowledge representation, which is output as the information extraction result.

2. The method for extracting structured information from multilingual natural scene text images according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Input the multilingual text information into the encoding module and use the mT5 encoder to Process and obtain The embedding representation matrix for each position in ,in, express Middle The embedding representation vector of the position, express The number of characters in represents the embedding dimension; Step 3.2: Input into the Transformer layer, and after multiple stacked multi-head attention mechanisms, feedforward operations and residual connections, we get The semantic feature matrix ,in, express Middle Semantic features of each position; Step 3.3: Input the text information extractor for prediction to obtain linearized knowledge representation ,in, express The corresponding linearized knowledge representation characters, Indicates the number of characters in the linearized knowledge representation; Step 3.4: Transformation into linear knowledge representation based on tree-structured rules , and then with the predicted linear knowledge representation A comparison is performed to construct a loss function, which is used to backpropagate the text branch to update the network parameters in the text branch, thereby obtaining a pre-trained text branch.

3. The method for extracting structured information from multilingual natural scene text images according to claim 2, characterized in that: The step 4 comprises the following steps: Step 4.1: Transfer the weights of each node of the text branch network trained in step 3 to the nodes at the corresponding positions of the multilingual multimodal information extraction network; Step 4.2: Input the multilingual image and text detection and recognition module in the visual branch to perform text detection and recognition, and obtain the text area position coordinates respectively , crop the image in the text area and recognition results ; Step 4.3: and Input the multilingual visual information encoding module for processing to obtain the visual embedding representation of the entire text image and visual embedding representation of cropped images of text regions ,in, express The A representation vector, express The A representation vector, express The number of vectors in , express The number of vectors in ; Step 4.4: Based on and The parts in The position in and The parts in The corresponding position in and ; Will and After splicing at the corresponding positions, the overall visual feature encoding of the text image is obtained. ; Will and After splicing at the corresponding positions, the visual feature encoding of the cropped image of the text area is obtained. ; Step 4.5: Input the multilingual text information into the encoding module for processing to obtain text embedding representation ,in, express The represents a vector, Represents the number of embedded vectors; based on The parts in The relative position in is calculated The position codes of the parts in ; Will and After splicing, we get the text feature encoding ; Step 4.6: , , Input the pre-trained multimodal Transformer layer for processing and output multilingual text images Corresponding semantically enhanced visual feature encoding , crop the image in the text area Corresponding semantically enhanced visual feature encoding , recognition results Corresponding semantically enhanced text feature encoding ; Step 4.7: Input the image description module for processing to obtain Image description sentence ; Step 4.8: The pre-trained text is input into the branch and processed by the multilingual text information encoding module and the Transformer layer in turn, and then output Corresponding semantically enhanced text feature encoding ,in, express Middle The encoding vector, Indicates the number of vectors; Step 4.9: , , and Input the multimodal information extractor to extract information of multilingual text images, and obtain the prediction result of linearized knowledge representation of text images ; Step 4.10: Transformed into the i-th linearized knowledge representation through tree-based rules , and the linearized knowledge representation of the i-th prediction Compare to construct a loss function for structured information extraction network Back propagation is performed to update the network parameters to obtain the trained structured information extraction model.

4. The method for extracting structured information from multilingual natural scene text images according to claim 3, characterized in that: The step 4.9 comprises the following steps: Step 4.9.1: The multimodal information fusion module , , and Fusion is performed to obtain the feature encoding after the i-th fusion ,in, express No. The fused feature vectors, Indicates the number of fused feature vectors; Step 4.9.2: Input into the decoding module for prediction, and obtain the linearized knowledge representation of the i-th prediction ,in, express The corresponding linearized knowledge representation characters; Indicates the total number of characters.

5. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the structured information extraction method according to any one of claims 1 to 4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the structured information extraction method according to any one of claims 1 to 4 are performed.

Images