End-to-end document image translation method and device based on feature conduction

Abstract

The application provides an end-to-end document image translation method and device based on feature conduction, which comprises the following steps: obtaining a document image to be translated; extracting geometric joint features of a character recognition result of the document image to be translated, wherein the geometric joint features comprise geometric layout features and text features; obtaining logical layout features of the document image to be translated based on the geometric joint features; performing feature fusion on the logical layout features and the text features in the geometric joint features to obtain translation guide features of the document image to be translated; and obtaining a document image translation result based on the translation guide features. The method provided by the application explicitly models geometric layout and logical layout information, and fuses them with text features, thereby improving the translation capability of the document image with a complex layout. Meanwhile, a unified joint framework based on feature conduction is provided, and the collaborative optimization of the layout module and the translation module is enhanced.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an end-to-end document image translation method and apparatus based on feature propagation. Background Technology

[0002] Unlike plain text, text in document images is organized and arranged according to a specific geometric layout and logical order. To more accurately understand the layout of document images and perform accurate text translation of the text within each page, existing technologies generally employ an end-to-end method with an image encoder-translation decoder structure. This existing end-to-end method first encodes the document image using an image encoder to extract the text and layout features, and then uses a translation decoder to generate the target language translation.

[0003] While existing end-to-end methods are simple to implement, their approach of implicitly extracting layout features based on image encoders cannot handle complex layout structures well, thus their translation accuracy needs to be improved. Summary of the Invention

[0004] This invention provides an end-to-end document image translation method and apparatus based on feature propagation, which solves the problem that the existing method of implicitly extracting layout features based on image encoders cannot handle complex layout structures well.

[0005] This invention provides an end-to-end document-image translation method based on feature propagation, comprising:

[0006] Obtain the image of the document to be translated;

[0007] Extract the geometric joint features of the character recognition results of the document image to be translated, wherein the geometric joint features include geometric layout features and text features;

[0008] Based on the geometric joint features, the logical layout features of the document image to be translated are obtained;

[0009] The logical layout features and the text features in the geometric joint features are fused to obtain the translation guidance features of the document image to be translated;

[0010] Based on the translation-oriented features, the document image to be translated is translated into text, resulting in a document image translation.

[0011] According to the present invention, an end-to-end document image translation method based on feature propagation is provided, wherein the logical layout features of the document image to be translated are obtained based on the geometric joint features, and then the method includes:

[0012] Based on the logical layout features, the logical reading order of the document image to be translated is predicted;

[0013] The step of fusing the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated includes:

[0014] Based on the logical reading order, the text features are sorted to obtain rearranged text features;

[0015] The logical layout features and the rearranged text features are fused to obtain the translation guidance features of the document image to be translated.

[0016] According to the present invention, an end-to-end document image translation method based on feature propagation is provided, wherein the method performs text translation on the document image to be translated based on the translation-guided features to obtain a document image translation result, including:

[0017] Based on the translation-guided features, predict the sentence boundary category of each word segment in the document image to be translated;

[0018] Based on the sentence boundary categories, the translation guidance features are divided into sentence-level segments to obtain sentence-level translation guidance features;

[0019] The sentence-level translation guidance features are decoded to obtain the document image translation result.

[0020] According to the end-to-end document image translation method based on feature propagation provided by the present invention, the character recognition result includes the recognized characters of each word segment, geometric layout coordinates, and one-dimensional position information;

[0021] The steps for obtaining the geometric layout features and the text features include:

[0022] The original text features of the identified characters, the original geometric layout features of the geometric layout coordinates, and the one-dimensional position features of the one-dimensional position information are extracted respectively.

[0023] The one-dimensional positional feature is superimposed on the original text feature and the original geometric layout feature to obtain the geometric layout feature and the text feature.

[0024] According to the end-to-end document image translation method based on feature propagation provided by the present invention, the step of extracting the geometric joint features of the character recognition results of the document image to be translated includes:

[0025] Contextual deep encoding is performed on the geometric layout features and the text features respectively to extract deep geometric layout features and deep text features;

[0026] The deep geometric layout features and the deep text features are concatenated, and the concatenated features are linearly mapped to obtain the geometric joint features.

[0027] According to the end-to-end document image translation method based on feature propagation provided by the present invention, the step of obtaining the character recognition result includes:

[0028] The document image to be translated is normalized to obtain a normalized document image;

[0029] Optical character recognition is performed on the normalized document image to obtain the character recognition result.

[0030] The present invention also provides an end-to-end document-image translation device based on feature propagation, comprising:

[0031] Get the unit, and get the image of the document to be translated;

[0032] The geometric layout unit extracts the geometric joint features of the character recognition results of the document image to be translated, and the geometric joint features include geometric layout features and text features;

[0033] Logical layout unit, based on the geometric joint features, obtains the logical layout features of the document image to be translated;

[0034] The fusion unit performs feature fusion between the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated;

[0035] The translation unit performs text translation on the document image to be translated based on the translation-oriented features, and obtains the document image translation result.

[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end document image translation method based on feature propagation as described above.

[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the end-to-end document image translation method based on feature propagation as described above.

[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the end-to-end document image translation method based on feature propagation as described above.

[0039] The present invention provides an end-to-end document image translation method and apparatus based on feature propagation. By using the geometric joint features of the character recognition results of the document image to be translated, the logical layout features of the document image are obtained. These logical layout features are then explicitly fused with the extracted text features to obtain translation-guided features for translation. This achieves explicit modeling of the geometric and logical layout information of the document image and fusion with text features, improving the translation capability for document images with complex layouts. Simultaneously, a unified joint framework based on feature propagation is provided to enhance the collaborative optimization between the layout module and the translation module. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0041] Figure 1 This is one of the flowcharts of the end-to-end document image translation method based on feature propagation provided by the present invention;

[0042] Figure 2 This is a schematic diagram of the document image to be translated according to the present invention;

[0043] Figure 3 This is a schematic diagram of the workflow of the logic layout module provided by the present invention;

[0044] Figure 4 This is a schematic diagram of the workflow of the translation module provided by the present invention;

[0045] Figure 5 This is a schematic diagram of the character recognition results provided by the present invention;

[0046] Figure 6 This is the second flowchart of the end-to-end document image translation method based on feature propagation provided by the present invention;

[0047] Figure 7 This is a comparative schematic diagram of existing document image translation methods provided by the present invention;

[0048] Figure 8 This is a schematic diagram of the end-to-end document image translation device based on feature propagation provided by the present invention;

[0049] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] Documents can be paper, images, or electronic files containing text. Document images refer to images of text on paper or physical surfaces generated through scanning, photographing, or other methods; they exist extensively in daily life and communication channels such as the internet. Document image translation aims to automatically translate text embedded in document images from the source language to the target language; it is one of the key technologies for automating document information processing.

[0052] It should be noted that, unlike plain text, text in document images is organized and arranged according to a specific geometric layout and logical order. Although the layout and logical order can be parsed using manually designed rules to extract the text content, and document image translation can be achieved through plain text machine translation, in practical applications, documents from different fields and with different formats often exhibit inconsistent layout structures and logical orders. This makes the current method unable to handle different types of document images simultaneously, resulting in very limited generalization. Therefore, an end-to-end document image translation method can better handle different types of document images.

[0053] However, existing end-to-end document image translation methods rely on implicit extraction of layout features using image encoders, resulting in unsatisfactory translation performance for documents with complex layouts. To address this issue, this invention provides an end-to-end document image translation method based on feature propagation, which explicitly incorporates layout information to improve the accuracy of translating documents with complex layouts. Figure 1 This is one of the flowcharts illustrating the end-to-end document-image translation method based on feature propagation provided by the present invention, such as... Figure 1 As shown, the method includes:

[0054] Step 110: Obtain the image of the document to be translated;

[0055] Here, the document image to be translated can be composed of text corresponding to multiple geometric layouts. These layouts and their corresponding texts must have a logical relationship, which could be a reading logic relationship. For example, Figure 2 This is a schematic diagram of the document image to be translated according to the present invention, such as... Figure 2As shown, the document image to be translated can be a task plan, where the overall task objective, sub-objectives, and sub-tasks at each stage each correspond to a separate page, containing the corresponding text content. It should be noted that since the text content in the document image may be divided into multiple pages, when translating the document image, it is necessary to consider not only the accuracy of the translation but also the text logic reflected in the page layout. Only by obtaining the correct text logic can accurate and effective document image translation be achieved.

[0056] Specifically, text on paper or a physical surface can be scanned or photographed to obtain an image, which can then be used as the document image to be translated. Alternatively, the document image to be translated can be obtained by accepting images containing text content uploaded by users. The language of the text in the document image to be translated can be any language, such as Chinese or English, so the target language of the document image to be translated can be any language different from the original language.

[0057] Step 120: Extract the geometric joint features of the character recognition results of the document image to be translated, wherein the geometric joint features include geometric layout features and text features;

[0058] Here, the character recognition results can reflect the character recognition results of each word in the document image to be translated, as well as the positional information of each word. The geometric joint features here refer to both geometric layout features and text features. Furthermore, there is a mapping relationship between geometric features and text features; this mapping relationship can be the mapping relationship between the geometric layout and text in the document image to be translated.

[0059] The geometric layout features here reflect the two-dimensional positional information of the text within the entire document image to be translated, such as the coordinates of the vertices of the bounding box surrounding the text. Additionally, the text features reflect the semantic information of the text and can be character-level text features.

[0060] Specifically, a layout encoder can be used to encode the geometric layout information in the character recognition results to extract the geometric layout features of the document image to be translated. Simultaneously, a text encoder can be used to encode the text in the character recognition results to extract the text features of the document image to be translated. Furthermore, the geometric layout features and text features can be concatenated and then linearly mapped to obtain joint geometric features.

[0061] Step 130: Based on the geometric joint features, obtain the logical layout features of the document image to be translated;

[0062] Here, logical layout features can be used to reflect the reading logic of each page in the document image to be translated. Specifically, the logical layout features can be obtained by inputting the geometric joint features into the logical layout module, and then performing forward computation on the geometric joint features through the logical layout module. The logical layout module can be obtained through an encoder and a decoder.

[0063] It should be noted that, for document images, the reading order of text within the image depends not only on the semantic information of the text but also on its layout. Therefore, based on geometric joint features—that is, textual features and geometric layout features—a logical layout feature reflecting the reading order can be obtained. This facilitates a more accurate understanding of complex layouts, thereby improving translation quality.

[0064] Step 140: Perform feature fusion between the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated;

[0065] Here, translation-oriented features can be used to reflect the semantic information of the text in the image of the document to be translated, as well as the reading logic information of the text. Specifically, the translation-oriented features can be obtained by fusing logical layout features and text features based on a dynamic gating mechanism through a bridging module.

[0066] It should be noted that, compared to existing document image translation methods that implicitly extract layout features based on image encoders, this method explicitly integrates logical layout features, which reflect geometric and logical layout information, into translation-oriented features used for document translation. This enhances the ability to understand the layout logic of document images with complex layout structures and logical sequences, thereby improving the translation capability of document images with complex layout structures and logical sequences.

[0067] Step 150: Based on the translation-guided features, perform text translation on the document image to be translated to obtain the document image translation result.

[0068] Specifically, translation-oriented features can be transmitted to the translation module, where a decoder performs sentence-by-sentence translation to generate document image translation results corresponding to the target language.

[0069] It should be noted that during the translation process of the document image to be translated, the initially extracted text features and geometric layout features are used for feature transfer to obtain the final translation-oriented features used for translation. This realizes an end-to-end document image translation method based on feature transfer, strengthening the interaction and connection between various modules. Furthermore, geometric layout information and logical layout information are fused to achieve joint optimization of various modules, avoiding the defects of error accumulation and lack of interaction caused by the independent existence of each module, thereby greatly improving the performance of document image translation.

[0070] The method provided in this invention obtains the logical layout features of the document image to be translated by using the geometric joint features of the character recognition results. The logical layout features are then explicitly fused with the extracted text features to obtain translation-oriented features for translation. This method explicitly models the geometric and logical layout information of the document image and fuses them with text features, improving the translation capability for document images with complex layouts. Simultaneously, a unified joint framework based on feature propagation is provided to enhance the collaborative optimization between the layout module and the translation module.

[0071] Based on any of the above embodiments, step 130 is followed by:

[0072] Based on the logical layout features, the logical reading order of the document image to be translated is predicted;

[0073] The step of fusing the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated includes:

[0074] Based on the logical reading order, the text features are sorted to obtain rearranged text features;

[0075] The logical layout features and the rearranged text features are fused to obtain the translation guidance features of the document image to be translated.

[0076] Here, logical reading order refers to a reasonable and easy-to-understand reading order determined based on the image content, layout, and context of the document to be translated. This order helps readers browse and understand the image content according to logical clues and the importance of information.

[0077] Understandably, when translating document images, following the logical structure of the text within the image allows for translation in the correct order, avoiding misunderstandings or omissions due to improper reading order, thereby improving translation accuracy. Furthermore, a logical reading order helps maintain the coherence and consistency of the translation and enhances the readability of the final document image translation.

[0078] Furthermore, the rearranged text features here refer to text features ordered according to the logical reading order. It's understandable that when extracting text features, a fixed order such as top-to-bottom or left-to-right might be used, resulting in text features that are not ordered according to the logical reading order.

[0079] Specifically, after obtaining the logical layout features through step 130, the reading order of each text segment can be predicted through the logical layout module to obtain an index sequence that reflects the reading order, i.e., the logical reading order here.

[0080] Furthermore, after obtaining the logical reading order of the document image to be translated, the text features of each word segment can be reordered based on this logical reading order to obtain rearranged text features that conform to the logical reading order. Next, based on a dynamic gating attention mechanism, the logical layout features and the rearranged text features are fused to obtain translation-oriented features. Here, feature fusion can be achieved using the following formula:

[0081] F trans =ε·F lgc +(1-ε)·F t

[0082] In the formula, F trans Indicates translation-oriented characteristics; ε represents the dynamic weighting factor; F lgc Indicates logical layout characteristics; F t This represents the rearranged text features. The dynamic weighting factor ε can be calculated using the following formula:

[0083] ε = Sigmoid(w lgc ·F lgc +w t ·F t +b)

[0084] In the formula, w lgc w t represents the learnable parameter; b represents the learnable bias term.

[0085] In one embodiment, Figure 3 This is a schematic diagram of the workflow of the logic layout module provided by the present invention, as shown below. Figure 3 As shown, the process includes: the input to the logic layout module includes word text t1, ..., t5, ..., t L Word bounding boxes b1, ..., b5, ..., b L Page layout and text features Initial order: idx1, ..., idx5, ..., idx L The layout-text features can be obtained by extracting features from word bounding boxes and word text separately, and the initial order can represent the recognition order of the text in the character recognition results. Additionally, BOS Tokens represent word text, and BOS Boxes represent word bounding boxes.

[0086] In detail, the word text, word bounding boxes, and corresponding layout-text features of each segmented word can be input into the logical layout module, and the logical layout module can output the corresponding logical layout features respectively. Furthermore, through classification tasks on each index category of L, the logical reading order idx1, ..., idx5, ..., idx is predicted based on logical layout features to obtain the word text and word bounding boxes corresponding to each segmentation. L .

[0087] It should be noted that during the training of the logical layout module, a loss function for the logical reading order prediction task can be constructed to iterate the parameters of the logical layout module, guiding it to better learn logical layout features and more accurately predict the logical reading order. The loss function here can be calculated using the following formula:

[0088]

[0089] In the formula, Loss lgc idx represents the prediction loss for logical reading order; L represents the total number of segmented texts in the document image to be translated, and i represents the sequence number of the segmented texts; i The label represents the logical order index of the i-th word segment; This represents the probability distribution of the logical reading order of the i-th word segment predicted by the logical layout module.

[0090] The method provided in this invention predicts a logical reading order based on logical layout features, reorders text features based on the logical reading order, and fuses the rearranged text features with logical layout features to obtain translation-oriented features for translation. By incorporating geometric layout information and logical layout information into the translation process, the method greatly improves the ability to understand the layout of document images with complex layouts, thereby improving the accuracy of document translation results.

[0091] Based on any of the above embodiments, step 150 includes:

[0092] Based on the translation-guided features, predict the sentence boundary category of each word segment in the document image to be translated;

[0093] Based on the sentence boundary categories, the translation guidance features are divided into sentence-level segments to obtain sentence-level translation guidance features;

[0094] The sentence-level translation guidance features are decoded to obtain the document image translation result.

[0095] Specifically, translation-guided features can be passed to the sentence segmentation module within the translation module. The sentence segmentation module then predicts the sentence boundary category for each segmented word. Here, the sentence boundary categories include the sentence start category (BOS) and the sentence internal category (IOS). The sentence start category refers to the segmented word at the beginning of the sentence, while the sentence internal category includes the segmented words in the middle and at the end of the sentence.

[0096] Next, the translation-oriented features can be segmented at the sentence level using sentence boundary categories, resulting in multiple feature subsequences corresponding to multiple sentences, i.e., multiple sentence-level translation-oriented features. For example, the word segments corresponding to the sentence beginning category (representing the sentence beginning boundary) and the sentence internal category (representing the category before the sentence beginning boundary) can be grouped into a single sentence. Thus, the translation-oriented features corresponding to these word segments constitute the sentence-level translation-oriented features for this sentence.

[0097] It should be noted that each sentence-level translation-oriented feature corresponds to a source language sentence. Therefore, during the training phase of the sentence segmentation module, a segmentation loss can be constructed to iterate the parameters of the sentence segmentation module, resulting in more accurate sentence segmentation. This segmentation loss can be calculated using the following formula:

[0098]

[0099] In the formula, Loss seg Indicates the splitting loss; bdr i P represents the sentence boundary category label of the i-th word in the word segmentation sequence of the document image to be translated; i seg L represents the probability distribution of the sentence boundary category of the i-th segment predicted by the sentence segmentation module; L represents the total number of segments in the document image to be translated, and i represents the segmentation sequence number.

[0100] Finally, the translation-guided features of each sentence can be decoded one by one using the decoder in the translation module to obtain the translation results for each sentence. A column search decoding strategy can be used to obtain the document image translation results according to the decoding order of the sentence-level translation-guided features.

[0101] It should also be noted that during the training phase of the translation module, the parameters of the translation module can be iterated by constructing a translation loss to improve the translation accuracy. The translation loss here can be calculated using the following formula:

[0102]

[0103] In the formula, Loss trans Indicates translation loss; Y k,j This represents the j-th word of the k-th translated sentence; Y represents the probability distribution of the target language translation result of the j-th word segment of the k-th translated sentence predicted by the translation module; M represents the total number of sentences; |Y k | indicates the number of word segments in the k-th sentence.

[0104] The method provided in this invention divides translation-oriented features into sentence-level segments during text translation, resulting in multiple sentence-level translation-oriented features. It is understood that decoding sentence-by-sentence translation-oriented features sequentially yields more accurate translation results compared to decoding individual, fragmented translation-oriented features.

[0105] In one embodiment, Figure 4 This is a schematic diagram of the workflow of the translation module provided by the present invention, as follows: Figure 4 As shown, where, Indicates the translation-oriented characteristics of each segmentation; The features used to predict the sentence boundary categories of each segmented word are represented; BOS and IOS represent the sentence start category and sentence internal category, respectively; R represents the feature sequence segmentation operation. This indicates sentence-level translation guidance features.

[0106] In detail, the translation-oriented features of each word segment are input into the sentence segmentation module, which predicts the sentence boundary category of each word segment. Then, based on the sentence boundary categories of each word segment, sentence-level segmentation is performed. Words corresponding to the sentence beginning category (representing the sentence's initial boundary) and the sentence internal category (representing the next sentence's internal boundary) are grouped into a single sentence. The translation-oriented features corresponding to these word segments are thus the sentence-level translation-oriented features of that sentence. For example, the translation-oriented features of each word segment can be... As a sentence-level translation guidance feature Next, the obtained sentence-level translation-guided features are input into the translation submodule, and the corresponding target sentences are translated by the translation submodule to obtain the final document image translation result.

[0107] Based on any of the above embodiments, the character recognition result includes the recognized characters of each word segment, geometric layout coordinates, and one-dimensional position information;

[0108] The steps for obtaining the geometric layout features and the text features include:

[0109] The original text features of the identified characters, the original geometric layout features of the geometric layout coordinates, and the one-dimensional position features of the one-dimensional position information are extracted respectively.

[0110] The one-dimensional positional feature is superimposed on the original text feature and the original geometric layout feature to obtain the geometric layout feature and the text feature.

[0111] Here, the recognized characters represent the text recognition results of each word segment; the geometric layout coordinates represent the two-dimensional position information of each word segment in the document image to be translated. The position of each word segment can be uniquely determined by the coordinates of the upper left and lower right corners of the bounding box surrounding it; the one-dimensional position information can reflect the position information of each word segment in each analysis sequence in the document image to be translated, for example, it can be the first word in the segmentation sequence.

[0112] Specifically, the encoder can be used to encode the features of the recognized characters, geometric layout coordinates, and one-dimensional position information respectively, and extract the original text features, original geometric layout, and one-dimensional position features of the recognized characters respectively.

[0113] Furthermore, the one-dimensional positional features are superimposed with the original text features and the original geometric layout features to obtain the final geometric layout features and text features.

[0114] The method provided in this invention integrates the one-dimensional position information of each word segment into text features and geometric features, and integrates fine-grained layout information into the word segmentation features and layout features of each word segment. This enables a joint understanding of the "text-layout" corresponding to the image of the document to be translated, significantly improving the ability to understand different formats and layout structures.

[0115] Based on any of the above embodiments, the extraction of geometric joint features of the character recognition results of the document image to be translated includes:

[0116] Contextual deep encoding is performed on the geometric layout features and the text features respectively to extract deep geometric layout features and deep text features;

[0117] The deep geometric layout features and the deep text features are concatenated, and the concatenated features are linearly mapped to obtain the geometric joint features.

[0118] Specifically, deep context encoding can be performed on geometric layout features and text features separately to extract deep geometric layout features and deep text features, resulting in layout features and text features containing contextual information. Next, the deep geometric layout features and deep text features are concatenated, and the concatenated features are linearly mapped to obtain joint geometric features. Thus, through the interaction and transmission between various features, a final joint geometric feature reflecting comprehensive and rich textual and layout information is obtained, thereby improving the end-to-end document image translation capability based on feature transmission.

[0119] Based on any of the above embodiments, the step of obtaining the character recognition result includes:

[0120] The document image to be translated is normalized to obtain a normalized document image;

[0121] Optical character recognition is performed on the normalized document image to obtain the character recognition result.

[0122] Specifically, firstly, the image of the document to be translated can be normalized. A pixel value threshold of 1000 is set, normalizing the height and width of the image to the range of [0, 1000] pixel values ​​to obtain the normalized document image. Next, OCR (Optical Character Recognition) processing is performed on the normalized document image to extract the recognized text and geometric coordinates of each word, thus obtaining the character recognition result. Figure 5 This is a schematic diagram of the character recognition results provided by the present invention, as shown below. Figure 5 As shown, it contains each word segment under each page of the document image to be translated, as well as the bounding box surrounding each word segment.

[0123] Based on any of the above embodiments Figure 6 This is the second flowchart of the end-to-end document image translation method based on feature propagation provided by the present invention, as shown below. Figure 6 As shown, the method includes: optical character recognition, layout understanding, and text translation.

[0124] First, in the optical character recognition module, OCR can be used to recognize the image of the document to be translated, obtaining text containing words and word bounding boxes.

[0125] Next, the identified word texts and word bounding boxes are input into the layout understanding module. In the layout understanding module, the word bounding boxes B and word text T are input into the geometric layout module. The layout encoder in the geometric layout module performs feature encoding on the word bounding boxes, extracting the layout features F for each word segmentation. l The text features F of each word segment are extracted by using the text encoder in the geometric layout module to perform feature encoding. t Next, the text features and layout features are concatenated and linearly mapped to obtain the layout-text feature F. lt Furthermore, the layout-text features are input into the logical layout module, and the logical layout features F are input through the logical layout module. lgc Based on logical layout features, the logical reading order is predicted. In the bridging module, the text features are rearranged based on this logical reading order. Then, the rearranged text features and the logical layout features are fused using a gating mechanism to obtain the translation guidance feature F. trans .

[0126] Furthermore, the translation-oriented features are input into the translation text module. Within the translation text module, these features are then input into the sentence segmentation module, where the translation-oriented features of each segment are divided into sentence-level features. Next, the sentence-level translation-oriented features are input into the translation submodules, which perform sentence-by-sentence translation to obtain their respective target sentences 1, ..., M.

[0127] Specifically, corresponding loss functions can be constructed for the logical layout module, sentence segmentation module, and translation sub-module. That is, the logical reading order prediction loss L... lgc Sentence segmentation loss L seg Translation loss L trans .

[0128] It should be noted that, Figure 7 This is a comparative schematic diagram of existing document image translation methods provided by the present invention, such as... Figure 7 As shown, existing document image translation methods include multiple independent models (cascaded), intuitive structures (end-to-end), and the end-to-end document image translation method based on feature propagation provided in this embodiment of the invention.

[0129] Among them, the translation method based on multiple independent models (cascaded) includes a layout parsing model, a logical order prediction model, a sentence segmentation model, and a translation model. Each model is designed independently, and the models are sequentially connected to obtain the layout understanding result and the translation result in turn. This type of method has good performance, but it suffers from defects such as error accumulation and independent modules without interaction, which cannot guarantee the cooperation and adaptation between modules, thus limiting its performance ceiling.

[0130] Furthermore, existing end-to-end methods first encode the document image using an image encoder to extract text and layout features, and then use a translation decoder to generate a translation in the target language. These methods are simple to implement and alleviate the error accumulation problem, but their method of implicitly extracting layout features based on an image encoder cannot handle complex layout structures well.

[0131] The present invention provides an end-to-end unified document image translation method based on feature propagation. First, the method designs a layout understanding module for document image translation, specifically including a geometric layout understanding module based on a dual-stream structure and a logical layout understanding module oriented towards logical reading order. This aims to explicitly integrate these two key types of layout information—geometric and logical layouts—into the document image translation process. Next, a layout bridging mechanism based on dynamic gating aims to fuse the output features of the geometric and logical layouts into translation-oriented features. Furthermore, a module collaboration mechanism based on feature propagation combines modules into a unified end-to-end model by transferring and fusing features between modules, thereby maximizing inter-module collaboration.

[0132] It should also be noted that the translation effect of the document image translation provided by the embodiments of the present invention can be verified on two document image translation datasets, DITrans-General and DITrans-Specific. The document images in DITrans-General belong to the general domain, while those in DITrans-Specific include four specific domains: reports, newspapers, advertisements, and photographic books. Combining the two datasets allows verification of the method's translation effect on document images of various layout types in both general and specific domains. Furthermore, to verify the advantages of the method provided by the present invention in multi-domain scenarios, multi-domain translation experiments were conducted on DITrans-Specific. By comparing existing cascaded methods and constructing an end-to-end method based on a layout-aware encoder-translation decoder, the end-to-end method provided by the present invention was compared. The evaluation metrics used were the two most commonly used evaluation metrics for translation tasks: BLEU and chrF++. The experimental results on the two datasets are shown in the table below:

[0133] Table 1 shows the results of different document image translation methods on the DITrans-General dataset, as shown in the table below:

[0134]

[0135]

[0136] Table 2 shows the results of different document image translation methods on the DITrans-Specific dataset, where B / F represents BLEU / chrF++, as shown in the table below:

[0137]

[0138] In Tables 1 and 2, DocHandler-1 / 2, MGTrans-DETR, and MGTrans-Conv are four existing cascaded document image translation methods; Bert-Dec, ShallowLayout-Dec, LayoutLM-Dec, LayoutXLM-Dec, LayoutLMv2-Dec, DocFormer-Dec, ErnieLayout-Dec, LayoutLMv3-Dec, LiLT-Dec, LayoutDIT, and BROS-Dec are end-to-end document image translation methods based on layout-text joint encoder-translation decoder constructed based on existing end-to-end methods; UniDIT is the end-to-end unified document image translation method based on feature propagation proposed in this embodiment of the invention.

[0139] The comparison reveals the following: 1) The proposed method achieves superior results on both datasets, demonstrating significant performance improvements over existing cascaded and end-to-end methods in both the general domain of the DITrans-General dataset and the four specific domains of the DITrans-Specific dataset. 2) The proposed method offers a significant advantage in terms of model parameter count, making it a parameter-efficient approach compared to most existing methods. 3) The proposed method demonstrates remarkable performance improvement in the multi-domain learning experimental setup on the DITrans-Specific dataset, indicating its significant advantages in multi-domain scenarios.

[0140] Based on any of the above embodiments Figure 8 This is a schematic diagram of the end-to-end document image translation device based on feature propagation provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0141] Get unit 810 to obtain the image of the document to be translated;

[0142] The geometric layout unit 820 extracts the geometric joint features of the character recognition results of the document image to be translated, the geometric joint features including geometric layout features and text features;

[0143] Logical layout unit 830 obtains the logical layout features of the document image to be translated based on the geometric joint features;

[0144] The fusion unit 840 performs feature fusion between the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated;

[0145] The translation unit 850 performs text translation on the document image to be translated based on the translation-oriented features, and obtains the document image translation result.

[0146] The apparatus provided in this invention obtains the logical layout features of the document image to be translated by using the geometric joint features of the character recognition results. It then explicitly fuses these logical layout features with the extracted text features to obtain translation-guided features for translation. This achieves the explicit integration of the geometric and logical layout of the document image into the document image translation method, improving the translation capability for document images with complex layouts. Simultaneously, it provides an end-to-end document graph translation method based on feature propagation, improving translation performance and the accuracy of the translation results.

[0147] Based on any of the above embodiments, the logical layout unit is specifically used for:

[0148] Based on the logical layout features, the logical reading order of the document image to be translated is predicted;

[0149] Based on any of the above embodiments, the fusion unit is specifically used for:

[0150] Based on the logical reading order, the text features are sorted to obtain rearranged text features;

[0151] The logical layout features and the rearranged text features are fused to obtain the translation guidance features of the document image to be translated.

[0152] Based on any of the above embodiments, the translation unit is specifically used for:

[0153] Based on the translation-guided features, predict the sentence boundary category of each word segment in the document image to be translated;

[0154] Based on the sentence boundary categories, the translation guidance features are divided into sentence-level segments to obtain sentence-level translation guidance features;

[0155] The sentence-level translation guidance features are decoded to obtain the document image translation result.

[0156] Based on any of the above embodiments, the character recognition result includes the recognized characters of each word segment, geometric layout coordinates, and one-dimensional position information;

[0157] Based on any of the above embodiments, the geometric layout unit is specifically used for:

[0158] The original text features of the identified characters, the original geometric layout features of the geometric layout coordinates, and the one-dimensional position features of the one-dimensional position information are extracted respectively.

[0159] The one-dimensional positional feature is superimposed on the original text feature and the original geometric layout feature to obtain the geometric layout feature and the text feature.

[0160] Based on any of the above embodiments, the geometric layout unit is further specifically used for:

[0161] Contextual deep encoding is performed on the geometric layout features and the text features respectively to extract deep geometric layout features and deep text features;

[0162] The deep geometric layout features and the deep text features are concatenated, and the concatenated features are linearly mapped to obtain the geometric joint features.

[0163] Based on any of the above embodiments, the geometric layout unit is further specifically used for:

[0164] The document image to be translated is normalized to obtain a normalized document image; optical character recognition is performed on the normalized document image to obtain the character recognition result.

[0165] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute an end-to-end document image translation method based on feature propagation. This method includes: acquiring a document image to be translated; extracting geometric joint features of the character recognition results of the document image to be translated, the geometric joint features including geometric layout features and text features; obtaining logical layout features of the document image to be translated based on the geometric joint features; fusing the logical layout features and the text features in the geometric joint features to obtain translation-oriented features of the document image to be translated; and performing text translation on the document image to be translated based on the translation-oriented features to obtain a document image translation result.

[0166] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the end-to-end document image translation method based on feature propagation provided by the above methods. The method includes: acquiring a document image to be translated; extracting geometric joint features of the character recognition results of the document image to be translated, the geometric joint features including geometric layout features and text features; obtaining logical layout features of the document image to be translated based on the geometric joint features; performing feature fusion of the logical layout features and the text features in the geometric joint features to obtain translation guidance features of the document image to be translated; and performing text translation on the document image to be translated based on the translation guidance features to obtain a document image translation result.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the end-to-end document image translation method based on feature propagation provided by the above methods. The method includes: acquiring a document image to be translated; extracting geometric joint features of character recognition results of the document image to be translated, the geometric joint features including geometric layout features and text features; obtaining logical layout features of the document image to be translated based on the geometric joint features; performing feature fusion of the logical layout features and the text features in the geometric joint features to obtain translation-oriented features of the document image to be translated; and performing text translation on the document image to be translated based on the translation-oriented features to obtain a document image translation result.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end document-image translation method based on feature propagation, characterized in that, include: Obtain the image of the document to be translated; Extract the geometric joint features of the character recognition results of the document image to be translated, wherein the geometric joint features include geometric layout features and text features; The geometric layout features are used to reflect the two-dimensional position information of the text within the entire image of the document to be translated; Based on the geometric joint features, the logical layout features of the document image to be translated are obtained; The logical layout features and the text features in the geometric joint features are fused to obtain the translation guidance features of the document image to be translated; Based on the translation-oriented features, the document image to be translated is translated into text to obtain the document image translation result; The logical layout features of the document image to be translated are obtained based on the geometric joint features, and then the process includes: Based on the logical layout features, the logical reading order of the document image to be translated is predicted; The step of fusing the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated includes: Based on the logical reading order, the text features are sorted to obtain rearranged text features; The logical layout features and the rearranged text features are fused to obtain the translation guidance features of the document image to be translated; The process of translating the document image to be translated based on the translation-oriented features to obtain the document image translation result includes: Based on the translation-guided features, predict the sentence boundary category of each word segment in the document image to be translated; Based on the sentence boundary categories, the translation guidance features are divided into sentence-level segments to obtain sentence-level translation guidance features; The sentence-level translation guidance features are decoded to obtain the document image translation result.

2. The end-to-end document image translation method based on feature propagation according to claim 1, characterized in that, The character recognition results include the recognized characters of each word segment, geometric layout coordinates, and one-dimensional position information; The steps for obtaining the geometric layout features and the text features include: The original text features of the identified characters, the original geometric layout features of the geometric layout coordinates, and the one-dimensional position features of the one-dimensional position information are extracted respectively. The one-dimensional positional feature is superimposed on the original text feature and the original geometric layout feature to obtain the geometric layout feature and the text feature.

3. The end-to-end document image translation method based on feature propagation according to claim 2, characterized in that, The geometric joint features of the character recognition results extracted from the document image to be translated include: Contextual deep encoding is performed on the geometric layout features and the text features respectively to extract deep geometric layout features and deep text features; The deep geometric layout features and the deep text features are concatenated, and the concatenated features are linearly mapped to obtain the geometric joint features.

4. The end-to-end document-image translation method based on feature propagation according to claim 2, characterized in that, The steps for obtaining the character recognition result include: The document image to be translated is normalized to obtain a normalized document image; Optical character recognition is performed on the normalized document image to obtain the character recognition result.

5. A feature-transduction-based end-to-end document image translation apparatus based on the feature-transduction-based end-to-end document image translation method according to any one of claims 1 to 4, characterized in that, include: Get the unit, and get the image of the document to be translated; The geometric layout unit extracts the geometric joint features of the character recognition results of the document image to be translated, and the geometric joint features include geometric layout features and text features; Logical layout unit, based on the geometric joint features, obtains the logical layout features of the document image to be translated; The fusion unit performs feature fusion between the logical layout features and the text features in the geometric joint features to obtain the translation guidance features of the document image to be translated; The translation unit performs text translation on the document image to be translated based on the translation-oriented features, and obtains the document image translation result.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the end-to-end document image translation method based on feature propagation as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the end-to-end document image translation method based on feature propagation as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the end-to-end document image translation method based on feature propagation as described in any one of claims 1 to 4.

Images