End-to-end document image translation method and device fusing layout information
By employing an end-to-end document image translation method, which combines word text, coordinates, and positional information for joint encoding and decoding, the problem of poor translation results in cascaded structures is solved, enabling efficient translation of different formats and layouts.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-14
- Publication Date
- 2026-03-27
AI Technical Summary
In existing technologies, cascaded document image translation methods suffer from error accumulation in the layout parsing module, resulting in poor translation quality. Furthermore, the lack of information exchange between modules makes it difficult to adapt to document images with different formats and layout structures.
Employing a Transformer Encoder-Decoder architecture, it achieves end-to-end document image translation by combining the text, two-dimensional coordinate information, and one-dimensional position information of each word for joint encoding, thereby enhancing the interactivity and adaptation of the encoding and decoding process.
It significantly improves the translation of document images with different formats and layouts, especially performing well in zero-shot transfer learning in cross-domain scenarios, and requires fewer parameters.
Smart Images
Figure CN117253239B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of natural language processing, and particularly relates to an end-to-end document image translation method and device fusing layout information. BACKGROUND
[0002] A document image refers to an image generated by scanning or photographing text on a paper or physical surface. Document image translation aims to automatically translate text embedded in a document image from a source language into a target language, and is one of key technologies for realizing automatic document information processing.
[0003] In view of the flexible and variable layout of a document image, most methods first use a deep learning-based layout analysis model and a logical order detection model to automatically analyze the layout and logical order of the document image, so as to extract source language text with logical order and use the text as input of a translation model to obtain a target language translation. However, these methods use a cascading manner to concatenate multiple modules such as layout analysis, logical order detection, sentence segmentation and translation, each module is independently trained on specific data, and there is lack of information interaction between modules, which is not conducive to the adaptation between modules. In addition, due to the cascading structure, errors of the layout analysis module are continuously accumulated and amplified in the forward process, which has a negative impact on the translation result. SUMMARY
[0004] The present application provides an end-to-end document image translation method and device fusing layout information, to solve the problem of poor translation effect of the document image analysis model with a cascading structure in the prior art.
[0005] The present application provides an end-to-end document image translation method fusing layout information, comprising:
[0006] Obtaining a character recognition result of a document image to be translated, the character recognition result comprising a plurality of words in the document image to be translated and two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on pixel values of the document image to be translated;
[0007] Based on the text corresponding to each word, the two-dimensional coordinate information of each word and one-dimensional position information of each word, a first feature vector is obtained, the one-dimensional position information being used to indicate the position of the word in a word sequence, the word sequence being used to indicate a one-dimensional sequence composed of all words recognized from the document image to be translated;
[0008] Decoding the first feature vector to obtain a translation text corresponding to the document image to be translated.
[0009] In some embodiments, the obtaining the first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word comprises:
[0010] encoding the text corresponding to each word to obtain a text feature vector;
[0011] encoding the two-dimensional coordinate information of each word to obtain a two-dimensional coordinate feature vector;
[0012] encoding the one-dimensional position information of each word to obtain a one-dimensional position feature vector;
[0013] performing feature fusion on the text feature vector, the two-dimensional coordinate feature vector, and the one-dimensional position feature vector to obtain the first feature vector.
[0014] In some embodiments, the decoding the first feature vector to obtain the translation text corresponding to the document image to be translated comprises:
[0015] decoding the first feature representation vector to obtain a first hidden layer vector corresponding to each word, the first hidden layer vector being used to indicate the reading order of each word;
[0016] determining a sentence boundary category label corresponding to each word based on the first hidden layer vector, the sentence boundary category label being used to indicate whether each word is a sentence starting word;
[0017] determining a semantic feature of each source language sentence corresponding to all words based on the first hidden layer vector and the sentence boundary category label;
[0018] determining the translation text corresponding to the document image to be translated based on the semantic feature of each source language sentence.
[0019] In some embodiments, the determining the sentence boundary category label corresponding to each word based on the first hidden layer vector comprises:
[0020] encoding the first hidden layer vector to obtain a second feature vector corresponding to each word, the second feature vector being used to indicate the sentence boundary of the source language sentence corresponding to each word;
[0021] determining the sentence boundary category label corresponding to each word based on the second feature vector.
[0022] In some embodiments, the determining the semantic feature of each source language sentence corresponding to all words based on the first hidden layer vector and the sentence boundary category label comprises:
[0023] Based on the sentence boundary category label corresponding to each word, the first hidden layer vector is divided to obtain the second hidden layer vector corresponding to each source language sentence;
[0024] The second hidden layer vector is used as the semantic feature of each source language sentence.
[0025] In some embodiments, determining the translated text corresponding to the document image to be translated based on the semantic features of each source language sentence includes:
[0026] The semantic features of each source language sentence are decoded to obtain the target language sentence corresponding to each source language sentence;
[0027] The target language sentences corresponding to each source language sentence are concatenated in sequence to obtain the translated text corresponding to the document image to be translated.
[0028] The present invention also provides an end-to-end document image translation device that integrates layout information, comprising:
[0029] The acquisition module is used to acquire the character recognition results of the document image to be translated. The character recognition results include multiple words in the document image to be translated and the two-dimensional coordinate information of each word. The two-dimensional coordinate information is determined based on the pixel values of the document image to be translated.
[0030] The encoding module is used to obtain a first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word. The one-dimensional position information is used to indicate the position of the word in the word sequence, and the word sequence is used to indicate the one-dimensional sequence composed of all the words identified from the document image to be translated.
[0031] The decoding module is used to decode the first feature vector to obtain the translated text corresponding to the document image to be translated.
[0032] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the end-to-end document image translation method for integrating layout information as described above.
[0033] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the end-to-end document image translation method for integrating layout information as described above.
[0034] The application further provides a computer program product comprising a computer program which, when executed by a processor, implements the end-to-end document image translation method of fusing layout information as described above.
[0035] The end-to-end document image translation method of fusing layout information provided by the application realizes joint coding of version information and text information of a document image to be translated by combining text corresponding to each word, two-dimensional coordinate information of each word, and one-dimensional position information of each word, to obtain a first feature vector; then the first feature vector is decoded to obtain a translation text corresponding to the document image to be translated, thereby enhancing interaction and adaptation between the coding and decoding processes and effectively improving the document image translation effect of different formats and layout structures. BRIEF DESCRIPTION OF DRAWINGS
[0036] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.
[0037] Figure 1 is a flowchart of the end-to-end document image translation method of fusing layout information provided by the application;
[0038] Figure 2 is one of optical character recognition diagrams of the end-to-end document image translation method of fusing layout information provided by the application;
[0039] Figure 3 is another of optical character recognition diagrams of the end-to-end document image translation method of fusing layout information provided by the application;
[0040] Figure 4 is a comparison diagram of a cascaded model structure and an end-to-end model structure of the end-to-end document image translation method of fusing layout information provided by the application;
[0041] Figure 5 is a model framework diagram of the end-to-end document image translation method of fusing layout information provided by the application;
[0042] Figure 6 is a structural diagram of the end-to-end document image translation device provided by the application;
[0043] Figure 7 is a structural diagram of the electronic device provided by the application. DETAILED DESCRIPTION
[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions in the present application with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0045] In the related art, a document refers to a paper, an image or an electronic file containing text, which exists in large quantities in daily life and communication channels such as the Internet.
[0046] Unlike pure text, the text in a document image is arranged in a certain layout and logical order. Although the layout and logical order can be parsed by manually designed rules to extract the text content, and the document image translation can be realized by pure text machine translation, in actual application, different fields and different formats of documents often present inconsistent layout structures and logical orders, so that the method cannot simultaneously process different types of document images, and the generalization is very limited.
[0047] The present application proposes an end-to-end document image translation method fusing layout information based on the Transformer Encoder-Decoder (Encoder-Decoder) architecture, which effectively improves the document image translation effect of different formats and layout structures.
[0048] The following will be described in detail with reference to the drawings. Figures 1-7 The present application describes an end-to-end document image translation method fusing layout information and a device.
[0049] Figure 1 FIG. 1 is a flowchart of the end-to-end document image translation method fusing layout information provided by the present application. With reference to FIG. 1, Figure 1 The end-to-end document image translation method fusing layout information provided by the present application includes steps 110, 120 and 130.
[0050] Step 110, obtaining a character recognition result of a document image to be translated, the character recognition result including a plurality of words in the document image to be translated and two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on pixel values of the document image to be translated;
[0051] Step 120, obtaining a first feature vector based on text corresponding to each word, two-dimensional coordinate information of each word and one-dimensional position information of each word, the one-dimensional position information being used to indicate a position of the word in a word sequence, the word sequence being used to indicate a one-dimensional sequence composed of all words recognized from the document image to be translated;
[0052] Step 130: Decode the first feature vector to obtain the translated text corresponding to the document image to be translated.
[0053] The execution subject of the end-to-end document image translation method integrating layout information provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc., and this invention does not impose specific limitations.
[0054] The following example, using a computer executing the end-to-end document image translation method with integrated layout information provided by this invention, illustrates the technical solution of this invention in detail.
[0055] In step 110, optical character recognition processing is performed on the document image to be translated to extract at least one word from the document image and the two-dimensional layout coordinates of each word.
[0056] In actual implementation, the pixel value threshold is set to 1000, and the height and width of the document image to be translated are normalized to the pixel value range of [0, 1000]. The pixel value threshold can be set according to actual needs and is not specifically limited here.
[0057] like Figure 2 As shown, the normalized document image 210 to be translated is processed using an optical character recognition engine to obtain the text and two-dimensional layout coordinates corresponding to each word. The two-dimensional layout coordinates are used to indicate the two-dimensional coordinate information of each word in the document image to be translated.
[0058] like Figure 3 As shown, each word in the processed document image 310 to be translated is surrounded by a rectangular bounding box. The rectangular bounding box identifies the word and its two-dimensional coordinate position as recognized by the optical character recognition engine. The two-dimensional coordinate information corresponding to the position of each word is uniquely determined by the coordinates of the upper left corner and the lower right corner of the rectangular bounding box surrounding it.
[0059] In step 120, the text corresponding to the word, the two-dimensional coordinate information and the one-dimensional position information are jointly encoded by using a layout-aware encoder to obtain a layout-aware feature representation, and the layout-aware feature representation is a first feature vector.
[0060] It can be understood that the word text identified from the document image to be translated is a one-dimensional sequence, and each word in the one-dimensional sequence can be assigned a position label starting from 1. The position label is one-dimensional sequence position information, that is, one-dimensional position information.
[0061] In step 130, based on the first feature vector (layout-aware feature representation), a multi-step transduction decoder is used to sequentially and orderly perform reading order decoding, sentence boundary decoding and translation decoding to obtain a final document image translation text, that is, a translation text corresponding to the document image to be translated.
[0062] The end-to-end document image translation method with fused layout information provided by the application realizes joint encoding of version information and text information of the document image to be translated by combining the text corresponding to each word, the two-dimensional coordinate information of each word and the one-dimensional position information of each word to obtain a first feature vector. Then, the first feature vector is decoded to obtain a translation text corresponding to the document image to be translated, thereby enhancing the interaction and adaptation between the encoding and decoding processes and effectively improving the document image translation effect of different formats and layout structures.
[0063] The above steps will be described in detail below. Figure 4 And Figure 5 The above steps will be described in detail below.
[0064] As Figure 4 shown, the existing cascaded model structure is to splice a plurality of independently trained modules such as a layout analysis module, a logical order detection module, a sentence segmentation module and a translation module, which cannot be jointly optimized for a translation target. The application provides an end-to-end model structure based on a layout-aware encoder-text decoder.
[0065] In some embodiments, step 120 can include:
[0066] encoding the text corresponding to each word to obtain a text feature vector;
[0067] encoding the two-dimensional coordinate information of each word to obtain a two-dimensional coordinate feature vector;
[0068] encoding the one-dimensional position information of each word to obtain a one-dimensional position feature vector;
[0069] performing feature fusion on the text feature vector, the two-dimensional coordinate feature vector and the one-dimensional position feature vector to obtain the first feature vector.
[0070] In actual execution, the text corresponding to each word, the two-dimensional coordinate information and the one-dimensional position information are respectively encoded by using respective embedding layers to obtain a text feature vector, a two-dimensional coordinate feature vector and a one-dimensional position feature vector.
[0071] The text feature vector, the two-dimensional coordinate feature vector and the one-dimensional position feature vector are added and then context encoding and feature fusion are performed by using a Transformer Encoder structure to obtain a layout perception feature representation, which is the first feature vector.
[0072] As shown in FIG. 1, the text representation corresponding to at least one word is "The"… "Strategy" "Vision"… "years"; Figure 5
[0073] The one-dimensional position information is represented as: i = 1… i = 5, i = 6… i = L; wherein L is the total number of recognized words.
[0074] The two-dimensional coordinate information is represented as: (x0, y0, x1, y1, w, h)1… (x0, y0, x1, y1, w, h)5, (x0, y0, x1, y1, w, h)6… (x0, y0, x1, y1, w, h) L ; wherein (x0, y0) is the upper left corner coordinate of the rectangular bounding box and (x1, y1) is the lower right corner coordinate, w is the width of the rectangular bounding box and h is the height of the rectangular bounding box.
[0075] The layout perception feature representation (the first feature vector) is represented as: X1 * … X5 * , X6 * … X L * .
[0076] The end-to-end document image translation method provided by the application fuses layout information, realizes joint encoding of text features and layout features by performing feature fusion on the text feature vector, the two-dimensional coordinate feature vector and the one-dimensional position feature vector, thereby realizing joint understanding of "text-layout" corresponding to a document image to be translated and significantly improving the translation capability of the model for document images to be translated in different formats and layout structures.
[0077] In some embodiments, step 130 can include:
[0078] The first feature representation vector is decoded to obtain a first hidden layer vector corresponding to each word, and the first hidden layer vector is used to indicate the reading order of each word.
[0079] Based on the first hidden layer vector, a sentence boundary category label corresponding to each word is determined, and the sentence boundary category label is used to indicate whether each word is a sentence starting word.
[0080] Based on the first hidden layer vector and the sentence boundary category label, a semantic feature of each source language sentence corresponding to all words is determined.
[0081] Based on the semantic feature of each source language sentence, a translation text corresponding to the document image to be translated is determined.
[0082] Based on the first feature vector (layout-aware feature representation), a reading order hidden layer vector and a reading order index value corresponding to each word are calculated using a reading order decoder in a multi-step transduction decoder in a self-recurrent manner, and a greedy decoding strategy is used for prediction.
[0083] The loss function designed for reading order decoding is:
[0084]
[0085] where Idx i represents the reading order index value label of the i-th word, represents the reading order probability distribution of the i-th word predicted by the reading order decoder, and L is the number of document words. The loss function is used to optimize the parameters of the reading order decoder, and the reading order hidden layer vector is the first hidden layer vector.
[0086] As shown in Figure 5 , the first hidden layer vector (decoder hidden layer vector) can be represented as H1 rsd …H5 rsd , H6 rsd …H L rsd . The reading order index value label of H1 rsd is Idx1…H5 rsd , the reading order index value label of H5 rsd is Idx5, the reading order index value label of H6 L …H rsd , and the reading order index value label of H L .
[0087] It should be noted that the reading order index value provides a supervision signal to train the reading order decoder so that the feature vector output by the reading order decoder is a feature vector corresponding to a correct reading order sequence, so as to facilitate subsequent sentence boundary decoding and translation decoding.
[0088] Based on the first hidden layer vector (reading order hidden layer vector), a sentence boundary vector corresponding to each word is obtained, and a sentence boundary category label of each word is predicted.
[0089] wherein the sentence boundary class label comprises {BOS, IOS}. Wherein BOS represents the beginning of sentence word, and IOS represents the non-beginning of sentence word.
[0090] Based on the predicted sentence boundary class label, the first hidden layer vector (reading order hidden layer vector) is divided into a plurality of sub-vector sequences corresponding to the semantic features of each source language sentence.
[0091] Based on the semantic features of each source language sentence, the translation decoder in the multi-step conduction decoder is used to generate the target language translation sentence by sentence, and finally the translation text corresponding to the to-be-translated document image is obtained.
[0092] In some embodiments, based on the first hidden layer vector, determining the sentence boundary class label corresponding to each word comprises:
[0093] encoding the first hidden layer vector to obtain a second feature vector corresponding to each word, the second feature vector being used to indicate the sentence boundary of the source language sentence corresponding to each word;
[0094] Based on the second feature vector, determine the sentence boundary class label corresponding to each word.
[0095] In actual execution, based on the first hidden layer vector (reading order hidden layer vector), the sentence boundary decoder in the multi-step conduction decoder is used to further encode the context of the first hidden layer vector to obtain the sentence boundary vector corresponding to each word, and the sentence boundary class label of each word is predicted. Wherein the sentence boundary vector is the second feature vector.
[0096] The loss function for the sentence boundary decoder can be designed as:
[0097]
[0098] wherein B i represents the sentence boundary class label of the i-th word in the sorted word sequence, represents the sentence boundary probability distribution of the i-th word predicted by the sentence boundary decoder, and L is the number of document words. The loss function is used to optimize the parameters of the sentence boundary decoder.
[0099] As Figure 5 shown, the sentence boundary vector (second feature vector) is represented as: H1 ssd …H5 ssd , H6 ssd …H L ssd . The semantic features of the source language sentence are H1 sent , H2 sent …HM sent .
[0100] In some embodiments, based on the first hidden layer vector and the sentence boundary category label, the semantic features of each source language sentence corresponding to all words are determined, including:
[0101] Based on the sentence boundary category label corresponding to each word, the first hidden layer vector is divided to obtain the second hidden layer vector corresponding to each source language sentence;
[0102] The second hidden layer vector is taken as the semantic feature of each source language sentence.
[0103] In actual execution, based on the sentence boundary label corresponding to each word, the first hidden layer vector (reading order hidden layer vector) is divided by using a sentence segmentation rule to obtain a plurality of subvector sequences, that is, the second hidden layer vector of each source language sentence is obtained, and it is taken as the semantic feature of the source language sentence.
[0104] As shown in Figure 5 H1 rsd to H5 rsd is a subvector sequence, H6 rsd is a subvector sequence…H L rsd is a subvector sequence.
[0105] In some embodiments, based on the semantic features of each source language sentence, the translation text corresponding to the to-be-translated document image is determined, including:
[0106] The semantic features of each source language sentence are decoded to obtain the target language sentence corresponding to each source language sentence;
[0107] The target language sentences corresponding to each source language sentence are spliced in sequence to obtain the translation text corresponding to the to-be-translated document image.
[0108] In actual execution, based on the semantic features of each source language sentence, the translation decoder in the multi-step conduction decoder is used to generate the target language sentence corresponding to each source language sentence, and a column search decoding strategy can be used during generation. All target language sentences are spliced in sequence to obtain the final document image chapter, that is, the translation text corresponding to the to-be-translated document image. The loss function for the translation decoder is designed:
[0109]
[0110] Wherein, Y k,j represents the jth word of the kth translation sentence, represents the word table probability distribution of the jth word of the kth translation sentence predicted by the translation decoder, M is the number of sentences, and |Yk | is the number of words in the kth sentence. The loss function can be used to optimize the parameters of the translation decoder.
[0111] As shown in Figure 5 , the semantic features of the source language sentence are H1 sent , H2 sent …H M sent , and the corresponding target language sentences (translated sentence) correspond to translated sentence 1, translated sentence 2 … translated sentence M .
[0112] In some embodiments, two document image translation data sets ReadingBank and DITrans are used to verify the effect of document image translation.
[0113] Among them, the document images of ReadingBank belong to the general field, and the document images of DITrans contain three special fields of government reports, news newspapers and advertisements, and the combination of the two can verify the translation effect of the method in various different layout types of document images in general and specific fields.
[0114] In addition, in order to verify the advantages of the method in the cross-field scene, the present application also carries out zero-shot cross-field translation experiment on DITrans.
[0115] The present application also compares the existing cascading method, and based on the layout-aware encoder-text decoder, an end-to-end document image translation method is constructed to compare the fusion of layout information, and the results are shown in Table 1 and Table 2.
[0116] Table 1 Results of different document image translation methods on ReadingBank data set
[0117]
[0118]
[0119] Table 2 Results of different document image translation methods on DITrans data set under two experimental settings
[0120]
[0121] From Table 1 and Table 2, it can be seen that DocHandler-1, DocHandler-2, MGTrans-DETR and MGTrans-Conv are four existing cascading document image translation methods in the related art.
[0122] LayoutLM-Dec and LiLT-Dec are two end-to-end document image translation methods based on layout-text joint encoder and text decoder with fused layout information constructed by the present application.
[0123] LayoutDIT is an end-to-end document image translation method with fused layout information proposed by the present application.
[0124] By comparison, it can be found that:
[0125] (1) The method proposed in the present application achieves good results on both datasets, whether in the general field of the Readingbank dataset or in the three specialized fields of the DITrans dataset. Compared with existing cascaded methods and end-to-end methods, the method has significantly improved performance.
[0126] (2) The method proposed in the present application has a significant advantage in model parameter quantity. Compared with most existing methods, it is a parameter-efficient method.
[0127] (3) The performance improvement brought by the method proposed in the present application in the experimental setting of zero-shot cross-domain transfer learning on the DITrans dataset is particularly significant, indicating that the method has more significant advantages in cross-domain settings.
[0128] Existing document image translation methods are difficult to achieve joint understanding of text and layout, and the cascaded combination of modules cannot be jointly optimized. The end-to-end document image translation method with fused layout information provided by the present application can jointly encode layout and text features. Its end-to-end modeling can utilize the translation target to jointly optimize all sub-modules, significantly improving the translation ability of the model for document images of different formats and layout structures.
[0129] The end-to-end document image translation device with fused layout information provided by the present application is described below. The end-to-end document image translation device with fused layout information described below can be referred to each other corresponding to the end-to-end document image translation method with fused layout information described above.
[0130] Figure 6 is a structural schematic diagram of the end-to-end document image translation device with fused layout information provided by the present application. Referring to Figure 6 , the end-to-end document image translation device with fused layout information provided by the present application comprises an acquisition module 610, an encoding module 620, and a decoding module 630.
[0131] The acquisition module 610 is configured to acquire a character recognition result of a document image to be translated, wherein the character recognition result comprises a plurality of words in the document image to be translated and two-dimensional coordinate information of each word, and the two-dimensional coordinate information is determined based on pixel values of the document image to be translated.
[0132] The encoding module 620 is configured to obtain a first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and one-dimensional position information of each word, wherein the one-dimensional position information is used to indicate the position of the word in a word sequence, and the word sequence is used to indicate a one-dimensional sequence composed of all words recognized from the document image to be translated.
[0133] The decoding module 630 is configured to decode the first feature vector to obtain a translation text corresponding to the document image to be translated.
[0134] The fusion version information end-to-end document image translation device provided by the application realizes the joint encoding of the version information and the text information of the document image to be translated by combining the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word, and obtains a first feature vector. Then, the first feature vector is decoded to obtain a translation text corresponding to the document image to be translated, thereby enhancing the interaction and adaptation between the encoding and decoding processes, and effectively improving the document image translation effect of different formats and layout structures.
[0135] In some embodiments, the encoding module 620 is specifically configured to:
[0136] encode the text corresponding to each word to obtain a text feature vector;
[0137] encode the two-dimensional coordinate information of each word to obtain a two-dimensional coordinate feature vector;
[0138] encode the one-dimensional position information of each word to obtain a one-dimensional position feature vector;
[0139] fuse the text feature vector, the two-dimensional coordinate feature vector, and the one-dimensional position feature vector to obtain the first feature vector.
[0140] In some embodiments, the decoding module 630 is specifically configured to:
[0141] decode the first feature vector to obtain a first hidden layer vector corresponding to each word, wherein the first hidden layer vector is used to indicate the reading order of each word;
[0142] determine a sentence boundary category label corresponding to each word based on the first hidden layer vector, wherein the sentence boundary category label is used to indicate whether each word is a sentence starting word;
[0143] determine a semantic feature of each source language sentence corresponding to all words based on the first hidden layer vector and the sentence boundary category label.
[0144] Based on the semantic features of each source language sentence, the translated text corresponding to the document image to be translated is determined.
[0145] In some embodiments, the decoding module 630 is specifically used for:
[0146] The first hidden layer vector is encoded to obtain a second feature vector corresponding to each word. The second feature vector is used to indicate the sentence boundary of the source language sentence corresponding to each word.
[0147] Based on the second feature vector, the sentence boundary category label corresponding to each word is determined.
[0148] In some embodiments, the decoding module 630 is specifically used for:
[0149] Based on the sentence boundary category label corresponding to each word, the first hidden layer vector is divided to obtain the second hidden layer vector corresponding to each source language sentence;
[0150] The second hidden layer vector is used as the semantic feature of each source language sentence.
[0151] In some embodiments, the decoding module 630 is specifically used for:
[0152] The semantic features of each source language sentence are decoded to obtain the target language sentence corresponding to each source language sentence;
[0153] The target language sentences corresponding to each source language sentence are concatenated in sequence to obtain the translated text corresponding to the document image to be translated.
[0154] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute an end-to-end document image translation method that integrates layout information. This method includes:
[0155] Obtain the character recognition results of the document image to be translated, the character recognition results including multiple words in the document image to be translated and the two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on the pixel values of the document image to be translated;
[0156] obtaining a first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and one-dimensional position information of each word, the one-dimensional position information being used to indicate a position of the word in a word sequence, the word sequence being used to indicate a one-dimensional sequence composed of all words recognized from the document image to be translated;
[0157] decoding the first feature vector to obtain a translation text corresponding to the document image to be translated.
[0158] In addition, the logic instructions in the memory 730 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0159] On the other hand, the present application also provides a computer program product, the computer program product comprising a computer program, the computer program being storable on a non-transitory computer readable storage medium, and the computer being capable of executing the end-to-end document image translation method fusing layout information provided by the above-mentioned methods when the computer program is executed by a processor, the method comprising:
[0160] obtaining a character recognition result of a document image to be translated, the character recognition result comprising a plurality of words in the document image to be translated and two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on pixel values of the document image to be translated;
[0161] obtaining a first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and one-dimensional position information of each word, the one-dimensional position information being used to indicate a position of the word in a word sequence, the word sequence being used to indicate a one-dimensional sequence composed of all words recognized from the document image to be translated;
[0162] decoding the first feature vector to obtain a translation text corresponding to the document image to be translated.
[0163] In yet another aspect, the present application also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the end-to-end document image translation method with layout information fusion provided by the above method, and the method comprises:
[0164] obtaining a character recognition result of a document image to be translated, the character recognition result comprising a plurality of words in the document image to be translated and two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on pixel values of the document image to be translated;
[0165] obtaining a first feature vector based on text corresponding to each word, two-dimensional coordinate information of each word, and one-dimensional position information of each word, the one-dimensional position information being used to indicate a position of the word in a word sequence, the word sequence being used to indicate a one-dimensional sequence formed by all words recognized from the document image to be translated;
[0166] decoding the first feature vector to obtain a translation text corresponding to the document image to be translated.
[0167] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.
[0168] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in terms of the contribution to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.
[0169] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An end-to-end document image translation method integrating layout information, characterized in that, include: Obtain the character recognition results of the document image to be translated, the character recognition results including multiple words in the document image to be translated and the two-dimensional coordinate information of each word, the two-dimensional coordinate information being determined based on the pixel values of the document image to be translated; Based on the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word, a first feature vector is obtained. The one-dimensional position information is used to indicate the position of the word in the word sequence, and the word sequence is used to indicate the one-dimensional sequence composed of all the words identified from the document image to be translated. Decoding the first feature vector yields the translated text corresponding to the document image to be translated; Decoding the first feature vector to obtain the translated text corresponding to the document image to be translated includes: Decoding the first feature vector yields the first hidden layer vector corresponding to each word, which is used to indicate the reading order of each word; Based on the first hidden layer vector, a sentence boundary category label corresponding to each word is determined, and the sentence boundary category label is used to indicate whether each word is the starting word of a sentence; Based on the first hidden layer vector and the sentence boundary category label, determine the semantic features of each source language sentence corresponding to all the words; Based on the semantic features of each source language sentence, the translated text corresponding to the document image to be translated is determined.
2. The end-to-end document image translation method integrating layout information according to claim 1, characterized in that, The first feature vector is obtained based on the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word, including: Encode the text corresponding to each word to obtain a text feature vector; The two-dimensional coordinate information of each word is encoded to obtain a two-dimensional coordinate feature vector; Encode the one-dimensional positional information of each word to obtain a one-dimensional positional feature vector; The text feature vector, the two-dimensional coordinate feature vector, and the one-dimensional position feature vector are fused to obtain the first feature vector.
3. The end-to-end document image translation method integrating layout information according to claim 1, characterized in that, The step of determining the sentence boundary category label corresponding to each word based on the first hidden layer vector includes: The first hidden layer vector is encoded to obtain a second feature vector corresponding to each word. The second feature vector is used to indicate the sentence boundary of the source language sentence corresponding to each word. Based on the second feature vector, the sentence boundary category label corresponding to each word is determined.
4. The end-to-end document image translation method integrating layout information according to claim 1, characterized in that, The step of determining the semantic features of each source language sentence corresponding to all words based on the first hidden layer vector and the sentence boundary category label includes: Based on the sentence boundary category label corresponding to each word, the first hidden layer vector is divided to obtain the second hidden layer vector corresponding to each source language sentence; The second hidden layer vector is used as the semantic feature of each source language sentence.
5. The end-to-end document image translation method integrating layout information according to claim 1, characterized in that, The step of determining the translated text corresponding to the document image to be translated based on the semantic features of each source language sentence includes: The semantic features of each source language sentence are decoded to obtain the target language sentence corresponding to each source language sentence; The target language sentences corresponding to each source language sentence are concatenated in sequence to obtain the translated text corresponding to the document image to be translated.
6. An end-to-end document image translation device integrating layout information, characterized in that, include: The acquisition module is used to acquire the character recognition results of the document image to be translated. The character recognition results include multiple words in the document image to be translated and the two-dimensional coordinate information of each word. The two-dimensional coordinate information is determined based on the pixel values of the document image to be translated. The encoding module is used to obtain a first feature vector based on the text corresponding to each word, the two-dimensional coordinate information of each word, and the one-dimensional position information of each word. The one-dimensional position information is used to indicate the position of the word in the word sequence, and the word sequence is used to indicate the one-dimensional sequence composed of all the words identified from the document image to be translated. The decoding module is used to decode the first feature vector to obtain the translated text corresponding to the document image to be translated; Decoding the first feature vector to obtain the translated text corresponding to the document image to be translated includes: Decoding the first feature vector yields the first hidden layer vector corresponding to each word, which is used to indicate the reading order of each word; Based on the first hidden layer vector, a sentence boundary category label corresponding to each word is determined, and the sentence boundary category label is used to indicate whether each word is the starting word of a sentence; Based on the first hidden layer vector and the sentence boundary category label, determine the semantic features of each source language sentence corresponding to all the words; Based on the semantic features of each source language sentence, the translated text corresponding to the document image to be translated is determined.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the end-to-end document image translation method that integrates layout information as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the end-to-end document image translation method that integrates layout information as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the end-to-end document image translation method that integrates layout information as described in any one of claims 1 to 5.