Translation method and apparatus, computer-readable storage medium, and computer program product

Abstract

A translation method and device, a computer readable storage medium and a computer program product are disclosed. The method comprises: inputting a source document image into an image encoder to obtain image encoding features, wherein the source document image comprises a source language text part and an illustration part; inputting the source document image into a semantic alignment encoder to obtain single-modal semantic encoding features; inputting the image encoding features and the single-modal semantic encoding features into a decoder to obtain a translation result of the source document image, wherein the translation result comprises a target language text part and an illustration part; and wherein a translation model is trained using the output of a multi-modal large model.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of natural language processing in general, and more particularly, to a translation method and device, computer readable storage medium and computer program product. BACKGROUND

[0002] The processing object of the document image translation technology is a document image, and the goal is to mine the text, layout and visual information contained in the document image, fuse and utilize multi-source information to translate the source language text in the document image into correct target language text.

[0003] At present, most of the document image translation systems are cascaded systems, and after inputting the document image, a series of steps such as layout analysis, optical character recognition, text translation and post-translation processing are needed to obtain the final translation text. The whole process is time-consuming and low in computing efficiency, and in addition, there are accumulated errors caused by transmission between different modules, complex system structure, deployment difficulties and other problems. In addition, the current document image translation system needs a large amount of supervised data training to achieve good results, but still has the problems of poor domain generalization, poor ability to process long document images and complex layout images. SUMMARY

[0004] Embodiments of the present disclosure provide a translation method and device, computer readable storage medium and computer program product, which can effectively solve the problems of poor generalization and long time consumption of the translation system in the prior art.

[0005] In one general aspect, there is provided a translation method, which is implemented based on a translation model including an image encoder, a semantic alignment encoder and a decoder, comprising: inputting a source document image into the image encoder to obtain image encoding features, wherein the source document image includes a source language text part and an illustration part; inputting the source document image into the semantic alignment encoder to obtain single-modal semantic encoding features; inputting the image encoding features and the single-modal semantic encoding features into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text part and an illustration part; and wherein the translation model is trained using the output of a multi-modal large model.

[0006] Optionally, the translation model is trained as follows: Multiple document image samples are acquired, and each document image sample is processed as follows: the current document image sample is input into an image encoder to obtain estimated image encoding features; the current document image sample is input into a semantic alignment encoder to obtain estimated single-modal semantic encoding features; the current document image sample and its corresponding source language text are input into a multimodal large model to obtain mixed-modal semantic encoding features; the cosine similarity between the estimated single-modal semantic encoding features and the mixed-modal semantic encoding features is obtained; the translation loss is obtained based on the estimated image encoding features and the estimated single-modal semantic encoding features; and the parameters of the translation model are adjusted by minimizing the cosine similarity and the translation loss.

[0007] Optionally, based on the estimated image coding features and the estimated unimodal semantic coding features, the translation loss is obtained, including: inputting the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain the translation result of the t-th character in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all characters translated before the t-th character, and t is a positive integer; and determining the translation loss as the logarithmic sum of the translation results of each character in the target language text corresponding to the current document image sample.

[0008] Optionally, the current document image sample and the corresponding source language text are input into a multimodal large model to obtain mixed-modality semantic coding features, including: inputting the current document image sample into the image encoder in the multimodal large model to obtain a first coding feature; inputting the corresponding source language text into the text embedding layer in the multimodal large model to obtain a second coding feature; and inputting the first coding feature and the second coding feature into multiple attention mechanism layers to obtain mixed-modality semantic coding features.

[0009] Optionally, the source document image is input into an image encoder to obtain image coding features, including: scaling the source document image to obtain an adjusted first image; and inputting the first image into the image encoder to obtain image coding features, wherein the image encoder employs a sliding window attention mechanism.

[0010] Optionally, the source document image is input into the semantic alignment encoder to obtain single-modal semantic coding features, including: scaling the source document image to obtain an adjusted second image; inputting the second image into the image encoder in the semantic alignment encoder to obtain a third coding feature, wherein the image encoder adopts a sliding window attention mechanism; and inputting the third coding feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain single-modal semantic coding features.

[0011] Optionally, after inputting the image encoding features and the unimodal semantic encoding features into the decoder to obtain the translation result of the source document image, the method further includes: rendering the target document image corresponding to the source document image based on the translation result.

[0012] In another general aspect, a translation apparatus is provided, which is based on a translation model including an image encoder, a semantic alignment encoder, and a decoder. The model includes: a first processing unit configured to input a source document image into the image encoder to obtain image-encoded features, wherein the source document image includes a source language text portion and an illustration portion; a second processing unit configured to input the source document image into the semantic alignment encoder to obtain unimodal semantic-encoded features; and a third processing unit configured to input the image-encoded features and the unimodal semantic-encoded features into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text portion and an illustration portion; wherein the translation model is trained using the output of a multimodal large model.

[0013] Optionally, the translation device further includes a training unit configured to train the translation model by: acquiring multiple document image samples and processing each document image sample as follows: inputting the current document image sample into an image encoder to obtain estimated image encoding features; inputting the current document image sample into a semantic alignment encoder to obtain estimated unimodal semantic encoding features; inputting the current document image sample and the corresponding source language text into a multimodal large model to obtain mixed-modal semantic encoding features; obtaining the cosine similarity between the estimated unimodal semantic encoding features and the mixed-modal semantic encoding features; obtaining the translation loss based on the estimated image encoding features and the estimated unimodal semantic encoding features; and adjusting the parameters of the translation model by minimizing the cosine similarity and the translation loss.

[0014] Optionally, the training unit is further configured to input the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain the translation result of the t-th character in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all characters translated before the t-th character, and t is a positive integer; the logarithmic sum of the translation results of each character in the target language text corresponding to the current document image sample is determined as the translation loss.

[0015] Optionally, the training unit is also configured to input the current document image sample into the image encoder in the multimodal large model to obtain the first encoding feature; input the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain the second encoding feature; and input the first encoding feature and the second encoding feature into multiple attention mechanism layers to obtain the hybrid modality semantic encoding feature.

[0016] Optionally, the first processing unit is further configured to scale the source document image to obtain an adjusted first image; and input the first image into an image encoder to obtain image encoding features, wherein the image encoder employs a sliding window attention mechanism.

[0017] Optionally, the second processing unit is further configured to scale the source document image to obtain an adjusted second image; input the second image into the image encoder in the semantic alignment encoder to obtain a third encoded feature, wherein the image encoder adopts a sliding window attention mechanism; and input the third encoded feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain a single-modal semantic encoded feature.

[0018] Optionally, the third processing unit is further configured to, after inputting the image encoding features and the single-modal semantic encoding features into the decoder to obtain the translation result of the source document image, render the target document image corresponding to the source document image based on the translation result.

[0019] In another general aspect, a computer-readable storage medium is provided for storing instructions, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform any of the translation methods described above.

[0020] In another general aspect, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform any of the translation methods described above.

[0021] In another general aspect, a computer program product is provided, comprising computer instructions that, when executed by a processor, implement any of the translation methods described above.

[0022] According to the translation method, apparatus, computer-readable storage medium, and computer program product disclosed herein, the translation model is trained using the output of a multimodal large model during forward inference, and its pre-trained knowledge is transferred to the translation model, thereby improving the generalization ability and translation quality of the translation model. Furthermore, the translation model does not require the participation of the multimodal large model during the inference phase, so the inference speed does not change significantly, thus achieving a balance between inference speed and translation quality. Therefore, this disclosure effectively solves the problems of poor generalization and long processing time in existing translation systems.

[0023] Further aspects and / or advantages of the general concept of this disclosure will be set forth in part in the description which follows, and in part will be clear from the description or may be learned by practice of the general concept of this disclosure. Attached Figure Description

[0024] The above and other objects and features of the embodiments of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings illustrating the embodiments, wherein:

[0025] Figure 1 This is a flowchart illustrating a translation method according to an embodiment of the present disclosure;

[0026] Figure 2 This is a schematic diagram illustrating document images of embodiments of this disclosure;

[0027] Figure 3 This is a schematic diagram illustrating a target document image corresponding to a document image in an embodiment of this disclosure;

[0028] Figure 4 This is a schematic diagram of the system structure illustrating the translation method of an embodiment of this disclosure;

[0029] Figure 5 This is a block diagram illustrating a translation apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0030] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and conciseness, descriptions of features known in the art may be omitted.

[0031] The features described herein may be implemented in different forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided only to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0032] As used herein, the term “and / or” includes any one of the associated listed items and any combination of any two or more.

[0033] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as the first component, first assembly, first region, first layer, or first part may also be referred to as the second component, second assembly, second region, second layer, or second part.

[0034] In the specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, the element may be directly "on" another element, directly "connected to," or "bonded to" the other element, or one or more other elements may be present in between. Conversely, when an element is described as being "directly on" another element, "directly connected to," or "directly bonded to" another element, no other elements may be present in between.

[0035] The terminology used herein is for the purpose of describing various examples only and is not intended to limit disclosure. Unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. The terms “comprising,” “including,” and “having” indicate the presence of the described features, quantities, operations, components, elements, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0037] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0038] Considering that multimodal large models can be pre-trained on a large number of document images of various types and have strong generalization ability, they can compensate for the shortcomings of existing end-to-end document image translation systems. Therefore, the basic idea of ​​this disclosure is to use the pre-trained knowledge (i.e., output) of the multimodal large model to train the translation model, and to transfer the pre-trained knowledge to the translation model through single-modal and mixed-modal alignment, thereby improving the generalization ability and translation quality of the translation model. During the inference stage, since only the translation model is involved and the multimodal large model does not need to run, the inference speed will not change significantly, achieving a balance between translation quality and inference speed.

[0039] Specifically, this disclosure can be divided into five steps: 1) Inputting the document image into an image encoder to obtain image encoding features; 2) Inputting the document image into a semantic alignment encoder to obtain unimodal semantic encoding features; 3) Inputting the document image and the source language text of the document image into a multimodal large model to obtain mixed-modal semantic encoding features; 4) In the training phase of the translation model, determining the cosine similarity loss of the unimodal semantic encoding features and the mixed-modal semantic encoding features, using the cosine similarity loss to guide the unimodal semantic encoding features output by the semantic alignment encoder to align to the mixed-modal semantic encoding features output by the multimodal large model, and combining the translation loss of the image encoding features and the unimodal semantic encoding features to optimize the overall translation model; 5) In the testing phase of the translation model, inputting the image encoding features and the unimodal semantic encoding features into the decoder to obtain the translation results.

[0040] The translation method and apparatus, computer-readable storage medium and computer program product of this disclosure are described in detail below with reference to the accompanying drawings.

[0041] This disclosure proposes a translation method. Figure 1 This is a flowchart illustrating a translation method according to an embodiment of this disclosure. (Refer to...) Figure 1 This translation method is based on a translation model, which includes an image encoder, a semantic alignment encoder, and a decoder. The translation model is trained using the output of a large multimodal model. The translation method includes the following steps:

[0042] In step S101, the source document image is input into the image encoder to obtain image encoding features, wherein the source document image includes the source language text portion and the illustration portion.

[0043] It should be noted that document images refer to a type of image that simultaneously contains multi-source information such as natural language text information, layout information, and visual information (such as illustrations) of a document, such as scanned documents, photographed documents, and PDF documents.

[0044] As an example, Figure 2An example is shown of a document image containing English text information and two illustrations. This document image is input into an image encoder for context encoding and fusion to obtain image encoded features.

[0045] According to embodiments of this disclosure, inputting a source document image into an image encoder to obtain image coding features may include: scaling the source document image to obtain an adjusted first image; and inputting the first image into the image encoder to obtain image coding features, wherein the image encoder employs a sliding window attention mechanism. Through this embodiment, by introducing scaling processing and employing a sliding window attention mechanism in the image encoder, superior image coding features can be extracted.

[0046] As an example, the input document image can be scaled, for instance, uniformly scaled to an image with a pixel count of 896*672. Then, a pre-trained image encoder based on the sliding window attention mechanism (Swin Transformer) structure can be used to perform context encoding and fusion on the scaled document image to obtain image encoding features with a sequence length of 588.

[0047] In step S102, the source document image is input into the semantic alignment encoder to obtain single-modal semantic coding features.

[0048] As an example, still using Figure 2 For example, by inputting the document image into the semantic alignment encoder, a single-modal semantic coding feature with the same dimension and length as the output of the multimodal large model can be obtained.

[0049] According to embodiments of this disclosure, inputting a source document image into a semantic alignment encoder to obtain unimodal semantic coding features may include: scaling the source document image to obtain an adjusted second image; inputting the second image into an image encoder within the semantic alignment encoder to obtain a third coding feature, wherein the image encoder employs a sliding window attention mechanism; and inputting the third coding feature into a two-layer feedforward neural network within the semantic alignment encoder to obtain unimodal semantic coding features. Through this embodiment, by using a two-layer feedforward neural network to align the third coding feature with the dimension and length of the output of a multimodal large model, superior unimodal semantic coding features are obtained.

[0050] As an example, the input document image can be scaled, for example, uniformly scaled to an image with a pixel count of 896*672; then, a pre-trained image encoder based on the sliding window attention mechanism (Swin Transformer) structure is used to perform context encoding and fusion on the scaled image to obtain image encoding features with a sequence length of 588; then, a two-layer feedforward neural network is used to align the obtained image encoding features with the output of the multimodal large model in terms of dimension and length, thereby obtaining better single-modal semantic encoding features.

[0051] In step S103, the image encoding features and unimodal semantic encoding features are input into the decoder to obtain the translation result of the source document image, wherein the translation result includes the target language text part and the illustration part.

[0052] As an example, the target language text mentioned above is the translated text from the source language text.

[0053] As an example, the image encoding features output by the image encoder and the unimodal semantic encoding features output by the semantic alignment encoder can be input into the corresponding cross-attention layer in the decoder, allowing the decoder to generate translation results autoregressively. It should be noted that the decoder translates character by character. Therefore, in this embodiment, predetermined information is also input into the decoder to obtain the translation result of the current character in the target language text corresponding to the document image. This predetermined information consists of the translation results of all characters translated before the current character.

[0054] As an example, image encoding features, unimodal semantic encoding features, and predetermined information are input into the decoder to obtain the translation result of the current character in the target language text corresponding to the document image. This process is repeated until all characters in the target language text are obtained. The predetermined information is the translation result of all characters translated before the current character.

[0055] According to embodiments of this disclosure, after inputting image encoding features and unimodal semantic encoding features into a decoder to obtain the translation result of the source document image, a target document image corresponding to the source document image can be rendered based on the translation result. Through this embodiment, a document image corresponding to the translated language text can be obtained.

[0056] As an example, the translation model can ultimately generate translation results in Markdown format, so a specialized renderer can be used to render the Markdown results into an image, i.e., the target document image, such as... Figure 3 As shown.

[0057] According to embodiments of this disclosure, the translation model can be trained as follows: Multiple document image samples are acquired, and each document image sample is processed as follows: the current document image sample is input into an image encoder to obtain estimated image encoding features; the current document image sample is input into a semantic alignment encoder to obtain estimated single-modal semantic encoding features; the current document image sample and the corresponding source language text are input into a multimodal large model to obtain mixed-modal semantic encoding features; the cosine similarity between the estimated single-modal semantic encoding features and the mixed-modal semantic encoding features is obtained; based on the estimated image encoding features and the estimated single-modal semantic encoding features, a translation loss is obtained; and the parameters of the translation model are adjusted by minimizing the cosine similarity and the translation loss. Through this embodiment, the translation model is trained using the output features of the multimodal large model during forward inference, transferring the pre-trained knowledge of the large model to the small model, effectively improving the translation performance of the translation model.

[0058] It should be noted that training of the translation model can be terminated when the total loss is less than a preset threshold or the number of iterations exceeds a preset number. The number of iterations can be comparable to the number of document image samples.

[0059] According to embodiments of this disclosure, obtaining translation loss based on estimated image coding features and estimated unimodal semantic coding features may include: inputting the estimated unimodal semantic coding features, estimated image coding features, and predetermined information into a decoder to obtain the translation result of the t-th character in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all characters translated before the t-th character, and t is a positive integer; and determining the translation loss as the logarithmic sum of the translation results of each character in the target language text corresponding to the current document image sample. This embodiment allows for convenient and rapid acquisition of relatively accurate translation loss.

[0060] According to embodiments of this disclosure, inputting the current document image sample and the corresponding source language text into a multimodal large model to obtain mixed-modal semantic coding features may include: inputting the current document image sample into an image encoder in the multimodal large model to obtain a first coding feature; inputting the corresponding source language text into a text embedding layer in the multimodal large model to obtain a second coding feature; and inputting the first coding feature and the second coding feature into multiple attention mechanism layers to obtain mixed-modal semantic coding features.

[0061] As an example, an image encoder in a pre-trained multimodal large model can be used to encode the input document image, and then a text embedding layer in a pre-trained multimodal large model can be used to encode the input source language text. The results of the first two steps of encoding, namely the first and second encoding features mentioned above, are then processed through several Transformer layers (attention mechanism layers) to extract the output sequence of the last layer as the mixed modality semantic encoding features.

[0062] To facilitate understanding of the above embodiments, the following is in conjunction with... Figure 4 Let's illustrate this with a specific example. Figure 4 The system architecture of the translation method disclosed herein is presented.

[0063] Suppose we are given a document image, such as Figure 2 As shown, using the document image as input to the translation model, the main steps in obtaining the translation result are as follows:

[0064] S1. Input the document image into the image encoder to obtain the image encoding features, which may include the following steps:

[0065] S11: Scaling the input document image to a uniform size of 896*672 pixels;

[0066] S12: Use a pre-trained image encoder based on the Swin Transformer structure to perform context encoding and fusion on the scaled image to obtain image encoding features with a sequence length of 588.

[0067] Step S1 can be formally represented as:

[0068] H image =Encoder image (I)

[0069] Where I represents the input document image, H image Encoder represents image encoding features. image This indicates an image encoder.

[0070] S2. Input the document image into the semantic alignment encoder to obtain single-modal semantic coding features, which may include the following steps:

[0071] S21: Perform scaling on the input document image, uniformly scaling it to an image with a pixel count of 896*672;

[0072] S22: Use a pre-trained image encoder based on the Swin Transformer structure to perform context encoding and fusion on the scaled image to obtain image encoding features with a sequence length of 588;

[0073] S23: Input the image encoding features into a two-layer feedforward neural network to obtain single-modal semantic encoding features. That is, use a two-layer feedforward neural network to align the obtained image encoding features with the output sequence of the multimodal large model in terms of dimension and length.

[0074] Step S2 can be formally represented as:

[0075] H swin =Swin(I)

[0076] H Align =FFN length (FFN dim (H swin ) T ) T

[0077] Where I represents the input document image, H swin H represents the output of the Swing Transformer. Alig n represents the single-modal semantic encoding feature, Swin represents the Swin Transformer, and FFN represents the feedforward neural network.

[0078] S3. Input the document image and the source language text in the document image into the multimodal large model to obtain the mixed modality semantic encoding features, which may include the following steps:

[0079] S31: Encode the input document image using the image encoder in the pre-trained multimodal large model to obtain the first encoded feature;

[0080] S32: The input source language text is encoded using a text embedding layer in a pre-trained multimodal large model to obtain the second encoded feature. It should be noted that the source language text can be identified from a document image through image recognition.

[0081] S33: After encoding the results of the first two steps, namely the first and second encoded features, through several Transformer layers, extract the output sequence of the last layer as the mixed modality semantic encoded features.

[0082] Step S3 can be formally represented as:

[0083] H MLLM =MLLM(I,X)

[0084] Where I represents the input document image, X represents the source language text, and H represents the source language text. MLLM MLLM represents mixed-modal semantic encoding features.

[0085] S4. During the training phase, cosine similarity loss can be used to guide the alignment of single-modal semantic coding features to mixed-modal semantic coding features, and translation loss can be used to optimize the overall model. Specifically, this can include the following steps:

[0086] S41: Calculate the cosine similarity between the single-modal semantic coding features output by the semantic alignment encoder and the mixed-modal semantic coding features output by the multimodal large model, and use this cosine similarity to guide the gradient optimization of the semantic alignment encoder. The alignment loss can be as follows:

[0087] L align =1-Cos(H MLLM H Align )

[0088] Among them, H MLLM H represents the semantic encoding features of mixed modalities. Align represents the unimodal semantic encoding feature, and Cos represents the calculation of the cosine similarity between two tensors.

[0089] S42: Calculate the translation loss of the decoder part, which can be formally represented as:

[0090] p(y t |y <t ,I,X;θ)=Decoder(y <t H Align H image )

[0091]

[0092] Among them, H Align H represents the semantic encoding features of a single modality. image Let θ represent the image encoding features, y represent the decoder, and θ represent the trainable parameters of the translation model, i.e., the trainable parameters including the image encoder, semantic alignment encoder, and decoder. t y represents the t-th character in the target language translated by the decoder. <t p(y) represents all characters generated before the current character is generated. t |y <t , I, X; θ) represent the probability distribution of the t-th character in the target language translated by the decoder on the vocabulary.

[0093] S43: The alignment loss and translation loss are combined to perform gradient optimization on the overall translation model. The overall loss can be as follows:

[0094] L=α×L align +L trans

[0095] Where α represents the hyperparameter, L align L represents the alignment loss. trans This indicates a loss in translation.

[0096] S5. During the testing phase, the image encoding features and unimodal semantic encoding features are input into the decoder to obtain the translation results. Specifically, this may include the following steps:

[0097] S51: The image encoding features output by the image encoder and the unimodal semantic encoding features output by the semantic alignment encoder are input into the corresponding cross-attention layer in the decoder, allowing the decoder to generate the translation result autoregressively. It should be noted that the decoder also receives predetermined information to obtain the translation result of the current character in the target language text corresponding to the document image. This predetermined information consists of the translation results of all characters translated before the current character.

[0098] The translation model ultimately generates the translation results in Markdown format, so a dedicated renderer can be used to render the Markdown text into an image, i.e., a target document image, such as... Figure 3 As shown.

[0099] In summary, this disclosure can translate any document image. Specifically, it utilizes the pre-trained knowledge of a multimodal large model to train the translation model, and transfers its own pre-trained knowledge to the translation model through mixed-modal and unimodal alignment. Moreover, during the inference stage, the multimodal large model is not required; only the translation model is needed, thus achieving a balance between inference speed and translation quality. Therefore, this disclosure overcomes the problem of poor generalization in existing end-to-end document image translation systems due to scarce training data, such as poor domain generalization, poor generalization of long text images, and poor generalization of complex layouts.

[0100] Figure 5 This is a block diagram illustrating a translation apparatus according to an embodiment of the present disclosure, such as... Figure 5 As shown, the translation device is implemented based on a translation model, which includes an image encoder, a semantic alignment encoder, and a decoder. The device includes a first processing unit 50, a second processing unit 52, and a third processing unit 54.

[0101] The first processing unit 50 is configured to input the source document image into an image encoder to obtain image encoding features, wherein the source document image includes a source language text portion and an illustration portion; the second processing unit 52 is configured to input the source document image into a semantic alignment encoder to obtain unimodal semantic encoding features; the third processing unit 54 is configured to input the image encoding features and the unimodal semantic encoding features into a decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text portion and an illustration portion; wherein the translation model is trained using the output of a multimodal large model.

[0102] According to embodiments of this disclosure, the translation apparatus further includes a training unit configured to train a translation model by: acquiring multiple document image samples and performing the following processing on each document image sample: inputting the current document image sample into an image encoder to obtain estimated image coding features; inputting the current document image sample into a semantic alignment encoder to obtain estimated single-modal semantic coding features; inputting the current document image sample and the corresponding source language text into a multimodal large model to obtain mixed-modal semantic coding features; obtaining the cosine similarity between the estimated single-modal semantic coding features and the mixed-modal semantic coding features; obtaining a translation loss based on the estimated image coding features and the estimated single-modal semantic coding features; and adjusting the parameters of the translation model by minimizing the cosine similarity and the translation loss.

[0103] According to embodiments of this disclosure, the training unit is further configured to input the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain the translation result of the t-th character in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all characters translated before the t-th character, and t is a positive integer; the logarithmic sum of the translation results of each character in the target language text corresponding to the current document image sample is determined as the translation loss.

[0104] According to embodiments of this disclosure, the training unit is further configured to input the current document image sample into the image encoder in the multimodal large model to obtain a first encoding feature; input the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain a second encoding feature; and input the first encoding feature and the second encoding feature into multiple attention mechanism layers to obtain mixed-modality semantic encoding features.

[0105] According to an embodiment of this disclosure, the first processing unit 50 is further configured to perform scaling processing on the source document image to obtain an adjusted first image; input the first image into an image encoder to obtain image encoding features, wherein the image encoder employs a sliding window attention mechanism.

[0106] According to an embodiment of this disclosure, the second processing unit 52 is further configured to perform scaling processing on the source document image to obtain an adjusted second image; input the second image into the image encoder in the semantic alignment encoder to obtain a third encoding feature, wherein the image encoder adopts a sliding window attention mechanism; and input the third encoding feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain a single-modal semantic encoding feature.

[0107] According to an embodiment of this disclosure, the third processing unit 54 is further configured to, after inputting image coding features and single-modal semantic coding features into the decoder to obtain the translation result of the source document image, render the target document image corresponding to the source document image based on the translation result.

[0108] According to embodiments of the present disclosure, a computer-readable storage medium for storing instructions is provided, wherein when the instructions are executed by at least one computing device, they cause at least one computing device to perform a translation method as described in any of the above embodiments.

[0109] According to embodiments of the present disclosure, a system is provided that includes at least one computing device and at least one storage device for storing instructions, wherein the instructions, when executed by at least one computing device, cause at least one computing device to perform a translation method as described in any of the above embodiments.

[0110] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents.

Claims

1. A translation method, characterized in that, The translation method is based on a translation model, which includes an image encoder, a semantic alignment encoder, and a decoder. The translation method includes: The source document image is input into the image encoder to obtain image encoding features, wherein the source document image includes a source language text portion and an illustration portion; The source document image is input into the semantic alignment encoder to obtain single-modal semantic coding features; The image encoding features and the unimodal semantic encoding features are input into the decoder to obtain the translation result of the source document image, wherein the translation result includes the target language text part and the illustration part; The translation model is trained using the output of a multimodal large model. The translation model is trained in the following manner: Obtain multiple document image samples, and perform the following processing on each document image sample: Input the current document image sample into the image encoder to obtain the estimated image coding features; The current document image sample is input into the semantic alignment encoder to obtain the estimated single-modal semantic coding features; The current document image sample and the corresponding source language text are input into the multimodal large model to obtain mixed modality semantic coding features; Obtain the cosine similarity between the estimated single-modal semantic coding features and the mixed-modal semantic coding features; Based on the estimated image coding features and the estimated single-modal semantic coding features, the translation loss is obtained; The parameters of the translation model are adjusted by minimizing the cosine similarity and the translation loss.

2. The translation method as described in claim 1, characterized in that, The process of obtaining translation loss based on the estimated image coding features and the estimated single-modal semantic coding features includes: The estimated single-modal semantic coding features, the estimated image coding features, and the predetermined information are input into the decoder to obtain the translation result of the t-th character in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation result of all characters translated before the t-th character, and t is a positive integer; The translation loss is determined by the logarithmic sum of the translation results of each character in the target language text corresponding to the current document image sample.

3. The translation method as described in claim 1, characterized in that, The step of inputting the current document image sample and the corresponding source language text into the multimodal large model to obtain mixed-modal semantic coding features includes: The current document image sample is input into the image encoder in the multimodal large model to obtain the first encoded feature; The source language text corresponding to the current document image sample is input into the text embedding layer in the multimodal large model to obtain the second encoded feature; The first and second encoding features are input into multiple attention mechanism layers to obtain the hybrid modality semantic encoding features.

4. The translation method as described in claim 1, characterized in that, The step of inputting the source document image into the image encoder to obtain image encoding features includes: The source document image is scaled to obtain the adjusted first image; The first image is input into the image encoder to obtain image encoded features, wherein the image encoder employs a sliding window attention mechanism.

5. The translation method as described in claim 1, characterized in that, The step of inputting the source document image into the semantic alignment encoder to obtain single-modal semantic coding features includes: The source document image is scaled to obtain the adjusted second image; The second image is input into the image encoder in the semantic alignment encoder to obtain the third encoded feature, wherein the image encoder adopts a sliding window attention mechanism; The third encoding feature is input into the two-layer feedforward neural network in the semantic alignment encoder to obtain the single-modal semantic encoding feature.

6. The translation method as described in claim 1, characterized in that, After inputting the image encoding features and the unimodal semantic encoding features into the decoder to obtain the translation result of the source document image, the method further includes: Based on the translation results, the target document image corresponding to the source document image is rendered.

7. A translation device, characterized in that, The translation device is based on a translation model, which includes an image encoder, a semantic alignment encoder, and a decoder, comprising: The first processing unit is configured to input a source document image into the image encoder to obtain image encoding features, wherein the source document image includes a source language text portion and an illustration portion; The second processing unit is configured to input the source document image into the semantic alignment encoder to obtain single-modal semantic coding features; The third processing unit is configured to input the image encoding features and the unimodal semantic encoding features into the decoder to obtain the translation result of the source document image, wherein the translation result includes the target language text part and the illustration part; The translation model is trained using the output of a multimodal large model. The translation device also includes a training unit configured to train a translation model by: acquiring multiple document image samples and processing each document image sample as follows: inputting the current document image sample into the image encoder to obtain estimated image encoding features; inputting the current document image sample into the semantic alignment encoder to obtain estimated single-modal semantic encoding features; inputting the current document image sample and the corresponding source language text into the multimodal large model to obtain mixed-modal semantic encoding features; obtaining the cosine similarity between the estimated single-modal semantic encoding features and the mixed-modal semantic encoding features; obtaining a translation loss based on the estimated image encoding features and the estimated single-modal semantic encoding features; and adjusting the parameters of the translation model by minimizing the cosine similarity and the translation loss.

8. A computer-readable storage medium for storing instructions, characterized in that, When the instruction is executed by at least one computing device, it causes the at least one computing device to perform the translation method as described in any one of claims 1 to 6.

9. A system comprising at least one computing device and at least one storage device for storing instructions, characterized in that, When the instruction is executed by the at least one computing device, it causes the at least one computing device to perform the translation method as described in any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the translation method as described in any one of claims 1 to 6.

Images