Translation method and device, computer readable storage medium and computer program product

Abstract

Disclosed are a translation method and apparatus, a computer readable storage medium and a computer program product, the method comprising: inputting a source document image into an image encoder to obtain an image encoding feature, the source document image comprising a source language text portion and an illustration portion; inputting the source document image into a semantic alignment encoder to obtain a single-mode semantic encoding feature; the image coding features and the single-mode semantic coding features are input into a decoder, a translation result of the source document image is obtained, and the translation result comprises a target language text part and an illustration part; wherein the translation model is trained by utilizing the output of the multi-modal large model.

Description

Technical Field

[0001] The present disclosure generally relates to the field of natural language processing technology, and more specifically, to a translation method and apparatus, a computer-readable storage medium, and a computer program product. Background Art

[0002] The processing object of document image translation technology is document image. Its goal is to mine the text, layout, visual and other information contained in the document image, and integrate and utilize multi-source information to translate the source language text in the document image into the correct target language text.

[0003] At present, most document image translation systems are cascade systems. After the document image is input, it needs to go through a series of steps such as layout analysis, optical character recognition, text translation, and post-translation processing in order to obtain the final translation text. The whole process is time-consuming and computationally inefficient. In addition, there are problems such as cumulative errors caused by transmission between different modules, complex system structure, and difficult deployment. In addition, the current document image translation system requires a large amount of supervised data training to achieve better results, but there are still problems such as poor domain generalization and poor ability to process long document images and images with complex layouts. Summary of the invention

[0004] The embodiments of the present disclosure provide a translation method and apparatus, a computer-readable storage medium, and a computer program product, which can effectively solve the problems of poor generalization and long time consumption of translation systems in the prior art.

[0005] In a general aspect, a translation method is provided. The translation method is implemented based on a translation model, and the translation model includes an image encoder, a semantic alignment encoder and a decoder, including: inputting a source document image into the image encoder to obtain image coding features, wherein the source document image includes a source language text part and an illustration part; inputting the source document image into the semantic alignment encoder to obtain unimodal semantic coding features; inputting the image coding features and the unimodal semantic coding features into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text part and an illustration part; wherein the translation model is trained using the output of a multimodal large model.

[0006] Optionally, the translation model is trained in the following manner: multiple document image samples are obtained, and the following processing is performed on each document image sample: the current document image sample is input into an image encoder to obtain estimated image coding features; the current document image sample is input into a semantic alignment encoder to obtain estimated unimodal semantic coding features; the current document image sample and the source language text corresponding to the current document image sample are input into a multimodal large model to obtain mixed modal semantic coding features; the cosine similarity of the estimated unimodal semantic coding features and the mixed modal semantic coding features is obtained; the translation loss is obtained based on the estimated image coding features and the estimated unimodal semantic coding features; and the parameters of the translation model are adjusted by minimizing the cosine similarity and the translation loss.

[0007] Optionally, based on the estimated image coding features and the estimated unimodal semantic coding features, the translation loss is obtained, including: inputting the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into a decoder to obtain a translation result of the t-th word in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all words translated before the t-th word, and t is a positive integer; and determining the logarithm sum of the translation results of each word in the target language text corresponding to the current document image sample as the translation loss.

[0008] Optionally, the current document image sample and the source language text corresponding to the current document image sample are input into the multimodal large model to obtain mixed modal semantic coding features, including: inputting the current document image sample into the image encoder in the multimodal large model to obtain a first coding feature; inputting the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain a second coding feature; inputting the first coding feature and the second coding feature into multiple attention mechanism layers to obtain mixed modal semantic coding features.

[0009] Optionally, the source document image is input into an image encoder to obtain image coding features, including: scaling the source document image to obtain an adjusted first image; and inputting the first image into an image encoder to obtain image coding features, wherein the image encoder adopts a sliding window attention mechanism.

[0010] Optionally, the source document image is input into a semantic alignment encoder to obtain a unimodal semantic coding feature, including: scaling the source document image to obtain an adjusted second image; inputting the second image into an image encoder in the semantic alignment encoder to obtain a third coding feature, wherein the image encoder adopts a sliding window attention mechanism; inputting the third coding feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain a unimodal semantic coding feature.

[0011] Optionally, after inputting the image coding features and the unimodal semantic coding features into the decoder to obtain the translation result of the source document image, the method further includes: rendering to obtain a target document image corresponding to the source document image based on the translation result.

[0012] In another general aspect, a translation device is provided, which is implemented based on a translation model, the translation model includes an image encoder, a semantic alignment encoder and a decoder, including: a first processing unit, configured to input a source document image into the image encoder to obtain image encoding features, wherein the source document image includes a source language text portion and an illustration portion; a second processing unit, configured to input the source document image into the semantic alignment encoder to obtain unimodal semantic encoding features; a third processing unit, configured to input the image encoding features and the unimodal semantic encoding features into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text portion and an illustration portion; wherein the translation model is trained using the output of a multimodal large model.

[0013] Optionally, the translation device also includes a training unit, which is configured to train the translation model in the following manner: obtain multiple document image samples, and perform the following processing on each document image sample: input the current document image sample into an image encoder to obtain estimated image coding features; input the current document image sample into a semantic alignment encoder to obtain estimated unimodal semantic coding features; input the current document image sample and the source language text corresponding to the current document image sample into a multimodal large model to obtain mixed modal semantic coding features; obtain the cosine similarity of the estimated unimodal semantic coding features and the mixed modal semantic coding features; obtain the translation loss based on the estimated image coding features and the estimated unimodal semantic coding features; and adjust the parameters of the translation model by minimizing the cosine similarity and the translation loss.

[0014] Optionally, the training unit is further configured to input the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain the translation result of the t-th word in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all words translated before the t-th word, and t is a positive integer; and the logarithm sum of the translation results of each word in the target language text corresponding to the current document image sample is determined as the translation loss.

[0015] Optionally, the training unit is also configured to input the current document image sample into the image encoder in the multimodal large model to obtain a first encoding feature; input the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain a second encoding feature; input the first encoding feature and the second encoding feature into multiple attention mechanism layers to obtain a mixed modal semantic encoding feature.

[0016] Optionally, the first processing unit is further configured to scale the source document image to obtain an adjusted first image; input the first image into an image encoder to obtain image encoding features, wherein the image encoder adopts a sliding window attention mechanism.

[0017] Optionally, the second processing unit is further configured to scale the source document image to obtain an adjusted second image; input the second image into an image encoder in a semantic alignment encoder to obtain a third encoding feature, wherein the image encoder adopts a sliding window attention mechanism; input the third encoding feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain a unimodal semantic encoding feature.

[0018] Optionally, the third processing unit is further configured to, after inputting the image coding features and the unimodal semantic coding features into the decoder to obtain the translation result of the source document image, render a target document image corresponding to the source document image based on the translation result.

[0019] In another general aspect, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to perform any of the translation methods described above.

[0020] In another general aspect, a system is provided comprising at least one computing device and at least one storage device storing instructions, wherein the instructions, when executed by the at least one computing device, cause the at least one computing device to perform any of the translation methods described above.

[0021] In another general aspect, a computer program product is provided, comprising computer instructions, which implement any of the above translation methods when executed by a processor.

[0022] According to the translation method and device, computer-readable storage medium and computer program product disclosed in the present invention, the output of the multimodal large model during forward reasoning is used to train the translation model, and the pre-trained knowledge is transferred to the translation model, thereby improving the generalization ability and translation quality of the translation model; and the translation model does not need the participation of the multimodal large model in the reasoning stage, so the reasoning speed will not change significantly, thereby achieving a balance between reasoning speed and translation quality. Therefore, through the present disclosure, the problem of poor generalization and long time consumption of the translation system in the prior art can be effectively solved.

[0023] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other objects and features of the embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings showing the embodiments, in which:

[0025] Figure 1 is a flowchart illustrating a translation method of an embodiment of the present disclosure;

[0026] Figure 2 is a schematic diagram showing a document image according to an embodiment of the present disclosure;

[0027] Figure 3 is a schematic diagram showing a target document image corresponding to a document image according to an embodiment of the present disclosure;

[0028] Figure 4 is a schematic diagram showing the system structure of the translation method of the embodiment of the present disclosure;

[0029] Figure 5 is a block diagram showing a translation device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0031] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0032] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0033] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0034] In the specification, when an element (such as a layer, a region, or a substrate) is described as being “on”, “connected to”, or “coupled to” another element, the element may be directly “on”, “connected to”, or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to”, or “directly coupled to” another element, there may be no other elements present therebetween.

[0035] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0036] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0037] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0038] Considering that the multimodal large model can be pre-trained on a large number of document images of various types, it has strong generalization and can make up for the defects of the existing end-to-end document image translation system. Therefore, the basic idea of ​​the present invention is to use the pre-trained knowledge (that is, output) of the multimodal large model to train the translation model, and transfer the pre-trained knowledge to the translation model by aligning the single modality and the mixed modality, so as to improve the generalization ability of the translation model and the translation quality. In the reasoning stage, since only the translation model is involved and the multimodal large model does not need to run, the reasoning speed will not change significantly, achieving a balance between translation quality and reasoning speed.

[0039] Specifically, the present disclosure can be divided into five steps: 1) inputting a document image into a picture encoder to obtain image coding features; 2) inputting a document image into a semantic alignment encoder to obtain unimodal semantic coding features; 3) inputting the document image and the source language text of the text image into a multimodal large model to obtain mixed-modal semantic coding features; 4) in the training phase of the translation model, determining the cosine similarity loss of the unimodal semantic coding features and the mixed-modal semantic coding features, using the cosine similarity loss to guide the unimodal semantic coding features output by the semantic alignment encoder to be aligned to the mixed-modal semantic coding features output by the multimodal large model, and combining the translation losses of the image coding features and the unimodal semantic coding features to optimize the translation model as a whole; 5) in the testing phase of the translation model, inputting the image coding features and the unimodal semantic coding features into the decoder to obtain the translation result.

[0040] The translation method and apparatus, computer-readable storage medium, and computer program product disclosed herein are described in detail below with reference to the accompanying drawings.

[0041] The present invention proposes a translation method. Figure 1 is a flowchart showing a translation method according to an embodiment of the present disclosure. Figure 1 The translation method is implemented based on a translation model, the translation model includes an image encoder, a semantic alignment encoder and a decoder, the translation model is trained using the output of a multimodal large model, and the translation method includes the following steps:

[0042] In step S101, a source document image is input into an image encoder to obtain image coding features, wherein the source document image includes a source language text portion and an illustration portion.

[0043] It should be noted that a document image refers to a type of image that contains multiple source information such as the document's natural language text information, page layout information, and visual information (such as illustrations), such as scanned documents, photographed documents, PDF documents, etc.

[0044] As an example, Figure 2An example of a document image is shown, which includes English text information and two illustrations. The document image is input into an image encoder for context encoding and fusion to obtain image encoding features.

[0045] According to an embodiment of the present disclosure, inputting a source document image into an image encoder to obtain image coding features may include: performing scaling processing on the source document image to obtain an adjusted first image; and inputting the first image into the image encoder to obtain image coding features, wherein the image encoder adopts a sliding window attention mechanism. Through this embodiment, scaling processing is introduced and the image encoder adopts a sliding window attention mechanism, so that better image coding features can be extracted.

[0046] As an example, the input document image can be scaled, for example, uniformly scaled to an image with a pixel number of 896*672, and then a pre-trained image encoder based on a sliding window attention mechanism (Swin Transformer) structure is used to contextually encode and fuse the scaled document image to obtain image coding features with a sequence length of 588.

[0047] In step S102, the source document image is input into a semantic alignment encoder to obtain a unimodal semantic encoding feature.

[0048] As an example, Figure 2 For example, by inputting the document image into the semantic alignment encoder, we can obtain the unimodal semantic encoding features that are consistent with the dimension and length of the multimodal large model output.

[0049] According to an embodiment of the present disclosure, inputting a source document image into a semantic alignment encoder to obtain a unimodal semantic coding feature may include: scaling the source document image to obtain an adjusted second image; inputting the second image into an image encoder in the semantic alignment encoder to obtain a third coding feature, wherein the image encoder adopts a sliding window attention mechanism; inputting the third coding feature into a two-layer feedforward neural network in the semantic alignment encoder to obtain a unimodal semantic coding feature. Through this embodiment, the third coding feature is aligned with the dimension and length of the output of the multimodal large model through a two-layer feedforward neural network, thereby obtaining a relatively excellent unimodal semantic coding feature.

[0050] As an example, the input document image can be scaled, for example, uniformly scaled to an image with a pixel count of 896*672; then, a pre-trained image encoder based on a sliding window attention mechanism (Swin Transformer) structure is used to contextually encode and fuse the scaled image to obtain image coding features with a sequence length of 588; and then a two-layer feedforward neural network is used to align the obtained image coding features with the output of the multimodal large model in terms of dimension and length, thereby obtaining better unimodal semantic coding features.

[0051] In step S103, the image coding features and the unimodal semantic coding features are input into a decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text part and an illustration part.

[0052] As an example, the target language text is a text translated from the source language text.

[0053] As an example, the image encoding features output by the image encoder and the unimodal semantic encoding features output by the semantic alignment encoder can be input into the corresponding cross-attention layer in the decoder, so that the decoder can generate the translation result autoregressively. It should be noted that the decoder translates word by word, so in this embodiment, predetermined information is also input into the decoder to obtain the translation result of the current word in the target language text corresponding to the document image, wherein the predetermined information is the translation results of all words translated before the current word.

[0054] As an example, image coding features, unimodal semantic coding features and predetermined information are input into the decoder to obtain the translation result of the current word in the target language text corresponding to the document image, and so on, until all the words of the target language text are obtained, wherein the predetermined information is the translation results of all words translated before the current word.

[0055] According to an embodiment of the present disclosure, after inputting the image coding features and the unimodal semantic coding features into the decoder to obtain the translation result of the source document image, the target document image corresponding to the source document image can also be rendered based on the translation result. Through this embodiment, the document image corresponding to the translated language text can be obtained.

[0056] As an example, the translation model can finally generate translation results in the markdown format, so a special renderer can be used to render the markdown results into an image, that is, the target document image, such as Figure 3 shown.

[0057] According to the embodiments of the present disclosure, the translation model can be trained in the following manner: obtain multiple document image samples, and perform the following processing on each document image sample: input the current document image sample into the image encoder to obtain the estimated image coding features; input the current document image sample into the semantic alignment encoder to obtain the estimated unimodal semantic coding features; input the current document image sample and the source language text corresponding to the current document image sample into the multimodal large model to obtain the mixed modal semantic coding features; obtain the cosine similarity of the estimated unimodal semantic coding features and the mixed modal semantic coding features; obtain the translation loss based on the estimated image coding features and the estimated unimodal semantic coding features; adjust the parameters of the translation model by minimizing the cosine similarity and the translation loss. Through this embodiment, the translation model is trained using the output features of the multimodal large model during forward reasoning, and the pre-trained knowledge of the large model is transferred to the small model, which effectively improves the translation effect of the translation model.

[0058] It should be noted that the training of the translation model may be terminated when the total loss is less than a preset threshold or the number of iterations exceeds a preset number, wherein the number of iterations may be equivalent to the number of document image samples.

[0059] According to an embodiment of the present disclosure, obtaining translation loss based on estimated image coding features and estimated unimodal semantic coding features may include: inputting estimated unimodal semantic coding features, estimated image coding features, and predetermined information into a decoder to obtain a translation result of the t-th word in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all words translated before the t-th word, and t is a positive integer; and determining the logarithm sum of the translation results of each word in the target language text corresponding to the current document image sample as the translation loss. Through this embodiment, a relatively accurate translation loss can be obtained conveniently and quickly.

[0060] According to an embodiment of the present disclosure, the current document image sample and the source language text corresponding to the current document image sample are input into a multimodal large model to obtain a mixed modal semantic coding feature, which may include: inputting the current document image sample into an image encoder in the multimodal large model to obtain a first coding feature; inputting the source language text corresponding to the current document image sample into a text embedding layer in the multimodal large model to obtain a second coding feature; inputting the first coding feature and the second coding feature into multiple attention mechanism layers to obtain a mixed modal semantic coding feature.

[0061] As an example, the image encoder in the pre-trained multimodal large model can be used to encode the input document image, and then the text embedding layer in the pre-trained multimodal large model can be used to encode the input source language text. Then, the results of the first two steps of encoding, that is, the first encoding features and the second encoding features mentioned above, are passed through several Transformer layers (attention mechanism layers), and the output sequence of the last layer is extracted as the mixed modal semantic encoding feature.

[0062] In order to facilitate the understanding of the above embodiments, Figure 4 Let's take a specific example to illustrate. Figure 4 The system structure of the translation method disclosed in the present invention is given.

[0063] Suppose a document image is given, such as Figure 2 As shown, the document image is used as the input of the translation model. The process of obtaining the translation result mainly includes the following steps:

[0064] S1. Inputting the document image into an image encoder to obtain image encoding features may specifically include the following steps:

[0065] S11: Scaling the input document image to uniformly scale it into an image with a pixel number of 896*672;

[0066] S12: Use a pre-trained image encoder based on the Swin Transformer structure to perform context encoding and fusion on the scaled image to obtain image coding features with a sequence length of 588.

[0067] Step S1 can be formally expressed as:

[0068] H image =Encoder image (I)

[0069] Where I represents the input document image, H image Represents image encoding features, Encoder image Represents an image encoder.

[0070] S2. Inputting the document image into a semantic alignment encoder to obtain a unimodal semantic encoding feature, which may specifically include the following steps:

[0071] S21: Scaling the input document image to uniformly scale it into an image with a pixel number of 896*672;

[0072] S22: Use the pre-trained image encoder based on the Swin Transformer structure to perform context encoding and fusion on the scaled image to obtain image coding features with a sequence length of 588;

[0073] S23: The image coding features are input into a two-layer feedforward neural network to obtain unimodal semantic coding features, that is, using a two-layer feedforward neural network, the obtained image coding features are aligned with the output sequence of the multimodal large model in dimension and length respectively.

[0074] Step S2 can be formally expressed as:

[0075] H swin =Swin(I)

[0076] H Align =FFN length (FFN dim (H swin ) T ) T

[0077] Where I represents the input document image, H swin represents the output of Swin Transformer, H Alig n represents unimodal semantic encoding features, Swin represents Swin Transformer, and FFN represents feedforward neural network.

[0078] S3, inputting the document image and the source language text in the document image into the multimodal large model to obtain mixed modal semantic encoding features, which may specifically include the following steps:

[0079] S31: Encode the input document image using the image encoder in the pre-trained multimodal large model to obtain a first encoding feature;

[0080] S32: Encode the input source language text using the text embedding layer in the pre-trained multimodal large model to obtain a second encoding feature. It should be noted that the source language text can be identified from the document image through image recognition.

[0081] S33: After the results of the first two steps of encoding, i.e., the first encoding feature and the second encoding feature, pass through several Transformer layers, the output sequence of the last layer is extracted as the mixed modal semantic encoding feature.

[0082] Step S3 can be formally expressed as:

[0083] H MLLM =MLLM(I,X)

[0084] Where I represents the input document image, X represents the source language text, and H MLLM It represents mixed modal semantic encoding features and MLLM represents multimodal large model.

[0085] S4. During the training phase, the cosine similarity loss can be used to guide the alignment of single-modal semantic encoding features to mixed-modal semantic encoding features, and the translation loss can be used to optimize the model as a whole. Specifically, the following steps can be included:

[0086] S41: Calculate the cosine similarity between the single-modal semantic coding features output by the semantic alignment encoder and the mixed-modal semantic coding features output by the multimodal large model, and guide the semantic alignment encoder to perform gradient optimization based on the cosine similarity, wherein the alignment loss can be as follows:

[0087] L align =1-Cos(H MLLM , H Align )

[0088] Among them, H MLLM represents the mixed modal semantic encoding feature, H Align Represents unimodal semantic encoding features, and Cos represents the calculation of the cosine similarity between two tensors.

[0089] S42: Calculate the translation loss of the decoder part, which can be formally expressed as:

[0090] p(y t |y <t , I, X; θ) = Decoder(y <t , H Align , H image )

[0091]

[0092] Among them, H Align represents the unimodal semantic encoding feature, H image represents the image encoding features, Decoder represents the decoder, θ represents the trainable parameters of the translation model, that is, the trainable parameters of the image encoder, semantic alignment encoder and decoder, y t represents the tth word in the target language translated by the decoder, y <t represents all the words generated before the current word is generated, p(y t |y <t , I, X; θ) represents the probability distribution of the tth word in the target language translated by the decoder in the vocabulary.

[0093] S43: Combine the alignment loss and translation loss to perform gradient optimization on the entire translation model. The overall loss can be as follows:

[0094] L = α × L align +L trans

[0095] Among them, α represents the hyperparameter, L align represents the alignment loss, L trans Indicates translation loss.

[0096] S5. In the testing phase, the image coding features and the unimodal semantic coding features are input into the decoder to obtain the translation result, which may specifically include the following steps:

[0097] S51: Input the image encoding features output by the picture encoder and the unimodal semantic encoding features output by the semantic alignment encoder into the corresponding cross attention layer in the decoder, and let the decoder autoregressively generate the translation result. It should be noted that the decoder will also input predetermined information to obtain the translation result of the current word in the target language text corresponding to the document image, wherein the predetermined information is the translation results of all words translated before the current word.

[0098] The translation model will eventually generate translation results in markdown format, so a dedicated renderer can be used to render the markdown text into an image, i.e. the target document image, such as Figure 3 shown.

[0099] In summary, the present disclosure can translate any document image. Specifically, the pre-trained knowledge of the multimodal large model is used to train the translation model, and the pre-trained knowledge is transferred to the translation model in the form of hybrid modality and single modality alignment. In addition, the multimodal large model does not need to be involved in the reasoning stage, but only the translation model needs to be involved, thereby achieving a balance between reasoning speed and translation quality. Therefore, the present disclosure overcomes the problem of poor generalization of the existing end-to-end document image translation system due to the scarcity of training data, such as poor domain generalization, poor generalization of long text images, and poor generalization of complex layouts.

[0100] Figure 5 is a block diagram showing a translation device according to an embodiment of the present disclosure, Figure 5 As shown, the translation device is implemented based on a translation model, the translation model includes an image encoder, a semantic alignment encoder and a decoder, and the device includes a first processing unit 50, a second processing unit 52 and a third processing unit 54.

[0101] The first processing unit 50 is configured to input the source document image into the image encoder to obtain image coding features, wherein the source document image includes a source language text portion and an illustration portion; the second processing unit 52 is configured to input the source document image into the semantic alignment encoder to obtain unimodal semantic coding features; the third processing unit 54 is configured to input the image coding features and the unimodal semantic coding features into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text portion and an illustration portion; wherein the translation model is trained using the output of the multimodal large model.

[0102] According to an embodiment of the present disclosure, the translation device also includes a training unit, which is configured to train a translation model in the following manner: obtain multiple document image samples, and perform the following processing on each document image sample: input the current document image sample into an image encoder to obtain estimated image coding features; input the current document image sample into a semantic alignment encoder to obtain estimated unimodal semantic coding features; input the current document image sample and the source language text corresponding to the current document image sample into a multimodal large model to obtain mixed modal semantic coding features; obtain the cosine similarity of the estimated unimodal semantic coding features and the mixed modal semantic coding features; obtain the translation loss based on the estimated image coding features and the estimated unimodal semantic coding features; and adjust the parameters of the translation model by minimizing the cosine similarity and the translation loss.

[0103] According to an embodiment of the present disclosure, the training unit is further configured to input the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain the translation result of the t-th word in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all words translated before the t-th word, and t is a positive integer; the logarithm sum of the translation results of each word in the target language text corresponding to the current document image sample is determined as the translation loss.

[0104] According to an embodiment of the present disclosure, the training unit is also configured to input the current document image sample into the image encoder in the multimodal large model to obtain a first encoding feature; input the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain a second encoding feature; input the first encoding feature and the second encoding feature into multiple attention mechanism layers to obtain a mixed modal semantic encoding feature.

[0105] According to an embodiment of the present disclosure, the first processing unit 50 is further configured to scale the source document image to obtain an adjusted first image; input the first image into an image encoder to obtain image encoding features, wherein the image encoder adopts a sliding window attention mechanism.

[0106] According to an embodiment of the present disclosure, the second processing unit 52 is also configured to scale the source document image to obtain an adjusted second image; input the second image into the image encoder in the semantic alignment encoder to obtain a third encoding feature, wherein the image encoder adopts a sliding window attention mechanism; input the third encoding feature into the two-layer feedforward neural network in the semantic alignment encoder to obtain a unimodal semantic encoding feature.

[0107] According to an embodiment of the present disclosure, the third processing unit 54 is further configured to, after inputting the image coding features and the unimodal semantic coding features into the decoder to obtain the translation result of the source document image, render the target document image corresponding to the source document image based on the translation result.

[0108] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions is provided, wherein when the instructions are executed by at least one computing device, the at least one computing device is prompted to execute the translation method as described in any of the above embodiments.

[0109] According to an embodiment of the present disclosure, a system including at least one computing device and at least one storage device storing instructions is provided, wherein the instructions, when executed by the at least one computing device, prompt the at least one computing device to execute the translation method as any of the above embodiments.

[0110] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A translation method, characterized in that: The translation method is implemented based on a translation model, wherein the translation model includes an image encoder, a semantic alignment encoder and a decoder, and the translation method includes: Inputting a source document image into the image encoder to obtain image encoding features, wherein the source document image includes a source language text portion and an illustration portion; Inputting the source document image into the semantic alignment encoder to obtain a unimodal semantic encoding feature; Inputting the image coding feature and the unimodal semantic coding feature into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text portion and the illustration portion; The translation model is trained using the output of a large multimodal model.

2. The translation method according to claim 1, characterized in that: The translation model is trained as follows: Obtain multiple document image samples, and perform the following processing on each document image sample: Inputting the current document image sample into the image encoder to obtain estimated image coding features; Inputting the current document image sample into the semantic alignment encoder to obtain an estimated unimodal semantic encoding feature; Inputting the current document image sample and the source language text corresponding to the current document image sample into the multimodal large model to obtain a mixed modality semantic encoding feature; Obtaining the cosine similarity of the estimated unimodal semantic coding feature and the mixed modal semantic coding feature; Obtaining a translation loss based on the estimated image coding features and the estimated unimodal semantic coding features; Parameters of the translation model are adjusted by minimizing the cosine similarity and the translation loss.

3. The translation method according to claim 2, characterized in that: The obtaining of translation loss based on the estimated image coding feature and the estimated unimodal semantic coding feature includes: Inputting the estimated unimodal semantic coding features, the estimated image coding features, and predetermined information into the decoder to obtain a translation result of the t-th word in the target language text corresponding to the current document image sample, wherein the predetermined information is the translation results of all words translated before the t-th word, and t is a positive integer; The logarithm sum of the translation results of each word in the target language text corresponding to the current document image sample is determined as the translation loss.

4. The translation method according to claim 2, characterized in that: The step of inputting the current document image sample and the source language text corresponding to the current document image sample into the multimodal large model to obtain a mixed modality semantic coding feature includes: Inputting the current document image sample into an image encoder in the multimodal large model to obtain a first encoding feature; Inputting the source language text corresponding to the current document image sample into the text embedding layer in the multimodal large model to obtain a second encoding feature; The first encoding feature and the second encoding feature are input into multiple attention mechanism layers to obtain the mixed modality semantic encoding feature.

5. The translation method according to claim 1, characterized in that: The step of inputting the source document image into the image encoder to obtain image encoding features includes: Scaling the source document image to obtain an adjusted first image; The first image is input into the image encoder to obtain image encoding features, wherein the image encoder adopts a sliding window attention mechanism.

6. The translation method according to claim 1, characterized in that: The step of inputting the source document image into the semantic alignment encoder to obtain a unimodal semantic encoding feature comprises: Scaling the source document image to obtain an adjusted second image; Inputting the second image into an image encoder in the semantic alignment encoder to obtain a third encoding feature, wherein the image encoder adopts a sliding window attention mechanism; The third encoding feature is input into the two-layer feedforward neural network in the semantic alignment encoder to obtain the unimodal semantic encoding feature.

7. The translation method according to claim 1, characterized in that: After inputting the image coding feature and the unimodal semantic coding feature into the decoder to obtain the translation result of the source document image, the method further includes: Based on the translation result, a target document image corresponding to the source document image is rendered.

8. A translation device, characterized in that: The translation device is implemented based on a translation model, and the translation model includes an image encoder, a semantic alignment encoder and a decoder, including: A first processing unit is configured to input a source document image into the image encoder to obtain an image encoding feature, wherein the source document image includes a source language text portion and an illustration portion; A second processing unit is configured to input the source document image into the semantic alignment encoder to obtain a unimodal semantic encoding feature; a third processing unit, configured to input the image coding feature and the unimodal semantic coding feature into the decoder to obtain a translation result of the source document image, wherein the translation result includes a target language text part and the illustration part; The translation model is trained using the output of a large multimodal model.

9. A computer-readable storage medium storing instructions, characterized in that: When the instructions are executed by at least one computing device, the at least one computing device is prompted to perform the translation method according to any one of claims 1 to 7.

10. A system comprising at least one computing device and at least one storage device storing instructions, characterized in that: When the instructions are executed by the at least one computing device, the at least one computing device is prompted to perform the translation method according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the translation method according to any one of claims 1 to 7 is implemented.

Images