Training Methods for Text-Image Translation Models Based on Multimodal Codebooks

Abstract

This application proposes a training method for a text-image translation model based on a multimodal codebook. The method includes training a text encoder and a text decoder using the language text corresponding to the images in the training sample set and the target language text; training the multimodal codebook using the language text corresponding to the images in the training sample set; training the image encoder and the multimodal codebook using the images and the language text corresponding to the images; and fine-tuning the text-image translation model using the images, the language text corresponding to the images, the optical character recognition text, and the target language text to obtain a trained text-image translation model. Thus, by associating related text with the input images, effective supplementary information is provided for the translation process, resulting in better translation performance.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a training method for a text-image translation model based on a multimodal codebook and a computer device. Background Technology

[0002] In related technologies, text-image translation aims to translate source text embedded in an image into a target translation. Current text-image translation models mainly adopt a cascaded approach, that is, first use an optical character recognition model to recognize the text in the image, and then input the recognized text into the translation model for translation. However, this cascaded approach has the problem of error propagation, in which the recognized text may contain errors, leading to subsequent translation errors. In addition, the amount of data for text-image translation tasks is relatively small, which limits the performance of the model. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the aforementioned technologies. To this end, one objective of this application is to propose a training method for a text-image translation model based on a multimodal codebook. This model includes a multimodal codebook that uses the input image to associate relevant text, thereby providing effective supplementary information for the translation process and achieving better translation results.

[0004] The second objective of this application is to propose a computer device.

[0005] To achieve the above objectives, a first aspect of this application proposes a training method for a text-image translation model based on a multimodal codebook. The method includes the following steps: obtaining a training sample set, wherein the training sample set includes multiple training sample pairs, each training sample pair including an image, the corresponding language text, optical character recognition text, and target language text; constructing a text-image translation model, wherein the text-image translation model includes a text encoder, an image encoder, a multimodal codebook, and a text decoder; and training the text-image translation model using the training sample set to obtain a trained text-image translation model.

[0006] According to the training method of the text-image translation model based on a multimodal codebook according to the embodiments of this application, firstly, a training sample set is obtained, wherein the training sample set includes multiple training sample pairs, each training sample pair including an image, the corresponding language text of the image, optical character recognition text, and target language text; a text-image translation model is constructed, wherein the text-image translation model includes a text encoder, an image encoder, a multimodal codebook, and a text decoder; the text-image translation model is trained using the training sample set to obtain a trained text-image translation model; thereby, the input image is associated with related text, thus providing effective supplementary information for the translation process, so as to obtain better translation results.

[0007] In addition, the training method for the text-image translation model based on a multimodal codebook proposed in the above embodiments of this application may also have the following additional technical features:

[0008] Optionally, training the text-image translation model using the training sample set to obtain a trained text-image translation model includes: training the text encoder and the text decoder using the language text corresponding to the images and the target language text in the training sample set; training the multimodal codebook using the language text corresponding to the images in the training sample set; training the image encoder and the multimodal codebook using the images and the language text corresponding to the images; and fine-tuning the text-image translation model using the images, the language text corresponding to the images, the optical character recognition text, and the target language text to obtain a trained text-image translation model.

[0009] Optionally, the text encoder is built on top of a Transformer encoder, stacking L... e The image encoder consists of three identical layers, each comprising a self-attention sublayer and a feedforward network sublayer; the image encoder is constructed using ViT and includes L... v The multimodal codebook comprises K code elements, which quantize the hidden state sequences of the text encoder and the visual feature sequences of the image encoder. The text decoder is constructed based on a Transformer decoder and has L... d The system comprises three identical layers, including a self-attention sublayer, a feedforward network sublayer, and a cross-attention sublayer, to generate the final target language text based on the hidden state sequence output by the text encoder and the code elements output by the multimodal codebook.

[0010] Optionally, the hidden state sequence of the text encoder and the visual feature sequence of the image encoder are quantized using the following formula:

[0011]

[0012] Among them, e k This represents the k-th symbol. Indicates the lth generation of the text encoder e The i-th hidden state in the hidden state sequence output by the layer. Represents the Lth generation of the image encoder. v The j-th feature in the visual feature sequence output by the layer, e k′ express Mapped to the embedding representation of the k′-th symbol in the multimodal codebook, e k"" express It is mapped to the embedding representation of the k″-th symbol in the multimodal codebook.

[0013] Optionally, the probability distribution for generating the next word in the text decoder at the t-th decoding time step is defined as follows:

[0014]

[0015] in, Indicates the Lth d The hidden state sequence output by the layer text decoder, where v is the input image. The input is optical character recognition text, y is the target language text, θ represents the parameters of the entire text image translation model, and W... o and b o These are the trainable model parameters.

[0016] Optionally, the text encoder and the text decoder are trained using the following training objectives:

[0017]

[0018] Where x is the language text corresponding to the image, and θ te θ is a parameter of the text encoder. td These are the parameters for the text decoder.

[0019] Optionally, when training the image encoder and the multimodal codebook using the image and the corresponding language text, the multimodal codebook is updated using an exponential moving average method, and the decay factor determines the degree to which past values ​​affect the current average value.

[0020] Optionally, the image encoder and the multimodal codebook are trained using the following training objectives:

[0021]

[0022] Where sg represents the gradient cutoff operation, θ ie These are the parameters in the image encoder besides the ViT module. Indicates the Lth v The visual feature sequence output by the layer image encoder. Indicates the Lth e The hidden state sequence output by the layer text encoder The calculation method is as follows The calculation method is as follows Representing the semantic information of images and text respectively, through The loss term, which quantizes image and text representations into the same code as much as possible, uses commitment loss. To ensure that the latent state output by the text encoder and the visual features output by the image encoder remain close to the selected symbol embedding representation, and to prevent frequent fluctuations between symbols during mapping, α is the control... Hyperparameters of the loss effect.

[0023] Optionally, the entire text-image translation model can be optimized using the following training objectives:

[0024]

[0025] in, The commitment loss is designed for the text encoder, where β is a hyperparameter quantifying its effectiveness. This represents the text-image translation loss.

[0026] To achieve the above objectives, a third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method for a text-image translation model based on a multimodal codebook as described above.

[0027] According to the computer device of the present application embodiment, a training program for a text-image translation model based on a multimodal codebook is stored in a memory. When the training program for the text-image translation model based on a multimodal codebook is executed by the processor, the above-described training method for the text-image translation model based on a multimodal codebook is implemented. As a result, when the text-image translation model generates a translation, it can utilize images to obtain effective supplementary information and improve the translation quality. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the training method of a text-image translation model based on a multimodal codebook according to an embodiment of this application.

[0029] Figure 2 This is a schematic diagram of the overall network structure of the text-image translation model according to an embodiment of this application;

[0030] Figure 3 This is a schematic diagram of the overall training framework according to an embodiment of this application. Detailed Implementation

[0031] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0032] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0033] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0034] Figure 1 This is a flowchart illustrating the training method of a text-image translation model based on a multimodal codebook according to an embodiment of this application, as shown below. Figure 1 As shown, the training method for the text-image translation model based on a multimodal codebook in this application includes the following steps:

[0035] S101, Obtain the training sample set, wherein the training sample set includes multiple training sample pairs, and each training sample pair includes an image, the corresponding language text of the image, optical character recognition text, and target language text.

[0036] It should be noted that the image contains the language text to be translated. The language text corresponding to the image is the text composed of the characters on the image. The optical character recognition text is the text obtained after recognizing the characters on the image using OCR recognition. The target language text is the text translated based on the language text corresponding to the image.

[0037] S102, Construct a text-image translation model, which includes a text encoder, an image encoder, a multimodal codebook, and a text decoder.

[0038] In other words, such as Figure 2As shown, the constructed text-image translation model includes four modules: text encoder, image encoder, multimodal codebook, and text decoder.

[0039] As one example, the text encoder is built on top of the Transformer encoder, stacking L... e Each of the three identical layers consists of a self-attention sublayer and a feedforward sublayer; using Let N represent the hidden state sequence output by the l-th layer text encoder, where N is the hidden state sequence output by the l-th layer text encoder. e is the length of the hidden state sequence.

[0040] As a specific embodiment, this application adopts a 6-layer text encoder structure.

[0041] As an example, the image encoder is built using ViT, which, similar to the Transformer encoder, includes L v Each stacked layer consists of a self-attention sublayer and a feedforward network sublayer. A key difference between the Transformer encoder and ViT is the placement of layer normalization, where ViT employs a pre-normalization method. Let N represent the visual feature sequence output by the l-th layer image encoder, where N is the number of layers in the image encoder. v is the length of the visual feature sequence.

[0042] As a specific embodiment, this application adopts a 12-layer image encoder structure, and adds a linear transformation sublayer and a cross-attention sublayer at the top layer to transform the visual feature sequence dimension to be consistent with the hidden state sequence dimension of the text encoder.

[0043] As an example, the multimodal codebook includes K code elements, which can be used to quantize the hidden state sequence of the text encoder and the visual feature sequence of the image encoder.

[0044] As a specific embodiment, the hidden state sequence of the text encoder and the visual feature sequence of the image encoder are quantized using the following formula:

[0045]

[0046] Among them, e k This represents the k-th symbol. Indicates the Lth level of the text encoder e The i-th hidden state in the hidden state sequence output by the layer. Represents the Lth generation of the image encoder. v The j-th feature in the visual feature sequence output by the layer, e k′ express Mapped to the embedding representation of the k′-th symbol in the multimodal codebook, e k″ express It is mapped to the embedding representation of the k″-th symbol in the multimodal codebook.

[0047] It should be noted that, in the above manner, both text and image representations are located in the shared semantic space of the mapped symbols.

[0048] As an example, the text decoder is built on top of the Transformer decoder, having L d The system consists of three identical layers, including a self-attention sublayer, a feedforward network sublayer, and a cross-attention sublayer, to generate the final target language text based on the hidden state sequence output by the text encoder and the code elements output by the multimodal codebook.

[0049] As a specific embodiment, using Let N represent the hidden state sequence output by the l-th layer text decoder, where N is the hidden state sequence output by the l-th layer text decoder. d Let be the length of the hidden state sequence; at the t-th decoding time step, the probability distribution for generating the next word is defined as follows:

[0050]

[0051] in, This indicates that at the t-th decoding time step, the L-th... d The hidden state sequence output by the layer text decoder, where v is the input image. The input is optical character recognition text, y is the target language text, θ represents the parameters of the entire text image translation model, and W... o and b o These are the trainable model parameters.

[0052] In conclusion, the text-image translation model of this application includes: (1) a text encoder, which converts the input text into a state sequence; (2) an image encoder, which initializes the input image using a pre-trained model and encodes the input image into a visual vector sequence; (3) a multimodal codebook, which is used to represent the input image and then outputs code elements containing the correct text or related text information; and (4) a text decoder, which uses the hidden state output by the text encoder and the code elements output by the multimodal codebook to generate the final translation. The text-image translation model of this application can use the input image to predict code elements related to the input text through the multimodal codebook, providing supplementary information for subsequent translation and mitigating the error propagation problem.

[0053] S103, the text-image translation model is trained using the training sample set to obtain a trained text-image translation model.

[0054] As one embodiment, the text-image translation model is trained using a training sample set to obtain a trained text-image translation model, including: training the text encoder and text decoder using the language text corresponding to the images and the target language text in the training sample set; training the multimodal codebook using the language text corresponding to the images in the training sample set; training the image encoder and the multimodal codebook using the images and the language text corresponding to the images; and fine-tuning the text-image translation model using the images, the language text corresponding to the images, optical character recognition text, and the target language text to obtain a trained text-image translation model.

[0055] In other words, such as Figure 3 As shown, the text-image translation model includes four stages during training.

[0056] Among them, such as Figure 3 (a) illustrates the first stage of this application, in which the text encoder and text decoder are pre-trained on a large-scale bilingual corpus in a plain translation manner. Formally, for each parallel sentence pair (x, y), this application defines the following training objective for this stage:

[0057]

[0058] Where x is the language text corresponding to the image, and θ te θ is a parameter of the text encoder. td These are the parameters for the text decoder.

[0059] In addition, such as Figure 3 (b) illustrates the second stage of this application, in which a multimodal codebook is pre-trained using monolingual data, wherein the monolingual data is taken from the source language portion of the bilingual data in the first stage; through training in this stage, a clustered representation is learned for each m symbol of the multimodal codebook; and the multimodal codebook is updated using an exponential moving average method, wherein a decay factor determines the degree to which past values ​​influence the current average. Formally, the embedding representation of the k-th symbol is updated as follows:

[0060]

[0061]

[0062] n k ←γn k +(1-γ)c k ,

[0063]

[0064] in, It is an indicator function, where γ is a decay factor set to 0.99; ck Calculate the number of hidden states of the text encoder clustered into the k-th symbol, h. k n represents the sum of these hidden states. k c represents past and current values. k The exponentially weighted average.

[0065] In addition, such as Figure 3 The dashed box in (c) shows the third stage of this application, in which an additional optical character recognition data and image-text alignment task are introduced to further train the image encoder and multimodal codebook. Through training in this stage, the multimodal codebook is given the initial ability to associate related text with images. Given an image-text training instance (v, x), the training objective of this stage is defined as follows:

[0066]

[0067] Where sg represents the gradient cutoff operation, θ ie These are the parameters in the image encoder besides the ViT module. Indicates the Lth v The visual feature sequence output by the layer image encoder. Indicates the Lth e The hidden state sequence output by the layer text encoder The calculation method is as follows The calculation method is as follows Representing the semantic information of images and text respectively, through The loss term, which quantizes image and text representations into the same code as much as possible, uses commitment loss. To ensure that the latent state output by the text encoder and the visual features output by the image encoder remain close to the selected symbol embedding representation, and to prevent frequent fluctuations between symbols during mapping, α is the control... The hyperparameters of the loss effect; in this stage, the multimodal codebook is updated in the same way as in the second stage.

[0068] In addition, such as Figure 3 (c) shows the training process in the fourth stage of this method, in which... The loss term still participates, maintaining training consistency and making fine-tuning smoother; given a text-image translation training instance. The entire model is optimized with the following objectives:

[0069]

[0070] in, The commitment loss is designed for the text encoder, where β is a hyperparameter quantifying its effectiveness. This represents the text-image translation loss.

[0071] It is important to note that Just as The input is used to ensure consistency between model training and inference, while x is used as the input for the image-text alignment task to train the multimodal codebook's ability to associate the input image with the correct text. In addition, the multimodal codebook is still updated using the exponential moving average method. Through this stage of training, the final text-image translation model based on the multimodal codebook is obtained, which can be used for model testing and practical applications.

[0072] In other words, the training process of the text-image translation model of this application includes: (1) pre-training the text encoder and text decoder using a large-scale bilingual corpus through a common text translation task; (2) pre-training the newly added multimodal codebook using monolingual data from the large-scale bilingual corpus through an exponential average update method; (3) pre-training the image encoder and multimodal codebook using an additional optical character recognition dataset through an image-text alignment task to enhance the ability of the multimodal codebook to associate images with relevant text; (4) fine-tuning all modules on the text-image translation dataset to obtain the final text-image translation model. After training, the text-image translation model can translate the target language text by inputting an image and recognizing the text using optical characters.

[0073] In conclusion, the training method for the text-image translation model based on a multimodal codebook proposed in this application includes a multimodal codebook module to associate relevant text with the input image, thereby providing effective supplementary information for the translation process. The multimodal codebook includes a fixed number of code elements, each of which is a clustering representation. During training, the model maps the image and the corresponding correct text to the semantic space of the same code elements. During inference, the model maps the image to code elements containing relevant text information through the multimodal codebook, thereby providing supplementary information for translation and obtaining better translation results.

[0074] In addition, this application also proposes a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the training method of the text image translation model based on the multimodal codebook as described above.

[0075] According to the computer device of the present application embodiment, a training program for a text-image translation model based on a multimodal codebook is stored in a memory. When the training program for the text-image translation model based on the multimodal codebook is executed by the processor, the above-described training method for the text-image translation model based on the multimodal codebook is implemented. As a result, when the text-image translation model generates a translation, it can utilize images to obtain effective supplementary information, thereby improving the translation quality.

[0076] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0081] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0082] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0083] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0084] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0085] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0087] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A training method for a text-image translation model based on a multimodal codebook, characterized in that, Includes the following steps: Obtain a training sample set, wherein the training sample set includes multiple training sample pairs, and each training sample pair includes an image, the corresponding language text of the image, optical character recognition text, and target language text; Construct a text-image translation model, wherein the text-image translation model includes a text encoder, an image encoder, a multimodal codebook, and a text decoder; The text-image translation model is trained using the training sample set to obtain a trained text-image translation model; The text encoder is built based on the Transformer encoder and stacks L... e The image encoder consists of three identical layers, each comprising a self-attention sublayer and a feedforward network sublayer; the image encoder is constructed using ViT and includes L... v The multimodal codebook comprises K code elements, which quantize the hidden state sequences of the text encoder and the visual feature sequences of the image encoder. The text decoder is constructed based on a Transformer decoder and has L... d The same layer includes a self-attention sublayer, a feedforward network sublayer, and a cross-attention sublayer, so as to generate the final target language text based on the hidden state sequence output by the text encoder and the code elements output by the multimodal codebook; The hidden state sequence of the text encoder and the visual feature sequence of the image encoder are quantized using the following formula: Among them, e k This represents the k-th symbol. Indicates the Lth level of the text encoder e The i-th hidden state in the hidden state sequence output by the layer. Represents the Lth generation of the image encoder. v The j-th feature in the visual feature sequence output by the layer, e k′ express Mapped to the embedding representation of the k′-th symbol in the multimodal codebook, e k″ express It is mapped to the embedding representation of the k″-th symbol in the multimodal codebook.

2. The training method for the text-image translation model based on a multimodal codebook as described in claim 1, characterized in that, The text-image translation model is trained using the training sample set to obtain a trained text-image translation model, including: The text encoder and the text decoder are trained using the language text and target language text corresponding to the images in the training sample set; The multimodal codebook is trained using the language text corresponding to the images in the training sample set; The image encoder and the multimodal codebook are trained using the image and the corresponding language text. The text image translation model is fine-tuned using the image, the corresponding language text, the optical character recognition text, and the target language text to obtain a trained text image translation model.

3. The training method for the text-image translation model based on a multimodal codebook as described in claim 2, characterized in that, The probability distribution for generating the next word in the text decoder at the T-th decoding time step is defined as follows: in, Indicates the Lth d The hidden state sequence output by the layer text decoder, where v is the input image. The input is optical character recognition text, y is the target language text, θ represents the parameters of the entire text image translation model, and W... o and b o These are the trainable model parameters.

4. The training method for the text-image translation model based on a multimodal codebook as described in claim 3, characterized in that, The text encoder and text decoder are trained using the following training objectives: Where x is the language text corresponding to the image, and θ te θ is a parameter of the text encoder. td These are the parameters for the text decoder.

5. The training method for the text-image translation model based on a multimodal codebook as described in claim 4, characterized in that, When training the image encoder and the multimodal codebook using the image and the corresponding language text, the multimodal codebook is updated using the exponential moving average method, and the decay factor determines the degree to which past values ​​affect the current average value.

6. The training method for the text-image translation model based on a multimodal codebook as described in claim 5, characterized in that, The image encoder and the multimodal codebook are trained using the following training objectives: Where sg represents the gradient cutoff operation, θ ie These are the parameters in the image encoder besides the ViT module. Indicates the Lth v The visual feature sequence output by the layer image encoder. Indicates the Lth e The hidden state sequence output by the layer text encoder The calculation method is as follows The calculation method is as follows Representing the semantic information of images and text respectively, through The loss term, which quantizes image and text representations into the same code as much as possible, uses commitment loss. To ensure that the latent state output by the text encoder and the visual features output by the image encoder remain close to the selected symbol embedding representation, and to prevent frequent fluctuations between symbols during mapping, α is the control... Hyperparameters of the loss effect.

7. The training method for the text-image translation model based on a multimodal codebook as described in claim 6, characterized in that, The following training objectives are used to optimize the entire text-image translation model: in, The commitment loss is designed for the text encoder, where β is a hyperparameter quantifying its effectiveness. This represents the text-image translation loss.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the training method for the text image translation model based on a multimodal codebook as described in any one of claims 1-7.

Images