A Multimodal Machine Translation Training Method Combining Knowledge Graph, Large Language Model and Visual Imagination Mechanism

Abstract

A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism, belonging to the field of artificial intelligence technology. To optimize the image generation quality of the multi-modal machine translation method, the present invention includes manually setting the sentences to be translated to construct Training Set 1; applying a text processing library to process the sentences to be translated in Training Set 1 to obtain text triples, inputting the sentences to be translated in Training Set 1 into an image generation model to generate pictures, and then using an image triple extraction model to process them to obtain image triples. Training the image generation model using the distance properties of the two sets of triples to obtain a trained image generation model; inputting the sentences to be translated in Training Set 1 into the trained image generation model to perform encoding processing on the images corresponding to the sentences to be translated, and then splicing them with the corresponding sentences to be translated to construct Training Set 2, and using Training Set 2 to train the large language model to optimize the multi-modal machine translation performance of the large language model.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism. Background Art

[0002] Large language models are large-scale, pre-trained statistical language models based on neural networks. They are usually pre-trained on large-scale corpora, containing hundreds to hundreds of billions of parameters. Compared with traditional language models, they are larger in scale, have stronger language understanding and generation abilities, and also have emergent abilities that small-scale language models do not possess. For this reason, they have also become the base models for many natural language processing methods.

[0003] A knowledge graph is a way of knowledge representation in natural language processing tasks, which can represent a large amount of knowledge as a graph structure. A knowledge graph consists of triples, and the structure of a triple is (h, r, t), where h, r, and t represent the head entity, relationship, and tail entity respectively. Compared with natural language, the triple structure has the characteristics of simplicity and high knowledge density. In addition, the distance between different triples can also be compared through the embedding technology of triples.

[0004] The visual imagination mechanism refers to such a phenomenon: that is, during the reasoning process of a large language model, generating pictures related to the question through a picture generation model can improve the efficiency and accuracy of the large language model.

[0005] Multi-modal machine translation technology refers to machine translation technology that combines non-text modalities such as pictures and voices.

[0006] Extracting triples from natural language is a very mature technology, which can be processed by relying on common natural language processing toolkits; recently, the technology of extracting triples from pictures has also become the focus of research.

[0007] However, existing multi-modal machine translation methods cannot generate high-quality corresponding pictures according to the sentence to be translated, thereby improving the translation quality. Therefore, it is very meaningful to construct a model that can generate high-quality pictures according to the sentence to be translated through existing technologies. Summary of the Invention

[0008] The problem to be solved by the present invention is to optimize the image generation quality of multi-modal machine translation methods, and propose a multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism.

[0009] To achieve the above object, the present invention is realized through the following technical solutions:

[0010] A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism, comprising the following steps:

[0011] S1. Manually set the sentences to be translated and construct training set 1;

[0012] S2. For the training set 1 obtained in step S1, apply a text processing library to process the sentences to be translated in training set 1 to obtain text triples. After inputting the sentences to be translated in training set 1 into an image generation model to generate pictures, use an image triple extraction model to process them to obtain image triples. Then, apply the distance properties of the two sets of triples to train the image generation model to obtain a trained image generation model;

[0013] S3. Input the sentences to be translated in training set 1 obtained in step S1 into the trained image generation model in step S2 to obtain the generated images corresponding to the sentences to be translated;

[0014] S4. Encode the generated images corresponding to the sentences to be translated obtained in step S3, and then splice them with the corresponding sentences to be translated to construct training set 2. Use training set 2 to train the large language model to optimize the multi-modal machine translation performance of the large language model.

[0015] Further, the specific implementation method of step S2 includes the following steps:

[0016] S2.1. Apply a text processing library to process the sentence to be translated x in training set 1, iterate through each data in training set 1, extract the verbs and nouns in the sentence to be translated according to the grammatical relationship, process the sentence to be translated language, and extract a set of text triples LSG=(h l1 , r l1 , t l1 ), (h l2 , r l2 , t l2 ), …, (h ln , r ln , t ln ), where h ln is the nth head entity in the set of text triples, r ln is the nth relation in the set of text triples, and t ln is the nth tail entity in the set of text triples;

[0017] S2.2. Input the sentences to be translated in training set 1 into an image generation model to generate pictures. The image generation model is the Stable Diffusion model, and automatically generate the corresponding pictures I of the sentences to be translated;

[0018] S2.3. Input the image obtained in step S2.2 into the image triple extraction model to automatically obtain the image triple set VSG = (h v1 , r v1 , t v1 ), (h v2 , r v2 , t v2 ), …, (h vm , r vm , t vm ), where h vm is the m-th head entity in the image triple set, r vm is the m-th relation in the image triple set, and t vm is the m-th tail entity in the image triple set;

[0019] S2.4. Calculate the distance between the members of the text triple set obtained in step S2.1 and the image triple set obtained in step S2.4 as the image generation score Sim(x, I) of the data. The specific formula is:

[0020]

[0021] Among them, Score is the scoring function for the similarity between the members of the text triple set and the image triple set;

[0022] The specific calculation method of Score is:

[0023] Score(LSG i , VSG) = max(D(LSG i , VSG1), …, D(LSG i , VSG m ))

[0024] Among them, D is the triple similarity function,

[0025] The specific calculation method of D is:

[0026]

[0027] Among them, SIM is the function for calculating the distance between natural language words;

[0028] S2.5. Use the image generation score obtained in step S2.4 as the performance score of the model, and apply the DDPO training method to optimize the image generation model. The optimization formula is:

[0029]

[0030] Among them, θ is the parameter of the Stable Diffusion model, J DDPO(θ) is the optimization objective of the Stable Diffusion model parameters, and r(x, I) is a manually set reward function used to quantify the matching degree between the sentence x to be translated and the corresponding image I. It means the expected value of the likelihood estimation of the reward function when the sentence x to be translated satisfies the probability distribution p θ (x|c) of the given parameters θ and the given text condition c, and the image I satisfies the probability distribution p(I) of the image itself.

[0031] Furthermore, the text processing library applied in step S2.1 is the SpaCY natural language text processing library or the NLTK natural language toolkit.

[0032] Furthermore, the image generation model is the Stable Diffusion model.

[0033] Furthermore, the image triple extraction model in step S2.3 is the Scene-Graph-Benchmark model or the Relationformer model.

[0034] Furthermore, the calculation method of SIM in step S2.4 is to first apply a large language model for encoding and then apply cosine similarity for calculation. The specific formula is:

[0035]

[0036] where Encode represents the process of automatically encoding words using a large language model, and A and B are two natural language words to be calculated.

[0037] Furthermore, the specific implementation method of step S4 includes the following steps:

[0038] S4.1. Encode the image I′ corresponding to the sentence to be translated obtained in step S3 to obtain the image encoding C(I′). The expression for the processing of the encoder is:

[0039] C(I′) = W′I′ + b′

[0040] where W′ and b′ are the first weight matrix and the second weight matrix respectively, and the encoder selects the CLIP model;

[0041] S4.2. Subsequently, apply the following formula to the obtained image encoding C(I′) to obtain the final encoding C′(I′) of the image:

[0042] C′(I′) = WC(I′) + b

[0043] where W and b are the third weight matrix and the fourth weight matrix respectively;

[0044] S4.3. Concatenate the final compilation of the image to the front of the sentence to be translated to construct Training Set 2, which is used as the input for training the large language model to obtain the final machine translation result;

[0045] S4.4. Calculate the loss between the final machine translation result w′ and the sentence w after manual translation to obtain the translation loss Loss LM The expression for this is:

[0046]

[0047] where C is the large language model parameter;

[0048] Then calculate the overall loss Loss of the large language model. The expression is:

[0049] Loss = Loss LM + Sim(x, I).

[0050] Furthermore, in step S4.3, the large language model is the Llama2 model or the Llama3 model or the Vicuna model further trained based on the Llama2 model. The large language model used is modeled as:

[0051]

[0052] where t represents the current time step, and j t represents the t-th word in the discourse segment, which is generated based on the large model step by step in time. One word is generated in one time step. Then p(w′) represents the probability of generating w′, and p(j t |j <t ) represents the probability of generating the t-th word after generating the previous t - 1 words. T is the total time.

[0053] Advantages of the present invention:

[0054] A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to the present invention focuses on using the triple structure of the knowledge graph in the training of the image generation module. Compared with existing models that use the imagination mechanism for large language model reasoning, during the training process of the image generation module, the present invention can utilize the triple structure to significantly improve the quality and efficiency of image generation; during the performance optimization training process of the large language model, through the combination of loss functions, the large language model can adapt to the changes in the image generation module, thereby optimizing the machine translation performance of the large language model. Description of the Drawings

[0055] Figure 1Flowchart of a multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to the present invention;

[0056] Figure 2 Training flowchart of the image generation module of the present invention;

[0057] Figure 3 Flow of the optimized training of the large language model of the present invention. Detailed implementation manners

[0058] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners. It should be understood that the specific implementation manners described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific implementation manners described are only a part of the implementation manners of the present invention, rather than all the specific implementation manners. Usually, the components of the specific implementation manners of the present invention described and shown in the drawings here can be arranged and designed in various different configurations, and the present invention can also have other implementation manners.

[0059] Therefore, the detailed description of the specific implementation manners of the present invention provided in the drawings below is not intended to limit the scope of the claimed present invention, but only represents the selected specific implementation manners of the present invention. All other specific implementation manners obtained by those skilled in the art based on the specific implementation manners of the present invention without creative efforts belong to the scope of protection of the present invention.

[0060] To further understand the content, features and effects of the present invention, the following specific implementation manners are exemplified and are accompanied by the attached Figure 1 - Attached Figure 3 The details are as follows:

[0061] Example 1:

[0062] A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism includes the following steps:

[0063] S1. Manually set the sentences to be translated and construct training set 1;

[0064] S2. For the training set 1 obtained in step S1, use a text processing library to process the sentences to be translated in the training set 1 to obtain text triples, input the sentences to be translated in the training set 1 into an image generation model to generate pictures, and then use an image triple extraction model to process to obtain image triples. Then, use the distance properties of the two groups of triples to train the image generation model to obtain a trained image generation model;

[0065] Further, the specific implementation method of step S2 includes the following steps:

[0066] S2.1. Process the sentence x to be translated in Training Set 1 using a word processing library. Iterate through each data in Training Set 1, extract the verbs and nouns in the sentence of the language to be translated according to the grammatical relationship, process the sentence of the language to be translated, and extract the set of literal triples LSG=(h l1 , r l1 , t l1 ), (h l2 , r l2 , t l2 ), …, (h ln , r ln , t ln ), where h ln is the nth head entity in the set of literal triples, r ln is the nth relationship in the set of literal triples, and t ln is the nth tail entity in the set of literal triples;

[0067] Further, the word processing library used in step S2.1 is the SpaCY natural language text processing library or the NLTK natural language toolkit;

[0068] S2.2. Input the sentence to be translated in Training Set 1 into an image generation model to generate a picture. The image generation model is the Stable Diffusion model, and automatically generate the corresponding picture I of the sentence to be translated;

[0069] Further, the image generation model is the Stable Diffusion model;

[0070] S2.3. Input the picture I obtained in step S2.2 into an image triple extraction model, and automatically obtain the set of image triples VSG=(h v1 , r v1 , t v1 ), (h v2 , r v2 , t v2 ), …, (h vm , r vm , t vm ), where h vm is the mth head entity in the set of image triples, r vm is the mth relationship in the set of image triples, and t vm is the mth tail entity in the set of image triples;

[0071] Further, the image triple extraction model in step S2.3 is the Scene-Graph-Benchmark model or the Relationformer model;

[0072] S2.4. Calculate the distance between the members of the text triple set obtained in step S2.1 and the image triple set obtained in step S2.4 as the image generation score Sim(x, I) of the data. The specific formula is:

[0073]

[0074] Among them, Score is the scoring function for the similarity between the members of the text triple set and the image triple set;

[0075] The specific calculation method of Score is:

[0076] Score(LSG i , VSG) = max(D(LSG i , VSG1), …, D(LSG i , VSG m ))

[0077] Among them, D is the triple similarity function,

[0078] The specific calculation method of D is:

[0079]

[0080] Among them, SIM is the function for calculating the distance between natural language words;

[0081] Furthermore, the calculation method of SIM in step S2.4 is to first apply the large language model for encoding and then apply cosine similarity for calculation. The specific formula is:

[0082]

[0083] Among them, Encode represents the process of automatically encoding words using the large language model, and A and B are two natural language words to be calculated;

[0084] S2.5. Use the image generation score obtained in step S2.4 as the performance score of the model, and apply the DDPO training method to optimize the image generation model. The optimization formula is:

[0085]

[0086] Among them, θ is the parameter of the Stable Diffusion model, J DDPO (θ) is the optimization objective of the Stable Diffusion model parameters, and r(x, I) is the reward function set manually, which is used to quantify the matching degree between the sentence x to be translated and the corresponding picture I. It means that when the sentence x to be translated satisfies the probability distribution p of the given parameter θ and the given text condition cθ (x|c), and the expected likelihood estimate of the reward function when the image I satisfies the probability distribution p(I) inherent to the image.

[0087] Furthermore, the meaning of the above formula is to optimize the likelihood estimate of the reward function under the condition of the given text and the generated image. In this formula, the given text and the generated image are the above-mentioned sentence to be translated and the image that is consistent with the description of the sentence to be translated.

[0088] S3. Input the sentence to be translated in the training set 1 obtained in step S1 into the image generation model trained in step S2 to obtain the generated image corresponding to the sentence to be translated;

[0089] S4. Encode the generated image corresponding to the sentence to be translated obtained in step S3, and then splice it with the corresponding sentence to be translated to construct training set 2, and use training set 2 to train the large language model to optimize the multi-modal machine translation performance of the large language model.

[0090] Furthermore, the specific implementation method of step S4 includes the following steps:

[0091] S4.1. Encode the generated image I′ corresponding to the sentence to be translated obtained in step S3 to obtain the image encoding C(I′), and the expression for the processing of the encoder is:

[0092] C(I′) = W′I′ + b′

[0093] where W′ and b′ are the first weight matrix and the second weight matrix respectively, and the encoder selects the CLIP model;

[0094] S4.2. Subsequently, apply the following formula to the obtained image encoding C(I′) to obtain the final encoding C′(I′) of the image:

[0095] C′(I′) = WC(I′) + b

[0096] where W and b are the third weight matrix and the fourth weight matrix respectively;

[0097] S4.3. Splice the final encoding of the image in front of the sentence to be translated to construct training set 2, and use it as the input for training the large language model to obtain the final machine translation result;

[0098] Furthermore, in step S4.3, the large language model is the Llama2 model or the Llama3 model or the Vicuna model further trained on the basis of the Llama2 model, and the large language model used is modeled as:

[0099]

[0100] Among them, t represents the current time step, jt represents the t-th word of the segment, which is generated by the large model step by step in time. One word is generated in one time step. Then p(w′) represents the probability of generating w′, and p(j t |j ＜t ) represents the probability of generating the t-th word after generating the previous t - 1 words. T is the total time.

[0101] S4.4. Calculate the loss between the final machine translation result w′ and the sentence w after manual translation to obtain the translation loss Loss LM The expression of which is:

[0102]

[0103] Among them, C is the parameter of the large language model;

[0104] Then calculate the overall loss Loss of the large language model, and the expression is:

[0105] Loss = Loss LM + Sim(x, I).

[0106] Example 2:

[0107] The actual operation using the method of Example 1 is as follows: The large model used is the Vicuna model, the image encoding module uses the CLIP model, and the image triple extraction method used is Scene - Graph - Benchmark.

[0108] First, improve the training explanation of the image generation module. Assume that there is only one sentence in the given dataset (Three women are standing under a white wall). First, rely on SpaCY to extract the triples in this sentence to obtain two text triples: (Three women, standing under, the wall) and (The wall, is, white), and then save them in the hardware for later use. Subsequently, input the sentence "Three women are standing under a white wall" into the Stable Diffusion model to generate a picture corresponding to the description. Subsequently, rely on the image triple extraction module Scene - Graph - Benchmark to process the obtained picture to get about 100 triples, such as: (Woman, standing under, the wall), (Woman, running, )... Subsequently, read the previously extracted text triples, and apply the above - mentioned formula to calculate their scores with the picture triples. The score obtained in the example is 0.235. Subsequently, rely on the DDPO training method to optimize the performance of the Stable Diffusion model.

[0109] Secondly, improve the explanation of the process for optimizing the training of the Vicuna model. Assume that there is only one sentence in the given dataset: "There is an apple on the table." First, rely on SpaCY to extract the triples in the sentence "There is an apple on the table", obtaining the text triple "(an apple, on, the table)", and then save it in the hardware for later use. Subsequently, input the sentence "There is an apple on the table" into the Stable Diffusion model to generate a picture corresponding to the description. Subsequently, rely on the image triple extraction module Scene-Graph-Benchmark to process the obtained picture, obtaining approximately 100 triples, such as: "(the table, is, brown)", "(the apple, is, red)",... Subsequently, read the previously extracted text triple, apply the formula mentioned above to calculate its score with the picture triples, and the score obtained in the embodiment is 0.107. Subsequently, rely on the CLIP model to encode the generated picture, and then apply to splice the encoding in front of the sentence to be translated and input it into the Vicuna model for inference, obtaining the translation "An apple is on the table." Rely on the Loss LM formula to calculate its difference from "There is an apple on the table.", which is 0.605 in this example. Subsequently, add the 0.107 and 0.605 calculated above as the overall loss to optimize the performance of the Vicuna model.

[0110] Example 3:

[0111] The actual operation using the method of Example 1 is as follows: The large model used is the Llama2 model, the image encoding module uses the CLIP model, and the image triple extraction method used is Relationformer.

[0112] First, improve the training interpretation of the image generation module. Assume that there is only one sentence in the given data set (three women standing under a white wall). First, rely on the NLTK library to extract the triples in this sentence, and obtain three text triples (woman, standing, under the wall), (wall, is, white), and (woman, is, three), and then save them in the hardware for standby. Then, input the sentence "three women standing under the white wall" into the Stable Diffusion model to generate a picture corresponding to the description. Then, rely on the image triple extraction module Relationformer to process the obtained picture and obtain about 100 triples, such as: (woman, standing, under the wall), (woman, is, running)... Then, read the previously extracted text triples, and apply the above-mentioned formula to calculate its score with the picture triples. The score obtained in the embodiment is 0.105. Then, rely on the DDPO training method to optimize the performance of the Stable Diffusion model.

[0113] Secondly, the explanation of the Llama2 model optimization training process is improved. Assume that there is only one sentence in the given data set (There is an apple on the table). First, rely on NLTK to extract the triples in the sentence "There is an apple on the table", obtaining two text triples (apple, on, table) and (apple, is, one), which are then stored in the hardware for future use. Subsequently, the sentence "There is an apple on the table" is input into the StableDiffusion model to generate a corresponding image description. Subsequently, the image is processed by the image triple extraction module Scene-Graph-Benchmark to obtain approximately 100 triples, such as: (table, is, brown), (apple, is, red) ... Then, read the previously extracted text triples and use the above-mentioned formula to calculate their score with the image triples. The score obtained in the embodiment is 0.086. Subsequently, the generated image is encoded using the CLIP model, and then the encoding is spliced before the sentence to be translated and input into the Vicuna model for reasoning, resulting in the translation "An apple is on the table." Rely on Loss LM The formula is used to calculate the difference between it and “There is an apple on the table.” In this case, it is 0.605. Then, the 0.086 calculated above is added to 0.605 as the overall loss to optimize the performance of the Vicuna model.

[0114] Some necessary terms of the present invention are as follows:

[0115] 1. Stable Diffusion Model: An artificial intelligence image generation model that can generate images corresponding to the given text descriptions.

[0116] 2. CLIP Model: An artificial intelligence encoding model that can encode images and convert them into vector representations.

[0117] 3. SpaCY: An open-source natural language processing library.

[0118] 4. NLTK: An open-source natural language processing library.

[0119] 5. Scene-Graph-Benchmark Method: A method for extracting triples from images.

[0120] 6. Relationformer Method: A method for extracting triples from images.

[0121] 7. Vicuna Model: The Vicuna model is a large language model fine-tuned for multi-turn conversations, with good content generation capabilities.

[0122] 8. DDPO Training Method: A reinforcement learning method that can optimize the performance of diffusion models by artificially setting reward functions.

[0123] It should be noted that relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variation thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0124] Although the present application has been described above with reference to specific embodiments, various modifications can be made thereto and components thereof can be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in the present application can be combined with each other in any way, and the exhaustive description of these combinations is not given in this specification only for the sake of saving space and resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism, characterized in that, It includes the following steps: S1. Manually set the sentence to be translated and construct training set 1; S2. For the training set 1 obtained in step S1, apply a text processing library to process the sentences to be translated in training set 1 to obtain text triples. Input the sentences to be translated in training set 1 into an image generation model to generate pictures, and then use an image triple extraction model to process them to obtain image triples. Then, apply the distance property of the two sets of triples to train the image generation model to obtain a trained image generation model; The specific implementation method of step S2 includes the following steps: S2.

1. Process the sentence x to be translated in Training Set 1 using a word processing library, iterate through each data in Training Set 1, extract the verbs and nouns in the sentence of the language to be translated according to the grammatical relationship, process the sentence of the language to be translated, and extract the literal triple set LSG = (h l1 , r l1 , t l1 ), (h l2 , r l2 , t l2 ), …, (h ln , r ln , t ln ), where h ln is the nth head entity in the literal triple set, r ln is the nth relationship in the literal triple set, and t ln is the nth tail entity in the literal triple set; S2.

2. Input the sentences to be translated in training set 1 into an image generation model to generate pictures. The image generation model is the Stable Diffusion model, and automatically generate the corresponding pictures I of the sentences to be translated; S2.

3. Input the image I obtained in step S2.2 into the image triple extraction model to automatically obtain the image triple set VSG = (h v1 , r v1 , t v1 ), (h v2 , r v2 , t v2 ), …, (h vm , r vm , t vm ), where h vm is the m-th head entity in the image triple set, r vm is the m-th relation in the image triple set, and t vm is the m-th tail entity in the image triple set; S2.

4. Calculate the distance between the members of the text triple set obtained in step S2.1 and the image triple set obtained in step S2.4 as the image generation score Sim(x, I) of the data. The specific formula is: where Score is the scoring function of the similarity between the members of the text triple set and the image triple set; The specific calculation method of Score is: Score(LSG i ,VSG) = max(D(LSG i ,VSG1),…,D(LSG i ,VSG m )) where D is the triple similarity function, The specific calculation method of D is: where SIM is the function for calculating the distance between natural language words; S2.

5. Take the image generation score obtained in step S2.4 as the performance score of the model, and apply the DDPO training method to optimize the image generation model. The optimization formula is: where θ is a parameter of the Stable Diffusion model, and J DDPO (θ) is the optimization objective of the Stable Diffusion model parameters, and r(x, I) is a manually set reward function used to quantify the matching degree between the sentence x to be translated and the corresponding image I. refers to the expected value of the likelihood estimation of the reward function when the sentence x to be translated satisfies the probability distribution p θ (x|c) of the given parameter θ and the given text condition c, and the image I satisfies the probability distribution p(I) of the image itself. S3. Input the sentences to be translated in training set 1 obtained in step S1 into the trained image generation model in step S2 to obtain the generated images corresponding to the sentences to be translated; S4. Encode the generated images corresponding to the sentences to be translated obtained in step S3, and then splice them with the corresponding sentences to be translated to construct training set 2. Use training set 2 to train the large language model to optimize the multi-modal machine translation performance of the large language model.

2. The multimodal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 1, wherein, The text processing library applied in step S2.1 is the SpaCY natural language text processing library or the NLTK natural language toolkit.

3. A multimodal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 2, characterized in that The image generation model is the Stable Diffusion model.

4. A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 3, characterized in that The image triple extraction model in step S2.3 is the Scene-Graph-Benchmark model or the Relationformer model.

5. A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 4, characterized in that, The calculation method of SIM in step S2.4 is to first encode using the large language model and then calculate using the cosine similarity. The specific formula is: where Encode represents the process of automatically encoding words using the large language model, and A and B are two natural language words to be calculated.

6. A multimodal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 5, characterized in that The specific implementation method of step S4 includes the following steps: S4.

1. Encode the generated image I′ corresponding to the sentence to be translated obtained in step S3 to obtain the image encoding C(I′). The expression of the encoder's processing is: C(I′) = W′I′ + b′ where W′ and b′ are the first weight matrix and the second weight matrix respectively, and the encoder selects the CLIP model; S4.

2. Subsequently, process the obtained image encoding C(I′) using the following formula to obtain the final encoding C′(I′) of the image: C′(I′) = WC(I′) + b where W and b are the third weight matrix and the fourth weight matrix respectively; S4.

3. Concatenate the final encoding of the image to the front of the sentence to be translated to construct training set 2, and use it as the input for the large language model for training to obtain the final machine translation result; S4.

4. Calculate the loss between the final machine translation result w′ and the sentence w after manual translation to obtain the translation loss Loss LM The expression is as follows: where C is the parameter of the large language model; Then calculate the overall loss Loss of the large language model, and the expression is: Loss=Loss LM +Sim(x, I).

7. A multi-modal machine translation training method combining a knowledge graph, a large language model, and a visual imagination mechanism according to claim 6, characterized in that In step S4.3, the large language model is the Llama2 model or the Llama3 model or the Vicuna model further trained on the basis of the Llama2 model, and the large language model used is modeled as: Among them, t represents the current time step, j t Represents the tth word in the paragraph, which is generated based on the large model according to the time step. One word is generated in one time step, then p(w′) represents the probability of generating w′, and p(j t |j <t ) represents the probability of generating the t-th word after generating the first t-1 words, and T is the total time.

Images