Multi-modal translation method and system
Through the combination of ResNet, LSTM, GPT and Seq2Seq models, the problems of low translation efficiency of multimodal data and poor adaptability of personalized texts are solved, and efficient and accurate cross-cultural background translation is achieved.
Patent Information
- Application Number
- CN202510493782.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-18
- Publication Date
- 2025-08-12
AI Technical Summary
The existing multimodal data translation methods are inefficient and have poor adaptability to personalized texts, making it difficult to adapt to the translation needs of different cultural backgrounds.
Image subtitles are generated using ResNet and LSTM models, the GPT model extracts long text summary, the Seq2Seq model is translated, and the GPT model is fine-tuned through the classification model to optimize the translated text.
It improves the efficiency and accuracy of multimodal text translation, enhances adaptability to different cultural backgrounds, and improves the user's reading experience.
Smart Images
Figure CN120471067A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a multimodal translation method and system. Background Art
[0002] In the context of globalization, the cross-border dissemination of cultural content has become increasingly important. With the development of internet technology, and particularly the popularity of social media, more and more cultural content needs to transcend language and cultural barriers to achieve global dissemination and exchange. However, the differences between languages and cultures pose challenges to the dissemination of this content.
[0003] For multimodal data translation, corresponding models are currently used for processing. For example, images are usually processed using a combination of convolutional neural networks (CNN) and recurrent neural networks (RNN) to generate subtitles, long text summary extraction uses statistical methods or shallow neural networks, and the translation module uses rule-based machine translation or statistical machine translation. This type of multimodal translation method not only has low translation efficiency but also has poor adaptability to personalized text. Summary of the Invention
[0004] In view of this, embodiments of the present invention provide a multimodal translation method and system for solving the current problems of low efficiency and poor adaptability of multimodal text translation.
[0005] In a first aspect of an embodiment of the present invention, a multimodal translation method is provided, comprising: Get the image input by the user, extract the image features through the ResNet model, input the image features into the LSTM model, and generate image captions through the LSTM model; Obtaining a long text input by a user, processing the context of the long text through a GPT model, and extracting summary information of the long text, wherein the long text is a text with more than a preset number of characters; Input image captions and long text summary information into the Seq2Seq model and output the translated target language text; The style of the long text is determined by a classification model, the GPT model is fine-tuned according to the style of the long text, and the style of the target language text is optimized based on the fine-tuned GPT model.
[0006] In a second aspect of an embodiment of the present invention, a multimodal translation system is provided, comprising: The subtitle generation module is used to obtain the image input by the user, extract the image features through the ResNet model, input the image features into the LSTM model, and generate image subtitles through the LSTM model; A summary extraction module is used to obtain long text input by the user, process the context of the long text through the GPT model, and extract summary information of the long text, where the long text is a text with more than a preset number of characters; The text translation module is used to input image captions and long text summary information into the Seq2Seq model and output the translated target language text; The style optimization module is used to determine the style of the long text through the classification model, fine-tune the GPT model according to the style of the long text, and optimize the style of the target language text based on the fine-tuned GPT model.
[0007] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable by the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.
[0008] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.
[0009] In the embodiment of the present invention, the combination of ResNet and LSTM improves the accuracy and efficiency of image caption generation, avoiding the gradient vanishing and long-term dependency problems in traditional methods; the GPT model enables efficient extraction of long text summaries, avoiding the problem that traditional methods cannot capture the semantics of long texts; the Seq2Seq model enables accurate translation of multilingual and multimodal data with high real-time performance and reliable results; and the personalized fine-tuning of the GPT model improves the adaptability of text in different cultural contexts, enhancing the user's reading experience. As a result, not only can the efficiency of multimodal text translation be improved, but also the accuracy and reliability of translation are guaranteed, and at the same time, it can adapt to the translation needs of different cultural backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0011] Figure 1 A schematic flow chart of a multimodal translation method provided by one embodiment of the present invention; Figure 2 A schematic diagram of the structure of a multimodal translation system provided by one embodiment of the present invention; Figure 3 The present invention provides a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION
[0012] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0013] It should be understood that the terms "including" and similar expressions in the specification, claims, and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, or apparatus comprising a series of steps or units is not limited to the listed steps or units. Furthermore, the terms "first" and "second" are used to distinguish between different objects and are not intended to describe a specific order.
[0014] See also Figure 1 , a flowchart of a multimodal translation method provided by an embodiment of the present invention includes: S101. Obtain an image input by a user, extract image features through a ResNet model, input the image features into an LSTM model, and generate image captions through the LSTM model; After the user inputs an image, the ResNet model and LSTM model can generate a description text of the image content. The image caption is also a description of the image content.
[0015] The ResNet model is a lightweight residual neural network that addresses the training degradation problem in deep neural networks by introducing residual learning. The LSTM (Long Short-Term Memory) model is a time-recurrent recurrent neural network. Its core lies in the introduction of memory cells and a gating mechanism. By dynamically controlling the flow of information, it achieves a balance between long-term memory and short-term updates, and can capture feature dependencies in time or text sequences.
[0016] The ResNet model is used as the encoder. In the residual block of the ResNet model, batch normalization and ReLU activation function are used to optimize feature extraction. The feature gradients in the residual block are directly propagated through skip connections, and the model parameters are adjusted based on the Adam optimizer.
[0017] Design the residual block, which is in the form of , where F(x) represents the residual function and H(x) represents the convolutional layer combination.
[0018] In the residual block, batch normalization and ReLU activation function are used to optimize feature extraction; skip connections allow gradients to be directly propagated, allowing for training deeper networks.
[0019] During the training process, the Adam optimizer is used to adjust the model parameters and improve the convergence speed. The trained ResNet model can extract the key features of the image as the input of the decoder.
[0020] Among them, the LSTM model is used as a decoder, and the generated image captions are optimized through the Beam Search algorithm.
[0021] LSTM acts as a decoder to decode the data encoded by the ResNet model. It processes sequence data through a gating mechanism (forget gate, input gate, output gate) and generates subtitle text by capturing long-distance dependencies in image descriptions.
[0022] The Beam Search algorithm is a heuristic search algorithm used to find the optimal or nearly optimal output sequence in sequence generation. It limits the number of nodes expanded at each step and balances the search space and computational efficiency. Based on the Beam Search algorithm, the generated text sequence can be optimized.
[0023] S102: Obtain a long text input by a user, process the context of the long text using a GPT model, and extract summary information of the long text, wherein the long text is a text with more than a preset number of characters; Long text generally refers to text with more than a preset number of characters. This can refer to coherent content of thousands or even millions of words that the model needs to understand at once. After the user enters long text, the GPT model needs to simplify it, that is, extract text summary information.
[0024] The GPT (Generative Pre-trained Transformer) model is a natural language processing model based on the Transformer architecture. It generates coherent and natural summary text by learning the statistical laws of language. The GPT model can extract summary information from long texts.
[0025] Among them, the context information of long texts is processed through the self-attention mechanism of the GPT model, which includes multi-head attention; Based on the long text sequence, the probability distribution of the next keyword is predicted through the autoregressive method, and the word with the highest probability is selected as the output to iteratively generate a complete summary.
[0026] Given a text sequence as input, GPT processes the context of the long text using a self-attention mechanism. It then generates a probability distribution for the next word using autoregression, selects the word with the highest probability as output, and iteratively generates a complete summary. When training the GPT model, it uses a large-scale corpus for pre-training to improve the model's generalization capabilities.
[0027] In this embodiment, the introduction of multi-head attention in the self-attention mechanism can enhance the expressive power of the model.
[0028] S103, input the image captions and long text summary information into the Seq2Seq model, and output the translated target language text; The Seq2Seq model is a deep learning model for processing sequential data. Its core component is an encoder and a decoder, converting an input sequence into an output sequence. By training the Seq2Seq model, it can translate input text into different languages. Using image captions and long text summaries as source text, the Seq2Seq model can output text in the target language, for example, converting English to Chinese.
[0029] Among them, the encoder based on the Seq2Seq model encodes the source language text into a fixed-length vector, and the source language text is the summary information of the image subtitles and long text; the context information of the source language text is captured through the bidirectional LSTM; and the decoder based on the Seq2Seq model decodes the fixed-length vector into the target language text.
[0030] A bidirectional LSTM (or BiLSTM) runs two LSTMs simultaneously at the same time step: one processes the data in the forward direction of the sequence, and the other processes it in the reverse direction. Through this bidirectional processing, the BiLSTM can simultaneously obtain the contextual information of each element in the sequence, thereby more comprehensively understanding the context of the sequence.
[0031] The Seq2Seq model uses an attention mechanism to dynamically focus on the encoder's output. The attention mechanism allows the decoder to look at specific parts of the input sequence when generating each output word, thereby improving translation accuracy and contextual relevance.
[0032] Preferably, unregistered words appearing in the text are processed by using subword segmentation technology.
[0033] Word segmentation technology breaks words into several consecutive segments. It is used in natural language processing (NLP) to address data sparsity and improve model generalization. Subword segmentation techniques include BPE (Byte Pair Encoding) and WordPiece. Unknown words are words that are not included in the word segmentation vocabulary or model training vocabulary, but need to be recognized and processed in real applications.
[0034] S104: Determine the style of the long text through the classification model, fine-tune the GPT model according to the style of the long text, and optimize the style of the target language text based on the fine-tuned GPT model.
[0035] Long text style refers to the text expression style. In different regions, the same language may have different expression styles. For example, there may be some differences in word order and the use of some modal particles. Therefore, text styles can be divided into Europe, Asia, America, etc., or divided according to specific regions or countries.
[0036] The GPT model is used as the core, and adaptive adjustments are made based on the text style. The text style is determined by training a classification model, calculating the probability of the text belonging to each style, and using the softmax function to obtain its probability distribution.
[0037] Fine-tune the GPT model based on the text style. During fine-tuning, use a small learning rate to prevent overfitting. Fine-tuning involves optimizing model parameters based on a pre-trained large model using domain-specific data to adapt to specific task requirements. Fine-tuning the GPT model optimizes translated text to suit the text styles of different regions, while also reducing model training costs.
[0038] Among them, in the self-attention mechanism of the GPT model, position encoding is added to capture the sequential information of the text; the attention probability is calculated based on the mask mechanism, and the contextual representation of the text is generated based on the tensor of attention probability to value.
[0039] The self-attention mechanism can consider the context and preferences of different cultural backgrounds when generating text. In the self-attention mechanism, positional encoding is introduced to capture the sequential information of the text, achieving culturally sensitive and personalized text transfer.
[0040] Specifically, define the SelfAttention class, initialize parameters such as the number of attention heads, input size, and hidden size, initialize the linear transformation layers for query, key, and value, use the Xavier initialization method to ensure the stability of the parameter distribution, and define the transpose_for_scores method to convert the input tensor to the multi-head attention format and calculate the attention score.
[0041] The softmax function is used to calculate the attention probability. When calculating the attention probability, a mask mechanism is introduced to prevent the model from focusing on invalid positions. The attention probability is applied to the value tensor to generate a context representation, and the final result is output through linear transformation and layer normalization.
[0042] In this embodiment, image captions are generated through the RetNet model and the LSTM model, and long texts are summarized and extracted through the GPT model. The text is translated based on the Seq2Seq model, and the translated text is optimized through the fine-tuned GPT model, thereby improving the efficiency of multimodal text translation and enhancing the adaptability of text translation in different regions and cultural backgrounds. It avoids the problems of gradient vanishing, low translation efficiency, and poor adaptability of traditional models, and ensures the accuracy and real-time performance of text translation.
[0043] It should be understood that the sequence numbers of the steps in the above embodiments do not imply a specific order of execution; the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0044] Figure 2 A schematic diagram of the structure of a multimodal translation system provided in an embodiment of the present invention, the system comprising: The subtitle generation module 210 is used to obtain an image input by a user, extract image features through a ResNet model, input the image features into an LSTM model, and generate image subtitles through the LSTM model; The extracting of image features by the ResNet model includes: The ResNet model is used as the encoder. In the residual block of the ResNet model, batch normalization and ReLU activation function are used to optimize feature extraction. The feature gradients in the residual block are directly propagated through skip connections, and the model parameters are adjusted based on the Adam optimizer.
[0045] Preferably, the LSTM model is used as a decoder, and the generated image captions are optimized by the Beam Search algorithm.
[0046] Summary extraction module 220, for obtaining a long text input by a user, processing the context of the long text using a GPT model, and extracting summary information of the long text, wherein the long text is a text with more than a preset number of characters; Among them, the context information of long texts is processed through the self-attention mechanism of the GPT model, which includes multi-head attention; Based on the long text sequence, the probability distribution of the next keyword is predicted through the autoregressive method, and the word with the highest probability is selected as the output to iteratively generate a complete summary.
[0047] A text translation module 230 is used to input image captions and long text summary information into a Seq2Seq model and output the translated target language text; The text translation module 230 includes: An encoder, configured to encode a source language text into a fixed-length vector based on a Seq2Seq model, wherein the source language text is summary information of an image caption and a long text; Feature extraction unit, used to capture the contextual information of the source language text through bidirectional LSTM; Decoder, which is used to decode the fixed-length vector into target language text based on the Seq2Seq model.
[0048] Preferably, unregistered words appearing in the text are processed by using subword segmentation technology.
[0049] The style optimization module 240 is used to determine the style of the long text through the classification model, fine-tune the GPT model according to the style of the long text, and perform style optimization on the target language text based on the fine-tuned GPT model.
[0050] Wherein, fine-tuning the GPT model according to the style of the long text further includes: In the self-attention mechanism of the GPT model, position encoding is added to capture the sequential information of the text; The attention probability is calculated based on the mask mechanism, and the contextual representation of the text is generated based on the tensor of attention probability to value.
[0051] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0052] Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is used for multimodal text translation. Figure 3 As shown, the electronic device 3 of this embodiment includes: a memory 310, a processor 320 and a system bus 330, and the memory 310 includes an executable program 3101 stored thereon. It can be understood by those skilled in the art that Figure 3 The electronic device structure shown in the figure does not constitute a limitation to the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0053] The following combination Figure 3 A detailed introduction to the various components of electronic equipment: Memory 310 can be used to store software programs and modules. Processor 320 executes the software programs and modules stored in memory 310 to perform various functional applications and data processing of the electronic device. Memory 310 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback). The data storage area may store data generated based on the use of the electronic device (such as cached data). Memory 310 may also include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state memory device.
[0054] The memory 310 includes an executable program 3101 for the interface generation method. The executable program 3101 can be divided into one or more modules / units, which are stored in the memory 310 and executed by the processor 320 to implement multimodal translation, etc. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the executable program 3101 in the electronic device 3. For example, the executable program 3101 can be divided into functional modules such as a subtitle generation module, a summary extraction module, a text translation module, and a style optimization module.
[0055] The processor 320 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and lines. By running or executing software programs and / or modules stored in the memory 310 and accessing data stored in the memory 310, it performs various functions of the electronic device and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, application programs, etc., and the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 320.
[0056] The system bus 330 connects the various functional components within the computer and can transmit data, address information, and control information. It can be a PCI bus, ISA bus, or CAN bus, for example. Instructions from the processor 320 are transmitted to the memory 310 via the bus, and the memory 310 feeds data back to the processor 320. The system bus 330 is responsible for the exchange of data and instructions between the processor 320 and the memory 310. Of course, the system bus 330 can also connect to other devices, such as network interfaces and display devices.
[0057] In an embodiment of the present invention, the executable program executed by the processing 320 included in the electronic device includes: Get the image input by the user, extract the image features through the ResNet model, input the image features into the LSTM model, and generate image captions through the LSTM model; Obtaining a long text input by a user, processing the context of the long text through a GPT model, and extracting summary information of the long text, wherein the long text is a text with more than a preset number of characters; Input image captions and long text summary information into the Seq2Seq model and output the translated target language text; The style of the long text is determined by a classification model, the GPT model is fine-tuned according to the style of the long text, and the style of the target language text is optimized based on the fine-tuned GPT model.
[0058] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0059] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.
[0060] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multimodal translation method, characterized in that: include: Get the image input by the user, extract the image features through the ResNet model, input the image features into the LSTM model, and generate image captions through the LSTM model; Obtaining a long text input by a user, processing the context of the long text through a GPT model, and extracting summary information of the long text, wherein the long text is a text with more than a preset number of characters; Input image captions and long text summary information into the Seq2Seq model and output the translated target language text; The style of the long text is determined by a classification model, the GPT model is fine-tuned according to the style of the long text, and the style of the target language text is optimized based on the fine-tuned GPT model.
2. The method according to claim 1, characterized in that Extracting image features through the ResNet model includes: The ResNet model is used as the encoder. In the residual block of the ResNet model, batch normalization and ReLU activation function are used to optimize feature extraction. The feature gradients in the residual block are directly propagated through skip connections, and the model parameters are adjusted based on the Adam optimizer.
3. The method according to claim 1, characterized in that Generating image captions by the LSTM model further includes: The LSTM model is used as a decoder and the generated image captions are optimized using the Beam Search algorithm.
4. The method according to claim 1, wherein The GPT model is used to process the context of the long text and extract the summary information of the long text, including: Processing the contextual information of long texts through the GPT model's self-attention mechanism, which includes multi-head attention; Based on the long text sequence, the probability distribution of the next keyword is predicted through the autoregressive method, and the word with the highest probability is selected as the output to iteratively generate a complete summary.
5. The method according to claim 1, wherein The image captions and long text summary information are input into the Seq2Seq model, and the translated target language text is outputted as follows: The encoder based on the Seq2Seq model encodes the source language text into a fixed-length vector, wherein the source language text is the summary information of the image caption and the long text; Capture the contextual information of the source language text through bidirectional LSTM; The decoder based on the Seq2Seq model decodes the fixed-length vector into the target language text.
6. The method according to claim 1, characterized in that The step of inputting the image captions and long text summary information into the Seq2Seq model and outputting the translated target language text further includes: The unregistered words in the text are processed by subword segmentation technology.
7. The method according to claim 1, characterized in that Fine-tuning the GPT model according to the style of the long text includes: In the self-attention mechanism of the GPT model, position encoding is added to capture the sequential information of the text; The attention probability is calculated based on the mask mechanism, and the contextual representation of the text is generated based on the tensor of attention probability to value.
8. A multimodal translation system, characterized in that: include: The subtitle generation module is used to obtain the image input by the user, extract the image features through the ResNet model, input the image features into the LSTM model, and generate image subtitles through the LSTM model; A summary extraction module is used to obtain long text input by the user, process the context of the long text through the GPT model, and extract summary information of the long text, where the long text is a text with more than a preset number of characters; The text translation module is used to input image captions and long text summary information into the Seq2Seq model and output the translated target language text; The style optimization module is used to determine the style of the long text through the classification model, fine-tune the GPT model according to the style of the long text, and optimize the style of the target language text based on the fine-tuned GPT model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the multimodal translation method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the steps of the multimodal translation method according to any one of claims 1 to 7 are implemented.