A method for generating optimized multilingual speech translation through reasoning cues of a thought tree

Abstract

The present application relates to a kind of by the inference prompt of thinking tree, and the method for generating optimization of multilingual speech translation belongs to speech translation technical field.The present application includes the following steps: constructing including automatic speech recognition module, thinking tree inference module, large model fine-tuning module;Transcription text corresponding to source language is obtained by automatic speech recognition module;High-quality translation result is obtained by thinking tree inference;The multi-stage inference and evaluation capability of large model based on thinking tree is enhanced by fine-tuning to large model;Structural inference framework based on thinking tree is constructed, and the output of large model is generated by multi-step prompt to guide structure, and then the inference path of large model is determined by introducing translation evaluation index based on semantic perception;An reorderer is designed to select the optimal translation result of large model;The parameter of large model is efficiently fine-tuned, and the ability of thinking tree inference and evaluation of large model on speech translation task can be enhanced.The present application improves speech translation performance.

Description

Technical Field

[0001] This invention relates to a method for optimizing multilingual speech translation generation through reasoning prompts using mind trees, belonging to the field of speech translation technology. Background Technology

[0002] Traditional speech translation methods typically employ a cascaded architecture, which is simple to implement but prone to error accumulation, impacting translation performance. End-to-end speech translation methods aim to directly translate speech in the source language into text in the target language, effectively avoiding error accumulation. However, their training process usually relies on large parallel corpora, facing modeling challenges, especially with multilingual and multi-domain data.

[0003] In recent years, with the rapid development of large language models, various large-model-based speech translation methods have emerged. Existing large-model-based translation methods mainly employ thought chain reasoning, which follows a linear reasoning process, but suffers from the following limitations: First, linear reasoning methods cannot effectively explore the translation policy space. Thought chain-based reasoning models rely on a single contextual input for sequential derivation at each step, limiting their ability to generate multiple reasoning paths in parallel or to compare multiple reasoning paths. Second, in complex translation tasks, thought chains are susceptible to error propagation, leading to unstable translation quality. Furthermore, early misjudgments in the thought chain can affect subsequent reasoning processes. While subsequent steps can be optimized, such as through backtracking mechanisms, effectively correcting this error remains challenging and increases time costs. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a method for optimizing multilingual speech translation generation through reasoning prompts using mind trees, thereby improving the performance of speech translation based on large models. This invention constructs an explicit mind tree reasoning structure to replace the thought chain in large language models. This structure can generate multiple candidate translation paths at different levels, thereby exploring the reasoning space more comprehensively and improving overall translation performance.

[0005] The technical solution of this invention is: a method for optimizing multilingual speech translation generation through reasoning prompts using mind trees, the method comprising:

[0006] Step 1: Construct a speech recognition module, which includes an acoustic modeling submodule and a decoding submodule;

[0007] The acoustic modeling submodule extracts features from the input speech signal to obtain an acoustic feature representation.

[0008] The decoding submodule decodes the acoustic features into source language transcribed text and outputs the transcribed text to subsequent modules.

[0009] Step 2: Construct a reasoning framework based on a mind tree. The reasoning module based on the mind tree includes a candidate generation submodule, a node evaluation submodule, and a path search submodule. The candidate generation submodule generates multiple candidate translation results based on pre-designed prompts. The node evaluation submodule is used to estimate the quality of each node or part of the path in the mind tree and return an evaluation score for search. The path search submodule is used to search for several high-quality complete translation paths on the mind tree, guided by the scores provided by the node evaluation submodule, and outputs N-best candidates.

[0010] Step 3: Build a reorderer module. Use the built reorderer module to filter the translation results of the large model and prepare high-quality training data for efficient fine-tuning of the large model.

[0011] Step 4: Fine-tune the large model using LoRA, save the trained speech translation model, package it and deploy it on the server, and establish an API so that other applications can interact with the model to achieve online translation of speech input.

[0012] Furthermore, prior to Step 1, data collection is required, including:

[0013] The FLEURS and CoVoST-2 datasets were used; the FLEURS dataset is a low-resource speech translation dataset released by Google; CoVoST-2 is a dataset extended from CommonVoice, suitable for evaluating the generalization of models; these datasets were divided into three subsets: a training set for model training; a validation set for hyperparameter tuning and model validation; and a test set for performance evaluation.

[0014] Furthermore, Step 1 includes:

[0015] Input a single-channel audio raw speech waveform with a sampling rate of 16kHz; convert the audio into an 80-dimensional log-Mel spectrum; use a Transformer encoder to map the acoustic features into a high-level speech representation; output the transcribed text using an autoregressive Transformer decoder.

[0016] Furthermore, Step 2 includes:

[0017] The candidate generation submodule for any node and text The next node is generated according to formula (1). of One candidate:

[0018]

[0019] in, Indicates based on the current state Generate the next thinking strategy The probability, This represents the parameters of the current LLM, and each candidate thinking strategy. Each translation is generated independently from a different mental space, thus ensuring the diversity of translation results and improving translation performance.

[0020] Furthermore, Step 2 also includes:

[0021] (1) Use the node evaluation submodule to perform quality estimation on each node of the mind tree, as shown in formulas (2) to (4): ; ; ; Wherein, formulas (2) and (3) respectively represent Formulas for calculating semantic similarity; Indicates calculation The maximum N-gram order used for fractions; For vector dimensions, Indicates the index of a vector component; Both semantic similarity scores are normalized to [0, 1]; Formula (2) uses N-Gram to match the reference translation with the LLM translation results and calculates... Score; Speech recognition text representation as vector The translated text is represented as a vector. Formula (3) is calculated using cosine similarity. and The semantic similarity between them; finally, formula (4) is obtained by weighted fusion calculation. This represents the overall score of a single candidate node, for the . For each sample, its true score is denoted as . .

[0022] (2) Using the path search submodule as a guide, guided by the scores provided by the node evaluation submodule, several high-quality complete translation paths are searched and N-best candidates are output; Let... This represents the transcribed text obtained after Step 1. As the initial state, for each step (from arrive ), calculate all possible extended states The calculation method involves generating candidates through the candidate generation submodule in Step 2. A new extension and attach it to Above, a new state is formed. Next, all new states are evaluated through the node evaluation submodule of Step 2 to obtain... Select the one with the highest evaluation value. each state As the set of states for the next step; finally, choose... The state with the highest evaluation value This yields the final translation result.

[0023] Furthermore, Step 3 includes:

[0024] The reorderer module uses Multilingual-MiniLM as the encoder backbone, employing a 12-layer Transformer encoder with 384 hidden layers. A linear regression head is added to the CLS vectors to predict translation performance variations. The input consists of concatenated transcribed text and corresponding candidate translations, in the following format: The maximum input length is set to 512; the training loss function is shown in formula (5):

[0025]

[0026] in, It is the true score of the i-th sample obtained according to formula (4). It represents the prediction score of the reorderer module, where N is the total number of training samples; MSE Loss represents minimizing the model's predicted value. Compared with the true value The "mean square distance" between them.

[0027] Furthermore, the specific steps of Step 4 are as follows:

[0028] Step 4.1: Fine-tune the large model using LoRA, save the trained model as a ".pth" file, and deploy it to the local server. Use Sanic technology to convert the model into an application interface.

[0029] Step 4.2: Call the application interface of the model deployed on the local server from the web interface, and directly call the model through the web page to output the retrieved results to the front-end interface for display.

[0030] The present invention also provides a system for optimizing multilingual speech translation generation through reasoning prompts from mind trees, the system comprising: a module for executing the method for optimizing multilingual speech translation generation through reasoning prompts from mind trees.

[0031] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for optimizing multilingual speech translation generation through reasoning prompts based on mind trees.

[0032] The beneficial effects of this invention are:

[0033] This invention constructs a speech recognition module, a mind tree reasoning module, a reorderer module, and a fine-tuning module. In the speech recognition stage, the input speech signal is subjected to text recognition to obtain the transcription corresponding to the source language. The recognition result is translated using a mind tree to generate multiple candidate reasoning results, avoiding the shortcomings of traditional thought chains. The optimal reasoning path is selected through a trained reorderer. LoRA fine-tuning is used to enhance the mind tree reasoning ability of the employed LLM, thereby improving its performance in speech translation. Attached Figure Description

[0034] Figure 1 This is the overall design block diagram of the present invention. Detailed Implementation

[0035] Example 1: As Figure 1 As shown, a method for optimizing multilingual speech translation generation through reasoning prompts using mind trees is described, the method comprising:

[0036] Step 1: Construct a speech recognition module, which includes an acoustic modeling submodule and a decoding submodule;

[0037] Prior to Step 1, data collection is required, including:

[0038] The FLEURS and CoVoST-2 datasets were used; the FLEURS dataset is a low-resource speech translation dataset released by Google; CoVoST-2 is a dataset extended from CommonVoice, suitable for evaluating the generalization of models; these datasets were divided into three subsets: a training set for model training; a validation set for hyperparameter tuning and model validation; and a test set for performance evaluation.

[0039] The acoustic modeling submodule extracts features from the input speech signal to obtain an acoustic feature representation.

[0040] The decoding submodule decodes the acoustic features into source language transcribed text and outputs the transcribed text to subsequent modules.

[0041] Furthermore, Step 1 includes:

[0042] Input a single-channel audio raw speech waveform with a sampling rate of 16kHz; convert the audio into an 80-dimensional log-Mel spectrum; use a Transformer encoder to map the acoustic features into a high-level speech representation; output the transcribed text using an autoregressive Transformer decoder.

[0043] Step 2: Construct a reasoning module based on a mind tree. This module includes a candidate generation submodule, a node evaluation submodule, and a path search submodule. The candidate generation submodule generates multiple candidate translation results based on pre-designed prompts. The node evaluation submodule performs quality estimation on each node or part of the path in the mind tree and returns an evaluation score for the search. The path search submodule searches for several high-quality complete translation paths on the mind tree, guided by the scores provided by the node evaluation submodule, and outputs N-best candidates.

[0044] Furthermore, Step 2 includes:

[0045] The candidate generation submodule for any node and text The next node is generated according to formula (1). of One candidate:

[0046]

[0047] in, Indicates based on the current state Generate the next thinking strategy The probability, This represents the parameters of the current LLM, and each candidate thinking strategy. Each translation is generated independently from a different mental space, thus ensuring the diversity of translation results and improving translation performance.

[0048] Furthermore, Step 2 also includes:

[0049] (1) Use the node evaluation submodule to perform quality estimation on each node of the mind tree, as shown in formulas (2) to (4): ; ; ; Wherein, formulas (2) and (3) respectively represent Formulas for calculating semantic similarity; Indicates calculation The maximum N-gram order used for fractions; For vector dimensions, Indicates the index of a vector component; Both semantic similarity scores are normalized to [0, 1]; Formula (2) uses N-Gram to match the reference translation with the LLM translation results and calculates... Score; Speech recognition text representation as vector The translated text is represented as a vector. Formula (3) is calculated using cosine similarity. and The semantic similarity between them; finally, formula (4) is obtained by weighted fusion calculation. This represents the overall score of a single candidate node, for the . For each sample, its true score is denoted as . .

[0050] (2) Using the path search submodule as a guide, guided by the scores provided by the node evaluation submodule, several high-quality complete translation paths are searched and N-best candidates are output; Let... This represents the transcribed text obtained after Step 1. As the initial state, for each step (from arrive ), calculate all possible extended states The calculation method involves generating candidates through the candidate generation submodule in Step 2. A new extension and attach it to Above, a new state is formed. Next, all new states are evaluated through the node evaluation submodule of Step 2 to obtain... Select the one with the highest evaluation value. each state As the set of states for the next step; finally, choose... The state with the highest evaluation value This yields the final translation result.

[0051] Step 3: Build a reorderer module. Use the built reorderer module to filter the translation results of the large model and prepare high-quality training data for efficient fine-tuning of the large model.

[0052] Furthermore, Step 3 includes:

[0053] The reorderer module uses Multilingual-MiniLM as the encoder backbone, employing a 12-layer Transformer encoder with 384 hidden layers. A linear regression head is added to the CLS vectors to predict translation performance variations. The input consists of concatenated transcribed text and corresponding candidate translations, in the following format: The maximum input length is set to 512; the training loss function is shown in formula (5):

[0054]

[0055] in, It is the true score of the i-th sample obtained according to formula (4). It represents the prediction score of the reorderer module, where N is the total number of training samples; MSE Loss represents minimizing the model's predicted value. Compared with the true value The "mean square distance" between them.

[0056] Step 4: Fine-tune the large model using LoRA, save the trained speech translation model, package it and deploy it on the server, and establish an API so that other applications can interact with the model to achieve online translation of speech input.

[0057] Furthermore, Step 4 includes:

[0058] Step 4.1: Fine-tune the large model using LoRA, save the trained model as a ".pth" file, and deploy it to the local server. Use Sanic technology to convert the model into an application interface.

[0059] Step 4.2: Call the application interface of the model deployed on the local server from the web interface, and directly call the model through the web page to output the retrieved results to the front-end interface for display.

[0060] This invention provides a system for optimizing multilingual speech translation generation through reasoning prompts using mind trees, the system comprising:

[0061] The speech recognition module includes an acoustic modeling submodule and a decoding submodule.

[0062] It is used to extract features from the input speech signal through the acoustic modeling submodule to obtain an acoustic feature representation;

[0063] This module is used to decode acoustic features into source language transcribed text via a decoding submodule, and then output the transcribed text to subsequent modules.

[0064] The reasoning module based on the mind tree includes a candidate generation submodule, a node evaluation submodule, and a path search submodule.

[0065] This is used to enable a large model to generate multiple candidate translation results based on pre-designed prompts through a candidate generation submodule;

[0066] The node evaluation submodule is used to perform quality estimation on each node or part of the path in the mind tree and return an evaluation score for the search.

[0067] The path search submodule is used to search for several high-quality complete translation paths on the mind tree, guided by the scores provided by the node evaluation submodule, and output N-best candidates.

[0068] The reorderer module is used to filter the translation results of large models by using a pre-built reorderer, thus preparing high-quality training data for efficient fine-tuning of large models.

[0069] The large model fine-tuning module is used to fine-tune large models using LoRA, save the trained speech translation model, encapsulate and deploy it on the server, and establish an API so that other applications can interact with the model to achieve online translation of speech input.

[0070] The present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the method for optimizing multilingual speech translation generation through reasoning prompts based on mind trees.

[0071] This invention includes comparative experiments with other schemes to demonstrate the effectiveness of the proposed method. Under identical experimental parameters, the proposed method is compared with a baseline model to verify its effectiveness. The experimental results are shown in Table 1.

[0072] Table 1 shows the experimental results comparing the proposed solution with other methods.

[0073] Analysis of Table 1 shows that the method of this invention demonstrates a significant improvement in BLEU scores for translation tasks on the FLEURS and CoVoST-2 datasets. For the FLEURS dataset, Table 1 shows that the method of this invention outperforms the strongest cascaded baseline, TGCoT. For example, in the Greek-English and Vietnamese-English language directions, the BLEU scores of the method of this invention are 1.6 and 2.9 higher than TGCoT, respectively. Although the method of this invention scores relatively lower in the Ukrainian-English translation task, the overall results still indicate that the method is effective in handling low-resource language scenarios.

[0074] Furthermore, the method of this invention also achieves state-of-the-art performance on the CoVoST-2 dataset. Specifically, in the Chinese-English and Japanese-English translation tasks, the method of this invention improves the BLEU scores by 1.9 and 2.0, respectively. For other language pairs, such as German-English and French-English, the method of this invention also demonstrates stable performance improvements.

[0075] In summary, the method of this invention achieved the highest overall performance among all comparative methods, with average BLEU scores of 36.1 and 42.8 on the FLEURS and CoVoST-2 datasets, respectively. These results strongly demonstrate the effectiveness and robustness of the method of this invention.

[0076] To further evaluate the impact of the mind tree reasoning module of this invention on translation performance, the following ablation experiments were conducted. The results are shown in Table 2. When only the initial translation step was used, the BLEU score significantly decreased from 42.8 to 38.2. When steps 2 and 3 were removed, the BLEU scores decreased to 38.3 and 38.4, respectively. These results highlight the importance of the polishing and comparison steps in the candidate generation submodule during the reasoning process. Further removal of the evaluation step leads to a further decrease in translation quality, thus validating the necessity of the node evaluation submodule.

[0077] Table 2 shows the results of the ablation experiment.

[0078] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for optimizing multilingual speech translation generation through reasoning prompts using mind trees, characterized in that: The method includes: Step 1: Construct a speech recognition module, which includes an acoustic modeling submodule and a decoding submodule; The acoustic modeling submodule extracts features from the input speech signal to obtain an acoustic feature representation. The decoding submodule decodes the acoustic features into source language transcribed text and outputs the transcribed text to subsequent modules. Step 2: Construct a reasoning module based on a mind tree. This module includes a candidate generation submodule, a node evaluation submodule, and a path search submodule. The candidate generation submodule generates multiple candidate translation results based on pre-designed prompts. The node evaluation submodule performs quality estimation on each node in the mind tree and returns an evaluation score for search. The path search submodule searches for several complete translation paths on the mind tree, guided by the scores provided by the node evaluation submodule, and outputs N-best candidates. Step 3: Build a reorderer module. Use the built reorderer module to filter the translation results of the large model. Specifically, this includes filtering by MSE Loss prediction score using the Multilingual-MiniLM reorderer to prepare training data for LoRA fine-tuning of the large model. Step 4: Fine-tune the large model using LoRA, save the trained speech translation model, package and deploy it on the server, and establish an API so that other applications can interact with the model to achieve online translation of speech input; Step 2 includes: The candidate generation submodule for any node and text The next node is generated according to formula (1). of One candidate: ； in, Indicates based on the current state Generate the next thinking strategy The probability, This represents the parameters of the current LLM, and each candidate thinking strategy. They are all generated independently from different thought spaces; Step 2 also includes: Using the path search submodule as a guide, guided by the scores provided by the node evaluation submodule, several complete translation paths are searched and N-best candidates are output; let... This represents the transcribed text obtained after Step 1. As the initial state, for each step ,from arrive Calculate all possible extended states The calculation method is generated through the candidate generation submodule in Step 2. A new extension and attach it to Above, a new state is formed. Next, all new states are evaluated through the node evaluation submodule of Step 2 to obtain... Select the one with the highest evaluation value. each state As the set of states for the next step; finally, choose... The state with the highest evaluation value This yields the final translation result. The node evaluation submodule is used to perform a comprehensive score when estimating the quality of each node in the mind tree. The scores and semantic similarity were calculated using a weighted fusion method.

2. The method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in claim 1, is characterized in that: Prior to Step 1, data collection is required, including: The FLEURS and CoVoST-2 datasets were used; the FLEURS dataset is a low-resource speech translation dataset released by Google; CoVoST-2 is a dataset extended from CommonVoice, suitable for evaluating the generalization of models; these datasets were divided into three subsets: a training set for model training; a validation set for hyperparameter tuning and model validation; and a test set for performance evaluation.

3. The method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in claim 1, is characterized in that: Step 1 includes: inputting a single-channel audio raw speech waveform with a sampling rate of 16kHz; converting the audio into an 80-dimensional log-Mel spectrum; using a Transformer encoder to map the acoustic features into a high-level speech representation; and using an autoregressive Transformer decoder to output the transcribed text.

4. The method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in claim 1, is characterized in that: Step 2 also includes: The node evaluation submodule is used to perform quality estimation for each node of the mind tree, as shown in formulas (2) to (4): ； ； ； Wherein, formulas (2) and (3) respectively represent Formulas for calculating semantic similarity; Indicates calculation The maximum N-gram order used for fractions; For vector dimensions, Indicates the index of a vector component; Both semantic similarity scores are normalized to [0, 1]; Formula (2) uses N-Gram to match the reference translation with the LLM translation results and calculates... Score; Speech recognition text representation as vector The translated text is represented as a vector. Formula (3) is calculated using cosine similarity. and The semantic similarity between them; finally, formula (4) is calculated by weighted fusion. This represents the overall score of a single candidate node, for the . For each sample, its true score is denoted as . .

5. The method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in claim 1, is characterized in that: Step 3 includes: The reorderer module uses Multilingual-MiniLM as the encoder backbone network, employing a 12-layer Transformer encoder with a hidden layer dimension of 384; a linear regression head is added to the CLS vector to predict the performance changes of the translation; the input consists of the transcribed text and the corresponding candidate translations concatenated, with a maximum input length of 512; the training loss function is shown in Equation (5): ； in, It is the first one obtained according to formula (4) The true score of each sample It represents the prediction score of the reorderer module, where N is the total number of training samples; MSE Loss represents minimizing the model's predicted value. Compared with the true value The average squared distance between them.

6. The method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in claim 1, is characterized in that: The specific steps of Step 4 are as follows: Step 4.1: Fine-tune the large model using LoRA, save the trained model as a ".pth" file, and deploy it to the local server. Use Sanic technology to convert the model into an application interface. Step 4.2: Call the application interface of the model deployed on the local server from the web interface, and directly call the model through the web page to output the retrieved results to the front-end interface for display.

7. A system for optimizing multilingual speech translation generation through reasoning prompts using mind trees, characterized in that, The system includes a module for performing a method for optimizing multilingual speech translation generation using reasoning prompts based on mind trees, as described in any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for optimizing multilingual speech translation generation through reasoning prompts based on mind trees, as described in any one of claims 1 to 6.

Images