Speech translation method, electronic equipment and storage medium
By converting speech segments into text sequences and calculating semantic coherence confidence, and combining acoustic features and neural network models to determine sentence boundaries, the problem of inaccurate sentence boundary judgment and high latency in real-time speech translation is solved, thus improving translation quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GOERTEK INC
- Filing Date
- 2025-12-18
- Publication Date
- 2026-04-21
AI Technical Summary
In current real-time speech translation, inaccurate sentence boundary detection and high latency result in poor translation quality.
The speech segment to be translated is converted into a text sequence, the semantic coherence confidence score is calculated, and the boundary judgment model of the pre-trained neural network structure is combined with the acoustic features input to determine the timing of the translation result output based on the confidence score.
It improves the accuracy of sentence boundary judgment, reduces judgment delay, and enhances the quality of real-time speech translation.
Smart Images
Figure CN121905149A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech processing technology, and in particular to a speech translation method, electronic device, and storage medium. Background Technology
[0002] With the development and advancement of technology, scenarios for real-time voice translation using voice translation software or devices are gradually increasing.
[0003] Currently, real-time speech translation typically relies on Voice Activity Detection (VAD) technology to determine sentence boundaries, i.e., detecting where there is sound and where there is silence. It segments continuous speech streams based on pauses in speech, but it cannot understand semantic logic, cannot distinguish whether a pause is a breath within a sentence or a boundary between sentences, and is prone to misjudgment. Furthermore, in order to reduce missed detections, VAD needs to accumulate a certain amount of audio data to verify whether it is a true pause, which directly leads to judgment delays and makes it difficult to adapt to the low latency requirements of real-time translation.
[0004] In summary, how to avoid the problem of poor overall real-time translation performance caused by inaccurate sentence boundary judgment and high judgment delay in speech translation has become a technical problem that urgently needs to be solved in this field.
[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0006] The main purpose of this application is to provide a speech translation method, electronic device and storage medium, which aims to avoid the problem of poor overall real-time translation effect caused by inaccurate judgment of sentence boundaries and high judgment delay in speech translation.
[0007] To achieve the above objectives, this application proposes a speech translation method, which includes: The speech segment to be translated is converted into a text sequence, and the semantic coherence confidence of the text sequence is calculated; The acoustic features of the speech segment to be translated and the semantic coherence confidence are input into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated. The boundary determination model is a neural network structure model and is obtained through pre-training. Based on the confidence level determined by the boundary, the timing for outputting the translation result of the sentence defined by the candidate sentence boundary is determined.
[0008] In one embodiment, the step of calculating the semantic coherence confidence of the text sequence includes: The text sequence is input into a language model to obtain the perplexity of the text sequence, wherein the language model is a Transformer (a deep learning model structure) structure model and is obtained through pre-training; The perplexity level is converted to obtain the semantic coherence confidence level.
[0009] In one embodiment, before the step of inputting the acoustic features of the speech segment to be translated and the semantic coherence confidence into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated, the method further includes: Extract the acoustic features of the speech segment to be translated; An attention mechanism is used to weight and fuse the acoustic features and the semantic coherence confidence to obtain a multimodal fusion vector; The step of inputting the acoustic features of the speech segment to be translated and the semantic coherence confidence into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated includes: The multimodal fusion vector is input into the boundary determination model to obtain the candidate sentence boundaries and the corresponding boundary determination confidence scores in the speech segment to be translated.
[0010] In one embodiment, the step of determining the timing of outputting the translation result of the statement defined by the candidate sentence boundary based on the confidence level of the boundary includes: The boundary determination confidence level is compared with a preset confidence threshold. If the confidence level of the boundary determination is higher than or equal to the confidence threshold, then the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary determination is lower than the confidence threshold, the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in a preset delayed translation window.
[0011] In one embodiment, the step of determining the timing of outputting the translation result of the statement defined by the candidate sentence boundary based on the confidence level of the boundary includes: The confidence level of the boundary determination is compared with a preset confidence threshold, and it is determined whether the boundary text sequence of the statement defined by the candidate sentence boundary conforms to the preset grammatical integrity rule. If the confidence level of the boundary determination is higher than or equal to the confidence threshold, and the boundary text sequence conforms to the grammatical integrity rule, then the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary determination is lower than the confidence threshold, or if the boundary text sequence does not conform to the grammatical integrity rule, then the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in a preset delayed translation window.
[0012] In one embodiment, after the step of storing the process state data of the statement into a preset delay window, the method further includes: Before the preset delay time is reached, a new speech segment to be translated is received, and the new speech segment to be translated is concatenated with the statement in the delayed translation window to obtain a concatenated speech segment. If the confidence level of the boundary determination of the concatenated speech segment is higher than or equal to the confidence threshold, the translation result of the concatenated speech segment is output.
[0013] In one embodiment, the method further includes: The process state data associated with each candidate sentence boundary is stored in a preset buffer window, wherein the process state data includes the sentence boundary timestamp, audio segment, recognized text, multimodal fusion vector and translation result of the sentence; In response to a speech correction instruction, the sentence to be corrected is determined from the buffer window, and the translation result of the sentence to be corrected is corrected according to the speech correction instruction and the process status data of the sentence to be corrected.
[0014] In one embodiment, after the step of correcting the translation result of the sentence to be corrected based on the speech correction instruction and the process state data of the sentence to be corrected, the method further includes: Based on the correction information of the statement to be corrected, the decision-making process for the boundary determination model and / or the timing of the translation result output is optimized, wherein the correction information includes the correction content, number of corrections, and statement identifier of the statement to be corrected.
[0015] Furthermore, to achieve the above objectives, this application also proposes a speech translation device, which includes: A coherence confidence calculation module is used to convert the speech segment to be translated into a text sequence and calculate the semantic coherence confidence of the text sequence; The sentence boundary determination module is used to input the acoustic features of the speech segment to be translated and the semantic coherence confidence into the boundary determination model to obtain the candidate sentence boundaries in the speech segment to be translated and the boundary determination confidence corresponding to the candidate sentence boundaries. The boundary determination model is a neural network structure model and is obtained through pre-training. The translation result output module is used to determine the confidence level based on the boundary and to determine the timing of outputting the translation result of the sentence defined by the candidate sentence boundary.
[0016] In addition, to achieve the above objectives, this application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech translation method as described above.
[0017] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the speech translation method described above.
[0018] One or more technical solutions proposed in this application have at least the following technical effects: In this application, the speech segment to be translated is converted into a text sequence and semantic coherence confidence is calculated. Then, the acoustic features of the speech segment are input into a pre-trained neural network boundary determination model to obtain candidate sentence boundaries and their corresponding boundary determination confidence. Finally, the timing of the translation result output is determined based on the confidence. By introducing semantic coherence confidence combined with acoustic features, the boundary determination model can understand the speech content at the semantic level and more accurately determine sentence boundaries. Furthermore, unlike VAD, it does not require a sufficiently long silence to determine the end of a sentence. For semantically clear sentences, it can proactively and proactively determine sentence boundaries when the acoustic pause is indistinct or very short, thus significantly reducing the inherent delay in the judgment process. This solves the problem of poor overall real-time translation performance caused by inaccurate sentence boundary judgment and high judgment delay in speech translation, thereby improving the quality of real-time speech translation. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 This is a flowchart illustrating an embodiment of the speech translation method of this application. Figure 2 This is a schematic diagram of the implementation process of the speech translation method of this application in Embodiment 1; Figure 3 This is a flowchart illustrating Embodiment 2 of the speech translation method of this application; Figure 4 This is a schematic diagram of the module structure of the voice translation device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the speech translation method in the embodiments of this application.
[0022] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0023] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0024] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0025] With the development and advancement of technology, scenarios for real-time voice translation using voice translation software or devices are gradually increasing.
[0026] Currently, real-time speech translation typically relies on speech activity detection (VAD) technology to determine sentence boundaries, i.e., detecting where there is sound and where there is silence. It segments continuous speech streams based on pauses in speech, but it cannot understand semantic logic, cannot distinguish whether the pause is a breath within a sentence or a boundary between sentences, and is prone to misjudgment. Furthermore, in order to reduce missed detections, VAD needs to accumulate a certain amount of audio data to verify whether it is a true pause, which directly leads to judgment delays and makes it difficult to adapt to the low latency requirements of real-time translation.
[0027] In summary, how to avoid the problem of poor overall real-time translation performance caused by inaccurate sentence boundary judgment and high judgment delay in speech translation has become a technical problem that urgently needs to be solved in this field.
[0028] To address the aforementioned technical problems, this application's embodiments convert the speech segment to be translated into a text sequence and calculate its semantic coherence confidence score. Then, the acoustic features of the speech segment are input into a pre-trained neural network boundary determination model to obtain candidate sentence boundaries and their corresponding boundary determination confidence scores. Finally, the timing of the translation result output is determined based on the confidence scores. By introducing semantic coherence confidence scores in conjunction with acoustic features, the boundary determination model can understand the speech content at a semantic level and more accurately determine sentence boundaries. Furthermore, unlike VAD, it does not require a sufficiently long silence segment to determine the end of a sentence. For semantically clear sentences, it can proactively and proactively determine sentence boundaries when the acoustic pause is indistinct or very short, significantly reducing the inherent delay in the judgment process. This solves the problem of poor overall real-time translation performance caused by inaccurate sentence boundary judgment and high judgment delays in speech translation, thus improving the quality of real-time speech translation.
[0029] It should be noted that the executing entity in this embodiment can be an electronic device with data processing, network communication, and program execution functions. The following description uses a translation system as an example to illustrate this embodiment and the subsequent embodiments.
[0030] The following presents a first embodiment of the speech translation method of this application. (Refer to...) Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the speech translation method of this application.
[0031] In this embodiment, the speech translation method includes steps S10 to S30: Step S10: Convert the speech segment to be translated into a text sequence and calculate the semantic coherence confidence of the text sequence; It should be noted that the speech segment to be translated refers to the audio segment initially segmented from a continuous speech stream using speech activity detection technology; the text sequence is the text content obtained by converting speech signals using automatic speech recognition technology; and the semantic coherence confidence score is a probabilistic indicator used to quantitatively evaluate whether the current text sequence constitutes a complete and fluent sentence. This indicator is derived by analyzing the semantic integrity of the text sequence using a language model.
[0032] The system receives the input speech stream in real time, performs preliminary segmentation using speech activity detection technology to obtain the speech segments to be translated, and then uses a speech recognition engine to convert the speech segments to be translated into the corresponding text sequence. The system also calculates the semantic coherence confidence of the text sequence to characterize the completeness of the current text in terms of grammatical structure and semantic expression.
[0033] In one feasible embodiment, step S10 may include steps S101 to S102: Step S101: Input the text sequence into the language model to obtain the perplexity of the text sequence. The language model is a Transformer structure model and is obtained through pre-training. It should be noted that the language model refers to a pre-trained model based on the Transformer structure, such as lightweight language models like DistilBERT; perplexity is a measure of the uncertainty of a language model's prediction of the probability distribution of a given text sequence. The lower the perplexity, the more the text sequence conforms to the language rules of the model's training corpus.
[0034] The text sequence obtained from speech recognition is input into a pre-trained Transformer language model. The perplexity value of the text sequence is obtained through the forward computation of the model. The lower the perplexity value, the higher the semantic confidence.
[0035] Step S102: Convert the perplexity to obtain the semantic coherence confidence.
[0036] The perplexity is converted into semantic coherence confidence using a preset conversion function. The reverse conversion method is usually used, that is, the lower the perplexity, the higher the semantic coherence confidence. The conversion function can be an exponential decay function or a linear transformation function, and finally the semantic coherence confidence score is obtained.
[0037] For example, low perplexity (high semantic coherence confidence) means that the text sequence is more like a normal, complete sentence fragment; high perplexity (low semantic coherence confidence) means that the text sequence is awkward, illogical, or grammatically strange. For instance, the language model would give a high perplexity (low semantic coherence confidence) to the text sequence "I'm going to the office today" because "go to the office" is incomplete, while the text sequence "I'm going to the company today" would give a low perplexity (high semantic coherence confidence).
[0038] Therefore, by using the Transformer language model to evaluate the semantic coherence of the text sequence of the speech segment to be translated, semantic feature input is provided for subsequent sentence boundary determination, thereby improving the entire translation system's ability to understand semantic structure.
[0039] Step S20: Input the acoustic features and semantic coherence confidence of the speech segment to be translated into the boundary determination model to obtain the candidate sentence boundary and the boundary determination confidence corresponding to the candidate sentence boundary in the speech segment to be translated. The boundary determination model is a neural network structure model and is obtained through pre-training. It should be noted that acoustic features refer to physical feature parameters extracted from speech signals (i.e., the speech segments to be translated in this embodiment), including but not limited to pitch, short-time energy, and zero-crossing rate; the boundary determination model is a classification model based on a neural network structure, which is trained on a large amount of labeled data and can comprehensively determine sentence boundaries by combining acoustic features and semantic features; the candidate sentence boundary is the possible sentence boundary position identified by the boundary determination model, including the start and end timestamps of the sentence boundary; the boundary determination confidence is the degree of certainty of the model's determination result for each candidate sentence boundary.
[0040] Acoustic features are extracted from the speech segment to be translated. These acoustic features, along with semantic features (i.e., the previously calculated semantic coherence confidence score), are input into a pre-trained boundary determination model. This model, through comprehensive analysis of the acoustic and semantic features, outputs one or more candidate sentence boundaries and their corresponding boundary determination confidence scores. These candidate sentence boundaries may be consistent with the segmentation results of speech activity detection, or they may identify new sentence boundaries within the speech segment to be translated or negate existing sentence boundaries. If the model determines that the speech segment to be translated is incomplete, it either does not output candidate sentence boundaries, outputs a "no boundary" signal, or outputs a very low boundary determination confidence score.
[0041] In one feasible embodiment, steps A10 to A20 may be included before step S20: Step A10: Extract the acoustic features of the speech segment to be translated; Acoustic features such as pitch, short-time energy, and zero-crossing rate are extracted from the speech segments to be translated. Furthermore, the extracted acoustic features can be dimensionality reduced and standardized using Mel-Frequency Cepstral Coefficients (MFCC) or Perceptual Linear Prediction (PLP) methods to form acoustic features suitable for model input.
[0042] Step A20: The acoustic features and semantic coherence confidence are weighted and fused using an attention mechanism to obtain a multimodal fusion vector; Acoustic features and semantic coherence confidence are input into an attention network. The network performs a weighted sum of the two features using the learned weight matrix to generate a multimodal fusion vector that integrates acoustic and semantic features. The attention network used is a resource allocation mechanism based on neural networks, which can be normalized using the softmax function (normalized exponential function) to obtain the attention weight distribution of acoustic features and semantic coherence confidence.
[0043] Based on this, step S20 may include step S201: Step S201: Input the multimodal fusion vector into the boundary determination model to obtain the candidate sentence boundaries in the speech segment to be translated and the boundary determination confidence corresponding to the candidate sentence boundaries.
[0044] The multimodal fusion vectors obtained above are input into a pre-trained boundary determination model. This model analyzes the multimodal fusion vectors and outputs one or more candidate sentence boundaries and their corresponding boundary determination confidence scores. These candidate sentence boundaries may be consistent with the segmentation results of speech activity detection, or they may identify new sentence boundaries or negate existing sentence boundaries within the speech segment to be translated. If the model determines that the speech segment to be translated is incomplete, it will not output candidate sentence boundaries, or it will output a "no boundary" signal, or it will output a very low boundary determination confidence score.
[0045] Thus, by using the attention mechanism, an adaptive weighted fusion of acoustic and semantic features is achieved, enabling the boundary determination model to flexibly adjust the degree of dependence on different modal features according to specific situations. The generated multimodal fusion vector can more comprehensively reflect the boundary features of speech segments, thereby improving the accuracy and robustness of sentence boundary determination.
[0046] For example, in one feasible implementation, the multimodal fusion vector is nonlinearly transformed using the ReLU activation function in the hidden computation layer of the boundary determination model to learn the complex correlation pattern between acoustic features and semantic confidence. Then, the sigmoid activation function is used in the output layer of the model to output two types of information: candidate sentence boundaries in the speech segment to be translated and the boundary determination confidence corresponding to the candidate sentence boundaries. For example, the first neuron in the output layer outputs a boundary existence confidence p, indicating the probability that a sentence boundary exists at that position; subsequent neurons in the output layer output the offsets Δt_start and Δt_end of the boundary position relative to the start and end times of the speech segment to be translated; when the boundary existence confidence p is higher than a preset threshold, the start time of the speech segment to be translated + Δt_start is used as the start timestamp of the candidate sentence boundary, and the end time of the speech segment to be translated + Δt_end is used as the end timestamp of the candidate sentence boundary, while p is used as the boundary determination confidence of the candidate sentence boundary; if the boundary existence confidence p is lower than the preset threshold, it is determined that there is no reliable sentence boundary in the speech segment to be translated.
[0047] Step S30: Determine the timing of outputting the translation results of the sentences defined by the candidate sentence boundaries based on the confidence level of the boundary judgment.
[0048] The decision to output the translation result immediately or delay it depends on the confidence level of the boundary judgment. Specifically, when the confidence level of the boundary judgment is high, the translation output is triggered immediately to ensure real-time performance. When the confidence level of the boundary judgment is low, the output is delayed to collect subsequent speech information. The translation is then performed in combination with the subsequent speech information to ensure semantic integrity. The duration of the delay can be set based on the actual application scenario, but this embodiment does not impose a specific limitation on it.
[0049] Thus, by integrating acoustic features and semantic coherence confidence in multimodal analysis, the accuracy of sentence boundary judgment is improved. At the same time, the dynamic output decision mechanism based on confidence effectively balances the real-time and accuracy requirements of the translation system.
[0050] For example, in a feasible implementation scenario, such as Figure 2 As shown, the boundary determination model may output the following results: Case 1: The boundary determination result is the same as VAD.
[0051] Preliminary sentence boundary position: Output a timestamp pair, i.e., the [start timestamp, end timestamp] of the speech segment to be translated.
[0052] Semantic confidence score: The semantic confidence score of the speech segment to be translated is a score given by combining acoustic features and semantic coherence confidence to determine whether "this part is suitable as the end of a sentence".
[0053] Case 2: The boundary determination model identifies n sentences from a speech segment to be translated.
[0054] This is because when a speaker speaks quickly without a sufficiently long pause to trigger VAD segmentation, but the text contains multiple complete sentences, the boundary determination model will search for potential boundary points within the text stream. For example, for the text "...shopping mall. [here] I will tomorrow...", the model will identify the position after "shopping mall" as a candidate boundary, even though there may not be an obvious acoustic pause.
[0055] Preliminary sentence boundary position: Output multiple timestamp pairs. That is, [start TS1, end TS1], [start TS2, end TS2], ..., [start TSn, end TSn], where end TS1 and start TS2 may be adjacent or even overlap in time.
[0056] Semantic confidence score: Each sentence bounded by a sentence boundary has an independent semantic confidence score. For example, if the text sequence of the semantic segment to be translated is "I'm going to the mall today. I'm going to school tomorrow.", the model analysis shows that the confidence score of "I'm going to the mall" is 0.9, while the confidence score of "I'm going to school tomorrow" is 0.7 due to unclear pronunciation.
[0057] Scenario 3: VAD considers the speech segment to be translated to be a complete sentence, but the boundary judgment model considers it to be an incomplete sentence.
[0058] Initial sentence boundary position: Do not output candidate sentence boundaries, or output a "no boundary" signal, or output a candidate sentence boundary but give a very low semantic confidence score.
[0059] Semantic confidence score: Output a low score below the confidence threshold, such as 0.3.
[0060] Subsequent actions: Refuse to output the translation result of the speech segment to be translated at the current point, and continue to wait and collect subsequent speech streams. This process will continue until: enough subsequent speech content is collected so that a new, later candidate boundary point obtains a high confidence score; or, the waiting time exceeds the maximum delay threshold, and the translation system is forced to output the translation at the current undesirable position to ensure the real-time performance of the translation.
[0061] Furthermore, if the boundary judgment model identifies multiple sentences in a speech segment to be translated, but the confidence of the earlier sentences is low while that of the later sentences is high, this may be due to external noise interference or unclear user expression, rather than true semantic incompleteness. In this case, the earlier and later sentences can be translated and output together to avoid a coherent dialogue being abruptly cut off due to local quality issues.
[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, step S30 may include steps S301 to S303: Step S301: Compare the boundary determination confidence level with a preset confidence threshold. It should be noted that the confidence threshold is a pre-set critical value used to determine whether the sentence boundary is reliable, and can be set according to the requirements of real-time performance and accuracy in the application scenario.
[0063] The confidence level of the boundary determination output by the boundary determination model is compared with the preset confidence threshold. The confidence threshold can be adjusted according to different application scenarios. For example, it can be set to 0.8 for real-time conferencing scenarios and 0.7 for cross-border interaction scenarios.
[0064] Step S302: If the confidence level of the boundary determination is higher than or equal to the confidence threshold, then output the translation result of the sentence defined by the candidate sentence boundary. When the confidence level of the boundary judgment is higher than or equal to the confidence threshold, the translation result of the sentence segment defined by the candidate sentence boundary is immediately output to ensure low-latency output under high confidence conditions.
[0065] It is worth mentioning that the translation result of the semantic segment to be translated can be obtained immediately after the segment is acquired by the translation engine, and the translation result can be output after the confidence level is determined to be higher than or equal to the confidence threshold. Alternatively, the translation result can be obtained by the translation engine after the confidence level is determined to be higher than or equal to the confidence threshold. In this embodiment, the time node for performing speech translation is not specifically limited.
[0066] Step S303: If the confidence level of the boundary determination is lower than the confidence level threshold, the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in the preset delayed translation window.
[0067] It should be noted that the semantically incomplete state is an internal system classification label used to identify that the current statement is not yet complete; the delayed translation window is a data buffer used to temporarily store statements that are yet to be completed.
[0068] When the confidence level of the boundary determination is lower than the confidence level threshold, the system will not immediately trigger the translation result output. Instead, it will mark the statement as semantically incomplete and store all its process state data (including the statement's sentence boundary timestamp, audio segment, recognized text, multimodal fusion vector, and translation result) in a preset delayed translation window for subsequent processing.
[0069] Thus, by comparing confidence thresholds and using a delayed translation mechanism, effective filtering and temporary storage of low-quality sentence boundaries are achieved, avoiding semantic confusion caused by prematurely outputting incomplete sentence translations, and providing the possibility for subsequent sentence completion translation processing, thereby achieving a balance between translation quality and output delay.
[0070] In one feasible embodiment, step S30 may include steps S304 to S306: Step S304: Compare the boundary determination confidence with the preset confidence threshold, and determine whether the boundary text sequence of the statement defined by the candidate sentence boundary conforms to the preset grammatical integrity rule. It should be noted that grammatical integrity rules refer to a set of grammatical rules used to assess the structural integrity of sentences, including but not limited to whether a sentence ends with a complete terminator (such as a period or question mark), whether it contains an incomplete clause structure, and whether it ends with a conjunction or punctuation mark indicating enumeration or contrast.
[0071] The confidence score of the boundary judgment output by the boundary judgment model is compared with the preset confidence score threshold. The confidence score threshold can be adjusted according to different application scenarios. For example, it can be set to 0.8 in real-time conferencing scenarios and 0.7 in cross-border interaction scenarios. At the same time, the grammatical integrity analysis is performed on the text sequence corresponding to the statement defined by the candidate sentence boundary (i.e., the boundary text sequence) to check whether it conforms to the preset grammatical integrity rules.
[0072] Thus, the accuracy of the output decision is ensured through a dual verification mechanism. For example, for the boundary text sequence "e.g. apple, banana", even if its boundary judgment confidence is high, it will still be judged as an incomplete sentence because it ends with an enumeration conjunction, which does not conform to the grammatical integrity rule.
[0073] Step S305: If the confidence level of the boundary determination is higher than or equal to the confidence threshold, and the boundary text sequence conforms to the grammatical integrity rule, then output the translation result of the sentence defined by the candidate sentence boundary. If the confidence level of the boundary judgment is higher than or equal to the confidence threshold, and the boundary text sequence conforms to the grammatical integrity rule, then the translation result of the sentence defined by the candidate sentence boundary will be output. That is, the translation result of the sentence fragment defined by the candidate sentence boundary will be output immediately only when both conditions are met, so as to ensure the dual guarantee of semantic reliability and grammatical integrity of the output sentence.
[0074] Step S306: If the confidence level of the boundary determination is lower than the confidence threshold, or the boundary text sequence does not conform to the grammatical integrity rule, then the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in the preset delayed translation window.
[0075] When the confidence level of the boundary determination is lower than the confidence threshold, or when the boundary text sequence does not conform to the grammatical integrity rule, the system will not immediately trigger the translation result output. Instead, it will mark the statement as semantically incomplete and store all its process state data (including the sentence boundary timestamp, audio segment, recognized text, multimodal fusion vector and translation result, etc.) in a preset delayed translation window for subsequent processing.
[0076] Thus, by employing a dual criterion mechanism of confidence threshold comparison and grammatical integrity check, both the semantic reliability of the statement and the integrity of the grammatical structure are considered. This effectively avoids the problem of premature output of incomplete statements that may result from relying solely on confidence, thereby ensuring the accuracy of the translation output at a deeper level.
[0077] In one feasible embodiment, after step S303 or step S306, steps S307 to S308 may also be included: Step S307: Before the preset delay time is reached, a new speech segment to be translated is received, and the new speech segment to be translated is concatenated with the statement in the delayed translation window to obtain a concatenated speech segment. It should be noted that the preset delay duration refers to the maximum time range during which the system is allowed to wait for subsequent voice data of statements marked as semantically incomplete. This delay duration can be set based on the application scenario. For example, the delay duration for real-time conferencing scenarios is set to 500ms, and the delay duration for cross-border interaction scenarios is set to 800ms.
[0078] Within the set maximum delay time range, it continuously receives subsequent input speech segments, that is, it receives new speech segments to be translated, and splices these newly received speech segments with the sentences temporarily stored in the delayed translation window at the audio level to form a spliced speech segment containing more semantic context.
[0079] Step S308: If the confidence level of the boundary determination of the concatenated speech segment is higher than or equal to the confidence threshold, then the translation result of the concatenated speech segment is output.
[0080] The spliced speech segment is re-extracted for acoustic features, speech recognition, semantic coherence confidence calculation, and boundary determination model analysis. When the newly obtained boundary determination confidence is higher than or equal to the preset threshold, the translation output of the entire spliced speech segment is immediately triggered.
[0081] Thus, by dynamically splicing speech segments and re-evaluating semantic confidence, the translation system can automatically correct semantic incompleteness caused by inaccurate initial sentence boundary judgment, and make full use of subsequent speech information to improve semantic understanding, thereby improving the translation accuracy of long and complex sentences while ensuring real-time performance.
[0082] In one feasible embodiment, such as Figure 3 As shown, the speech translation method of this application may further include steps B10 to B20: Step B10: Store the process state data associated with the sentences defined by the boundaries of each candidate sentence in a preset buffer window. The process state data includes the sentence boundary timestamp, audio segment, recognized text, multimodal fusion vector, and translation result. It should be noted that the buffer window is a data storage area organized in chronological order, used to store information related to statements processed within a recent period (e.g., within 10 minutes); process status data refers to various intermediate data generated during the processing of a statement, including the statement's sentence boundary timestamps, audio segments, recognized text, multimodal fusion vectors, and translation results.
[0083] The system maintains a data storage structure based on a linked list or circular buffer, storing the process state data of each candidate sentence boundary in chronological order, and assigning a unique identifier to each data unit. Only one candidate sentence boundary determined by the boundary judgment model and its process state data are stored in a data unit. At the same time, when the stored data in the buffer window exceeds the preset duration or capacity threshold, the earliest data is automatically removed in chronological order.
[0084] Step B20: In response to the speech correction instruction, determine the sentence to be corrected from the buffer window, and correct the translation result of the sentence to be corrected according to the speech correction instruction and the process status data of the sentence to be corrected.
[0085] It should be noted that voice correction commands are translation correction requests issued by users through voice commands or other feasible methods. Voice correction commands are in the form of "correct [incorrect content keywords]" or "modify [timestamp / fragment description]". The statement to be corrected is the target statement that needs to be translated. The statement to be corrected can be a statement that has already output a translation result or a statement that has not yet output a translation result.
[0086] When a user submits a translation correction request, the system matches the data in the buffer window, finds the corresponding sentence to be corrected, and calls the process status data of the sentence to be corrected to re-execute the recognition and translation process, generating the corrected translation result.
[0087] Thus, by establishing a process state data buffering mechanism for statements, efficient location and local correction of translation errors were achieved, improving the system's interactive performance and user experience.
[0088] In one feasible embodiment, such as Figure 3 As shown, step B20 may be followed by step B30: Step B30: Based on the correction information of the statement to be corrected, optimize the decision-making process for the boundary determination model and / or the timing of the translation result output. The correction information includes the correction content, number of corrections, and statement identifier of the statement to be corrected.
[0089] The system collects correction information generated by each correction operation, including the identifier of the corrected statement, the number of corrections, and a comparison of the content before and after correction. Then, it synchronizes this information to the boundary decision model and the output decision module through a feedback interface. The output decision module is used to execute the output control logic based on confidence threshold comparison. If a statement or a type of statement with certain features is corrected multiple times, the system will automatically lower the initial semantic confidence threshold of that type of statement or assign a higher attention weight to that type of feature in the boundary decision model, thereby adopting a more cautious strategy in subsequent processing, such as extending the observation time or raising the output threshold.
[0090] For example, the system continuously collects correction information through a dedicated feedback interface and stores it in a historical correction database. For the boundary judgment model, the system performs feature backtracking analysis based on the correction records to identify common patterns in the acoustic features or semantic fusion vectors of frequently corrected sentences. For instance, if sentences ending with specific conjunctions (such as "but" or "however") are found to be corrected multiple times, the system will add a penalty term for such patterns to the model's loss function or assign them higher weights in the attention mechanism, making the model's subsequent judgments of such boundaries more accurate. Regarding the decision logic for the timing of translation result output, the system establishes a correlation rule between correction frequency and confidence threshold. When the correction frequency of a certain type of sentence (identified by the clustering of its semantic fusion vector in the feature space) exceeds the preset number of corrections, the system will automatically increase the confidence threshold referenced when making the output decision. This essentially raises the threshold for outputting the translation result and simultaneously extends its delay waiting time, forcing the system to collect more sentence information for this type of sentence before making an output decision.
[0091] Thus, by establishing a closed-loop feedback path from the correction result to the front-end processing, the system can continuously learn from the user's correction behavior, adaptively optimize the boundary judgment model parameters and output decision logic, and continuously improve the translation accuracy of the system.
[0092] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the speech translation method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.
[0093] This application also provides a voice translation device; please refer to... Figure 4 The voice translation device includes: The coherence confidence calculation module 10 is used to convert the speech segment to be translated into a text sequence and calculate the semantic coherence confidence of the text sequence. The sentence boundary determination module 20 is used to input the acoustic features and semantic coherence confidence of the speech segment to be translated into the boundary determination model to obtain the candidate sentence boundaries in the speech segment to be translated and the boundary determination confidence corresponding to the candidate sentence boundaries. The boundary determination model is a neural network structure model and is obtained through pre-training. The translation result output module 30 is used to determine the confidence level based on the boundary and to determine the timing of outputting the translation result of the sentence defined by the candidate sentence boundary.
[0094] Optionally, the coherence confidence calculation module 10 is also used for: The text sequence is input into the language model to obtain the perplexity of the text sequence. The language model is a Transformer structure model and is obtained through pre-training. The perplexity level is converted to obtain the semantic coherence confidence level.
[0095] Optionally, the sentence boundary determination module 20 is also used for: Extract the acoustic features of the speech segment to be translated; An attention mechanism is used to weight and fuse acoustic features and semantic coherence confidence to obtain a multimodal fusion vector; By inputting the multimodal fusion vector into the boundary determination model, the candidate sentence boundaries in the speech segment to be translated and the boundary determination confidence corresponding to the candidate sentence boundaries are obtained.
[0096] Optionally, the translation result output module 30 is also used for: The boundary determination confidence level is compared with a preset confidence threshold. If the confidence level of the boundary determination is higher than or equal to the confidence threshold, the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary determination is lower than the confidence threshold, the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in the preset delayed translation window.
[0097] Optionally, the translation result output module 30 is also used for: The boundary determination confidence is compared with the preset confidence threshold, and it is determined whether the boundary text sequence of the statement defined by the candidate sentence boundary conforms to the preset grammatical integrity rule. If the confidence level of the boundary judgment is higher than or equal to the confidence threshold, and the boundary text sequence conforms to the grammatical integrity rule, then the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary judgment is lower than the confidence threshold, or if the boundary text sequence does not conform to the grammatical integrity rule, the statement defined by the candidate sentence boundary will be marked as semantically incomplete, and the statement will be stored in the preset delayed translation window.
[0098] Optionally, the translation result output module 30 is also used for: Before the preset delay time is reached, a new speech segment to be translated is received, and the new speech segment to be translated is concatenated with the statement in the delayed translation window to obtain a concatenated speech segment. If the confidence level of the boundary determination of the concatenated speech segment is higher than or equal to the confidence threshold, the translation result of the concatenated speech segment is output.
[0099] Optionally, the voice translation device also includes a correction module (not shown), which is used for: The process state data associated with each candidate sentence boundary is stored in a preset buffer window. The process state data includes sentence boundary timestamps, audio segments, recognized text, multimodal fusion vectors, and translation results. In response to a speech correction command, the sentence to be corrected is determined from the buffer window, and the translation result of the sentence to be corrected is corrected according to the speech correction command and the process status data of the sentence to be corrected.
[0100] Optionally, the correction module is also used for: Based on the correction information of the statement to be corrected, the decision-making process for the boundary determination model and / or the timing of translation result output is optimized. The correction information includes the correction content, number of corrections, and statement identifier of the statement to be corrected.
[0101] This application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the status indication method described above.
[0102] The following is for reference. Figure 5 The diagram illustrates an electronic device suitable for implementing embodiments of this application. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0103] like Figure 5 As shown, the electronic device may include a processing unit 1001 (e.g., a DSP processor), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: an input device 1007 including, for example, a microphone, an accelerometer, etc.; an output device 1008 including, for example, a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 allows the electronic device to exchange data wirelessly or via wired communication with other devices. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. It can be implemented alternatively or with more or fewer systems.
[0104] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0105] Compared with the prior art, the beneficial effects of the electronic device provided in this application embodiment are the same as those of the speech translation method provided in the above embodiment, and other technical features of the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0106] It should be understood that the various parts disclosed in the embodiments of this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0107] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0108] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the speech translation method in the above embodiments.
[0109] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0110] The aforementioned computer-readable storage medium may be included in an electronic device or may exist independently without being assembled into an electronic device.
[0111] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the functions defined in the methods of the embodiments disclosed in this application.
[0112] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0113] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0114] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0115] The readable storage medium provided in this application embodiment is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-described speech translation method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application embodiment are the same as the beneficial effects of the speech translation method provided in the above-described embodiments, and will not be repeated here.
[0116] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the status indication method described above. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the status indication method provided in the above embodiments, and will not be repeated here.
[0117] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A speech translation method, characterized in that, The speech translation method includes: The speech segment to be translated is converted into a text sequence, and the semantic coherence confidence of the text sequence is calculated; The acoustic features of the speech segment to be translated and the semantic coherence confidence are input into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated. The boundary determination model is a neural network structure model and is obtained through pre-training. Based on the confidence level determined by the boundary, the timing for outputting the translation result of the sentence defined by the candidate sentence boundary is determined.
2. The speech translation method as described in claim 1, characterized in that, The step of calculating the semantic coherence confidence of the text sequence includes: The text sequence is input into a language model to obtain the perplexity of the text sequence, wherein the language model is a Transformer structure model and is obtained through pre-training; The perplexity level is converted to obtain the semantic coherence confidence level.
3. The speech translation method as described in claim 1, characterized in that, Before the step of inputting the acoustic features of the speech segment to be translated and the semantic coherence confidence into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated, the method further includes: Extract the acoustic features of the speech segment to be translated; An attention mechanism is used to weight and fuse the acoustic features and the semantic coherence confidence to obtain a multimodal fusion vector; The step of inputting the acoustic features of the speech segment to be translated and the semantic coherence confidence into the boundary determination model to obtain the candidate sentence boundaries and the boundary determination confidence corresponding to the candidate sentence boundaries in the speech segment to be translated includes: The multimodal fusion vector is input into the boundary determination model to obtain the candidate sentence boundaries and the corresponding boundary determination confidence scores in the speech segment to be translated.
4. The speech translation method as described in claim 1, characterized in that, The step of determining the timing of outputting the translation result of the statement defined by the candidate sentence boundary based on the confidence level of the boundary includes: The boundary determination confidence level is compared with a preset confidence threshold. If the confidence level of the boundary determination is higher than or equal to the confidence threshold, then the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary determination is lower than the confidence threshold, the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in a preset delayed translation window.
5. The speech translation method as described in claim 1, characterized in that, The step of determining the timing of outputting the translation result of the statement defined by the candidate sentence boundary based on the confidence level of the boundary includes: The confidence level of the boundary determination is compared with a preset confidence threshold, and it is determined whether the boundary text sequence of the statement defined by the candidate sentence boundary conforms to the preset grammatical integrity rule. If the confidence level of the boundary determination is higher than or equal to the confidence threshold, and the boundary text sequence conforms to the grammatical integrity rule, then the translation result of the sentence defined by the candidate sentence boundary is output. If the confidence level of the boundary determination is lower than the confidence threshold, or if the boundary text sequence does not conform to the grammatical integrity rule, then the statement defined by the candidate sentence boundary is marked as semantically incomplete, and the statement is stored in a preset delayed translation window.
6. The speech translation method as described in any one of claims 4 or 5, characterized in that, After the step of storing the process state data of the statement into a preset delay window, the method further includes: Before the preset delay time is reached, a new speech segment to be translated is received, and the new speech segment to be translated is concatenated with the statement in the delayed translation window to obtain a concatenated speech segment. If the confidence level of the boundary determination of the concatenated speech segment is higher than the confidence threshold, the translation result of the concatenated speech segment is output.
7. The speech translation method as described in claim 1, characterized in that, The method further includes: The process state data associated with each candidate sentence boundary is stored in a preset buffer window, wherein the process state data includes the sentence boundary timestamp, audio segment, recognized text, multimodal fusion vector and translation result of the sentence; In response to a speech correction instruction, the sentence to be corrected is determined from the buffer window, and the translation result of the sentence to be corrected is corrected according to the speech correction instruction and the process status data of the sentence to be corrected.
8. The speech translation method as described in claim 7, characterized in that, After the step of correcting the translation result of the sentence to be corrected based on the speech correction instruction and the process state data of the sentence to be corrected, the method further includes: Based on the correction information of the statement to be corrected, the decision-making process for the boundary determination model and / or the timing of the translation result output is optimized, wherein the correction information includes the correction content, number of corrections, and statement identifier of the statement to be corrected.
9. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech translation method as described in any one of claims 1 to 8.
10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the speech translation method as described in any one of claims 1 to 8.