Real-time speech translation method and system based on large model

Abstract

The embodiment of the invention provides a real-time speech translation method and system based on a large model, and belongs to the technical field of natural languages. The method comprises the following steps: acquiring original voice data and performing feature extraction to obtain a voice frame feature sequence; and inputting the voice frame feature sequence into a rhythm boundary recognition model to obtain a boundary confidence sequence. Performing phrase segmentation on the original voice data according to the boundary confidence sequence and the segmentation threshold to obtain a candidate phrase set; performing speech recognition according to the candidate phrase set to obtain a text transcription sequence set; wherein the text transliteration sequence set comprises a plurality of text transliteration sequences. And scoring the text transcription sequence according to the semantic density scoring function to obtain a semantic density score. And selecting in the text transcription sequence set according to the semantic density score to obtain an input text sequence set. And inputting the input text sequence set into the translation model to obtain a translation result, thereby improving the accuracy of real-time speech translation.

Description

Technical Field

[0001] The present application relates to the field of natural language technology, and in particular to a real-time speech translation method and system based on a large model. Background Art

[0002] Speech translation systems have evolved from early statistically or rule-based two-stage architectures to end-to-end systems integrating speech recognition (ASR), natural language processing (NLP), and neural machine translation (NMT). Representative solutions, such as Whisper and SeamlessM4T, already possess the capabilities to recognize, translate, and even re-speechize multilingual speech, and are widely used in scenarios such as simultaneous interpretation in conferences, real-time captioning, and cross-border communications. However, existing translation systems still face key technical shortcomings in real-world continuous, real-time speech translation. Most translation methods rely on static sliding windows, sentence buffering, or time-controlled input organization. These methods struggle to adapt to the fluctuating speech rate, unstructured pauses, and semantic changes found in natural speech, often leading to mistranslations, syntactical gaps, or repeated translations. Therefore, improving the accuracy of real-time speech translation has become a pressing technical challenge. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a real-time speech translation method and system based on a large model, aiming to improve the accuracy of real-time speech translation.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a real-time speech translation method based on a large model, the method comprising: Get the original voice data; Extracting features from the original speech data to obtain a speech frame feature sequence; Inputting the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; Segmenting the original speech data according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; Performing speech recognition on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes a plurality of text transcription sequences; Scoring the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; Selecting from the text transcription sequence set according to the semantic density score to obtain an input text sequence set; The input text sequence set is input into a preset translation model to obtain a translation result.

[0005] In some embodiments, inputting the input text sequence into a preset translation model to obtain a translation result includes: Scoring the input text sequence according to a preset trigger scoring function to obtain a trigger score; If the trigger score is greater than a preset trigger threshold, the input text sequence set is input into a preset translation model to obtain a translation result.

[0006] In some embodiments, scoring the input text sequence according to a preset trigger scoring function to obtain a trigger score includes: The trigger scoring function is: ; in, Indicates trigger rating, represents the semantic density score corresponding to the input text sequence, Indicates the number of words in the th text transcription sequence, and represent weighting parameters respectively.

[0007] In some embodiments, extracting features based on the original speech data to obtain a speech frame feature sequence includes: Performing division processing according to the original voice data to obtain a voice frame stream; Extracting features based on the speech frame stream to obtain short-time energy features, pitch fundamental frequency features, zero-crossing rate features, and cepstral coefficient features; The speech frame feature sequence is obtained by splicing the short-time energy feature, the pitch fundamental frequency feature, the zero-crossing rate feature and the cepstral coefficient feature; In some embodiments, selecting from the set of text transcription sequences according to the semantic density score to obtain the set of input text sequences includes: Scoring the text transcription sequence according to a preset retention scoring function and the semantic density score to obtain a retention score; The text transcription sequence set is selected according to the retention score and a preset upper limit of the input length to obtain the input text sequence set.

[0008] In some embodiments, after selecting from the text transcription sequence set according to the semantic density score to obtain the input text sequence set, the method further includes: Obtain updated candidate segments; Performing speech recognition based on the updated candidate segments to obtain an updated text transcription sequence; Scoring the updated text transcription sequence according to the retention scoring function to obtain an updated retention score; Obtaining the input text sequence with the lowest retention score in the input text sequence set to obtain a target input text sequence; If the updated retention score is greater than the retention score of the target input text sequence, the updated text transcription sequence is replaced with the target input text sequence to obtain an updated input text sequence set.

[0009] In some embodiments, scoring the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score includes: The semantic density scoring function is: ; in, Indicates the Semantic density scores of text transcription sequences, Indicates the The result of a text transcription sequence processed by a specific semantic encoder, Indicates the The result of a text transcription sequence processed by a specific semantic encoder, Indicates the The number of words in the text transcription sequence, represents the mean number of words in the adjacent text transcription sequence, Indicates the The mean boundary confidence across frames of the text transcription sequence, 、 and are weight coefficients respectively.

[0010] To achieve the above objectives, a second aspect of the embodiments of the present application provides a real-time speech translation system based on a large model, the system comprising: An acquisition module is used to obtain original voice data; An extraction module, configured to extract features from the original speech data to obtain a speech frame feature sequence; A boundary recognition module, configured to input the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; a segmentation module, configured to segment the original speech data into segments according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; A speech recognition module, configured to perform speech recognition based on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes a plurality of text transcription sequences; A scoring module, configured to score the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; A selection module, configured to select from the text transcription sequence set according to the semantic density score to obtain an input text sequence set; The translation module is used to input the input text sequence set into a preset translation model to obtain a translation result.

[0011] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0012] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0013] The present application proposes a large-scale model-based real-time speech translation method and system, which obtains original speech data, performs feature extraction based on the original speech data, and obtains a speech frame feature sequence. The speech frame feature sequence is input into a preset prosody boundary recognition model to obtain a boundary confidence sequence. The original speech data is segmented according to the boundary confidence sequence and a preset segmentation threshold to obtain a candidate segment set. Speech recognition is performed on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes multiple text transcription sequences. The text transcription sequence is scored according to a preset semantic density scoring function to obtain a semantic density score. The text transcription sequence set is selected according to the semantic density score to obtain an input text sequence set. The input text sequence set is input into a preset translation model to obtain a translation result, thereby improving the accuracy of real-time speech translation. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is a flow chart of a large model-based real-time speech translation method provided in an embodiment of the present application; Figure 2 yes Figure 1 Flowchart of step S102 in FIG. Figure 3 yes Figure 1 Flowchart of step S107 in FIG. Figure 4 yes Figure 1 Flowchart of step S108 in FIG. Figure 5 is a flowchart of a large model-based real-time speech translation method provided by another embodiment of the present application; Figure 6 Schematic diagram of the structure of a real-time speech translation system based on a large model provided in an embodiment of the present application; Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0016] It should be noted that although the system diagrams illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the system or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0017] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0018] Existing translation systems still face key technical shortcomings in real-world continuous, real-time speech translation tasks. Most translation methods rely on static sliding windows, sentence buffering, or time-controlled input strategies. These methods struggle to adapt to the fluctuating speech rates, unstructured pauses, and sudden semantic shifts found in natural speech. This often leads to mistranslations, syntactical discontinuities, or repeated translations, resulting in low accuracy in real-time speech translation.

[0019] Based on this, the embodiment of the present application provides a real-time speech translation method and system based on a large model, which aims to introduce a rhythm-aware prosodic boundary recognition mechanism, use acoustic features such as pitch, energy and spectrum to model the natural pause positions in the speech signal, and achieve accurate segmentation without text support. At the same time, a semantic density modeling method that integrates semantic mutation scoring and boundary confidence adjustment factor is proposed, which can quantify the content value of each transcribed segment in the context and support precision-controlled segment optimization. On this basis, a flexible context caching mechanism combining token budget and semantic scoring is constructed to dynamically generate a set of high-value segments that meet the input constraints of the large model, thereby improving the semantic carrying capacity of the context structure. Finally, a translation trigger control strategy driven by semantic density trend and structural change amplitude is designed to achieve response optimization of the system at the translation call level, effectively avoiding model calculation redundancy and translation granularity fragmentation problems. Through the synergistic combination of the above four modules, the present invention forms a real-time speech translation input construction scheme with unified structural decoupling, signal fusion, input drive and policy control, which is suitable for large-model intelligent translation tasks in complex speech environments and improves the accuracy of real-time speech translation.

[0020] The large-model-based real-time speech translation method and system provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the large-model-based real-time speech translation method in the embodiments of the present application is described.

[0021] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0022] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] The real-time speech translation method based on a large model provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running on a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the real-time speech translation method based on a large model, etc., but is not limited to the above forms.

[0024] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0025] Please refer to Figure 1 , Figure 1 is a flowchart of a large model-based real-time speech translation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S108.

[0026] Step S101, obtaining original voice data; Step S102, performing feature extraction based on the original speech data to obtain a speech frame feature sequence; Step S103, inputting the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; Step S104, segmenting the original speech data into segments according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; Step S105: performing speech recognition based on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes multiple text transcription sequences; Step S106, scoring the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; Step S107, selecting from the text transcription sequence set according to the semantic density score to obtain an input text sequence set; Step S108: inputting the input text sequence set into a preset translation model to obtain a translation result.

[0027] In step S101 of some embodiments, the raw voice data is the original speech uttered by the user in a real communication scenario, collected by a hardware terminal deployed on the user's side. Examples include a smart conference microphone (such as the Jabra Speak 750), a voice assistant hardware module (such as a device equipped with a MEMS microphone array), or a matching remote call terminal. The sampling rate can be set to 16kHz, 16-bit linear quantization format, and a mono channel. The sampled data forms continuous raw voice data.

[0028] See also Figure 2 In some embodiments, step S102 further includes but is not limited to steps S201 to S203: Step S201, dividing and processing the original voice data to obtain a voice frame stream; Step S202, performing feature extraction based on the speech frame stream to obtain short-time energy features, pitch fundamental frequency features, zero-crossing rate features, and cepstral coefficient features; Step S203, combining the short-time energy feature, the pitch fundamental frequency feature, the zero-crossing rate feature, and the cepstral coefficient feature to obtain a speech frame feature sequence; In step S201 of some embodiments, the system has a built-in low-latency audio buffer module to divide and process the original voice data. For example, the original voice data (sampled signal) is divided into a frame every 25 milliseconds and continuously slides in steps of 10 milliseconds to form a voice frame stream. For example, a 3-second speech segment will be divided into approximately 296 frames, each containing 400 samples, with 15 milliseconds of overlap between frames.

[0029] In step S202 of some embodiments, the speech frame stream includes a plurality of speech frames , each speech frame All of them will be sent to the feature extraction module to extract four types of rhythm-sensitive features, which will form the basic signals of the subsequent context control strategy. The first type is short-term energy features. , which is used to measure the strength of the current speech segment. This feature is obtained by averaging the squares of the amplitudes of all sampling points in the frame, as shown in the following formula (1): , (1) in, Represents the short-time energy characteristics, Represents a speech frame The first The second type is the pitch fundamental frequency feature. , used to reflect the tone trend, this value is obtained by Perform a fast Fourier transform (FFT) and search for the main frequency peak in the range of 60Hz to 500Hz, as shown in the following formula (2): , (2) in, Indicates the fundamental frequency characteristics of the pitch. Indicates the speech frame after FFT transformation In frequency The range of 60Hz to 500Hz is selected based on engineering experience of covering the pitch range of adult male and female voices.

[0030] The third type is the zero-crossing rate feature , used to identify sudden change areas such as friction sounds and plosive sounds. The specific method is to check whether the symbols of adjacent sampling points have changed, and count the proportion of the number of changes to the frame length. The fourth category is the 13-dimensional Mel frequency cepstral coefficient feature , using the standard speech signal processing process: first the speech frame Pre-emphasis and window function processing (such as Hamming window) are performed, and then FFT is performed to obtain the power spectrum. Then, it is filtered through 40 Mel filters, and discrete cosine transform (DCT) is applied to the filtered logarithmic energy. Finally, the first 13-dimensional cepstrum components are retained as the spectral structure description of the frame.

[0031] In step S203 of some embodiments, the short-time energy feature, pitch fundamental frequency feature, zero-crossing rate feature and cepstral coefficient feature are uniformly normalized and then concatenated into a speech frame feature vector , each , corresponding to energy, pitch, zero-crossing rate and 13-dimensional cepstral coefficients respectively. All speech frame feature vectors are combined in sequence to form a speech frame feature sequence, as shown in the following formula (3): , (3) At the same time, the system shifts the time according to the frame, e.g. Calculate the start timestamp of each frame in milliseconds. The timestamp of the frame is , forming a time series, as shown in the following formula (4): , (4) In real-world applications, for example, a user says in a meeting, "Hello everyone, today's topic is project progress." This sentence lasts approximately 2.1 seconds, and the system generates approximately 210 frames. The speech frame feature sequence is a 210×16 matrix, and the system also outputs 210 corresponding frame start times in milliseconds. The entire feature extraction process is performed locally, with a single-frame processing latency of less than 5 milliseconds, ensuring support for real-time system operation.

[0032] Through steps S201 to S203, the raw speech data uttered by the user in a real-world communication scenario is converted in real time into a rhythmically sensitive speech frame feature sequence, providing an operational input foundation for subsequent prosodic boundary identification. Unlike preprocessing methods in common speech recognition systems that prioritize modeling speech content, the feature design in this step specifically targets the implicit rhythmic structure and potential semantic pause locations within speech, providing signal-level support for determining the core question of "where to begin translation."

[0033] In step S103 of some embodiments, the prosodic boundary recognition model is a lightweight neural network module for prosodic feature modeling. The input is fed into a pre-trained rhythm boundary recognition model to identify the rhythm mutation location. The rhythm boundary recognition model consists of two structures: The one-dimensional convolutional encoder (1D-CNN) consists of two layers. The first layer has a convolution kernel size of 5, a stride of 1, and 32 output channels, which is used to locally capture short-term fluctuations in energy and pitch. The second layer has a convolution kernel size of 3 and 64 output channels, which further compresses features and enhances rhythmic pattern recognition. All convolution layers are followed by ReLU and BatchNorm operations.

[0034] Bidirectional GRU module (Bi-GRU), with input as the temporal feature sequence output by CNN, hidden unit dimension 64, models context change trends in forward and backward directions respectively, and is used to understand the frame-level rhythm sequence at the sequence level. The output hidden state Corresponding frame The comprehensive rhythm representation of .

[0035] Next, each It is fed into a single-layer fully connected classifier, which outputs the probability value of whether the frame is a "prosodic boundary point" , as shown in the following formula (5): , (5) in, represents the linear transformation weight, represents the bias term, Represents the Sigmoid function, the output of which constitutes the frame-level boundary confidence sequence , reflecting the probability that each frame is a semantic pause or sentence change position.

[0036] In step S104 of some embodiments, in order to Converted into actual segmentation points, the system sets a dynamic segmentation threshold (The recommended value is 0.5), and a sliding window strategy is used to identify local maxima. satisfy And it is the largest in the neighborhood, then As the segmentation point. The system also introduces a minimum segment length limit (such as 300ms) to avoid over-fragmentation. For example, if the user says "Please open the meeting record and then start the report", the system detects a combined decrease in energy and pitch between "record" and "then". The value reaches 0.81, corresponding to a timestamp of 1220ms, which meets the segmentation condition. The system uses this time point as the boundary for segmenting the speech.

[0037] After the segment division is completed, the system will convert the original speech data into Cut into several non-overlapping candidate segments to obtain a set of candidate segments . Each Indicates the time interval in the original speech data A collection of continuous speech frames, the length of which is generally controlled between 0.5 seconds and 3 seconds, is used for subsequent text transcription and semantic modeling.

[0038] The system uses a pre-trained model to complete the boundary recognition task. During the training process, the pause points marked in the public Chinese speech emotion corpus (such as AISHELL-3) are used as supervision targets, and the parameters are obtained through frame-level binary classification cross entropy loss training. 、 , supports prosodic segmentation modeling in zero semantic situations and is language-independent.

[0039] In some embodiments, steps S105 to S106 undertake two key tasks: first, the candidate segment set Transcribing into a set of text transcription sequences that can be processed through a speech recognition system Second, perform semantic density evaluation on the transcription results and construct a continuous semantic density score , used to determine whether the current node is an information transition, thereby assisting in triggering large-model translation in subsequent steps. Unlike the common ASR+text analysis process, this step focuses on the core issue of real-time speech translation: how to control the large-model input window. This creates a fine-grained solution that not only ensures high recognition accuracy but also captures the semantic flow structure, demonstrating clear creative and structural value.

[0040] The candidate segment set includes multiple candidate segments, each candidate segment Recognition is performed through a streaming automatic speech recognition (ASR) module. This module uses a lightweight model based on the Conformer structure. Its encoder consists of a 5-layer convolution and multi-head attention hybrid block, with each layer dimension of 256. The decoder is a Transformer decoder, and the output unit is Pinyin syllables (Chinese) or WordPiece subwords (English). For each segment, the ASR module generates a text transcription sequence. , and comes with phoneme alignment information to support sentence length normalization for semantic density modeling.

[0041] After the transcription is completed, the text transcription sequence set is obtained , the system transcribes all text sequences The text is then fed into the semantic encoder for vectorization. The encoder uses the MiniBERT structure, a 4-layer Transformer with a hidden layer dimension of 256, which is dedicated to fast embedding construction. Each text transcription sequence corresponds to a vector representation. Different from the common practice, this step introduces a density modeling mechanism based on the “inter-sentence mutation structure” and constructs the semantic density scoring function through the following formula (6): , (6) in, Indicates the Semantic density scores of text transcription sequences, semantic density score sequences , each It is a composite score that integrates semantic jump, syntactic structure and prosodic trust. The higher the value, the more likely it is to be a trigger point for large-scale model translation. Indicates the Text transcription sequence The result after processing by a specific semantic encoder is Text transcription sequence The input is fed into a MiniBERT encoder consisting of a four-layer Transformer. After processing through the embedding layer, positional encoding, multi-head attention mechanism, and feedforward network, the final output [CLS] vector or average pooling result is taken as the semantic vector representation of the text transcription sequence. This representation is not the original text itself, but an abstract expression of its semantic features by the encoder. Indicates the Text transcription sequence The result after being processed by a specific semantic encoder, Indicates the The number of words in the text transcription sequence, Represents the mean number of words in adjacent text transcription sequences. Indicates the The average boundary confidence of the frames of a text transcription sequence is used to measure whether the text transcription sequence as a whole is at a semantic transition or pause position. 、 and are weight coefficients respectively, and the recommended settings are , , .

[0042] It should be noted that in formula (6), the first term is the standard cosine distance (normalized), which represents the magnitude of the change in semantic theme. The second term is the length jump term, which, after taking its logarithm, expresses the degree of "sentence length jump" and can assist in determining whether it is a paragraph switching point. The third term is the prosodic confidence adjustment term, which is used to reduce the density score of rhythmically stable areas to prevent phrasal jitter from being mistakenly identified as semantic mutation.

[0043] In one example, when the speech content is "Now I will talk about two aspects. The first is the budget issue," there is a significant semantic jump between the first and second segments, the cosine distance is large, and the sentence length is significantly shortened (syntactic rupture). It shows a high-low reversal, under the joint action of the three factors The value of the sentence increases significantly; when the sentence is just a simple supplementary explanation, such as "We have submitted the application and are now waiting for the approval", The value of is low and translation is not easily triggered.

[0044] In addition, to enhance the stability in multi-round spoken input scenarios, the system introduces a "context residual smoothing mechanism" that performs exponentially weighted moving average processing on continuous density values ​​through a sliding window. It is more robust in fast conversation or self-correction scenarios. indivual value and multiply it by the attenuation factor The inference overhead is extremely low.

[0045] See also Figure 3 In some embodiments, step S107 further includes but is not limited to steps S301 to S302: Step S301, scoring the text transcription sequence according to a preset retention scoring function and semantic density score to obtain a retention score; Step S302 : selecting from the text transcription sequence set according to the retention score and the preset upper limit of the input length to obtain an input text sequence set.

[0046] In step S301 of some embodiments, each text transcription sequence is transcribed by retaining the scoring function Calculating retention scores , the retention score function is shown in the following formula (7): , (7) in, Represents a text transcription sequence The retention score, and Represents the weighting coefficient, the recommended value is , . Represents a text transcription sequence The number of words, that is, the token length (calculated by tokenizer). Represents a text transcription sequence The semantic density score of the sentence is denoted by . In formula (7), the first term encourages the selection of segments with strong information mutation, while the second term penalizes lengthy segments, preventing long texts from occupying a large token budget with a small contribution. This scoring calculation process can be implemented in engineering using conventional embedding similarity and text length functions, with low inference cost and deployability.

[0047] In step S302 of some embodiments, all retention scores are calculated. After that, the system transcribes the text into a sequence set according to The values ​​are sorted in descending order, and their token lengths are accumulated in sequence until the total length approaches the maximum input limit of the model. (e.g. 2048). Finally, the selected text transcription sequences are concatenated in chronological order to generate the input text sequence set. , as shown in the following formula (8): , (8) in, Represents a set of input text sequences, Represents the sequence index selected after sorting, and the splicing order maintains the consistency of the timeline to ensure context coherence. , which indicates the position of the input text sequence in the cache in the current input, and can be used for subsequent output alignment and tracking control. Indicates the total number of selected text transcription sequences, Indicates the token length of the text transcription sequence, Indicates the maximum input limit of the model.

[0048] For example, suppose the current cache has the following text transcription sequence: : "We are going on a business trip next week", , ; : "Please organize the information", , ; : "I have already booked the transportation ticket", , ; Then we can calculate: , , .

[0049] After sorting , is retained, was excluded. For: "We are going on a business trip next week. Please organize the information." Through the above steps S301 to S302, the input content adapted to the large language model (such as GPT) is constructed. Under the limit of the model token number budget (such as 2048 tokens), the most semantically valuable fragments are intelligently selected from the existing text fragments and spliced ​​to generate the final input text sequence set. Traditional methods often use time windows, fixed lengths, or FIFO mechanisms, but these methods often ignore dimensions such as semantic jumps and content novelty, resulting in the omission of important content or the repeated introduction of redundant content. To this end, this embodiment proposes a sorting mechanism that integrates "semantic density drive" and "token length regularization" to give priority to fragments with high semantic contribution and reasonable token overhead.

[0050] See also Figure 3 In some embodiments, step S108 further includes but is not limited to steps S401 to S402: Step S401, scoring the input text sequence according to a preset trigger scoring function to obtain a trigger score; Step S402: If the trigger score is greater than a preset trigger threshold, the input text sequence set is input into a preset translation model to obtain a translation result.

[0051] In step S401 of some embodiments, after obtaining the input text sequence set and its index collection After that, it determines whether the translation should be triggered and directly calls the translation model to generate the translation result when the conditions are met. , as the final external output operation action.

[0052] Specifically, in order to ensure that the translation trigger mechanism has immediate response capabilities and is information-driven, the system adopts a lightweight calculation method to directly score the semantic density of the input text sequence set. Changes in length of preceding and following segments Perform combined scoring to obtain trigger scores That is, the trigger scoring function is as shown in the following formula (9): , (9) in, Indicates trigger rating, Represents an input text sequence The corresponding semantic density score, Indicates the The number of words in the text transcription sequence, Indicates the The number of words in the text transcription sequence, and They represent weighting parameters, with recommended values ​​of 0.7 and 0.3.

[0053] In step S402 of some embodiments, the trigger threshold Can be set to 0.6, when When the translation model is called immediately, the input text sequence set Translate and generate translation results , the result is directly presented as the final output without further judgment or cache delay. For example, when the two latest input text sequences in the cache are: "Budget adjustment completed" and "Please start the next round of advancement as soon as possible", their semantic density and length difference are large, When the trigger threshold is exceeded, the system immediately triggers translation and returns the English sentence "The budget adjustment has been completed. Please proceed with the next phase as soon as possible." Through the above steps S401 to S402, a trigger score is introduced and a trigger threshold is set to control the translation timing. The real-time translation of the input text sequence set is not performed directly. This is to balance the response time and computing resource efficiency in real-time voice scenarios, and avoid the delay and burden caused by excessively frequent model calls. Specifically, continuous speech input is often highly fragmented and semantically incomplete. If the system immediately translates every small piece of text, it may lead to insufficient semantic integrity. Although some input segments have been transcribed, they are still in the middle of a sentence or a topic. Rash translation will cause semantic fragmentation and lack of context, reducing the translation quality. This is a serious waste of computing resources. Each inference of a large language model requires significant computing resources. Frequent calls will cause the overall system response to slow down, making it difficult to meet real-time requirements. The user experience is poor. Too much invalid or incoherent output may interfere with user understanding and even affect the coherence of the interaction process. Therefore, by integrating the two factors of semantic density and sentence length jump to construct a trigger score, we can determine whether the current input has formed an expression with sufficient information value. Translation is triggered only when there is a real semantic mutation or important information turning point, realizing a more intelligent and rhythm-adaptive translation control strategy.

[0054] In actual applications, in order to improve the consistency of user-side feedback, the system synchronously outputs a marker bit , indicating whether model inference has actually been executed (1 for executed, 0 for not triggered). If not executed, the system maintains the original cache state and enters the next window to wait for new content changes.

[0055] In steps S101 to S108 shown in the embodiment of the present application, original speech data is obtained, and feature extraction is performed based on the original speech data to obtain a speech frame feature sequence. The speech frame feature sequence is input into a preset prosody boundary recognition model to obtain a boundary confidence sequence. The original speech data is segmented according to the boundary confidence sequence and the preset segmentation threshold to obtain a candidate segment set. Speech recognition is performed based on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes multiple text transcription sequences. The text transcription sequence is scored according to a preset semantic density scoring function to obtain a semantic density score. Selection is performed in the text transcription sequence set based on the semantic density score to obtain an input text sequence set. The input text sequence set is input into a preset translation model to obtain a translation result, thereby improving the accuracy of real-time speech translation.

[0056] Please refer to Figure 5 In some embodiments, after step S302, the large model-based real-time speech translation method may further include, but is not limited to, steps S501 to S505: Step S501, obtaining updated candidate segments; Step S502: performing speech recognition based on the updated candidate segments to obtain an updated text transcription sequence; Step S503, scoring the updated text transcription sequence according to the retention score function to obtain an updated retention score; Step S504: obtaining the input text sequence with the lowest retained score in the input text sequence set to obtain a target input text sequence; Step S505 : If the updated retention score is greater than the retention score of the target input text sequence, the updated text transcription sequence is replaced with the target input text sequence to obtain an updated input text sequence set.

[0057] In some embodiments, in step S501 to step S502, when an updated candidate segment is obtained, speech recognition is performed on the updated candidate segment using the above method to obtain an updated text transcription sequence. .

[0058] In some embodiments, in steps S503 to S505, the updated text transcription sequence is transcribed according to the retention score function. Rating to get updated retention rating Get the input text sequence with the lowest retention score in the input text sequence set, get the target input text sequence, and update the retention score Retention score with target input text sequence Compare, if And the length of the token after replacement does not exceed the maximum input limit of the translation model , then the updated text transcription sequence is replaced with the target input text sequence to obtain an updated input text sequence set.

[0059] Through the above steps S501 to S505 , it is ensured that the input text sequence set is dynamically updated over time and can respond to real-time changes in the input content structure.

[0060] See also Figure 6 The present application also provides a large-model-based real-time speech translation system, which can implement the above-mentioned large-model-based real-time speech translation method. The system includes: Acquisition module 601, used to acquire original voice data; Extraction module 602, used to extract features based on the original speech data to obtain a speech frame feature sequence; The boundary recognition module 603 is used to input the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; A segmentation module 604 is configured to segment the original speech data into segments according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; A speech recognition module 605 is configured to perform speech recognition based on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes a plurality of text transcription sequences; Scoring module 606, configured to score the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; A selection module 607 is configured to select from the text transcription sequence set based on the semantic density score to obtain an input text sequence set; The translation module 608 is used to input the input text sequence set into a preset translation model to obtain a translation result.

[0061] The specific implementation of the large model-based real-time speech translation system is basically the same as the specific embodiment of the large model-based real-time speech translation method described above, and will not be repeated here.

[0062] The present application also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the large model-based real-time speech translation method. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0063] See also Figure 7 , Figure 7 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes: The processor 701 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application. Memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). Memory 702 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in memory 702 and is called by processor 701 to execute the large model-based real-time speech translation method of the embodiments of this application. Input / output interface 703, used to implement information input and output; Communication interface 704, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.); Bus 705 , which transmits information between various components of the device (e.g., processor 701 , memory 702 , input / output interface 703 , and communication interface 704 ); The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0064] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large model-based real-time speech translation method.

[0065] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0066] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0067] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0068] The system embodiment described above is merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0069] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0070] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0071] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of the above-mentioned units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0073] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0076] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A real-time speech translation method based on a large model, characterized in that: The method comprises: Get the original voice data; Extracting features from the original speech data to obtain a speech frame feature sequence; Inputting the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; Segmenting the original speech data according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; Performing speech recognition on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes a plurality of text transcription sequences; Scoring the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; Selecting from the text transcription sequence set according to the semantic density score to obtain an input text sequence set; The input text sequence set is input into a preset translation model to obtain a translation result.

2. The method according to claim 1, characterized in that Inputting the input text sequence into a preset translation model to obtain a translation result includes: Scoring the input text sequence according to a preset trigger scoring function to obtain a trigger score; If the trigger score is greater than a preset trigger threshold, the input text sequence set is input into a preset translation model to obtain a translation result.

3. The method according to claim 2, characterized in that Scoring the input text sequence according to a preset trigger scoring function to obtain a trigger score includes: The trigger scoring function is: ； in, Indicates trigger rating, represents the semantic density score corresponding to the input text sequence, Indicates the The number of words in the text transcription sequence, Indicates the The number of words in the text transcription sequence, and represent weighting parameters respectively.

4. The method according to claim 1, wherein The extracting features based on the original speech data to obtain a speech frame feature sequence includes: Performing division processing according to the original voice data to obtain a voice frame stream; Extracting features based on the speech frame stream to obtain short-time energy features, pitch fundamental frequency features, zero-crossing rate features, and cepstral coefficient features; The speech frame feature sequence is obtained by splicing the short-time energy feature, the pitch fundamental frequency feature, the zero-crossing rate feature and the cepstral coefficient feature.

5. The method according to claim 1, wherein The selecting from the text transcription sequence set according to the semantic density score to obtain an input text sequence set includes: Scoring the text transcription sequence according to a preset retention scoring function and the semantic density score to obtain a retention score; The text transcription sequence set is selected according to the retention score and a preset upper limit of the input length to obtain the input text sequence set.

6. The method according to claim 5, characterized in that After selecting from the text transcription sequence set according to the semantic density score to obtain the input text sequence set, the method further includes: Obtain updated candidate segments; Performing speech recognition based on the updated candidate segments to obtain an updated text transcription sequence; Scoring the updated text transcription sequence according to the retention scoring function to obtain an updated retention score; Obtaining the input text sequence with the lowest retention score in the input text sequence set to obtain a target input text sequence; If the updated retention score is greater than the retention score of the target input text sequence, the updated text transcription sequence is replaced with the target input text sequence to obtain an updated input text sequence set.

7. The method according to claim 1, characterized in that Scoring the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score includes: The semantic density scoring function is: ； in, Indicates the Semantic density scores of text transcription sequences, Indicates the The result of a text transcription sequence processed by a specific semantic encoder, Indicates the The result of a text transcription sequence processed by a specific semantic encoder, Indicates the The number of words in the text transcription sequence, represents the mean number of words in the adjacent text transcription sequence, Indicates the The mean boundary confidence across frames of the text transcription sequence, 、 and are weight coefficients respectively.

8. A real-time speech translation system based on a large model, characterized in that: The system comprises: An acquisition module is used to obtain original voice data; An extraction module, configured to extract features from the original speech data to obtain a speech frame feature sequence; A boundary recognition module, configured to input the speech frame feature sequence into a preset prosodic boundary recognition model to obtain a boundary confidence sequence; a segmentation module, configured to segment the original speech data into segments according to the boundary confidence sequence and a preset segmentation threshold to obtain a set of candidate segments; A speech recognition module, configured to perform speech recognition based on the candidate segment set to obtain a text transcription sequence set; wherein the text transcription sequence set includes a plurality of text transcription sequences; A scoring module, configured to score the text transcription sequence according to a preset semantic density scoring function to obtain a semantic density score; A selection module, configured to select from the text transcription sequence set according to the semantic density score to obtain an input text sequence set; The translation module is used to input the input text sequence set into a preset translation model to obtain a translation result.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the large model-based real-time speech translation method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the real-time speech translation method based on a large model according to any one of claims 1 to 7 is implemented.

Images