Audio stream processing method and device

Abstract

The invention discloses an audio stream processing method and device, and the method comprises the steps: carrying out the voice activity detection of a received audio stream, obtaining a sentence segmentation result, and segmenting a target audio segment from the audio stream based on the sentence segmentation result; inputting the target audio clip into a pre-trained language translation large model to execute translation operation so as to obtain a translation result; outputting a translation result; thus, the continuous audio stream is subjected to sentence segmentation through voice activity detection, so that segmented segments are complete in semantics and moderate in length, meanwhile, a large model directly converts the audio into a translation result in a text form and belongs to an end-to-end model, the overall translation speed and accuracy are improved, and in addition, due to the fact that the audio is introduced into the model, the translation efficiency is improved. Extra modal information carried in the audio can be understood, and a translation result more suitable for the context is output.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio stream processing method and apparatus. Background Technology

[0002] In the field of speech translation, the traditional industry solution is usually a cascaded speech recognition plus machine translation (MT) framework. This approach first uses Automatic Speech Recognition (ASR) to transcribe the source language audio into text, and then uses machine translation (MT) to convert the text into the target language. Therefore, it is also known as a two-step framework of "audio-to-text plus text-to-text". Because this process is divided into two independent stages, it is not only time-consuming and has translation delays, but the connection of multiple models and the accumulation of errors can also affect the accuracy of the translation results. Summary of the Invention

[0003] This application provides an audio stream processing method, apparatus, computing device, storage medium, and computer program product to reduce translation latency, increase translation speed, and improve translation accuracy.

[0004] The technical solution adopted in this application is as follows.

[0005] In a first aspect, this application provides an audio stream processing method, comprising: performing speech activity detection on a received audio stream to obtain a sentence segmentation result; segmenting a target audio segment from the audio stream based on the sentence segmentation result; inputting the target audio segment into a language translation big data model to perform a translation operation to obtain a translation result corresponding to the audio stream, wherein: the translation result includes translated text and a sentence-level timestamp associated with the translated text; the language translation big data model is obtained through pre-training based on multiple training data corresponding to multiple language types, the training data including: training audio, reference text corresponding to the training audio, and a sentence-level timestamp associated with the reference text; and outputting the translation result corresponding to the audio stream.

[0006] Thus, this application performs speech activity detection on the received audio stream, segments the continuous audio stream into sentences, and then segments the target audio segments accordingly. This ensures that the segmented segments are semantically complete and of appropriate length, effectively avoiding semantic incompleteness or information loss, and ensuring the accuracy and completeness of subsequent translation processing. The appropriate length also ensures the timeliness of data processing and reduces latency. Simultaneously, the audio segments are input into a pre-trained large-scale language translation model. This model directly converts the audio into text-based translation results, belonging to an end-to-end model. Because the segments segmented by this application through speech activity detection are semantically complete and of appropriate length, the end-to-end large-scale language translation model is well-suited for semi-streaming translation scenarios. This model can directly translate the original audio information into text-based results, solving the latency problem of traditional cascaded speech recognition plus machine translation frameworks, as well as the accuracy problem caused by accumulated errors and information loss, thus improving the overall translation speed and accuracy. Furthermore, because the method in this application employs an end-to-end audio-to-text translation process, and since the model's input is no longer pure text but incorporates audio, the model can understand the additional modal information carried in the audio. This allows the translation results to better preserve the semantics, context, and speaker's expressive intent in the original audio, resulting in more context-appropriate translations and further improving translation accuracy. In summary, the method in this application can balance low latency and accuracy. Moreover, the diverse training audio in multiple languages ​​covers the phonetic features, sentence segmentation rules, and phonetic diversity of different languages, enabling the trained model to learn audio processing and sentence segmentation for multiple languages, thus expanding the model's applicability. Furthermore, because the model's training data incorporates sentence-level timestamps as part of the training output, the model learns sentence segmentation capabilities. When using the model for subsequent translation, sentence-level timestamps of the audio stream can be obtained, laying the foundation for aligning the target text with the corresponding audio stream and ensuring the synchronization of translated subtitles with the audio.

[0007] In conjunction with the first aspect, in one possible implementation, speech activity detection is performed on the received audio stream to obtain sentence segmentation results, and target audio segments are segmented from the audio stream based on the sentence segmentation results. This includes: storing the received audio stream data blocks into an audio buffer, and performing speech activity detection on the audio stream data blocks; when the detection finds that a first type of detection condition is met, retrieving all data blocks from the audio buffer as target audio segments, and clearing the audio buffer. The first type of detection condition includes the condition that the latest cached data block is the end of a sentence or the length of all cached data blocks reaches a preset length; when the detection finds that a second type of detection condition is met, retrieving all data blocks from the audio buffer as target audio segments. The second type of detection condition includes the condition that the latest cached data block is the end of a clause.

[0008] Thus, by storing the received audio stream data blocks into an audio buffer and performing real-time speech activity detection, when the end of a clause (second type of detection condition) or the end of a sentence / reaching a preset length (first type of detection condition) is detected, the target audio segment is promptly segmented from the audio stream. This segmentation strategy ensures that the segmented segments simultaneously maintain speech integrity and appropriate length. In particular, this application translates the case of the end of a clause promptly without clearing the audio buffer. The data in the buffer continues to participate in the translation of the next clause data, improving translation accuracy.

[0009] In conjunction with the first aspect, in one possible implementation, the first type of detection condition includes either a silence duration greater than a first duration, or the cumulative detection duration of all cached data blocks reaching a preset duration. The second type of detection condition includes either a silence duration greater than a second duration and less than a first duration, where the first duration is greater than the second duration.

[0010] Thus, by distinguishing different silence durations and cumulative detection durations, the speech boundaries between clauses and sentences are differentiated. This detection method reflects the pausing habits during speech and ensures the rationality of segment segmentation.

[0011] In conjunction with the first aspect, in one possible implementation, the method further includes: in response to a triggering event indicating that the translation needs to be interrupted, retrieving all data blocks from the audio cache as input to a pre-trained large language translation model to obtain the translation result.

[0012] Thus, when a triggering event indicating that translation needs to be interrupted is detected, the system can promptly retrieve all cached data blocks from the audio cache for translation. Moreover, in this case, the audio cache is handled in the same way as at the end of the clause, that is, the audio cache is not cleared, and the data in the cache will continue to participate in the translation of the next clause data, further improving the translation accuracy.

[0013] In conjunction with the first aspect, in one possible implementation, the training data also includes sentence-level timestamps associated with the reference text, and the translation results include the translated text and sentence-level timestamps associated with the translated text. The output translation result includes: extracting sentence-level timestamps before and after the translated text from the translation result; obtaining the target text based on all translated text in the translation result; and aligning the target text with the corresponding audio in the audio stream based on the extracted sentence-level timestamps before outputting it.

[0014] In this way, sentence-level timestamps are introduced as part of the training output, enabling the model to learn sentence segmentation capabilities. When using the model for translation later, sentence-level timestamps can be obtained. By aligning the target text with the corresponding audio in the audio stream using sentence-level timestamps, the synchronization between the translated text subtitles and the audio can be ensured, thus improving the user experience.

[0015] In conjunction with the first aspect, in one possible implementation, the training data further includes cue words; the method further includes: before the first target audio segment is input into the language translation big model, obtaining cue words corresponding to the audio stream, and injecting the cue words into the cue text of the language translation big model.

[0016] Thus, by injecting prompt words into the model's context text, the model is guided by these prompts in its translation direction and output, further improving translation accuracy. The prompt words can be adjusted according to different scenarios, domains, or specific contexts, allowing the model's translation strategy and word choice to adapt accordingly, ultimately enhancing the relevance and professionalism of the translation results. For example, when the prompt words contain specific domain terms, people, place names, or events, the model can more accurately identify and generate corresponding professional vocabulary, reducing the probability of omissions, mistranslations, and ambiguities.

[0017] In conjunction with the first aspect, in one possible implementation, the translation operation includes: encoding the input target audio segment into a speech embedding representation, encoding the prompt text into a text embedding representation, and inputting the speech embedding representation and the text embedding representation into a language model to obtain the translation result.

[0018] In conjunction with the first aspect, in one possible implementation, the pre-training includes: Obtain the training data and the corresponding prompt words for the training data; The training audio is input into the language translation model, and the prompt words corresponding to the training audio are injected into the prompt text of the language translation model. The translation result corresponding to the training audio is obtained by performing a translation operation using a large language translation model. The language translation model is trained based on a preset loss, which includes: the loss between the translated text corresponding to the training audio and the reference text corresponding to the training audio, and the loss between the sentence-level timestamp associated with the translated text corresponding to the training audio and the sentence-level timestamp associated with the reference text corresponding to the training audio.

[0019] Thus, introducing prompt words allows the model to learn the ability to adjust its output based on those prompts. By introducing sentence-level timestamps, the model not only focuses on the accuracy of the translated text during training but also simultaneously learns the ability to segment sentences and output timestamps. These timestamps can be used for subsequent text-to-audio alignment, ensuring synchronization between the translated subtitles and audio, and improving the user experience.

[0020] In conjunction with the first aspect, in one possible implementation, obtaining the prompt words corresponding to the training data includes: extracting a first prompt word from the reference text in the current training data; constructing a lexicon based on the first prompt words corresponding to all the training data, and randomly extracting a second prompt word from the lexicon; and constructing the prompt words corresponding to the current training data based on the first and second prompt words.

[0021] In this way, the prompts provided during training include both the prompts for the current sentence (first prompts) and prompts unrelated to the current sentence (second prompts), allowing the model to select relevant prompts on its own, thereby enhancing the robustness of prompt injection.

[0022] In conjunction with the first aspect, in one possible implementation, acquiring training data includes: obtaining reference text corresponding to the training audio and word-level timestamps associated with the reference text through automatic speech recognition; inputting the reference text and prompts corresponding to the training audio into a translation service interface to segment the reference text into multiple clauses; determining the start and end timestamps of each clause based on the word-level timestamps to obtain sentence-level timestamps; and embedding the sentence-level timestamps into the reference text to associate the sentence-level timestamps with the reference text.

[0023] In this way, by combining word-level timestamps with sentence segmentation results, it is possible to accurately annotate sentence-level timestamps of the reference text.

[0024] Secondly, this application provides an audio stream processing apparatus, which includes modules for performing the audio stream processing method of the first aspect or any optional implementation thereof. For example, the audio stream processing apparatus includes: The first module is used to perform speech activity detection on the received audio stream to obtain sentence segmentation results, and to segment target audio segments from the audio stream based on the sentence segmentation results; The second module is used to input the target audio segment into the language translation big model to perform a translation operation to obtain the translation result corresponding to the audio stream, wherein: the translation result includes the translated text and the sentence-level timestamp associated with the translated text; the language translation big model is obtained by pre-training based on multiple training data corresponding to multiple language types, and the training data includes: training audio, reference text corresponding to the training audio, and sentence-level timestamp associated with the reference text; The third module is used to output the translation result corresponding to the audio stream.

[0025] Thirdly, this application also provides a computing device. The computing device includes a memory and a processor, the memory being used to store computer programs or instructions that, when executed by the processor, implement the methods described in the first aspect or any possible implementation thereof.

[0026] Fourthly, this application provides a computer-readable storage medium. The storage medium stores a computer program or instructions that, when executed by a processor, implement the method described in the first aspect or any possible implementation thereof.

[0027] Fifthly, this application provides a computer program product. The computer program product includes a computer program or instructions that, when executed by a processor, implement the method described in the first aspect or any possible implementation thereof.

[0028] The beneficial effects of aspects two through five above can be referenced to aspect one or any possible implementation thereof, and will not be elaborated upon here. Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations.

[0029] Other advantages, objectives and features of this application will be partly apparent from the description below, and partly understood by those skilled in the art through study and practice of this application. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0031] Figure 1a This is one of the schematic diagrams illustrating the implementation environment of the audio stream processing method provided in the embodiments of this application; Figure 1b This is the second schematic diagram of the implementation environment of the audio stream processing method provided in the embodiments of this application; Figure 2 This is one of the flowcharts illustrating an audio stream processing method in an exemplary embodiment of this application; Figure 3 This is one of the sub-step diagrams of step S203; Figure 4 This is the second sub-step diagram of step S203; Figure 5This is a second flowchart illustrating an audio stream processing method in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the sub-steps of step S205; Figure 7 This is a flowchart of the pre-training process for a large-scale speech-language translation model; Figure 8 It is a flowchart for constructing training data based on a preset processing method; Figure 9 This is a schematic diagram illustrating the specific principle of the audio stream processing method provided in the embodiments of this application in a live streaming scenario; Figure 10a This is one of the schematic diagrams illustrating the use of a large-scale language translation model; Figure 10b This is the second illustration of how the large-scale language translation model is used; Figure 11 This is a schematic block diagram of the audio stream processing device provided in the embodiments of this application; Figure 12 This is a schematic block diagram of a computing device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0032] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0033] The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. In this application, "at least one" means one or more, and "more than one" means two or more. The terms "first," "second," and other ordinal terms used in this application may be used to describe various constituent elements, but these constituent elements are not limited by these terms. The purpose of using these terms is solely to distinguish one constituent element from others and should not be construed as indicating or implying relative importance. For example, without departing from the scope of this application, a first constituent element may be named a second constituent element, and similarly, a second constituent element may be named a first constituent element.

[0034] Before introducing the embodiments of this application, the technical terms involved in this application will be introduced first.

[0035] Audio stream: refers to a sequence of audio data input or transmitted continuously over a period of time. An audio stream continuously feeds audio data into a system or network in small segments or frames for real-time processing or playback. For example, in a live streaming scenario, the broadcaster's voice is continuously captured and transmitted to a server; this continuous and seamless audio data constitutes an audio stream.

[0036] Large-scale language translation models refer to neural network models trained using deep learning or artificial intelligence techniques, possessing a large number of parameters, and capable of performing language translation tasks. These large-scale models are typically based on the Transformer architecture, and through pre-training on massive amounts of multilingual data, they possess powerful cross-language understanding and generation capabilities, supporting multimodal (text + audio, etc.) collaborative translation.

[0037] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0038] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, large-scale speech processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0039] Key technologies in speech technology include Automatic Speech Recognition (ASR) and Speech Translation (ST). Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech being one of the most promising methods. Large-scale modeling has revolutionized speech technology; pre-trained models such as WavLM and UniSpeech, which utilize the Transformer architecture, possess strong generalization and versatility, enabling them to excel in various speech processing tasks.

[0040] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0041] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, digital twins, virtual humans, robots, AI-generated content (AIGC), conversational interaction, smart healthcare, smart customer service, and game AI. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0042] The solutions provided in this application relate to artificial intelligence-based speech translation (ST) technology, which will be described in detail in subsequent embodiments.

[0043] This application provides an innovative audio stream processing method for speech translation of audio streams, aiming to improve both translation speed and accuracy. The application scenarios for this speech translation technology include: 1. Automatic subtitles for cross-border live events: In multilingual live streaming scenarios such as international sports events, concerts, and press conferences, the solution proposed in this application can automatically translate foreign language audio into target language subtitles in real time, effectively eliminating language barriers and enhancing the viewing experience for global audiences.

[0044] 2. Game events and virtual anchor live streaming: For emerging live streaming scenarios such as e-sports events and virtual anchors, the solution in this application can automatically identify and translate multilingual live streaming content and generate synchronized subtitles to help viewers understand the progress of the event and interact with the anchor across languages.

[0045] 3. Online Education and International Classrooms: In online education scenarios such as multilingual remote teaching and international open courses, the solution proposed in this application can translate the lecturer's foreign language explanation into subtitles in the student's native language in real time, significantly improving the accessibility and learning effectiveness of cross-border education.

[0046] 4. Multilingual Enterprise Meetings and Remote Collaboration: In scenarios such as video conferencing and remote collaboration within multinational corporations, the solution proposed in this application can generate multilingual subtitles in real time, assisting team members in conducting efficient information exchange and decision-making.

[0047] 5. Multilingual reporting by news media: When news media conduct live coverage of international breaking events, they can use the solution in this application to achieve simultaneous multilingual subtitles, thereby improving the timeliness and global accessibility of the news.

[0048] 6. Cultural Exchange and Exhibition Activities: In international exhibitions, cultural exchange activities, and other similar settings, this application can provide real-time translation subtitles for multilingual explanations, interviews, and other sessions, promoting communication and interaction among people from different cultural and linguistic backgrounds.

[0049] Before introducing the embodiments of this application, the relevant technologies involved in this application will be introduced first.

[0050] In industry, speech translation technology mainly includes the traditional cascaded speech recognition and machine translation (ASR+MT) framework. The traditional cascaded framework typically involves two stages: first, automatic speech recognition (ASR) transcribes the source language audio into source language text; then, a machine translation (MT) module translates this text into the target language text, achieving an audio-to-text-to-text processing flow.

[0051] While this translation method has a clear structure and facilitates modular development and maintenance, it also has significant drawbacks. First, the entire process involves two main modules, ASR and MT, resulting in a lengthy processing path and high overall inference time. Furthermore, because the output of each stage directly becomes the input of the next, recognition errors in the ASR stage are propagated and amplified in the MT stage, making it difficult to guarantee the final translation quality. Especially in practical applications, ASR systems can be affected by various factors such as accent, noise, and speech rate, leading to mispronunciations and omissions. Once these errors enter the machine translation module, they often cause unpredictable translation biases and comprehension difficulties.

[0052] In summary, related technologies not only have a long overall processing time and translation delays, but the connection between multiple models and the accumulation of errors also affect the accuracy of the translation results. Therefore, this application provides an audio stream processing method that can improve both translation speed and accuracy.

[0053] The following first describes one or more exemplary operating environments to make it easier and clearer to understand the role and intent of the various implementation methods in the embodiments of this application.

[0054] like Figure 1aFigure 1 is a schematic diagram illustrating the implementation environment of the method provided in this application embodiment. The method provided in this application embodiment can be applied to the environment in Figure 1, which includes a client 101 and a server 102. The client 101 and the server 102 establish a communication connection via wired or wireless means. The device form of the client 101 includes, but is not limited to, smartphones, tablets (such as iOS / Android devices), PCs (such as iMacs), and smart wearable devices. The software carrier of the client 101 can be a native application (App) or its functional sub-modules, or it can be a web browser. The server 102 should be interpreted broadly as an entity capable of responding to external requests and providing data, resources, or services. The server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but is not limited to these. This application does not limit the number of servers.

[0055] refer to Figure 1b In a live streaming scenario, clients can be divided into viewer clients 101a and broadcaster clients 101b. The party that initiates the live stream is broadcaster client 101b, and the party watching the live stream is viewer client 101a. Clients with the application software (live streaming software) installed can obtain live streaming permissions and become broadcaster client 101b by enabling the live streaming function (installing the live streaming resource component). It is important to understand that whether a client is broadcaster client 101b does not depend on the client itself, but on whether it is the party that initiates the live stream. For example, if A logs into the live streaming application on client A and B logs into the live streaming application on client B, if A enters B's live stream room, then client A belongs to viewer client 101a, and client B belongs to broadcaster client 101b. However, if B enters A's live stream room, then client A belongs to broadcaster client 101b, and client B belongs to viewer client 101a. Server 102 can provide live streaming services for viewer client 101a and broadcaster client 101b.

[0056] Combination Figure 1a , Figure 1b The method of this application can be implemented by server 102, that is, the code corresponding to the method of this application can be deployed in server 102. Taking a live broadcast scenario as an example, the audio stream of the broadcaster 101b is sent to server 102. When server 102 obtains the audio stream sent by broadcaster 101b, it will execute the translation result of the method of this application and send it to the viewer 101a.

[0057] The technical solutions of this application are described below through several embodiments. It should be understood that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein.

[0058] refer to Figure 2 Firstly, embodiments of this application provide an audio stream processing method, wherein the executing entity of the method may be... Figure 1a , Figure 1b The server in the process includes steps S201-S205: S201: Perform speech activity detection on the received audio stream to obtain the sentence segmentation result, and segment the target audio segment from the audio stream based on the sentence segmentation result; S203: Input the target audio segment into the large-scale language translation model to perform translation operations and obtain the translation result corresponding to the audio stream; The language translation big model is pre-trained based on training data, which includes training audio and corresponding reference text. The training audio serves as the input data for the language translation big model, while the reference text corresponds to the output data of the language translation big model.

[0059] S205: Output the translation result corresponding to the audio stream.

[0060] Thus, this application performs speech activity detection on the received audio stream, segments the continuous audio stream into sentences, and then segments the target audio segments accordingly. This ensures that the segmented segments are semantically complete and of appropriate length, effectively avoiding semantic incompleteness or information loss, and ensuring the accuracy and completeness of subsequent translation processing. The appropriate length also ensures the timeliness of data processing and reduces latency. Simultaneously, the audio segments are input into a pre-trained large-scale language translation model. This model directly converts the audio into text-based translation results, belonging to an end-to-end model. Because the segments segmented by this application through speech activity detection are semantically complete and of appropriate length, the end-to-end large-scale language translation model is well-suited for semi-streaming translation scenarios. This model can directly translate the original audio information into text-based results, solving the latency problem of traditional cascaded speech recognition plus machine translation frameworks, as well as the accuracy problem caused by accumulated errors and information loss, thus improving the overall translation speed and accuracy. Furthermore, because the method in this application employs an end-to-end audio-to-text translation process, and since the model's input is no longer plain text but incorporates audio, the model can understand the additional modal information carried in the audio. This allows the translation result to better preserve the semantics, context, and speaker's expressive intent in the original audio, outputting a more context-appropriate translation and thus further improving translation accuracy. In summary, the method in this application can balance low latency and accuracy.

[0061] The following combination Figure 2-12The steps in steps S201-S205 and other optional steps are described in detail.

[0062] Regarding step S201 The received audio stream is subjected to speech activity detection to obtain sentence segmentation results, and the target audio segment is segmented from the audio stream based on the sentence segmentation results; Voice Activity Detection (VAD) is a signal processing technique used to detect when speech is present and when it is silence or noise in a speech signal. Specifically, VAD analyzes the input audio signal frame by frame to determine whether each frame contains speech.

[0063] Because users typically pause after uttering a sentence, this application proposes to achieve sentence segmentation of audio based on speech activity detection. Based on the segmentation results, semantically complete and appropriately sized target audio segments are extracted from the audio stream for translation. This ensures timely data processing and reduces latency.

[0064] In some embodiments, reference Figure 3 Step S201 specifically includes steps S301-S305: S301: Store the received audio stream data blocks into the audio buffer and perform voice activity detection on the audio stream data blocks; A data chunk is a smaller audio unit that is divided from a continuous audio stream according to a certain time length (such as 10ms, 50ms, 100ms, etc.). A data chunk can be a single audio frame or multiple audio frames.

[0065] Audio cache: The audio cache is a memory area used to temporarily store these "data blocks".

[0066] refer to Figure 9 The server provides live streaming and translation services based on its functions. When the live streaming service receives the raw audio stream from the broadcaster, it typically preprocesses it, dividing it into data blocks and sending them to the translation service. The translation service then places the data blocks into an audio buffer and performs VAD (Video Adaptation and Distributed Data) checks. It's important to understand that the data blocks continuously supplied by the live streaming service accumulate in the audio buffer; therefore, VAD checks are a continuous process, performing checks on the continuously arriving data blocks.

[0067] S303: When the detection finds that the first type of detection condition is met, retrieve all data blocks from the audio buffer as the target audio segment and clear the audio buffer; The first type of detection conditions includes the condition that the latest cached data block is the end of a complete sentence (which means that a complete sentence has been spoken, so it is necessary to translate it in time) or the condition that the length of all cached data blocks reaches the preset length (too long a data length will cause a delay, so this condition is used to ensure that the data is translated in time and that the data accumulated during the audio caching process is not cached).

[0068] A complete sentence is a grammatically and semantically complete unit of language that can independently express a complete meaning. A complete sentence may consist of multiple clauses; that is, a clause is a component of a complete sentence. A clause can express a relatively complete semantic unit, but may depend on the main clause to fully understand its meaning. For example, in the sentence "She was late because of traffic," "She was late," is one clause, and "because of traffic" is another clause. These two clauses are combined to form a complete sentence.

[0069] Specifically, the latest cached data block being the end of a complete sentence (such as after a pause, a topic change, or a long period of silence) can be represented by the duration of silence. The length of the latest cached data block reaching a preset length can be represented by the cumulative detection time. Therefore, in some embodiments, the first type of detection condition specifically includes the following conditions; satisfying any one of them is considered to meet the first type of detection condition: (a) The silence duration is longer than the first duration. For example, the first duration is typically in the hundreds of milliseconds range.

[0070] (b) The cumulative detection time of all cached data blocks reaches the preset time. For example, the preset time is generally in the second range.

[0071] As mentioned earlier, VAD detection is a continuous process. Therefore, the silence duration mentioned in condition (a) (denoted as T1) is actually the cumulative silence duration of multiple consecutive data blocks. That is, the silence duration of consecutive silent data blocks will be accumulated. Moreover, when voice activity is detected, the silence duration is reset to zero. Furthermore, the silence duration is also reset to zero every time the "target audio segment" is segmented. For example, assuming the first duration is 800 milliseconds and the duration corresponding to a data block is t1, when VAD detects silence starting from a certain data block, it considers the silence duration T to be t1. If the next data block is also silent, the silence duration T1 changes to 2*t1 (if the next block is not silent, T1 is reset to zero), and so on. When the silence duration T1 is greater than 800 milliseconds, it is considered that the first type of detection condition has occurred. At this time, it can be considered that the sentence before the silence is roughly a clause, which needs to be translated in time. Therefore, all the data blocks in the audio buffer are retrieved together to form a "target audio segment". The "target audio segment" is recognized and translated by the input language translation model to generate the corresponding subtitles.

[0072] The cumulative detection duration (denoted as T2) in condition (b) actually refers to the sum of the durations corresponding to all data blocks in the cache. The cumulative detection duration is reset to zero each time a "target audio segment" is segmented. For example, assuming the preset duration is 4 seconds, the duration corresponding to one data block is t1, and there are a total of n data blocks in the audio cache, then the cumulative detection duration T2 is considered to be n*t1. For each additional data block added to the audio cache, T2 changes to T2+t1, and so on. When T2=4 seconds, it is considered that the first type of detection condition has occurred. At this time, it can be considered that the data in the audio cache is long enough and needs to be translated in time. Therefore, all the data blocks in the audio cache are retrieved together to form a "target audio segment". The "target audio segment" will be recognized and translated by the input language translation model to generate corresponding subtitles.

[0073] When the first type of detection conditions are met, this application will not only send all data blocks in the audio cache into the model for translation, but also clear the cache and wait for the next "target audio segment".

[0074] In summary, the first type of detection condition ensures both the integrity of the sentence and prevents the system from waiting indefinitely, thus achieving low latency.

[0075] S305: When the detection finds that the second type of detection conditions are met, retrieve all data blocks from the audio buffer as the target audio segment; The second type of detection condition includes the condition that indicates the latest cached data block ends with a clause.

[0076] As mentioned earlier, a clause is a component of a complete sentence; therefore, its corresponding silence duration is shorter than the silence duration of the entire sentence. In some embodiments, the second type of detection conditions includes the following: (c) The silence duration is greater than the second duration but less than the first duration, and the first duration is greater than the second duration. For example, the second duration is generally in the hundreds of milliseconds range. A silence duration that is too short may be considered inconsistent with the sentence ending, while a silence duration that is too long will cause the subtitles to be delayed. It is generally recommended that the silence duration be between 200ms and 800ms, which is a suitable "sentence break" window that has been repeatedly verified in actual live streaming operations.

[0077] For example, assuming the second and first durations are 200 and 800 ms respectively, and the duration corresponding to a data block is t1, when VAD detects silence starting from a certain data block, it considers the silence duration T1 to be t1. If the next data block is also silent, the silence duration T1 changes to 2*t1, and so on. When the silence duration T1 is greater than 200 milliseconds but less than 800 milliseconds, it is considered that a second type of detection condition has occurred. At this point, the sentence before the silence can be considered to be a clause, which needs to be translated promptly. Therefore, all data blocks in the audio buffer (from the previous punctuation point to the present) are retrieved together to form a "target audio segment". The "target audio segment" is recognized and translated by the input language translation model to generate corresponding subtitles. However, unlike step S303, the audio buffer is not cleared at this time. Therefore, the data blocks in the audio buffer will be retained to participate in the composition of the next "target audio segment" when the first or second type of detection condition occurs again, thus making the next translation more accurate.

[0078] The following is combined Figure 9 Let's illustrate this with a concrete example. Suppose a streamer says during a live broadcast, "Okay, okay, goodbye. Let's talk again next time." The corresponding audio stream is transmitted to the server in real time, and the server's processing is as follows: At time Ta: At this time, the cache stores 3 data blocks C1-C3, and the condition (c) is met, indicating that the table has reached the end of a clause (this is actually because the silence duration after "okay," is 200-800ms). The system will combine the data blocks C1-C3 in the cache into a "target audio segment" and send it to the model for translation, which will result in "okay," and send it to the client. At this time, the cache is not cleared. At time Tb: At this time, the cache stores 6 data blocks C1-C6. When the condition (c) is detected again, it is considered that the table has the end of the clause (in fact, this is because the silence duration after "okay," is 200-800ms). The system will combine the data blocks C1-C6 in the cache into a "target audio segment" and send it to the model for translation, which will result in "okay, okay," and send it to the client. At this time, the cache is not cleared. At time Tc: At this point, the cache contains 8 data blocks C1-C8. If condition (a) is met, the entire sentence is considered to be over (this is actually because the silence duration after "Goodbye." is greater than 800ms). The system will combine the data blocks C1-C8 in the cache into a "target audio segment" and send it to the model for translation, resulting in "Okay, okay, goodbye.", which is then sent to the client, and the audio cache is cleared.

[0079] Therefore, the effect seen in the live stream will be: Ta moment: Good; TB moment: Okay, fine; Tc moment: Okay, okay, goodbye.

[0080] Thus, the visual effect would be that the current "Okay," subtitle keeps getting longer until the host starts saying the next sentence, "Let's talk again next time."

[0081] As can be seen, this application translates clauses promptly, ensuring timely translation, while the cache is not cleared to guarantee the accuracy of subsequent whole-sentence translations. Therefore, the translation is more accurate and natural, and subtitle delay is minimized, improving the viewer experience.

[0082] In a semi-streaming live automatic translation and subtitling system, audio data flows in continuously. The previously mentioned strategy relies on VAD (Voice Activity Detection) and other methods to automatically determine when to output a "target audio segment." However, in real-world business scenarios, human or system-level anomalies may occur, requiring immediate output of cached content without waiting for the second or first type of detection conditions to be met. Therefore, in some embodiments, refer to... Figure 4 Step S201 further includes step S401: S401: In response to a triggering event indicating that translation needs to be interrupted, retrieve all data blocks from the audio buffer as input to the pre-trained language translation model to obtain the translation result.

[0083] When a "translation interruption required" event is detected, regardless of the number of data blocks currently accumulated in the audio cache, all cached data blocks must be immediately retrieved as the target audio segment and fed into the pre-trained language translation model for translation to obtain the corresponding translation result. Furthermore, the audio cache is not cleared at this point because it is not a complete sentence; therefore, the cached data can be retained for the next translation, ensuring the accuracy of the next translation.

[0084] Specifically, events that trigger translation interruptions include: room shutdown, reconnection, etc.

[0085] For example, suppose the host says "Today we are going to introduce" and the room suddenly needs to reconnect due to an anomaly. This is a "trigger event indicating that the translation needs to be interrupted". At this time, the audio cache already contains the complete data block of the phrase "Today we are going to introduce". The system immediately retrieves all the data blocks from the audio cache, concatenates them into a target audio segment, and immediately sends it to the pre-trained language translation model for translation, outputting the text "Today we are going to introduce". Moreover, the cache is not cleared at this time. When the room is back to normal, the data block of the audio "A new product" is subsequently sent to the cache. The cache then contains the complete data block of "Today we are going to introduce a new product", so the translated text "Today we are going to introduce a new product" will be displayed in its entirety.

[0086] Regarding step S203 The target audio segment is input into a large-scale language translation model to perform a translation operation and obtain the translation result corresponding to the audio stream. The large-scale language translation model is pre-trained using various training data corresponding to multiple language types. The training data includes: training audio, reference text corresponding to the training audio, and sentence-level timestamps associated with the reference text. The training audio serves as the input data for the large-scale language translation model, while the reference text and sentence-level timestamps correspond to the output data. Further details regarding the pre-training of the large-scale model will be explained later.

[0087] The large model in this application directly converts audio into text, making it an end-to-end model. It directly translates raw audio information into text, solving the latency problem inherent in traditional cascaded speech recognition and machine translation frameworks, as well as the accuracy issues caused by accumulated errors and information loss, thus improving overall translation speed and accuracy. Furthermore, because the method in this application uses an end-to-end audio-to-text translation process, and the model's input is no longer pure text but incorporates audio, it can understand the additional modal information carried in the audio. This allows the translation result to better preserve the semantics, context, and speaker's expressive intent in the original audio, outputting a more context-appropriate translation, thereby further improving translation accuracy. In summary, the method in this application can balance low latency and high accuracy.

[0088] Specifically, refer to Figures 10a-10b The input to the large-scale language translation model consists of two parts: audio and prompt text. The target audio segment obtained in step S201 serves as the audio input to the model. In the large-scale language translation model, the prompt is used to instruct and guide the model to perform specific tasks. For example, when you want the model to perform a Chinese-to-English translation task, you can specify "Please translate the following content into English" in the prompt; or, when you want the model to give an emotionally charged response, you can give instructions like "Please answer with a humorous tone" in the prompt. For audio translation scenarios, prompts are often used to tell the model which target language to translate into, whether to output a timestamp, and whether to prioritize the use of specific proper nouns. Detailed examples of prompts will be provided later to aid understanding.

[0089] In some embodiments, reference Figure 5 Before step S201, the process also includes performing step S501 before the first target audio segment is input into the language translation big data model, or it can be understood as performing step S501 first when the audio stream is initially received: S501: Obtain the prompt words corresponding to the audio stream and inject the prompt words into the prompt text of the language translation big data model.

[0090] When acquiring an audio stream, the system first retrieves the corresponding prompt words, such as determining the live stream room based on the audio stream. It is understandable that the method described in this application is repeatedly executed during audio stream transmission, such as... Figure 5 As shown, before executing step S201 for the first time, it is necessary to determine the prompt words and inject them into the large model.

[0091] The number of prompts can be multiple. Specifically, prompts are generally provided by the business side and may include: the live stream room title, channel, trending topics, channel / program preset word library, high-frequency words from viewer comments and historical subtitles, output style, and other key information, as well as a priority vocabulary list manually specified by the business side. For example, in a medical live stream scenario, prompts may include various medical terms (such as "artery" and "blood pressure"); in a game live stream scenario, hot words may include character names and skill names.

[0092] As mentioned earlier, the large-scale language translation model is pre-trained based on training data, which includes training audio and corresponding reference text. Therefore, when introducing prompt words into the large-scale language translation model, it means that the model can understand the prompt words, thus requiring the inclusion of prompt words in the training data. Accordingly, in some embodiments, pre-training also includes simultaneously injecting prompt words into the prompt text of the large-scale language translation model when the training audio is input. Further details regarding the pre-training of the large-scale model will be discussed later.

[0093] Specifically, the prompt text includes: fixed commands, prompt words, and an identifier for the target language type to be translated. For example, if the target language type is Chinese, the identifier for Chinese is zh.

[0094] Thus, by injecting prompt words into the model's context text, the model is guided by these prompts in its translation direction and output, further improving translation accuracy. The prompt words can be adjusted according to different scenarios, domains, or specific contexts, allowing the model's translation strategy and word choice to adapt accordingly, ultimately enhancing the relevance and professionalism of the translation results. For example, when the prompt words contain specific domain terms, people, place names, or events, the model can more accurately identify and generate corresponding professional vocabulary, reducing the probability of omissions, mistranslations, and ambiguities.

[0095] refer to Figure 10a , Figure 10bThe translation operation mentioned in step S203 includes: encoding the input target audio segment into a speech embedding representation (audio embedding vector), encoding the prompt text into a text embedding representation (text embedding vector), and inputting the speech embedding representation and the text embedding representation into the language model to obtain the translation result.

[0096] The specific encoding implementation of text embedding representation can be based on deep modules (such as text encoders), such as... Figure 10a As shown, encoding can also be achieved using a pre-defined text-to-token mapping, such as... Figure 10b As shown.

[0097] refer to Figure 10a , Figure 10b The input to the large-scale language translation model includes the target audio segment and the prompt text. The target audio segment obtained in step S201 is input to the speech encoder to obtain a speech embedding representation (audio embedding vector), which is rich in audio representation. The prompt text is then processed by a text encoder or through a simple mapping to obtain a text embedding representation (text embedding vector). Both the speech and text embedding representations are input to the language model, which performs internal encoding and decoding to obtain the translation result (transcript), which is then output as the translation result (transcript).

[0098] Regarding step S205 Output the translation result corresponding to the audio stream.

[0099] The translation result includes the translated text and sentence-level timestamps associated with it. All translated text in the result is combined to form the target text, which is then output by the server to the user's client and ultimately displayed as subtitles. Sentence-level timestamps can be used for audio and subtitle (text) synchronization, which will be explained in detail later.

[0100] In live streaming automatic subtitle systems, the streaming nature of audio and the real-time requirements of subtitles are extremely high. Traditional whole-segment audio translation struggles to achieve precise alignment with the original audio, resulting in significant delays or desynchronization in subtitle display, impacting user experience. To address this, this application proposes using a pre-trained large-scale language translation model to annotate timestamps. By adding sentence-level timestamps to the translation results in the application, more detailed audio-translated text alignment information can be provided for live streaming services, enabling users to see translated text with finer temporal granularity while ensuring synchronization between audio and translated text. Therefore, in some embodiments, the translation results also include sentence-level timestamps associated with the translated text.

[0101] In order to output timestamps, the fixed command part of the Prompt text mentioned earlier needs to be added with "with timestamp". "With timestamp" in Prompt translates to "with attached timestamp", which enables the language translation model to have sentence-level timestamp capabilities.

[0102] The number of sentence-level timestamps corresponds to the number of translated text (clauses). Each sentence-level timestamp consists of a start time and an end time (the start and end times refer to the times within the audio stream), used to mark or record the start and end times of its corresponding clause (translated text) in the audio. Thus, sentence-level timestamps clearly indicate the start and end times of a sentence within the audio stream. Especially in live translation scenarios, sentence-level timestamps help ensure the translation matches the broadcaster's spoken language more naturally and fluently.

[0103] The specific method for associating sentence-level timestamps with translated text is to directly embed the sentence-level timestamps into the translated text. For example, the start time of the corresponding sentence-level timestamp can be inserted before the translated text, and the end time of the same sentence-level timestamp can be inserted after the translated text, forming a triplet data format, as follows: <|start_time|>Translate the text<|end_time|>; In the above triples, start_time and end_time belong to the start and end times of the same sentence-level timestamp, respectively. When the model outputs multiple translated texts, the corresponding triples are separated by a predetermined delimiter (e.g., <|sil|>), and the predetermined delimiter is also in triple form.

[0104] For example, refer to Figure 9 : At time Ta, the output of the large model includes "<|t1|><|sil|><|t2|><|t3|>Good,<|t4|>", which means that the translated text is a single clause, namely "Good," and the corresponding sentence-level timestamps are t3 and t4.

[0105] At time Tb, the output of the large model includes "<|t1|><|sil|><|t2|><|t3|>Good,<|t4|><|t5|><|sil|><|t6|><|t7|>Good,<|t8|>", which means that the translated text consists of two clauses, namely "Good," + "Good,", with corresponding sentence-level timestamps of t3 and t4, and t7 and t8, respectively. At time Tc, the output of the large model includes "<|t1|><|sil|><|t2|><|t3|>Good,<|t4|><|t5|><|sil|><|t6|><|t7|>Good,<|t8|><|t9|><|sil|><|t10|> "<|t11|> Goodbye." <|t12|><|t13|><|sil|><|t14|> means that the translated text consists of three clauses: "Okay," + "Good," + "Goodbye." The corresponding sentence-level timestamps are t3 and t4, t7 and t8, and t11 and t12, respectively.

[0106] Accordingly, refer to Figure 6 In some embodiments, step S205 includes steps S601-S603: S601: Extract sentence-level timestamps before and after the translated text from the translation results; For example, extract the timestamp from each clause triple. For example, continue to refer to... Figure 9 The timestamps extracted at time Ta are t3 and t4; the timestamps extracted at time Tb are t3 and t4, t7 and t8; and the timestamps extracted at time Tc are t3 and t4, t7 and t8, t11 and t12.

[0107] S603: Obtain the target text based on all translated text in the translation results; Understandably, if there is only one translation text, it is directly used as the target text. If there are multiple translation texts, they are merged into the target text in order (this merging can be based on sentence-level timestamps). For example, the timestamps before and after the translation texts in the output, as well as the timestamps before and after the delimiter <|sil|>, can be removed, and then the translation texts can be directly concatenated sequentially. Understandably, if there are no punctuation marks between two translation texts after removing the timestamps, punctuation marks (such as commas) can be added at these positions to separate the translation texts when concatenating them. (Reference) Figure 9 : At time Ta, after cleaning the timestamp and the segmentation character <|sil|>, there is only one translated text ("Okay,") obtained. At this time, directly output "Okay," as the target text; At time Tb, after cleaning the timestamp and the segmentation character <|sil|>, there are two translated texts ("Okay,", "Okay, sure."). At this time, it is necessary to combine "Okay," and "Okay, sure." into "Okay, Okay, sure." as one target text; At time Tc, after cleaning the timestamp and the segmentation character <|sil|>, there are three translated texts ("Okay,", "Okay, sure.", "Goodbye."). At this time, it is necessary to combine "Okay,", "Okay, sure.", and "Goodbye." into "Okay, Okay, sure., Goodbye." and output it as one target text.

[0108] S605: Based on the extracted sentence-level timestamps, align the target text with the audio corresponding to the audio stream and then output.

[0109] Specifically, it is sufficient to display the obtained target text within the time period starting from the first start moment of all the sentence-level timestamps extracted in step S601 (if this moment of the audio stream has passed, start from the current moment of the audio stream) to the last end moment.

[0110] For example Figure 9 in: When refreshing the subtitle at time Ta, when "Okay," appears within the time period between t3 and t4 of the audio stream, it is output together with the audio within this time period; when refreshing the subtitle at time Tb, "Okay, Okay, sure." will appear until the t8 moment of the audio stream; when refreshing the subtitle at time Tc, "Okay, Okay, sure., Goodbye." will appear until the t12 moment of the audio stream. ​​​​​​​​​​As mentioned earlier, the language translation big model is pre-trained based on training data. The training data includes training audio and corresponding reference text. The training audio serves as the input data for the language translation big model. The input data for the big model also includes prompt text, and the reference text corresponds to the output data of the language translation big model.

[0115] As mentioned above, the training data includes sentence-level timestamps associated with the reference text, and the translation results include the translated text and sentence-level timestamps associated with it. Furthermore, prompt words can be injected into the large model to guide the translation results. Accordingly, in some embodiments, the reference... Figure 7 The pre-training mentioned above includes S701-S707: S701: Obtain training data and corresponding prompt words for the training data; S703: Input the training audio into the language translation model and inject the prompt words corresponding to the training audio into the prompt text of the language translation model; In other words, when the training audio is input, the prompt words are also injected into the prompt text of the language translation model simultaneously, so as to inject the prompt words into the model. For details, please refer to the previous section on the method of injecting prompt words when using the model.

[0116] S705: Use a large language translation model to perform translation operations and obtain the translation results corresponding to the training audio; S707: Train a large-scale language translation model based on a preset loss; The preset loss includes: the loss between the translated text corresponding to the training audio and the reference text corresponding to the training audio, and the loss between the sentence-level timestamp associated with the translated text corresponding to the training audio and the sentence-level timestamp associated with the reference text corresponding to the training audio.

[0117] For details regarding the specific process of translation operations performed internally by the model, please refer to the previous section. Figure 10a , Figure 10b This application addresses two main aspects. First, it calculates the difference between the translated text output by the model and the reference text to optimize translation accuracy. Second, it compares the sentence-level timestamps output by the model with the real sentence-level timestamps annotated in the training data, thereby improving the model's sentence segmentation ability. The output sentence-level timestamps can be used for subsequent audio-text alignment. Based on these two loss functions, the speech encoding, text encoding, and language model in the model are jointly optimized, enabling the model to fully utilize the rich modal information in the audio. Ultimately, this achieves the ability to output high-quality translated text and accurately annotate sentence-level timestamps, meeting the needs of scenarios such as semi-streaming live subtitles.

[0118] As can be seen from the above, the data used to train the model includes: audio data (training audio) and text data (prompt text with prompt words) as input to the model, and text data (reference text and associated sentence-level timestamps) as output to the model.

[0119] In some embodiments, reference Figure 8 In some embodiments, obtaining training data mentioned in step S701 includes steps S801-S807: S801: Through automatic speech recognition, obtain the reference text corresponding to the training audio and the word-level timestamp associated with the reference text; Word-level timestamps: refer to the method of marking or recording the start and end times of each character (or the smallest linguistic unit) in audio. In this application, when recognizing a "sentence" during speech recognition, the sentence is segmented into individual characters (or tokens), and the start time and end time of the corresponding audio are recorded for each character.

[0120] Automatic speech recognition (ASR) is a technology that converts human speech into text. By inputting training audio into ASR software, you can obtain the corresponding reference text and word-level timestamps associated with it.

[0121] S803: Input the reference text and the prompt words corresponding to the training audio into the translation service interface to segment the reference text into multiple clauses; For example, in the stage of requesting the translation service interface (GPT4omini interface), the batch ASR text obtained in step S801 and the prompt words (also injected into the Prompt, which can be manually set or obtained through the HanLP tool) are input into GPT4omini to obtain the Chinese sentence segmentation translation results. The Chinese sentence segmentation translation results include the whole sentence and all clauses in the whole sentence.

[0122] In some embodiments, the obtained whole sentences and clauses can also be verified by translation data. If the splicing result of the translated text of the whole sentence and the translated text of the clause is consistent, the translation data verification is passed and the subsequent step S805 is continued.

[0123] S805: Determine the start and end timestamps of each clause based on the word-level timestamps to obtain the clause-level timestamps; Specifically, word-level timestamps also include start and end times. The start time of the first word of a clause and the end time of the last word of a clause can be used as the start and end times of that clause, thus obtaining the clause-level timestamp.

[0124] S807: Embed sentence-level timestamps into the reference text to establish a connection between sentence-level timestamps and the reference text.

[0125] Refer to the sentence-level timestamp section above for details. The time t in the timestamp can be represented as <|t|>, and combined with the reference text to form a triple format.

[0126] In some embodiments, obtaining the prompt words corresponding to the training data mentioned in step S701 includes: 1) Extract the first prompt word from the reference text in the current training data; 2) Construct a vocabulary based on the first prompt words corresponding to all training data, and randomly extract the second prompt words from the vocabulary. 3) Construct the prompt words corresponding to the current training data based on the first prompt word and the second prompt word.

[0127] That is, the training data corresponds to two prompt words: a first prompt word and a second prompt word. The first prompt word is extracted from the reference text in the current training data; the second prompt word is randomly selected from a vocabulary library, which consists of the first prompt words corresponding to all the training data.

[0128] For example, during pre-training, the first prompt word can be extracted from the reference text using the HanLP open-source toolkit. The NER pipeline is selected from HanLP, a widely used open-source Natural Language Processing (NLP) toolkit that supports multiple languages ​​and performs particularly well in Chinese word segmentation, part-of-speech tagging, and named entity recognition, making it suitable for automatic entity extraction tasks. The NER pipeline refers to the named entity recognition pipeline in NLP tools like HanLP, which can automatically identify various entities such as person names, place names, and organization names from the input text sentence by sentence. By inputting the N reference texts corresponding to the prepared N training data points into the NER pipeline, four categories of prompt words can be extracted for each training data point: Person, Location, Organization, and Miscellaneous. These extracted prompt words are used as the first prompt word for that training data point, and then the first prompt words from the N training data points are combined to form a vocabulary. For each training data point, a prompt word is randomly selected from the vocabulary library as the second prompt word for that training data point. In this way, a first prompt word and a second prompt word can be prepared for each training data point.

[0129] For example, the reference text for the first training data is "Li Lei attended the Artificial Intelligence Conference at Peking University." HanLP can automatically identify "Zhang San" (person's name) and "Peking University" (organization name) as the first prompt words for the first training data. The reference text for the second training data is "Li Si played in Shanghai for three days." HanLP can automatically identify "Li Si" (person's name) and "Shanghai" (place name) as the first prompt words for the second training data. The reference text for the third training data is "Wang Wu went to Shenzhen on a business trip." HanLP can automatically identify "Wang Wu" (person's name) and "Shenzhen" (place name) as the first prompt words for the third training data. And so on, ultimately combining the first prompt words of all training data results in a word library: {"Zhang San," "Peking University," "Li Si," "Shanghai," "Wang Wu," "Shenzhen," etc.}. Then, one (or multiple) prompt words are randomly selected from the word library for the first training data as the second prompt word. Assuming the extracted word is "Shanghai," the final prompt words for the first training data are: "Zhang San," "Peking University," and "Shanghai."

[0130] Thus, during training, the model must learn to focus on entity words closely related to the reference text in the current training data (first cue words), and also identify and distinguish interfering entity words unrelated to the reference text in the current training data (second cue words, from reference text in other training data). This design allows the model to automatically select cue words relevant to the current training data, enabling it to learn and enhance its ability to discriminate cue words, thereby increasing the robustness of cue word injection and ultimately improving the accuracy and practicality of the translation.

[0131] All data involved in this application was obtained through legal channels and has been anonymized to ensure user privacy. The data collection and use process complies with relevant laws and regulations and is used solely for training and optimizing the risk identification model; it is not linked to any specific personally identifiable information.

[0132] Based on the same technical concept, in the second aspect, refer to... Figure 11 This application also provides an audio stream processing device 1100, characterized in that it includes: The first module 1101 is used to perform speech activity detection on the received audio stream to obtain a sentence segmentation result, and to segment a target audio segment from the audio stream based on the sentence segmentation result; The second module 1102 is used to input the target audio segment into a language translation big model to perform a translation operation to obtain a translation result corresponding to the audio stream, wherein: the translation result includes translated text and sentence-level timestamps associated with the translated text; the language translation big model is obtained through pre-training based on multiple training data corresponding to multiple language types, and the training data includes: training audio, reference text corresponding to the training audio, and sentence-level timestamps associated with the reference text; The third module 1103 is used to output the translation result corresponding to the audio stream.

[0133] For more details, please refer to the method implementation examples in the first aspect, which will not be repeated here.

[0134] It should be noted that the various modules described herein are divided into modules for clarity. However, in actual implementation, the boundaries between modules may be blurred. For example, any or all functional modules in this application may share various hardware and / or software elements. As another example, any and / or all functional modules in this application may be wholly or partially implemented by a shared processor executing software instructions. Furthermore, various software sub-modules executed by one or more processors may be shared among various software modules. Accordingly, unless expressly required, the scope of this application is not limited by mandatory boundaries between various hardware and / or software elements.

[0135] Based on the same technical concept, in the third aspect, refer to Figure 12 This application embodiment also provides a computing device 1200, including a memory 1202, a processor 1201, a communication module 1203, etc. Optionally, the components can be connected and communicate with each other via a bus 1205. The memory 1202 is used to store computer programs or instructions; when the processor 1201 executes the computer program or instructions, it implements the method steps in any of the method embodiments of the first aspect. It should be noted that... Figure 12 The structure of the device 1200 shown is merely illustrative and does not constitute a limitation on the device to which the method provided in the embodiments of this application applies.

[0136] The specific entity of the computing device may be a server, used to implement the method steps in any of the method embodiments of the first or second aspect.

[0137] The memory 1202 can be used to store operating systems and computer programs or instructions, which, when invoked by the processor 1201, implement the methods shown in the embodiments of the first or second aspect of the present invention. The memory 1202 can also store programs for implementing other functions or services. The memory 1202 includes at least one type of computer-readable storage medium, including flash memory, hard disk, multimedia card, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disk, optical disk, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computing device, such as the hard disk or memory of the computing device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computing device, such as a plug-in hard disk, a secure digital card (SD card), a flash memory card, etc., provided on the computing device. Of course, the computer-readable storage medium can also include both internal storage units and external storage devices of the computing device. In this embodiment, the computer-readable storage medium is typically used to store software installed on the computing device, such as program code of the methods of the embodiments of the first or second aspect. Furthermore, the computer-readable storage medium can also be used to temporarily store various types of data that have been output or will be output.

[0138] The processor 1201 is connected to the memory 1202 via a bus 1205 and performs corresponding functions by calling the application programs stored in the memory 1202. In some embodiments, it may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. The processor 1201 is typically used to control the overall operation of the processing device, such as performing control and processing related to data interaction or communication with other entities. In this embodiment, the processor 1201 is used to run program code stored in the memory 1202 or process data.

[0139] The computing device 1200 can connect to a network via a communication module 1203 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as user terminals or servers) through the network and achieve data interaction, such as sending data to or receiving data from other devices. The communication module 1203 may include a wired network interface and / or a wireless network interface, meaning the communication module may include at least one of a wired communication module or a wireless communication module.

[0140] The computing device 1200 can connect to necessary input / output devices, such as a keyboard and display device, via the input / output interface 1204. The device 1200 itself may have a display device, and other display devices can also be connected externally via interface 1204. It is understood that the input / output interface 1204 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 1204 can be a component of the device 1200, or it can be an external device connected to the device 1200 when needed.

[0141] The bus 1205 used to connect the components may include a path for transmitting information between the components. The bus 1205 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Depending on its function, the bus 1205 may be divided into an address bus, a data bus, a control bus, etc.

[0142] Based on the same technical concept, this application also provides a computer-readable storage medium storing a computer program or instructions. When the computer program or instructions are executed by a processing device, they implement the method steps in any of the method embodiments of the first or second aspect. Further details can be found in the method embodiments, which will not be repeated here. In this embodiment, the computer-readable storage medium can be non-volatile or volatile. Computer-readable storage media include flash memory, hard disks, multimedia cards, random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), magnetic disks, optical disks, etc. In some embodiments, the computer-readable storage medium can be an internal storage unit of a computing device, such as the hard disk or memory of the computing device. In other embodiments, the computer-readable storage medium can also be an external storage device of the computing device, such as a plug-in hard disk, a secure digital card (SD card), a flash memory card, etc., equipped on the computing device. Of course, the computer-readable storage medium can also include both the internal storage unit and the external storage device of the computing device. In this embodiment, the computer-readable storage medium is typically used to store software installed on the computing device, such as program code of the methods of the embodiments of the first or second aspect. In addition, computer-readable storage media can also be used to temporarily store various types of data that have been output or will be output.

[0143] Based on the same technical concept, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the first or second aspect of the above-described method embodiments.

[0144] It should be noted that the order in which the embodiments are described in this application is not intended to limit the priority of the embodiments. The reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0145] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application and in its specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0146] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many forms under the guidance of this application without departing from the spirit and scope of protection of the claims. All equivalent transformations made under the inventive concept of this application using the content of this application's specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.

Claims

1. An audio stream processing method, characterized in that, include: Speech activity detection is performed on the received audio stream to obtain sentence segmentation results, and target audio segments are segmented from the audio stream based on the sentence segmentation results; The target audio segment is input into a large-scale language translation model to perform a translation operation to obtain a translation result corresponding to the audio stream, wherein: the translation result includes the translated text and a sentence-level timestamp associated with the translated text; the large-scale language translation model is obtained through pre-training based on multiple training data corresponding to multiple language types, and the training data includes: training audio, reference text corresponding to the training audio, and sentence-level timestamp associated with the reference text; Output the translation result corresponding to the audio stream.

2. The method according to claim 1, characterized in that, The step of performing speech activity detection on the received audio stream to obtain sentence segmentation results, and then segmenting the target audio segment from the audio stream based on the sentence segmentation results, includes: The received audio stream data blocks are stored in the audio buffer, and voice activity is detected on the audio stream data blocks; When the detection finds that the first type of detection condition is met, all data blocks are obtained from the audio cache as the target audio segment, and the audio cache is cleared. The first type of detection condition includes the condition that the latest cached data block is the end of a sentence or the length of all the cached data blocks reaches a preset length. When the detection finds that the second type of detection condition is met, all data blocks are obtained from the audio cache as the target audio segment. The second type of detection condition includes the condition that the latest cached data block is the end of a clause.

3. The method according to claim 2, characterized in that, The first type of detection conditions includes: the silence duration is greater than the first duration, or the cumulative detection duration of all cached data blocks reaches a preset duration; The second type of detection conditions includes a silence duration greater than a second duration and less than a first duration, wherein the first duration is greater than the second duration.

4. The method according to any one of claims 2-3, characterized in that, The method further includes: In response to a triggering event indicating that translation needs to be interrupted, all data blocks are retrieved from the audio cache and used as input to the pre-trained language translation model to obtain the translation result.

5. The method according to any one of claims 1-4, characterized in that, The output of the translation result includes: extracting sentence-level timestamps before and after the translated text from the translation result; obtaining the target text based on all the translated text in the translation result; and aligning the target text with the audio corresponding to the audio stream based on the extracted sentence-level timestamps before outputting it.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: before the first target audio segment is input into the language translation big model, obtaining a prompt word corresponding to the audio stream, and injecting the prompt word into the prompt text of the language translation big model.

7. The method according to claim 6, characterized in that, The translation operation includes: encoding the input target audio segment into a speech embedding representation, encoding the prompt text into a text embedding representation, and inputting the speech embedding representation and the text embedding representation into a language model to obtain the translation result.

8. The method according to any one of claims 1-7, characterized in that, The pre-training includes: Obtain the training data and the corresponding prompt words for the training data; The training audio is input into the language translation model, and the prompt words corresponding to the training audio are injected into the prompt text of the language translation model. The translation result corresponding to the training audio is obtained by performing a translation operation using a large language translation model. The language translation model is trained based on a preset loss, which includes: the loss between the translated text corresponding to the training audio and the reference text corresponding to the training audio, and the loss between the sentence-level timestamp associated with the translated text corresponding to the training audio and the sentence-level timestamp associated with the reference text corresponding to the training audio.

9. The method according to claim 8, characterized in that, The step of obtaining the prompt words corresponding to the training data includes: The first prompt word is extracted from the reference text in the current training data; A vocabulary is constructed based on the first prompt words corresponding to all the training data, and the second prompt word is randomly extracted from the vocabulary. The prompt words corresponding to the current training data are constructed based on the first prompt word and the second prompt word.

10. The method according to any one of claims 8-9, characterized in that, The acquisition of training data includes: Automatic speech recognition is used to obtain the reference text corresponding to the training audio and the word-level timestamp associated with the reference text. The reference text and the prompt words corresponding to the training audio are input into the translation service interface to segment the reference text into multiple clauses; The start and end timestamps of each clause are determined based on the word-level timestamps to obtain the sentence-level timestamps; The sentence-level timestamp is embedded into the reference text to establish a connection between the sentence-level timestamp and the reference text.

11. An audio stream processing device, characterized in that, include: The first module is used to perform speech activity detection on the received audio stream to obtain sentence segmentation results, and to segment target audio segments from the audio stream based on the sentence segmentation results; The second module is used to input the target audio segment into the language translation big model to perform a translation operation to obtain the translation result corresponding to the audio stream, wherein: the translation result includes the translated text and the sentence-level timestamp associated with the translated text; the language translation big model is obtained by pre-training based on multiple training data corresponding to multiple language types, and the training data includes: training audio, reference text corresponding to the training audio, and sentence-level timestamp associated with the reference text; The third module is used to output the translation result corresponding to the audio stream.

12. A computing device, characterized in that, It includes a memory and a processor, the memory being used to store computer programs or instructions; when the computer programs or instructions are executed by the processor, the method of any one of claims 1-10 is implemented.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a processor, implement the method of any one of claims 1-10.

14. A computer program product, characterized in that, The computing program product includes a computer program or instructions that, when executed by a processor, implement the method of any one of claims 1-10.

Images