Real-time interpretation method and device, electronic equipment, product and storage medium

Abstract

The application is suitable for the technical field of voice processing, and provides a real-time mutual translation method and device, electronic equipment, product and storage medium. The application first acquires audio data, performs endpoint detection on the audio data, and obtains a plurality of voice segments. Then, the translation processing is performed on the plurality of voice segments concurrently. A sequence token is set for each voice segment, the sequence token is used to indicate the arrangement order of each voice segment in the plurality of voice segments, a statistical counter is set for each user, and when the statistical counter indicates that the number of obtained translation results and the number of voice segments of the user are equal, the translation results of the plurality of voice segments are output in the order indicated by the sequence tokens of the plurality of voice segments. Even if the translation results of the plurality of voice segments are returned in disorder, the final output order is still consistent with the real speaking order of the speaker, so that semantic dislocation is avoided.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and in particular to a real-time translation method, apparatus, electronic device, product, and storage medium. Background Technology

[0002] In real-time translation scenarios involving two or more people, the speech is typically captured after the speaker finishes speaking, then sent as a whole to the cloud for translation processing. Once the translation is complete, the translated text is played back. Because the translation can only proceed after the entire speech has finished, the output latency is relatively high. To reduce latency, the entire speech can be broken down into multiple segments, and each segment can be translated and processed separately.

[0003] However, because each segment is translated separately, the completion time for each segment's translation is often uncertain, and the translated results are usually not returned in the order in which the speakers delivered the translations. If the translation results are broadcast directly in the order they arrive, it will lead to a disordered translation order, affecting semantic understanding. Summary of the Invention

[0004] This application provides a real-time translation method, apparatus, electronic device, product, and storage medium to enable the playback of translated text in the order of speaking and to prevent omissions of translated text.

[0005] To achieve the above objectives, this application adopts the following technical solution: Firstly, a real-time translation method is provided, the method comprising: Acquire the audio data to be processed; Endpoint detection is performed on the audio data to obtain detection results. The detection results include multiple first speech segments of the first user. Each first speech segment has a corresponding sequence token. The sequence token of the first speech segment is used to indicate the order of the first speech segment in the multiple first speech segments. The first user has a corresponding first statistical count. The first statistical count is used to indicate the number of first speech segments of the first user. The plurality of first speech segments are translated, and the translation processing of the plurality of first speech segments is performed concurrently; After obtaining the translation result corresponding to each of the first speech segments, update the first statistical count; When the number of translated results obtained after the updated first statistical count is equal to the number of the first speech segments, the translated results of the plurality of first speech segments are output in the order indicated by the sequence tokens of the plurality of first speech segments.

[0006] Optionally, the endpoint detection of the audio data to obtain the detection result includes: During the detection of the audio data, if the duration of the first user being in a speaking state is greater than or equal to a first time threshold, or if the duration of the first user entering a silent state after being detected as being in a speaking state is greater than or equal to a second time threshold, then the audio data in which the first user is in a speaking state is segmented into the first speech segment.

[0007] Optionally, the endpoint detection of the audio data to obtain the detection result includes: Endpoint detection is performed on the audio data to obtain multiple first speech segments arranged in chronological order. The translation processing of the plurality of first speech segments includes: For each of the first speech segments obtained, the first speech segment is translated.

[0008] Optionally, the translation processing of the first speech segment includes: The first speech segment is identified to obtain multiple identified sub-texts arranged in chronological order; For each identified subtext obtained, the identified subtext is translated to obtain the corresponding translated sub-result; Based on the translation sub-results corresponding to the multiple identified sub-texts, the translation result corresponding to the first speech segment is obtained.

[0009] Optionally, the audio data includes multiple split-channel voice streams, each of which corresponds to a user; The plurality of branch paths include a first branch path and a second branch path, wherein the first branch path corresponds to the first user and the second branch path corresponds to the second user; The detection result also includes multiple second speech segments of the second user: each second speech segment has a corresponding sequence token, the sequence token of the second speech segment is used to indicate the order of the second speech segment in the multiple second speech segments, and the second user has a corresponding second statistical count, the second statistical count is used to indicate the number of second speech segments of the second user; The method further includes: translating the plurality of second speech segments, wherein the translation processing of the plurality of second speech segments is performed concurrently; After obtaining the translation result corresponding to each of the second speech segments, update the second statistical count; When the number of translated results corresponding to the second speech segment obtained after the updated second statistical count is equal to the number of the second speech segments, the translated results of the multiple second speech segments are played in the order indicated by the sequence tokens of the multiple second speech segments.

[0010] Optionally, if the duration of the first user's speaking state is detected to be greater than or equal to a first time threshold, the audio data in which the first user is speaking is segmented into the first speech segment, including: If the duration of the first user's speaking state is detected to be greater than or equal to the first time threshold, then the audio data in which the first user is speaking is segmented into the first speech segment based on the segmentation time. The segmentation time satisfies at least one of the following conditions: The audio energy of the audio data at the segmentation time is lower than the audio energy of the audio data at other times in the adjacent time window of the target time; The segmentation time is the time corresponding to a short pause position of the first user in the time window adjacent to the target time, and the target time is the time when the duration of the first user's speaking state reaches a first time threshold.

[0011] Secondly, a real-time translation device is provided, the real-time translation device comprising: The acquisition module is used to acquire the audio data to be processed; The detection module is used to perform endpoint detection on the audio data and obtain detection results. The detection results include multiple first speech segments of the first user, each first speech segment having a corresponding sequence token. The sequence token of the first speech segment is used to indicate the order of the first speech segment in the multiple first speech segments. The first user has a corresponding first statistical count, which is used to indicate the number of first speech segments of the first user. A translation module is used to translate the plurality of first speech segments, wherein the translation processing of the plurality of first speech segments is performed concurrently; The update module is used to update the first statistical count after obtaining the translation result corresponding to each of the first speech segments; The output module is configured to output the translation results of the plurality of first speech segments in the order indicated by the sequence tokens of the plurality of first speech segments when the number of translated results obtained after the updated first statistical count is equal to the number of the first speech segments.

[0012] Thirdly, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the real-time translation method as described in any of the first aspects.

[0013] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the real-time translation method as described in any of the first aspects.

[0014] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the real-time translation method as described in any of the first aspects.

[0015] The real-time translation method provided in this application has the following advantages: First, audio data is acquired and endpoint detection is performed on the audio data to obtain multiple speech segments; then, translation processing is performed concurrently on the multiple speech segments; a sequence token is set for each speech segment, indicating the order of each speech segment among the multiple speech segments, and a statistical count is set for each user; when the number of translated results indicated by the statistical count is equal to the number of speech segments of the user, the translated results of the multiple speech segments are output in the order indicated by the sequence tokens of the multiple speech segments. Performing concurrent translation on multiple speech segments eliminates the need to wait for the entire round of dialogue to end before processing begins, shortening the translation output latency. Because a corresponding sequence token is maintained for each speech segment of each user, and the translated results of the multiple speech segments are output in the order indicated by the sequence tokens of the multiple speech segments, even if the translated results of the multiple speech segments are returned out of order, the final output order is still consistent with the speaker's actual speaking order, avoiding semantic misalignment. Because a corresponding statistical count is maintained for each user, a unified broadcast is triggered only when the number of translated results indicated by the statistical count equals the number of user's voice segments. This ensures that all submitted voice segments of the user have received translated results before outputting, preventing any omissions in the translated results. Since each voice segment has a corresponding sequence token, meaning each sequence token corresponds to a voice segment and a translated result, duplicate translated results are avoided. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application; Figure 2 A flowchart illustrating a real-time translation method provided in an embodiment of this application; Figure 3 This is a schematic diagram of a real-time translation device provided in an embodiment of this application. Detailed Implementation

[0017] It should be noted that the terminology used in the embodiments of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] In real-time translation scenarios involving two or more people, to achieve "speaking and translating simultaneously" and reduce waiting time, it is usually necessary to divide the entire audio segment into multiple audio segments and translate each segment separately.

[0020] Because the completion time of translation processing for each segment is often uncertain, the translated results are usually not returned in the original speaker's speaking order. For example, if segment 1 was generated earlier than segment 2, and segment 1 and segment 2 are translated sequentially, the translation processing time for segment 1 will be longer than that for segment 2, resulting in the translated result for segment 2 being obtained first, followed by the translated result for segment 1. If the translated results are broadcast directly in the order they arrive, the translation order will be disordered, affecting semantic understanding.

[0021] This application provides a real-time translation method that first acquires audio data and performs endpoint detection to obtain multiple speech segments. Then, it performs concurrent translation processing on these multiple speech segments. A sequence token is set for each speech segment, indicating its order within the multiple speech segments. A statistical count is set for each user. When the statistical count indicates that the number of translated results obtained is equal to the number of speech segments a user has, the translated results of the multiple speech segments are output in the order indicated by the sequence tokens. Performing concurrent translation on multiple speech segments eliminates the need to wait for the entire dialogue to end before processing begins, shortening the translation output latency. Because a corresponding sequence token is maintained for each speech segment of each user, and the translated results of the multiple speech segments are output in the order indicated by the sequence tokens, even if the translated results of the multiple speech segments are returned out of order, the final output order still matches the speaker's actual speaking order, avoiding semantic misalignment. Because a corresponding statistical count is maintained for each user, a unified broadcast is triggered only when the number of translated results indicated by the statistical count equals the number of user's voice segments. This ensures that all submitted voice segments of the user have received translated results before outputting, preventing any omissions in the translated results. Since each voice segment has a corresponding sequence token, meaning each sequence token corresponds to a voice segment and a translated result, duplicate translated results are avoided.

[0022] The real-time translation method provided in this application can be applied to electronic devices.

[0023] Figure 1 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application, such as... Figure 1 As shown in the embodiment of this application, an electronic device 10 is provided, which may be a mobile terminal or a server.

[0024] If the electronic device is a mobile terminal, then the electronic device can be a device such as a smartphone, tablet computer, or personal computer.

[0025] If the electronic device is a server, then the electronic device can be a standalone server, a server cluster, a cloud server, or other similar devices.

[0026] like Figure 1 As shown, the electronic device may include a processor 101, a memory 102, a communication bus 103, and a communication interface 104. The processor 101 is connected to the memory 102 and the communication interface 104 via the communication bus 103. It should be noted that... Figure 1 The illustrated device structure does not constitute a limitation on the electronic device 10. The electronic device 10 may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. This application embodiment does not limit this. The following is in conjunction with... Figure 1 A detailed description of each component of the electronic device 10 is provided.

[0027] The processor 101 can be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, used to execute relevant programs to perform the steps executed by the electronic device in the real-time translation method described in the embodiments of this application. The processor 101 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the real-time translation method of this application can be completed by the integrated logic circuits in the hardware of the processor 101 or by instructions in software form. The processor 101 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 102. Processor 101 reads information from memory 102 and, in conjunction with its hardware, executes the steps performed by the electronic device in the real-time translation method of this application embodiment. Memory 102 can be read-only memory (ROM), static storage medium, dynamic storage medium, or random access memory (RAM). Memory 102 can store programs and data, such as the program for the real-time translation method in this application embodiment. When the program stored in memory 102 is executed by processor 101, processor 101 executes the various steps performed by the electronic device in the real-time translation method of this application embodiment. Communication interface 104 uses a transceiver device, such as, but not limited to, a transceiver, to realize communication between electronic device 10 and other devices or communication networks.

[0028] Please see Figure 2 , Figure 2This is a flowchart illustrating a real-time translation method provided in an embodiment of this application. The following describes the application of the real-time translation method. Figure 1 Taking the electronic device shown as an example, this real-time translation method includes the following steps: S201. Obtain the audio data to be processed.

[0029] In one example, audio data can be acquired in real time via a voice acquisition device, which can be a translation headset or other acquisition device, such as a microphone.

[0030] In one example, throughout the session, the electronic device allocates an audio buffer for audio data and continuously appends newly acquired audio data to the end of the audio buffer, which stores continuous audio data from the start of the session to the current moment.

[0031] As an example, the audio data could be a user's voice stream during a session. Since the audio data is captured in real time, the audio data to be processed is constantly changing during the user's session, for example, new voice streams are continuously added.

[0032] S202. Perform endpoint detection on the audio data to obtain the detection results, which include multiple first speech segments of the first user.

[0033] In this system, multiple first speech segments are arranged sequentially in chronological order. Each first speech segment has a corresponding sequence token. The sequence token of each first speech segment is used to indicate the order of the first speech segment among the multiple first speech segments. The generation order of the multiple first speech segments can be determined by the sequence tokens of the multiple first speech segments. The first user has a first statistical count, which is used to indicate the number of first speech segments of the first user, or in other words, the number of first speech segments obtained by endpoint detection of the audio data.

[0034] In one example, endpoint detection is performed on the audio data sequentially according to the speaking time to obtain multiple first speech segments of the first user arranged in chronological order. Each time a first speech segment is obtained, a corresponding sequence token is generated for that first speech segment, and the first statistical count of the first user is updated.

[0035] In one example, multiple first speech segments from a first user are arranged sequentially in chronological order, with the sequence tokens corresponding to these segments increasing sequentially. For instance, the sequence token for the first speech segment is 1, and the sequence token for the second speech segment is 2. The time here can represent the duration of the user's speech.

[0036] In one example, each time a first voice segment is obtained, the first statistical count of the first user is incremented by 1, and the first statistical count can be used to indicate the actual number of first voice segments obtained from the first user.

[0037] In one example, the audio data is a user's voice stream, and the voice segments are audio clips of the user speaking, such as the first voice segment being audio clips of the first user speaking.

[0038] As an example, audio data within a preset time window can be detected according to a preset detection period, where the detection period indicates the frequency of detection, for example, 100ms. The preset time window indicates the duration of each detection.

[0039] By detecting whether the user is in a silent or speaking state within a preset time window of the audio data, the audio of the user in a speaking state is segmented to obtain detection results including multiple speech segments.

[0040] One example involves detecting the audio energy of audio data within a preset time window to determine whether a user is in a silent or speaking state. Speech signals typically have higher audio energy when speaking, while silent audio energy is lower. Therefore, the audio energy of the audio data within the preset time window can be calculated and compared to a preset energy threshold: if the audio energy is greater than or equal to the preset energy threshold, the user is determined to be speaking; if the audio energy is less than the preset energy threshold, the user is determined to be silent.

[0041] In one example, during the detection of audio data, if the duration of a first user's speaking state is greater than or equal to a first time threshold, the audio data in which the first user is speaking is segmented into a first speech segment. That is, audio data in which the user is speaking for a duration equal to the first time threshold is considered as a single first speech segment. When the speaker speaks continuously for an extended period, the length of a single speech segment is controlled within a preset upper limit (i.e., the first time threshold) to prevent the speech segment from growing indefinitely due to continuous speaking. This shortens the translation output latency and improves the real-time experience of "speaking and translating simultaneously." Limiting the length of a single speech segment by the first time threshold avoids increased backend processing burden and latency due to excessively large audio requests per request, thus reducing buffering and memory overhead.

[0042] In one example, the initial time threshold is 5000ms.

[0043] One example is that the first time threshold can be adjusted based on at least one of the following: current network conditions, processing engine load, and speaker's speech rate. For instance, when network conditions are poor or processing engine load is high, the first time threshold is decreased to reduce the size of the first speech segment; when the speaker's speech rate is fast, the first time threshold is increased to avoid excessively frequent segmentation.

[0044] Specifically, when a speaker pauses for an extended period and the duration of a speech segment reaches a certain first time threshold, the current speech segment must be forcibly segmented to prevent it from growing indefinitely. However, if segmentation is performed directly at the point where the first time threshold is reached, a word or semantic unit may be cut in the middle, causing it to be split into two speech segments. When subsequent segments are translated independently, the segmented word or semantic unit may be mistranslated, thus reducing the quality of the translation.

[0045] In one example, if the duration of the first user's speaking state is detected to be greater than or equal to a first time threshold, the audio data in which the first user is speaking is segmented into the first speech segment based on the segmentation time.

[0046] In one example, the audio energy at the segmentation point is lower than the audio energy at other points within a preset adjacent time window of the target time. This segmentation point corresponds to a gap or weakened segment of the user's pronunciation, where the segmentation causes minimal disruption to speech continuity.

[0047] Of course, in other examples, the segmentation time can also be the time corresponding to a short pause position of the user in a preset adjacent time window of the target time, such as the pause between different words or different phrases spoken by the user.

[0048] In other examples, the segmentation time can also be the time corresponding to the position of the punctuation mark within a preset adjacent time window of the target time. For example, if the audio data of the preset adjacent time window of the target time is identified, and the identified sentence contains punctuation marks, and the time corresponding to the position of the punctuation mark is within the preset adjacent time window, then the time corresponding to the position of the punctuation mark is the segmentation time.

[0049] In other examples, the segmentation time can also be the time corresponding to the prosodic boundary position within a preset adjacent time window of the target time. For example, by recognizing the audio data within the preset adjacent time window of the target time, the prosodic boundary position of the user's spoken sentence is identified, and the time corresponding to the prosodic boundary is the segmentation time.

[0050] In one example, during audio data detection, if a user enters a silent state after being detected speaking, and the duration of the silence is greater than or equal to a second time threshold, the audio data during the speaking state is segmented into a first speech segment. That is, the audio data during the speaking state is considered as one first speech segment. By detecting when a user enters a silent state and then determining whether the duration of the silence reaches the second time threshold, short pauses during speech can be avoided from being misjudged as speech segment boundaries, thus accurately identifying the speech segment corresponding to a single continuous speech. Speech segmentation is only triggered when it is confirmed that the user is speaking and the silence duration is greater than or equal to the second time threshold, avoiding excessive segmentation that increases the number of speech segments and reducing the processing burden.

[0051] One example is a second time threshold of 500 ms. In natural conversational flow, pauses within a sentence (breathing, hesitation, word cohesion) are typically in the range of 200–400 ms, while pauses between sentences or semantic units are generally greater than 500 ms. Setting the second time threshold to 500 ms falls precisely on the boundary between "intra-sentence pauses" and "inter-sentence pauses." Setting the second time threshold too short can lead to accidental segmentation mid-sentence, causing semantic fragmentation, excessive segmentation, and consequently lowering translation quality; setting it too long unnecessarily increases the delay of the initial translation.

[0052] One example is adjusting the second time threshold based on at least one of the following: ambient noise background, speaker's speaking speed, or historical pause distribution. For instance, when the ambient noise background is high, the second time threshold is increased to avoid misjudging noise as the end of a speech; when the speaker's speaking speed is fast, the second time threshold is decreased to reduce the delay in the translated output; based on the historical pause distribution, the duration patterns of the speaker's intra-sentence and inter-sentence pauses can be statistically analyzed, and a second time threshold that better matches the speaker's speaking habits can be set based on these duration patterns.

[0053] In one example, during endpoint detection of audio data, after a speech segment is detected in the detection window, the end position of the detection window is used as the starting position for the next endpoint detection, and endpoint detection of the audio data continues until the audio data detection is completed.

[0054] As an example, if audio data is stored in an audio buffer, when performing endpoint detection on the audio data, first, a detection snapshot of the audio data in the audio buffer is generated, and then endpoint detection is performed based on the detection snapshot.

[0055] In one example, during the continuous reception and storage of audio data in the audio buffer, a dirty flag is set in the audio buffer to indicate whether the audio data in the audio buffer has changed since the last detection snapshot was generated. When the dirty flag is set, it indicates that new audio data has been written to the audio buffer. The detection snapshot is then regenerated based on the audio data within a detection window of a preset duration at the end of the current audio data, and the dirty flag is cleared. Here, the preset duration detection window refers to a fixed duration window for audio energy detection, such as approximately 300ms.

[0056] When the dirty flag is not set, it indicates that the audio data in the audio buffer has not changed since the last detection snapshot was generated. The already generated detection snapshot is directly reused for audio energy detection, instead of being regenerated. This avoids a full data copy of the audio buffer in each detection cycle; the detection snapshot is only updated when the data actually changes, thus reducing computational and memory overhead.

[0057] S203. Translate multiple first speech segments concurrently.

[0058] In one example, audio data is detected sequentially according to speaking time to obtain multiple first speech segments of the first user arranged in chronological order. Each first speech segment is translated without waiting for the next first speech segment or the previous first speech segment of the first user to be translated. This allows for concurrent translation of multiple first speech segments of the first user, thereby improving translation efficiency and reducing translation latency.

[0059] In one example, after obtaining each first speech segment, the first speech segment is first identified to obtain the identified text, and then the identified text is translated to obtain the corresponding translation result.

[0060] One example is that after the identified text is obtained, it is displayed so that the first user or other users can see the progressively generated identified text as the first user speaks.

[0061] In one example, after obtaining each first speech segment, the first speech segment is first input into the speech recognition engine (SR) to obtain the recognized text, and then the recognized text is input into the machine translation engine (MT) to obtain the translated result.

[0062] The recognition engine can recognize multiple first speech segments in parallel, and the machine translation engine can translate multiple recognized texts in parallel.

[0063] As an example, the recognition engine and machine translation engine can be deployed locally, on a cloud server, or implemented by local devices in collaboration with the cloud.

[0064] In one example, after obtaining each first speech segment, the first speech segment is first recognized, resulting in multiple recognized sub-texts arranged in chronological order. Each recognized sub-text is then translated to obtain a corresponding translated sub-result. Based on the translated sub-results of the multiple recognized sub-texts, the translated result corresponding to the first speech segment is then obtained. That is, during the recognition of the first speech segment, it is divided into multiple sub-segments. After each sub-segment's recognized sub-text is identified, it is translated. The concurrent execution of the translation of multiple recognized sub-texts enables simultaneous recognition and translation of the first speech segment, further reducing translation latency.

[0065] For example, the first speech segment is "The weather is nice today. Let's go to the park this afternoon. Let's invite Xiao Li along." First, after recognizing "The weather is nice today," it is output as a recognized subtext and translated. Then, after recognizing "Let's go to the park this afternoon," it is output as a recognized subtext and translated, thus achieving concurrent execution of the translation of the two recognized subtexts.

[0066] In one example, a corresponding sequence token is assigned to each identified subtext. This sequence token indicates the order in which the identified subtext is arranged among multiple identified subtexts. Thus, after obtaining the translation sub-results of multiple identified subtexts, the translation sub-results are concatenated based on the sequence tokens of the multiple identified subtexts to obtain the translation result corresponding to the first speech segment.

[0067] In one example, each recognized subtext is determined based on sentence boundary information output during the recognition process. Sentence boundary information includes one or more of the following: period, question mark, exclamation mark, semicolon, and comma. Thus, during the recognition of the first speech segment, if sentence boundary information is recognized, a recognized subtext is obtained; that is, each recognized subtext includes at least one sentence, and that sentence includes at least one sentence boundary information.

[0068] S204. After obtaining the translation result corresponding to each first speech segment, update the first statistical count.

[0069] The process involves concurrent translation of multiple first speech segments to obtain translation results for each segment. Since the time required for translation of each segment differs, the order of the translation results obtained during concurrent processing can vary. After obtaining the translation result for each segment, the first statistical count is updated.

[0070] S205. When the number of translated results obtained after the updated first statistical count is equal to the number of first speech segments of the first user, output the translated results of multiple first speech segments in the order indicated by the sequence tokens of multiple first speech segments.

[0071] The sequence token of the first speech segment indicates the order in which the first speech segment is arranged among multiple first speech segments. The multiple first speech segments are arranged in chronological order or in the order in which the first user speaks. Therefore, the translation results of the multiple first speech segments are output in the order indicated by the sequence tokens of the multiple first speech segments, so that the output order of the translation results is the same as the actual speaking order of the first user.

[0072] One example is outputting the translation results of multiple first speech segments in the order indicated by sequence tokens of multiple first speech segments, including: playing the translation results of multiple first speech segments in the order indicated by sequence tokens of multiple first speech segments.

[0073] One example is outputting the translation results of multiple first speech segments in the order indicated by sequence tokens of the multiple first speech segments, including: The translations of multiple first speech segments are concatenated according to the sequence tokens indicated by these tokens, and the concatenated translation is output. For example, in a meeting minutes scenario, the concatenated translation can be written into the meeting minutes document.

[0074] In one example, after obtaining the translation result corresponding to each first speech segment, the first statistical count is decremented by one. When the first statistical count is 0, it indicates that the number of translation results is the same as the number of first speech segments, that is, the translation results of multiple first speech segments of the first user have been obtained.

[0075] In one example, when the first user finishes speaking in this round, and the updated first statistical count indicates that the number of translated results obtained is equal to the number of the first user's first speech segments, the translated results of the multiple first speech segments are played in the order indicated by the sequence tokens of the multiple first speech segments.

[0076] In one example, if at least one of the following conditions is met: the duration of the silent state is greater than or equal to the end determination threshold, a speech end indication is received, or the speech button corresponding to the voice channel is detected to change from a pressed state to a released state, then the first user's speech in this round is confirmed to have ended.

[0077] Thus, this application first acquires audio data and performs endpoint detection segmentation to obtain multiple speech segments. Then, it performs concurrent translation processing on these multiple speech segments to obtain the corresponding translation result for each segment. Simultaneously, a sequence token is set for each speech segment, and a statistical count is set for each user. When the statistical count indicates that the number of obtained translation results equals the number of speech segments for the user, the translation results of the multiple speech segments are played in the order indicated by the sequence tokens. Concurrent translation of multiple speech segments eliminates the need to wait for the entire dialogue round to end before processing begins, shortening the translation output latency. Because a corresponding sequence token is maintained for each speech segment of each user, and the translation results of the multiple speech segments are played in the order indicated by the sequence tokens, even if the translation results are returned out of order, the final playback order still matches the speaker's actual speaking order, avoiding semantic misalignment. Because a corresponding statistical count is maintained for each user, a unified broadcast is triggered only when the number of translated results indicated by the statistical count is equal to the number of user's voice segments. This ensures that all submitted voice segments of the user have received the final translation results before broadcasting, without omission or duplication.

[0078] The following example illustrates this using a single speaker delivering consecutive remarks.

[0079] The second time threshold is 500ms, and the first time threshold is 5000ms. User A speaks continuously through the voice channel corresponding to the left earphone, as follows: At t=0.0s, user A starts speaking, continuously acquiring the audio stream of that channel, obtaining the audio data to be processed, and writing it into the audio buffer. The segment start offset pointer points to 0.0s.

[0080] At t=2.3s, User A pauses speaking and enters a silent state.

[0081] At t=2.8s, the duration of the user's silence reaches the second time threshold of 500ms, triggering segment segmentation: the audio data from the segment start offset of 0.0s to the current end minus the position corresponding to the silence window is identified as speech segment 1; the sequence token for speech segment 1 is set to 1; speech segment 1 is translated; the statistical count of this segment is incremented by one, and the statistical count is now 1. Subsequently, the segment start offset pointer is updated to 2.8s, and the audio data stored in the audio buffer remains unchanged.

[0082] At t=2.9s, User A continues to speak, accumulating a new voice stream starting from the updated starting offset of 2.8s.

[0083] At t=4.5s, User A enters a silent state again from the speaking state. The duration of the silent state reaches the silence duration threshold of 500ms, triggering segment segmentation: the audio data from the starting offset of 2.8s to the current end minus the silence window is determined as speech segment 2; sequence token 2 is set for speech segment 2; speech segment 2 is translated; the statistical count of this segment is incremented by one, and the statistical count is now 2.

[0084] At t=5.5s, the translation result of speech segment 2 is returned first. When this branch receives the translation result of speech segment 2, it decrements the statistical count by one, and the statistical count is 1. Since the statistical count is not zero, the unified broadcast of this branch is not triggered, and the translation result of speech segment 2 is only cached according to sequence token 2.

[0085] At t=6.0s, User A releases the talk button, ending the current round of speaking. At this time, the statistical count of this channel is still not zero (it is 1), so only the broadcast of this voice channel is suspended, without affecting the broadcast of other voice channels.

[0086] At t=6.4s, the translation result of speech segment 1 is returned. The branch receives the translation result of speech segment 1 and decrements the statistical count by one. At this time, the statistical count returns to zero, indicating that the number of translation results is equal to the number of speech segments, triggering the unified broadcast of the branch: according to the sequence token order of each speech segment in the branch, that is, the translation result of speech segment 1 corresponding to sequence token 1 is broadcast first, and then the translation result of speech segment 2 corresponding to sequence token 2 is broadcast. The broadcast order is consistent with the actual speaking order of user A.

[0087] Therefore, even if speech segment 1 is processed slowly due to network issues and its translation result is returned later than that of speech segment 2, this branch can still broadcast the results in the order of sequence tokens after all the translated results have arrived, so that the broadcast order is consistent with the speaker's actual speaking order.

[0088] In real-time translation scenarios involving two or more people, existing solutions typically employ a time-sharing mechanism: by detecting which party is currently speaking, only one stream of speech is translated and processed at a time, switching to the other stream only after that stream finishes. Because only one stream is processed at a time, even if both parties speak almost simultaneously, they must queue and wait for the previous stream to finish before processing the other, failing to support true concurrent duplexing. In face-to-face natural conversations, overlapping speech times are common. If queuing is required, the translation of one party's speech must wait for the other party's speech to finish before being output, resulting in a significant lag in translation output compared to the actual conversation's pace and a noticeable decline in user experience.

[0089] In one example, the audio data to be processed includes multiple speech streams in branches (or channels), each branch corresponding to a speaker or user; that is, the speech stream of each branch is the speech stream of the corresponding speaker. For example, the multiple branches may include a first branch and a second branch, with the first branch corresponding to a first user and the second branch corresponding to a second user. The detection result also includes multiple second speech segments of the second user: each second speech segment has a corresponding sequence token, which indicates the order of the second speech segment among the multiple second speech segments. The second user has a corresponding second statistical count, which indicates the number of second speech segments for the second user. Then, the multiple second speech segments are translated concurrently. After obtaining the translation result for each second speech segment, the second statistical count is updated. When the updated second statistical count indicates that the number of obtained translation results is equal to the number of second speech segments, the translation results of the multiple second speech segments are played in the order indicated by the sequence tokens of the multiple second speech segments.

[0090] The translation processing of the second user's second speech segment can be found in the translation processing of the first user's first speech segment, and will not be repeated here.

[0091] In this way, the audio data to be processed includes multiple audio streams, with a sequence token and statistical count maintained for each stream independently. Each stream independently triggers the unified broadcast of its translation results when the number of translated texts corresponding to its statistical count equals the number of audio segments. If the number of translated texts corresponding to its statistical count does not equal the number of audio segments, the broadcast of that stream's translation results is suspended, without affecting the broadcasts already prepared on other streams. Thus, each stream can be processed independently and concurrently, achieving concurrent full-duplex processing.

[0092] If multiple users speak simultaneously, multiple audio streams from multiple users exist concurrently in multiple channels. During endpoint detection of the audio data, multiple users' audio segments are obtained, and then each user's audio segment is translated independently. The translation processing of multiple users' audio segments overlaps at least partially in time, or in other words, the translation processing of multiple users' audio segments is executed concurrently.

[0093] In one example, if the first and second speech segments overlap at least partially in time, the translation processing of the first and second speech segments is performed concurrently. When multiple users speak simultaneously, resulting in at least two separate speech segments existing at the same time, the speech segments of these at least two separate channels are translated independently, thereby enabling the simultaneous processing of multiple speech segments within the same time period and achieving concurrent full-duplex processing. Since the translation processing of each speech segment is performed concurrently, the translation processing of any channel can start and proceed without waiting for the translation processing of other channels to complete. Therefore, when two or more parties speak almost simultaneously, each party does not need to queue, and their respective translation results can be output promptly after their respective segments are processed, reducing the latency of translation output. In conversation scenarios involving three or more people, when multiple speech segments exist simultaneously, the translation processing of the multiple speech segments can be performed concurrently, with each channel not queuing or blocking the others, thereby improving the concurrent processing capability in real-time multi-user translation scenarios.

[0094] The following example illustrates a scenario of concurrent dialogue between two people.

[0095] The second time threshold is 500ms, and the first time threshold is 5000ms. User A corresponds to the left-side audio channel, and User B corresponds to the right-side audio channel. The specific implementation process is as follows: At t=0.0s, user A starts speaking through the left channel. The left channel continuously collects speech, obtains audio data, and writes it into its corresponding audio buffer. The left channel segment start offset pointer points to 0.0s.

[0096] At t=1.2s, User B begins speaking through the right channel, overlapping with User A's speech in time. The right channel independently acquires speech, obtains audio data, and writes it to its corresponding audio buffer. The right channel segment start offset pointer points to 1.2s.

[0097] At t=2.3s, the duration of user A's silence in the left channel reaches the second time threshold of 500ms, triggering the segmentation of the left channel: the audio data between the starting offset of the left channel segment 0.0s and the current end minus the silence window is determined as the left channel speech segment A1; a sequence token A1 is set for A1; A1 is translated; the left channel statistics count is incremented by one, at which point the left channel statistics count is 1.

[0098] At t=3.0s, the duration of the right sub-channel being in a silent state reaches the silence duration threshold of 500ms, triggering the right sub-channel segment segmentation: the audio data between the starting offset of the right sub-channel segment 1.2s and the current end minus the silence window is determined as the right sub-channel speech segment B1; a sequence token B1 is set for B1; B1 is translated; the right sub-channel statistical count is incremented by one, at which point the right sub-channel statistical count is 1.

[0099] At t=3.5s, the translation result of right-side speech segment B1 is returned first. Upon receiving the translation result of B1, the right-side statistical count is decremented by one, at which point the right-side statistical count reaches zero. Because the right-side statistical count is zero, a unified broadcast is triggered: the translation result of B1 is broadcast to the corresponding listener on the right-side according to the right-side sequence token order. The right-side broadcast is unaffected by the current processing state of the left-side.

[0100] At t=4.0s, User A ends their current round of speaking. At this time, the left branch's statistical count is still 1, not zero. Therefore, only the left branch's broadcast is suspended, without triggering the unified broadcast of the left branch, and it does not affect the broadcast that has already been completed on the right branch.

[0101] At t=4.6s, the translation result of speech segment A1 from the left channel is returned. Upon receiving the translation result of A1, the left channel decrements its statistical count by one, bringing the left channel statistical count to zero. Since the left channel statistical count is zero, a unified broadcast is triggered: the translation result of A1 is broadcast to the corresponding listener on the left channel according to the left channel sequence token order.

[0102] As can be seen from the above process, the left and right paths independently maintain their own sequence tokens and statistical counts. When the statistical count of either path reaches zero, a unified broadcast for that path is triggered without waiting for the other path to complete processing. If the statistical count of a path does not reach zero, only the broadcast of that path is suspended, without affecting the broadcast of the other path that is already ready. Therefore, when the speaking times of both sides overlap, the two paths can independently and concurrently process and broadcast separately, achieving concurrent full-duplex.

[0103] In one example, if the number of translated results indicated by the statistical count is not equal to the number of the first speech segments, the translated results are sorted according to the sequence tokens of the speech segments, and the sorted translated results are output. This sorting allows output to begin without waiting for all segment translations from that branch to be returned, thus shortening the initial output time.

[0104] In one example, if the number of translated results obtained by the statistical count is not equal to the number of the first speech segments and a preset condition is met, then the translated results obtained are sorted according to the sequence tokens of the speech segments, and the sorted translated results are output.

[0105] One example is that the preset condition includes the translation processing time of the audio data exceeding a processing time threshold.

[0106] The real-time translation device provided in the embodiments of this application will be described next.

[0107] Figure 3This is a schematic diagram of a real-time translation device provided in an embodiment of this application. This real-time translation device can be deployed in the aforementioned electronic equipment. For example... Figure 3 As shown, the real-time translation device includes: an acquisition module, a detection module, a translation module, an update module, and an output module.

[0108] The following is a detailed explanation.

[0109] The acquisition module is used to acquire the audio data to be processed; The detection module is used to perform endpoint detection on the audio data and obtain detection results. The detection results include multiple first speech segments of the first user, each first speech segment having a corresponding sequence token. The sequence token of the first speech segment is used to indicate the order of the first speech segment in the multiple first speech segments. The first user has a corresponding first statistical count, which is used to indicate the number of first speech segments of the first user. A translation module is used to translate the plurality of first speech segments, wherein the translation processing of the plurality of first speech segments is performed concurrently; The update module is used to update the first statistical count after obtaining the translation result corresponding to each of the first speech segments; The output module is configured to output the translation results of the plurality of first speech segments in the order indicated by the sequence tokens of the plurality of first speech segments when the number of translated results obtained after the updated first statistical count is equal to the number of the first speech segments.

[0110] Optionally, the endpoint detection of the audio data to obtain the detection result includes: During the detection of the audio data, if the duration of the first user being in a speaking state is greater than or equal to a first time threshold, or if the duration of the first user entering a silent state after being detected as being in a speaking state is greater than or equal to a second time threshold, then the audio data in which the first user is in a speaking state is segmented into the first speech segment.

[0111] Optionally, the endpoint detection of the audio data to obtain the detection result includes: Endpoint detection is performed on the audio data to obtain multiple first speech segments arranged in chronological order. The translation processing of the plurality of first speech segments includes: For each of the first speech segments obtained, the first speech segment is translated.

[0112] Optionally, the translation processing of the first speech segment includes: The first speech segment is identified to obtain multiple identified sub-texts arranged in chronological order; For each identified subtext obtained, the identified subtext is translated to obtain the corresponding translated sub-result; Based on the translation sub-results corresponding to the multiple identified sub-texts, the translation result corresponding to the first speech segment is obtained.

[0113] Optionally, the audio data includes multiple split-channel voice streams, each of which corresponds to a user; The plurality of branch paths include a first branch path and a second branch path, wherein the first branch path corresponds to the first user and the second branch path corresponds to the second user; The detection result also includes multiple second speech segments of the second user: each second speech segment has a corresponding sequence token, the sequence token of the second speech segment is used to indicate the order of the second speech segment in the multiple second speech segments, and the second user has a corresponding second statistical count, the second statistical count is used to indicate the number of second speech segments of the second user; The translation module is also used to translate the plurality of second speech segments, and the translation processing of the plurality of second speech segments is performed concurrently; The update module is also used to update the second statistical count after obtaining the translation result corresponding to each of the second speech segments; The output module is also used to play the translation results of the plurality of second speech segments in the order indicated by the sequence tokens of the plurality of second speech segments when the number of translation results corresponding to the second speech segments obtained after the updated second statistical count is equal to the number of second speech segments.

[0114] Optionally, the step of segmenting the audio data of the first user in the speaking state into the first speech segment if the duration of the first user in the speaking state is detected to be greater than or equal to the first time threshold includes: if the duration of the first user in the speaking state is detected to be greater than or equal to the first time threshold, segmenting the audio data of the first user in the speaking state into the first speech segment based on the segmentation time. The segmentation time satisfies at least one of the following conditions: The audio energy of the audio data at the segmentation time is lower than the audio energy of the audio data at other times in the adjacent time window of the target time; The segmentation time is the time corresponding to a short pause position of the first user in the time window adjacent to the target time, and the target time is the time when the duration of the first user's speaking state reaches a first time threshold.

[0115] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the real-time translation device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0116] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0117] This application provides a computer program product that, when run on a real-time translation device, enables the real-time translation device to implement the steps described in the above-described method embodiments.

[0118] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.

[0119] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0120] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

[0121] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time translation method, characterized in that, The method includes: Acquire the audio data to be processed; Endpoint detection is performed on the audio data to obtain detection results. The detection results include multiple first speech segments of the first user. Each first speech segment has a corresponding sequence token. The sequence token of the first speech segment is used to indicate the order of the first speech segment in the multiple first speech segments. The first user has a corresponding first statistical count. The first statistical count is used to indicate the number of first speech segments of the first user. The plurality of first speech segments are translated, and the translation processing of the plurality of first speech segments is performed concurrently; After obtaining the translation result corresponding to each of the first speech segments, update the first statistical count; When the number of translated results obtained after the updated first statistical count is equal to the number of the first speech segments, the translated results of the plurality of first speech segments are output in the order indicated by the sequence tokens of the plurality of first speech segments.

2. The method according to claim 1, characterized in that, The endpoint detection of the audio data to obtain the detection result includes: During the detection of the audio data, if the duration of the first user being in a speaking state is greater than or equal to a first time threshold, or if the duration of the first user entering a silent state after being detected as being in a speaking state is greater than or equal to a second time threshold, then the audio data in which the first user is in a speaking state is segmented into the first speech segment.

3. The method according to claim 1 or 2, characterized in that, The endpoint detection of the audio data to obtain the detection result includes: Endpoint detection is performed on the audio data to obtain multiple first speech segments arranged in chronological order. The translation processing of the plurality of first speech segments includes: For each of the first speech segments obtained, the first speech segment is translated.

4. The method according to claim 3, characterized in that, The translation process for the first speech segment includes: The first speech segment is identified to obtain multiple identified sub-texts arranged in chronological order; For each identified subtext obtained, the identified subtext is translated to obtain the corresponding translated sub-result; Based on the translation sub-results corresponding to the multiple identified sub-texts, the translation result corresponding to the first speech segment is obtained.

5. The method according to claim 1 or 2, characterized in that, The audio data includes multiple split-channel voice streams, each of which corresponds to a user; The plurality of branch paths include a first branch path and a second branch path, wherein the first branch path corresponds to the first user and the second branch path corresponds to the second user; The detection result also includes multiple second speech segments of the second user: each second speech segment has a corresponding sequence token, the sequence token of the second speech segment is used to indicate the order of the second speech segment in the multiple second speech segments, and the second user has a corresponding second statistical count, the second statistical count is used to indicate the number of second speech segments of the second user; The method further includes: translating the plurality of second speech segments, wherein the translation processing of the plurality of second speech segments is performed concurrently; After obtaining the translation result corresponding to each of the second speech segments, update the second statistical count; When the number of translated results corresponding to the second speech segment obtained after the updated second statistical count is equal to the number of the second speech segments, the translated results of the multiple second speech segments are played in the order indicated by the sequence tokens of the multiple second speech segments.

6. The method according to claim 2, characterized in that, If the duration of the first user's speaking state is detected to be greater than or equal to a first time threshold, then the audio data in which the first user is speaking is segmented into the first speech segment, including: If the duration of the first user's speaking state is detected to be greater than or equal to the first time threshold, then the audio data in which the first user is speaking is segmented into the first speech segment based on the segmentation time. The segmentation time satisfies at least one of the following conditions: The audio energy of the audio data at the segmentation time is lower than the audio energy of the audio data at other times in the adjacent time window of the target time; The segmentation time is the time corresponding to a short pause position of the first user in the time window adjacent to the target time, and the target time is the time when the duration of the first user's speaking state reaches a first time threshold.

7. A real-time translation device, characterized in that, The real-time translation device includes: The acquisition module is used to acquire the audio data to be processed; The detection module is used to perform endpoint detection on the audio data and obtain detection results. The detection results include multiple first speech segments of the first user, each first speech segment having a corresponding sequence token. The sequence token of the first speech segment is used to indicate the order of the first speech segment in the multiple first speech segments. The first user has a corresponding first statistical count, which is used to indicate the number of first speech segments of the first user. A translation module is used to translate the plurality of first speech segments, wherein the translation processing of the plurality of first speech segments is performed concurrently; The update module is used to update the first statistical count after obtaining the translation result corresponding to each of the first speech segments; The output module is configured to output the translation results of the plurality of first speech segments in the order indicated by the sequence tokens of the plurality of first speech segments when the number of translated results obtained after the updated first statistical count is equal to the number of the first speech segments.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images