Real-time duplex translation method and corresponding product based on multi-channel parallel processing

Abstract

The application relates to the field of real-time translation, and provides a real-time duplex translation method based on multi-channel parallel processing and a corresponding product.The method comprises the following steps: collecting multi-channel voice signals of at least two user groups in real time through a group of audio acquisition modules respectively; dynamically adjusting beam forming parameters of each audio acquisition module in the group of audio acquisition modules based on a sound source positioning result and feeding back to the corresponding audio acquisition module; monitoring voice activity of each audio channel corresponding to each audio acquisition module; when the voice activity of any audio channel reaches a predetermined condition, automatically activating a translation processing procedure of the audio channel and keeping the remaining audio channels in a monitoring state; using a parallel processing mechanism for the voice signals of the activated audio channel, simultaneously performing real-time translation of the currently activated audio channel and voice activity monitoring of the remaining audio channels; and transmitting the translation result of the current speaker to other users participating in the conversation in the user group, so that multi-channel data is synchronously coordinated.

Description

Technical Field

[0001] This application relates to the field of real-time translation, and in particular to a real-time duplex translation method and corresponding product based on multi-channel parallel processing. Background Technology

[0002] With the deepening of globalization, the demand for real-time cross-language communication is becoming increasingly urgent. Whether in international business negotiations, technical seminars, or in tourism and education, language barriers have become a key constraint on communication efficiency. As an important tool for eliminating language barriers, voice translation devices are being developed with the aim of achieving a more natural and fluent dialogue experience, enabling both parties to communicate seamlessly as if using their native language.

[0003] Currently, most real-time speech translation solutions on the market are based on a single-channel interaction mode. These solutions typically require users to control the speech acquisition and translation process through explicit interactive actions. For example, after user A speaks, the device needs to be manually passed to user B, or a specific button needs to be pressed to switch recording channels to instruct the device to start acquiring the other party's speech. This design treats the translation device as a "microphone" that needs to be operated alternately, and its workflow is fundamentally different from the continuous and overlapping characteristics of natural human conversation.

[0004] However, existing technical solutions of this kind have obvious drawbacks. First, the forced manual operation (transferring devices or pressing buttons) disrupts the natural rhythm of the conversation, causing frequent interruptions and pauses in the communication process, which seriously affects the fluency and efficiency of communication. Second, since the device can only process a single audio stream at any given time, it cannot effectively deal with the overlapping phenomenon of "turn-taking" common in conversations. That is, when one party has not finished speaking before the other begins speaking, the system may not be able to fully capture or correctly distinguish the speech signal, resulting in omissions or errors in the translation. Therefore, providing a real-time duplex translation method that can simulate natural human conversation and automatically identify and process alternating speech without human intervention has become an urgent technical problem to be solved in this field. Summary of the Invention

[0005] This application provides a real-time duplex translation method and corresponding product based on multi-channel parallel processing. Through hardware isolation and parallel processing, the accuracy, fluency, and user convenience of real-time duplex translation can be significantly improved.

[0006] On the one hand, this application provides a real-time duplex translation method based on multi-channel parallel processing, the method comprising:

[0007] A set of audio acquisition modules is used to acquire multiple audio signals from at least two user groups in real time. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology.

[0008] The acquired multiple audio signals are processed for sound source localization. Based on the localization results, the beamforming parameters of each audio acquisition module in the set of audio acquisition modules are dynamically adjusted, and the adjusted beamforming parameters are fed back to the corresponding audio acquisition module.

[0009] Using the adjusted beamforming parameters, the speech signals acquired by each audio acquisition module are enhanced in real time, and the speech activity of each audio channel corresponding to each audio acquisition module is monitored.

[0010] When the voice activity of any audio channel reaches a predetermined condition, the translation processing flow of the audio channel whose voice activity has reached the predetermined condition is automatically activated, and the monitoring status of the remaining audio channels is maintained.

[0011] The voice signal of the activated audio channel is processed in frames, and a parallel processing mechanism is used to simultaneously perform real-time translation of the voice content of the currently activated audio channel and voice activity monitoring of the other audio channels.

[0012] The real-time translation result of the speech content of the current user in the user group is transmitted to other users participating in the dialogue in the user group through the audio output module, and the synchronous coordination of multi-channel data is realized during the transmission process.

[0013] On the other hand, this application provides a real-time duplex translation device based on multi-channel parallel processing, the device comprising:

[0014] The acquisition module is used to acquire multiple audio signals from at least two user groups in real time through a set of audio acquisition modules. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology.

[0015] The adjustment module is used to perform sound source localization processing on the acquired multi-channel voice signals, dynamically adjust the beamforming parameters of each audio acquisition module in the group of audio acquisition modules based on the localization results, and feed back the adjusted beamforming parameters to the corresponding audio acquisition module.

[0016] The first processing module is used to perform real-time enhancement processing on the speech signals acquired by each audio acquisition module using the adjusted beamforming parameters and to monitor the speech activity of each audio channel corresponding to each audio acquisition module.

[0017] The second processing module is used to automatically activate the translation processing flow of the audio channel whose speech activity reaches the predetermined condition when the speech activity of any audio channel is detected to reach the predetermined condition, and to maintain the monitoring state for the remaining audio channels.

[0018] The third processing module is used to perform frame-by-frame processing on the voice signal of the activated audio channel. It adopts a parallel processing mechanism to simultaneously perform real-time translation of the voice content of the currently activated audio channel and voice activity monitoring of the other audio channels.

[0019] The coordination module is used to transmit the real-time translation results of the speech content of the current speaker in the user group to other users participating in the dialogue in the user group through the audio output module, and to realize the synchronous coordination of multi-channel data during the transmission process.

[0020] Thirdly, this application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described real-time duplex translation method based on multi-channel parallel processing.

[0021] Fourthly, this application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described real-time duplex translation method based on multi-channel parallel processing.

[0022] As can be seen from the technical solution provided in this application, on the one hand, by using a physically isolated set of audio acquisition modules to synchronously acquire multiple audio signals, and combining this with voice activity monitoring to automatically trigger the translation process, the reliance on manual transmission devices or button presses is eliminated. Participants can speak freely as in everyday conversation, and the device automatically recognizes and processes speaker switching, thus ensuring the continuity and natural fluency of the dialogue. On the other hand, by dynamically adjusting beamforming parameters through sound source localization processing, and using the adjusted parameters to enhance the audio signal in real time, the system can effectively focus on the target sound source, suppressing environmental noise and interference from the other party's voice. This provides a cleaner and clearer input signal for subsequent speech recognition and translation modules, laying the foundation for high-quality translation results. Thirdly, through a multi-channel data synchronization and coordination mechanism, the timing consistency and coordination of the translation output are ensured, effectively avoiding output errors caused by processing or transmission delays, and significantly improving the overall coherence of the dialogue and user experience. In summary, the technical solution of this application, through hardware isolation and parallel processing, can significantly improve the accuracy, fluency, and user convenience of real-time full-duplex translation. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a real-time duplex translation method based on multi-channel parallel processing provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of the structure of the real-time duplex translation device based on multi-channel parallel processing provided in the embodiments of this application;

[0026] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In this specification, adjectives such as "first" and "second" are used only to distinguish one element or action from another, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be construed as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0029] For ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.

[0030] As a crucial tool for eliminating language barriers, voice translation devices aim to achieve a more natural and fluent dialogue experience, enabling seamless communication as if both parties were using their native language. Currently, most real-time voice translation solutions on the market are based on a single-channel interaction mode. These solutions typically require users to control the voice acquisition and translation process through explicit interactive actions. For example, after user A speaks, they need to manually pass the device to user B, or switch recording channels by pressing specific buttons to instruct the device to start acquiring the other party's voice. This design treats the translation device as a "microphone" that needs to be operated alternately, and its workflow is fundamentally different from the continuous and overlapping characteristics of natural human conversation. However, such existing technological solutions have significant drawbacks. First, the forced manual operation (passing the device or pressing buttons) interrupts the natural rhythm of the conversation, causing frequent interruptions and pauses in the communication process, severely affecting the fluency and efficiency of communication. Secondly, because the device can only process a single audio stream at any given time, it cannot effectively handle the overlapping "turn-taking" phenomenon common in dialogues. That is, when one party begins speaking before the other has finished speaking, the system may not be able to fully capture or correctly distinguish the speech signal, resulting in omissions or errors in the translation. Therefore, providing a real-time duplex translation method that can simulate natural human dialogue and automatically identify and process alternating speech without human intervention has become a pressing technical problem in this field.

[0031] To address the aforementioned problems in the existing technology, this application proposes a real-time duplex translation method based on multi-channel parallel processing, the flowchart of which is attached. Figure 1 As shown, the main steps include S101 to S106, which are detailed below:

[0032] Step S101: Collect multiple audio signals from at least two user groups in real time using a set of audio acquisition modules. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology.

[0033] In real-time duplex translation systems, the voice signals of multiple users participating in a dialogue are prone to mutual interference (e.g., crosstalk and electromagnetic interference), leading to a decline in the quality of the acquired signal and affecting the accuracy of subsequent translation. To address this issue, using software algorithms for signal separation, such as blind source separation, may not completely eliminate electromagnetic interference at the hardware level, especially in complex environments where the separation effect is unstable and computational costs are high. While physically separated microphones without electromagnetic shielding may reduce interference due to physical distance, electromagnetic interference can still introduce noise through circuit coupling, resulting in incomplete signal isolation. To ensure that each audio acquisition module operates independently, reduce cross-interference, improve the purity and reliability of the voice signal, and provide high-quality input for subsequent processing, this application proposes a set of audio acquisition modules where each module achieves signal isolation at the hardware level using electromagnetic shielding technology. These modules acquire multiple voice signals from at least two user groups in real time. It should be noted that in the multiple voice signals of at least two user groups, one user in the user group corresponds to one voice signal. For example, if at least two user groups contain four users, namely user A, user B, user C and user D, then user A corresponds to the first voice signal, user B corresponds to the second voice signal, user C corresponds to the third voice signal, and user D corresponds to the fourth voice signal.

[0034] In the above embodiments, there are four possible implementation schemes for achieving signal isolation between the various audio acquisition modules in a group of audio acquisition modules at the hardware level through electromagnetic shielding technology:

[0035] Option 1: Physical structural isolation and shielding. The physical structural isolation is achieved by designing an independent metal shielding cavity for each audio acquisition module (i.e., the microphone array and its preamplifier circuit) inside the device housing. These cavities are physically separated by metal partitions. The shielding is achieved by adding a metal shielding cover to the sensitive analog circuits (especially the preamplifier) ​​at the circuit board level of each microphone array.

[0036] Option 2: Shielded cable and shielded connector. The shielded cable is implemented by using shielded cable, such as coaxial or twisted pair shielded cable, to connect the microphone unit and the main processing board. The shielded connector is implemented by using a connector with a metal shell (e.g., shielded USB connector, D-Sub connector) and ensuring good conductivity between the connector shell and the ground plane of the device housing or printed circuit board.

[0037] Option 3: Common-mode choke and filter circuit. This involves adding filter components to the power and signal lines. The common-mode choke is implemented by placing a π-type filter circuit or a common-mode choke at the power input of each audio acquisition module. The filter circuit is implemented by using an RC low-pass filter circuit before the analog signal enters the analog-to-digital converter.

[0038] Option 4: Independent power supply and grounding design. The independent power supply is implemented by providing each audio acquisition module with an independent, isolated power supply module, such as using an independent low dropout linear regulator or DC-DC isolated power supply module. The grounding design is implemented by routing the analog ground wires of all audio acquisition modules separately and finally converging to a common ground point (star point) on the motherboard, instead of arbitrarily daisy-chaining connections.

[0039] Step S102: Perform sound source localization processing on the acquired multi-channel voice signals respectively, dynamically adjust the beamforming parameters of each audio acquisition module in a set of audio acquisition modules based on the localization results, and feed back the adjusted beamforming parameters to the corresponding audio acquisition module.

[0040] In dynamic environments (e.g., user movement or environmental changes), fixed beamforming parameters cannot adapt to changes in sound source location, leading to inaccurate speech signal capture and affecting signal quality and subsequent translation. If fixed beamforming parameters are used to address this problem, they cannot adapt to sound source movement; when the user's location changes, signal capture quality deteriorates, potentially resulting in speech loss or increased noise. This application performs sound source localization processing on the acquired multiple speech signals separately and dynamically adjusts the beamforming parameters of each audio acquisition module based on the localization results. By dynamically adjusting the beamforming parameters through sound source localization processing, the audio acquisition modules can adapt to changes in sound source location, improving the accuracy and stability of speech signal capture and enhancing the system's robustness in dynamic environments.

[0041] Existing sound source localization methods are susceptible to noise and reverberation interference, leading to inaccurate target sound source location estimations. Conventional energy accumulation methods also struggle to distinguish spatially adjacent sound sources. To suppress noise interference with phase information and improve time delay measurement accuracy, spatial location is transformed into a weighted vector to achieve closed-loop control for sound source tracking. As an embodiment of this application, sound source localization processing for the acquired multi-channel speech signals can be performed as follows: Real-time calculation of the sound wave arrival time difference between each microphone unit corresponding to each audio acquisition module; calculation of time delay estimation using a frequency domain weighted cross-correlation time delay estimation method; generation of a sound field spatial distribution map based on the time delay estimation using a spatial spectrum phase transformation energy scanning localization method; and peak search of the sound field spatial distribution map to determine the point with the highest energy as the location of the main sound source. Specifically, the time delay estimation method using frequency domain weighted cross-correlation can be as follows: perform a short-time Fourier transform on the audio signals from any two microphone units to obtain their complex spectra; calculate the cross-power spectrum of the two complex spectra; apply a frequency domain weighting function to the cross-power spectrum to suppress the influence of noise and reverberation components on the phase information; perform an inverse Fourier transform on the weighted cross-power spectrum to obtain an enhanced cross-correlation function; find the peak with the largest absolute value in the enhanced cross-correlation function, and the time delay corresponding to this peak is the estimated time difference of arrival of the sound waves between the two microphone units. The specific steps for generating a sound field spatial distribution map using the spatial spectrum phase transformation energy scanning localization method are as follows: A search grid containing multiple candidate sound source locations is defined within the target spatial region; for each candidate sound source location in the search grid, the theoretical time delay required for the sound wave to propagate from that location to each microphone unit in the microphone array is calculated; based on the theoretical time delay, the cross-correlation value corresponding to that time delay is extracted from the pre-calculated and stored frequency domain weighted cross-correlation functions of all microphone pairs; the extracted cross-correlation values ​​of all microphone pairs are summed to obtain the spatial spectrum energy value of the candidate sound source location; after traversing all candidate sound source locations, all locations and their corresponding spatial spectrum energy values ​​are combined to generate a sound field spatial distribution map. It should be noted that, due to the significant impact of multipath interference and noise on the generated sound field spatial distribution map, in this embodiment, the sound wave arrival time difference is smoothed and filtered before generating the sound field spatial distribution map to suppress the influence of multipath interference and noise.

[0042] As another embodiment of this application, the sound source localization processing of the collected signals can also be as follows: for each of the multiple speech signals, extract a dual-path feature map based on phase and amplitude information; input the dual-path feature map into a pre-trained sound source localization neural network model; the sound source localization neural network model maps the input features into a two-dimensional or three-dimensional sound source direction probability map through an encoder-decoder architecture, wherein the intensity of each pixel represents the probability that the sound source comes from that direction; select one or more peak points in the sound source direction probability map whose probability value exceeds a preset threshold, and the corresponding coordinates are one or more peak points. The target direction estimation values ​​for multiple sound sources; specifically, for each of the multiple speech signals, extracting a dual-path feature map based on phase and amplitude information can be achieved by: performing a short-time Fourier transform on the audio signal of each pair of microphone channels to obtain a complex spectrum; calculating the amplitude spectrum of the complex spectrum and performing logarithmic compression and normalization to form an amplitude feature channel; calculating the phase spectrum of the complex spectrum and further calculating the phase difference between each pair of microphone channels to form a phase difference feature channel; and concatenating the amplitude feature channel and the phase difference feature channel along the feature dimension to form a dual-path feature map.

[0043] In the above embodiments, the specific implementation of dynamically adjusting the beamforming parameters of each audio acquisition module in a set of audio acquisition modules based on the positioning results can be as follows: Based on the positioning results and combined with the fixed geometric layout of the microphone array, the theoretical arrival time delay difference of the sound waves of each microphone unit in the array relative to the reference microphone is calculated. Then, based on the calculated theoretical arrival time delay difference, a spatial steering vector representing the direction of the target sound source is constructed. These spatial steering vectors are then used as core constraints and input into the beamforming weight optimization algorithm for solving, resulting in a set of weight coefficients. The digital signal processor of the audio acquisition module applies the newly loaded weight coefficients to perform a weighted summation operation on the multi-channel microphone signals acquired in real time, generating a beamforming output signal pointing towards the target sound source. Finally, the generated beamforming output signal is fed back to the sound source localization and subsequent voice activity monitoring modules as one of the input signals for subsequent localization and monitoring. Since the final generated beamforming output signal is fed back to the sound source localization and subsequent voice activity monitoring modules, a technical closed loop is formed, enabling the entire system to continuously adapt to sound source movement and environmental changes, thereby demonstrating the dynamic adjustment of the beamforming parameters of each audio acquisition module.

[0044] Step S103: Using the adjusted beamforming parameters, perform real-time enhancement processing on the speech signals acquired by each audio acquisition module and monitor the speech activity of each audio channel corresponding to each audio acquisition module.

[0045] Since speech signals acquired in everyday environments inevitably contain environmental noise, directly processing these signals can lead to misjudgments in speech activity monitoring (e.g., false activation or missed activation), thus affecting the triggering efficiency and accuracy of the translation process. Even using a fixed threshold for speech activity detection has drawbacks: improper threshold settings can lead to inaccurate detection and cannot adapt to changes in environmental noise. To ensure that only valid speech triggers translation, reduce false triggers, save computational resources, improve system reliability, and provide a decision-making basis for channel switching in full-duplex translation, this application utilizes adjusted beamforming parameters to perform real-time enhancement processing on the speech signals acquired by each audio acquisition module and monitor the speech activity of each audio channel corresponding to each audio acquisition module. Here, the speech activity of an audio channel specifically refers to the real-time judgment of a particular audio channel (e.g., the microphone array signal pointing to user A), i.e.: speech activity = 1 (activated), indicating that the current frame contains valid speech (translation needs to be triggered); speech activity = 0 (silent), indicating that the current frame contains only environmental noise or non-human voice (translation is not triggered).

[0046] As one embodiment of this application, monitoring the speech activity of each audio acquisition module corresponding to each audio channel can be achieved by: performing frame-by-frame processing on the signal from the main sound source direction and calculating the sound pressure level of each frame in real time; and dynamically updating the environmental noise baseline using an exponentially weighted moving average method, with the update formula being: ,in, For the first n The baseline estimate of ambient noise at frame time, For the first n The baseline estimate of ambient noise for frame -1 (i.e., the previous frame). As a smoothing factor, For the current frame (i.e., the first frame) n The noise energy value of the current frame is measured; the signal-to-noise ratio (SNR) between the sound pressure level of the current frame and the noise baseline is calculated. When the SNR exceeds a preset threshold, a speech activity indication signal is generated. The above formula for updating the environmental noise baseline shows that when the environmental noise changes slowly, the baseline can transition smoothly, while brief bursts of noise (e.g., keyboard typing) are smoothed out by a smoothing factor. The inhibitory effect does not significantly raise the baseline, and the smoothing factor can be adjusted. The value can be adjusted to optimize performance for different scenarios (e.g., stable meeting rooms, variable outdoor environments, etc.).

[0047] As another embodiment of this application, monitoring the speech activity of each audio channel corresponding to each audio acquisition module can also be achieved based on speech activity monitoring using deep learning and attention mechanisms. That is: continuous audio frames are obtained through the audio acquisition module; Mel spectrum features of each audio frame are extracted; the Mel spectrum features of multiple consecutive audio frames are concatenated into a feature vector and input into a pre-trained speech activity monitoring model, which includes a one-dimensional convolutional neural network layer and a long short-term memory network layer based on an attention mechanism; local frequency domain features at the frame level are extracted through the one-dimensional convolutional neural network layer of the speech activity monitoring model; the long short-term memory network layer based on an attention mechanism learns the long-range dependency of local frequency domain features in the time series, and the attention mechanism calculates the importance weight of each frame to the final classification decision; according to the output of the long short-term memory network and the attention weight, the time series information is weighted and fused, and finally the probability of the current frame being a speech frame or a non-speech frame is output by the classifier. The aforementioned speech activity monitoring scheme based on deep learning and attention mechanisms effectively avoids energy-induced misjudgments by automatically learning the semantic attributes of human voice (e.g., phoneme structure, formant patterns) through Mel-spectral features and neural networks. Furthermore, the mechanism of building long-term memory using long short-term memory networks and the dynamic weighting of keyframes through attention mechanisms can accurately identify speech start and end boundaries (e.g., the hesitation segment in "um...I think"). Therefore, compared to the static models of the aforementioned embodiments that rely on sound pressure levels, linear assumptions, and rules, the speech activity monitoring scheme based on deep learning and attention mechanisms has stronger semantic understanding capabilities and is more suitable for complex acoustic scenarios.

[0048] Step S104: When the voice activity of any audio channel reaches the predetermined condition, the translation processing flow of the audio channel whose voice activity reaches the predetermined condition is automatically activated, and the monitoring status of the remaining audio channels is maintained.

[0049] In duplex dialogue, multiple users may speak simultaneously or alternately. The system needs to intelligently determine which audio channel to activate for translation to avoid processing conflicts or missing speech, while maintaining real-time response. Processing the translation of two audio channels simultaneously demands high computational resources, potentially causing system overload, increased latency, and power consumption. Using a fixed priority or polling method to process the translation of two audio channels lacks flexibility, potentially delaying response or missing sudden speech, resulting in unnatural dialogue. To ensure timely response and avoid resource waste, achieving intelligent channel management, in this embodiment, when the speech activity of any audio channel reaches a predetermined condition, the translation processing flow of the audio channel whose speech activity reaches the predetermined condition is automatically activated, while the remaining audio channels are kept under monitoring. Specifically, automatically activating the translation processing flow of the audio channel whose speech activity reaches the predetermined condition and keeping the remaining audio channels under monitoring can be achieved by: generating an activation signal for the audio channel whose speech activity reaches the predetermined condition when speech activity reaches the predetermined condition; starting the translation engine based on the activation signal; and simultaneously sending a monitoring and holding instruction to the remaining audio channels.

[0050] Furthermore, considering the numerous turn-taking transitions in natural dialogue, users often speak simultaneously or briefly overlap. This is normal conversational behavior, not a system malfunction. Without a conflict resolution mechanism, the system might either be at a loss, resulting in both channels being processed, leading to chaotic translations; or it might choose randomly, potentially interrupting important speakers and severely disrupting the fluency and logic of the dialogue. To intelligently prioritize the channel most likely to dominate the conversation, mimicking the human tendency to subconsciously focus on clearer sounds in noisy environments, the aforementioned translation processing flow that automatically activates audio channels that meet predetermined conditions while monitoring the remaining audio channels introduces a conflict resolution mechanism (hereinafter referred to as the "first conflict resolution mechanism"). Specifically, when the speech activity of multiple audio channels simultaneously meets predetermined conditions, the speech activity intensity of these multiple audio channels is compared; the audio channel with higher speech activity intensity is activated first; the activation of the remaining audio channels is delayed, and a signal buffer is maintained during this period. The aforementioned conflict resolution mechanism, through a human-like and intelligent decision-making process, can gracefully handle conflict scenarios involving simultaneous speech, ensuring system stability while maximizing the integrity of dialogue information and the smoothness of user experience.

[0051] As another embodiment of this application, the above-mentioned scheme of automatically activating the translation processing flow of audio channels that meet predetermined conditions and maintaining a monitoring state for the remaining audio channels also introduces another conflict resolution mechanism (hereinafter referred to as the "second conflict resolution mechanism"), namely: performing real-time speech recognition in parallel on the speech signals of multiple audio channels to generate multiple temporary text streams; inputting the multiple temporary text streams into a lightweight intent classification model, outputting the intent category and contextual relevance score of each temporary text stream; comparing the intent priorities of the multiple temporary text streams based on predefined intent priority rules: if the intent priority of one temporary text stream is higher than the intent priority of the other temporary text streams, then activating the translation processing flow of the audio channel corresponding to the temporary text stream with the highest intent priority, while storing the speech signals of the remaining audio channels into a buffer queue; after completing the translation processing of the activated audio channels... The semantic relevance of speech signals in the buffer queue is analyzed based on the current dialogue context: if the semantic relevance exceeds a threshold, speech signals in the buffer queue with semantic relevance exceeding the threshold are inserted into the output queue; if the semantic relevance is lower than the threshold, speech signals in the buffer queue with semantic relevance lower than the threshold are discarded. When multiple temporary text streams are input into the lightweight intent classification model, the lightweight intent classification model performs the following operations: extracting keywords and sentence structure features from the multiple temporary text streams; determining the intent category based on a pre-trained classifier, which includes questioning, answering, stating, interrupting, and emergency instructions, etc.; calculating the semantic association score between the current temporary text stream and the last three rounds of dialogue; the intent priority rule can be that questioning intents have higher priority than answering intents, emergency instructions have higher priority than stating intents, and statements answering the current question have higher priority than statements initiating new topics. Compared to the first conflict resolution mechanism, which bases its decision on the intensity of speech activity obtained from which audio channel (in terms of which user's voice is louder), the second conflict resolution mechanism bases its decision on whose words are more important. Therefore, it is closer to the intelligent nature of human dialogue. In addition, the second conflict resolution mechanism closely integrates conflict resolution with the overall context of the dialogue, making the decision more accurate and reasonable. It can also avoid the discomfort that may be caused by simply choosing based on volume (for example, always letting the loudest and fastest speaker dominate), making the dialogue more balanced.

[0052] Step S105: Perform frame-by-frame processing on the voice signal of the activated audio channel, and use a parallel processing mechanism to simultaneously perform real-time translation of the voice content of the currently activated audio channel and voice activity monitoring of the other audio channels.

[0053] In the field of translation technology, real-time translation requires low-latency processing, but simultaneously necessitates continuous monitoring of another channel to avoid missing speech. This demands that the system possess parallel processing capabilities; otherwise, delays or omissions may occur. Sequential processing (translating first, then monitoring) increases processing latency, potentially leading to the neglect of speech in the other channel and disrupting dialogue continuity. Single-channel processing, on the other hand, cannot achieve true duplexing, failing to respond promptly when speech is available in the other channel. To ensure real-time performance and continuity, improve system throughput and response speed, and ensure smooth and natural dialogue, this application performs frame-based processing on the speech signal of the activated audio channel, employing a parallel processing mechanism to simultaneously perform real-time translation of the currently activated audio channel's speech content and monitor the speech activity of the remaining audio channels.

[0054] Specifically, as one embodiment of this application, employing a parallel processing mechanism to simultaneously perform real-time translation of the currently activated audio channel's speech content and voice activity monitoring of the remaining audio channels can be achieved by: setting a main processing channel and one or more auxiliary monitoring channels based on signals from multiple audio channels; performing real-time speech translation processing on the main processing channel; continuously monitoring voice activity on each auxiliary monitoring channel; and dynamically switching the processing roles of the main processing channel and the auxiliary monitoring channels based on the monitoring results. As can be seen from the above embodiment, based on real-time voice activity, the main processing channel and the auxiliary monitoring channels dynamically exchange roles (e.g., when an auxiliary channel detects a sudden increase in voice activity, it immediately becomes the main processing channel), achieving zero-delay speech transfer and simulating the instantaneous alternation of human dialogue. Furthermore, the continuous monitoring of voice activity by the auxiliary channels ensures that no speech segments are missed, thereby eliminating blind spots in the dialogue.

[0055] Considering the fixed allocation of computing resources, when the main processing channel experiences a surge in load (e.g., long sentence translation), processing latency occurs, while the resources of the auxiliary monitoring channels become idle. Meanwhile, highly complex speech (e.g., technical terms) can overload the main channel, causing system lag or even crashes. To accurately identify resource bottlenecks and ensure that high-load tasks are processed first, the aforementioned scheme, which employs a parallel processing mechanism to simultaneously perform real-time translation of the speech content of the currently active audio channel and voice activity monitoring of the remaining audio channels, can also include: real-time monitoring of the speech processing load of the main processing channel and each auxiliary monitoring channel; dynamic allocation of computing resources based on the processing load; and automatic allocation of more computing resources to the main processing channel when the processing load of the main processing channel exceeds a threshold.

[0056] Step S106: Transmit the real-time translation result of the speech content of the current speaker in the user group to other users participating in the dialogue in the user group through the audio output module, and realize the synchronous coordination of multi-channel data during the transmission process.

[0057] In the field of real-time translation technology, translation results need to be output synchronously to two users to avoid dialogue confusion or asynchrony (e.g., disordered output order or inconsistent delays), which would affect user experience and the naturalness of the dialogue. In other words, asynchronous output of translation results may lead to disordered output order, causing users to hear asynchronous dialogue and making comprehension difficult. If simple forwarding without a synchronization mechanism is used, network latency or packet loss cannot be handled, resulting in inconsistent output timing. To ensure that translation results are output in order and at the right time, maintain the coherence and naturalness of the dialogue, and improve user experience, this application transmits the translation result of the currently speaking user in the user group to other participating users in the dialogue through an audio output module, and achieves synchronous coordination of multi-channel data during transmission.

[0058] Specifically, as an embodiment of this application, the synchronization and coordination of multi-channel data during transmission in the above embodiment can be achieved by: adding a timestamp and sequence number to each translation result data packet; sorting the multi-channel data based on the timestamp and sequence number to obtain a data packet sequence arranged in chronological order; monitoring the current playback progress of the target user in real time; dynamically deciding the next data packet to be played based on the temporal relationship of the data packet sequence and the current playback progress of the target user; and immediately triggering the audio output of the data packet to be played whose temporal position is continuous with the current playback progress when it is detected that the temporal position of the data packet to be played is continuous with the current playback progress. Furthermore, considering that real-time translation systems are extremely sensitive to network fluctuations—for example, a 300ms delay in a translation scenario can disrupt the flow of the conversation—to shift from reactive response to proactive prevention, overcome the unpredictability of network fluctuations, and achieve a dynamic balance between data integrity (packet loss prevention) and low-latency transmission, the aforementioned scheme for synchronizing and coordinating multi-channel data during transmission can also include an intelligent buffer prediction mechanism. This involves establishing a network state prediction model to predict future network conditions based on historical network quality data; when a decline in network quality is predicted, increasing the buffer size in advance to prevent data loss; and when an improvement in network quality is predicted, decreasing the buffer size in advance to reduce transmission latency. Practice shows that this intelligent buffer prediction mechanism can predict network state changes 50-200ms in advance, providing a decision window for buffer adjustments, thereby effectively resolving the contradiction between transmission reliability and latency caused by network fluctuations in real-time full-duplex translation.

[0059] It should be noted that although the methods in the above embodiments primarily process speech signals, they can also process text signals in another scenario. That is, the methods in the above embodiments may further include: converting the text content of the multiple parties involved in the communication into corresponding multi-channel speech signals using a text-to-speech mechanism, after which the subsequent process can be used. Figure 1 The example method execution flow.

[0060] From the above appendix Figure 1As illustrated by the example of a real-time duplex translation method based on multi-channel parallel processing, firstly, by employing a physically isolated set of audio acquisition modules to simultaneously acquire multiple audio signals and automatically triggering the translation process in conjunction with voice activity monitoring, the reliance on manual transmission devices or button presses is eliminated. Participants can speak freely as in everyday conversation, and the device automatically identifies and handles speaker switching, thus ensuring the continuity and natural fluency of the dialogue. Secondly, by dynamically adjusting beamforming parameters through sound source localization processing and using the adjusted parameters to enhance the audio signal in real time, the method effectively focuses on the target sound source, suppressing environmental noise and interference from the other party's speech. This provides a cleaner and clearer input signal for subsequent speech recognition and translation modules, laying the foundation for high-quality translation results. Thirdly, through a multi-channel data synchronization and coordination mechanism, the temporal consistency and coordination of the translation output are ensured, effectively avoiding output errors caused by processing or transmission delays, and significantly improving the overall coherence of the dialogue and user experience. In summary, the technical solution of this application, through hardware isolation and parallel processing, can significantly improve the accuracy, fluency, and user convenience of real-time duplex translation.

[0061] Please see the appendix Figure 2 This application provides a real-time duplex translation device based on multi-channel parallel processing. The device may include an acquisition module 201, an adjustment module 202, a first processing module 203, a second processing module 204, a third processing module 205, and a coordination module 206, as detailed below:

[0062] The acquisition module 201 is used to acquire multiple voice signals from at least two user groups in real time through a set of audio acquisition modules. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology.

[0063] The adjustment module 202 is used to perform sound source localization processing on the acquired multi-channel voice signals, dynamically adjust the beamforming parameters of each audio acquisition module in a set of audio acquisition modules based on the localization results, and feed back the adjusted beamforming parameters to the corresponding audio acquisition module.

[0064] The first processing module 203 is used to perform real-time enhancement processing on the speech signals acquired by each audio acquisition module using the adjusted beamforming parameters and to monitor the speech activity of each audio channel corresponding to each audio acquisition module.

[0065] The second processing module 204 is used to automatically activate the translation processing flow of the audio channel whose speech activity reaches the predetermined condition when the speech activity of any audio channel is detected to reach the predetermined condition, and to maintain the monitoring state of the remaining audio channels.

[0066] The third processing module 205 is used to perform frame-by-frame processing on the speech signal of the activated audio channel, and adopts a parallel processing mechanism to simultaneously perform real-time translation of the speech content of the currently activated audio channel and monitoring of the speech activity of the other audio channels.

[0067] The coordination module 206 is used to transmit the real-time translation result of the speech content of the current speaker in the user group to other users participating in the dialogue in the user group through the audio output module, and to realize the synchronous coordination of multi-channel data during the transmission process.

[0068] From the above appendix Figure 2 As illustrated by the example of a real-time duplex translation device based on multi-channel parallel processing, firstly, by employing a physically isolated set of audio acquisition modules to simultaneously acquire multiple audio signals and automatically triggering the translation process in conjunction with voice activity monitoring, the reliance on manual transmission devices or button presses is eliminated. Participants can speak freely as in everyday conversation, and the device automatically identifies and handles speaker switching, thus ensuring the continuity and natural fluency of the dialogue. Secondly, by dynamically adjusting beamforming parameters through sound source localization processing and using the adjusted parameters to enhance the audio signal in real time, the device effectively focuses on the target sound source, suppressing environmental noise and interference from the other party's speech. This provides a cleaner and clearer input signal for subsequent speech recognition and translation modules, laying the foundation for high-quality translation results. Thirdly, through a multi-channel data synchronization and coordination mechanism, the temporal consistency and coordination of the translation output are ensured, effectively avoiding output errors caused by processing or transmission delays, and significantly improving the overall coherence of the dialogue and user experience. In summary, the technical solution of this application, through hardware isolation and parallel processing, can significantly improve the accuracy, fluency, and user convenience of real-time duplex translation.

[0069] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 3 As shown, the electronic device 3 in this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a real-time duplex translation method based on multi-channel parallel processing. When the processor 30 executes the computer program 32, it implements the steps in the above-described embodiment of the real-time duplex translation method based on multi-channel parallel processing, for example... Figure 1 The steps S101 to S106 are shown. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the acquisition module 201, adjustment module 202, first processing module 203, second processing module 204, third processing module 205, and coordination module 206 are shown.

[0070] For example, the computer program 32 of the real-time duplex translation method based on multi-channel parallel processing mainly includes: acquiring multiple audio signals from at least two user groups in real time through a set of audio acquisition modules, wherein each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology; performing sound source localization processing on the acquired multiple audio signals respectively, dynamically adjusting the beamforming parameters of each audio acquisition module in the set of audio acquisition modules based on the localization results, and feeding back the adjusted beamforming parameters to the corresponding audio acquisition module; and using the adjusted beamforming parameters to perform real-time enhancement of the audio signals acquired by each audio acquisition module. The system processes and monitors the speech activity of each audio channel corresponding to each audio acquisition module. When the speech activity of any audio channel reaches a predetermined condition, the translation processing flow of the audio channel whose speech activity reaches the predetermined condition is automatically activated, while the monitoring status of the remaining audio channels is maintained. The system performs frame processing on the speech signals of the activated audio channels and uses a parallel processing mechanism to simultaneously execute the real-time translation of the speech content of the currently activated audio channel and the speech activity monitoring of the remaining audio channels. The system transmits the real-time translation result of the speech content of the currently speaking user in the user group to other participating users in the dialogue group through the audio output module, and realizes the synchronous coordination of multi-channel data during the transmission process. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31 and executed by the processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 32 in the electronic device 3.For example, computer program 32 can be divided into the functions of acquisition module 201, adjustment module 202, first processing module 203, second processing module 204, third processing module 205, and coordination module 206 (a module in the virtual device). The specific functions of each module are as follows: Acquisition module 201 is used to acquire multiple audio signals from at least two user groups in real time through a set of audio acquisition modules, wherein each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology; Adjustment module 202 is used to perform sound source localization processing on the acquired multiple audio signals, dynamically adjust the beamforming parameters of each audio acquisition module in the set of audio acquisition modules based on the localization results, and feed back the adjusted beamforming parameters to the corresponding audio acquisition modules; First processing module 203 is used to utilize the adjusted... The beamforming parameters enhance the speech signals acquired by each audio acquisition module in real time and monitor the speech activity of each audio channel corresponding to each audio acquisition module. The second processing module 204 automatically activates the translation processing flow of the audio channel whose speech activity reaches the predetermined condition when the speech activity of any audio channel is detected to reach the predetermined condition, and keeps the monitoring state of the remaining audio channels. The third processing module 205 performs frame processing on the speech signals of the activated audio channels and uses a parallel processing mechanism to simultaneously perform real-time translation of the speech content of the currently activated audio channel and monitor the speech activity of the remaining audio channels. The coordination module 206 transmits the real-time translation result of the speech content of the current speaker in the user group to other participating users in the dialogue in the user group through the audio output module, and realizes the synchronous coordination of multi-channel data during the transmission process.

[0071] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0072] The processor 30 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0073] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0074] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed. That is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above-described device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0075] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0076] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0077] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program for the real-time duplex translation method based on multi-channel parallel processing can be stored in a storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above, namely, acquiring multiple audio signals from at least two user groups in real time through a set of audio acquisition modules, wherein each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology; performing sound source localization processing on the acquired multiple audio signals, dynamically adjusting the beamforming parameters of each audio acquisition module in the set of audio acquisition modules based on the localization results, and feeding back the adjusted beamforming parameters to the corresponding... The system includes an audio acquisition module; utilizing adjusted beamforming parameters, it performs real-time enhancement processing on the speech signals acquired by each audio acquisition module and monitors the speech activity of each corresponding audio channel; when the speech activity of any audio channel reaches a predetermined condition, it automatically activates the translation processing flow of that audio channel while maintaining monitoring of the remaining audio channels; it performs frame-by-frame processing on the speech signals of the activated audio channels, employing a parallel processing mechanism to simultaneously perform real-time translation of the speech content of the currently activated audio channel and monitor the speech activity of the remaining audio channels; it transmits the real-time translation results of the speech content of the currently speaking user in the user group to other participating users in the dialogue group through the audio output module, achieving synchronous coordination of multi-channel data during transmission. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. Storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of storage media can be appropriately added to or removed according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media may not include electrical carrier signals and telecommunication signals.

[0081] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application. The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the protection scope of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this invention.

Claims

1. A real-time duplex translation method based on multi-channel parallel processing, characterized in that, The method includes: A set of audio acquisition modules is used to acquire multiple audio signals from at least two user groups in real time. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology. The acquired multiple audio signals are processed for sound source localization. Based on the localization results, the beamforming parameters of each audio acquisition module in the set of audio acquisition modules are dynamically adjusted, and the adjusted beamforming parameters are fed back to the corresponding audio acquisition module. Using the adjusted beamforming parameters, the speech signals acquired by each audio acquisition module are enhanced in real time, and the speech activity of each audio channel corresponding to each audio acquisition module is monitored. When the voice activity of any audio channel reaches a predetermined condition, the translation processing flow of the audio channel whose voice activity has reached the predetermined condition is automatically activated, and the monitoring status of the remaining audio channels is maintained. The voice signal of the activated audio channel is processed in frames, and a parallel processing mechanism is used to simultaneously perform real-time translation of the voice content of the currently activated audio channel and voice activity monitoring of the other audio channels. The real-time translation result of the speech content of the current user in the user group is transmitted to other users participating in the dialogue in the user group through the audio output module, and the synchronization and coordination of multi-channel data is realized during the transmission process; When the speech activity of multiple audio channels simultaneously reaches a predetermined condition, the speech activity intensity of these multiple audio channels is compared; the audio channel with higher speech activity intensity is selected for priority activation; the activation of the remaining audio channels is delayed, and the signal buffer is maintained during this period.

2. The real-time duplex translation method based on multi-channel parallel processing as described in claim 1, characterized in that, The process of performing sound source localization processing on the acquired multi-channel speech signals includes: The time difference of sound wave arrival between each microphone unit corresponding to each audio acquisition module is calculated in real time, and the time delay is estimated by using the frequency domain weighted cross-correlation time delay estimation method. Based on the time delay estimation, a spatial distribution map of the sound field is generated using the spatial spectrum phase transformation energy scanning localization method. Peak search is performed on the spatial distribution map of the sound field to determine the point with the highest energy as the location of the main sound source.

3. The real-time duplex translation method based on multi-channel parallel processing as described in claim 2, characterized in that, Also includes: Before generating the spatial distribution map of the sound field, the arrival time difference of the sound waves is smoothed and filtered to suppress the effects of multipath interference and noise.

4. The real-time duplex translation method based on multi-channel parallel processing as described in claim 1, characterized in that, The parallel processing mechanism simultaneously performs real-time translation of the currently active audio channel's speech content and monitors the speech activity of the remaining audio channels, including: Based on signals from multiple audio channels, a main processing channel and one or more auxiliary monitoring channels are set up; Real-time speech translation is performed on the main processing channel; Continuous voice activity monitoring is conducted on each auxiliary monitoring channel; The processing roles of the main processing channel and the auxiliary monitoring channel are dynamically switched based on the monitoring results.

5. The real-time duplex translation method based on multi-channel parallel processing as described in claim 4, characterized in that, Also includes: Real-time monitoring of the voice processing load of the main processing channel and each auxiliary monitoring channel; Dynamically allocate computing resources based on the processing load; When the processing load of the main processing channel exceeds the threshold, more computing resources are automatically allocated to the main processing channel.

6. The real-time duplex translation method based on multi-channel parallel processing as described in claim 1, characterized in that, The synchronization and coordination of dual-channel data during transmission includes: Add a timestamp and sequence number to each translation result data packet; The multi-channel data is sorted based on timestamps and sequence numbers to obtain a sequence of data packets arranged in chronological order. Real-time monitoring of the target user's current playback progress; Based on the temporal relationship of the data packet sequence and the current playback progress of the target user, the next data packet to be played is dynamically determined; When the timing position of the data packet to be played is detected to be continuous with the current playback progress, the audio output of the data packet to be played that is continuous with the current playback progress is immediately triggered.

7. The real-time duplex translation method based on multi-channel parallel processing as described in claim 6, characterized in that, The method further includes: Establish a network status prediction model to predict future network conditions based on historical network quality data; When a network quality degradation is predicted, the buffer size is increased in advance to prevent data loss. When network quality improvement is anticipated, the buffer size is reduced in advance to decrease transmission latency.

8. A real-time duplex translation device based on multi-channel parallel processing, characterized in that, The device includes: The acquisition module is used to acquire multiple audio signals from at least two user groups in real time through a set of audio acquisition modules. Each audio acquisition module in the set of audio acquisition modules achieves signal isolation at the hardware level through electromagnetic shielding technology. The adjustment module is used to perform sound source localization processing on the acquired multi-channel voice signals, dynamically adjust the beamforming parameters of each audio acquisition module in the group of audio acquisition modules based on the localization results, and feed back the adjusted beamforming parameters to the corresponding audio acquisition module. The first processing module is used to perform real-time enhancement processing on the speech signals acquired by each audio acquisition module using the adjusted beamforming parameters and to monitor the speech activity of each audio channel corresponding to each audio acquisition module. The second processing module is used to automatically activate the translation processing flow of the audio channel whose speech activity reaches the predetermined condition when the speech activity of any audio channel is detected to reach the predetermined condition, and to maintain the monitoring state for the remaining audio channels. The third processing module is used to perform frame-by-frame processing on the voice signal of the activated audio channel. It adopts a parallel processing mechanism to simultaneously perform real-time translation of the voice content of the currently activated audio channel and voice activity monitoring of the other audio channels. The coordination module is used to transmit the real-time translation result of the speech content of the current speaker in the user group to other users participating in the dialogue in the user group through the audio output module, and to realize the synchronous coordination of multi-channel data during the transmission process; It also includes modules that perform the following functions: when the speech activity of multiple audio channels simultaneously reaches a predetermined condition, compare the speech activity intensity of the multiple audio channels; select the audio channel with higher speech activity intensity for priority activation; delay the activation of the remaining audio channels and maintain signal buffering during this period.

9. An electronic device, the device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images