Real-time voice translation method for independently capturing audio in conference software environment

Abstract

The invention belongs to the technical field of speech translation, and particularly relates to a real-time speech translation method for independently capturing audio in a conference software environment, which comprises the following steps of: creating an independent audio monitoring channel in an operating system kernel layer, and utilizing a virtual audio drive to capture a system audio mixing output signal; meanwhile, independent operation of the original microphone input channel of the conference software is kept; an intelligent shunting algorithm is adopted to separate audio streams, a main audio stream is transmitted to conference software, an auxiliary audio stream is processed by a noise suppression module and then input into a multi-language commercial hybrid translation recognition model, and real-time subtitles are generated and synchronously translated into a target language; and aligning the subtitle window with the core interaction area of the conference software interface through a dynamic coordinate matching algorithm to avoid shielding. According to the method, low-delay audio shunting is realized, multi-language translation is supported, sensitive information of a user is protected by a differential privacy desensitization technology, and the method is generally suitable for various mainstream software platforms.

Description

Technical Field

[0001] The present invention relates to the technical field of speech translation, and in particular to a real-time speech translation method for independently capturing audio in a conference software environment. Background Art

[0002] At present, the Chinese or foreign language audio and video materials on the market have Chinese characters or foreign languages or their corresponding subtitles superimposed on them. Generally, the Chinese or foreign languages in the Chinese or foreign language audio and video materials are converted into Chinese characters or foreign languages manually, and then handed over to the video screen or image screen subtitle superimposition machine to superimpose the Chinese subtitles expressing the Chinese meaning on the video screen or image screen. At the same time, Chinese is translated into a foreign language or vice versa through a translator, and then the voice is dubbed onto the synchronized video screen or image screen by a dubbing actor. Due to the existence of a large amount of real-time or non-real-time Chinese or foreign language audio and video materials from all over the world, including audio and video materials such as television recordings and movies.

[0003] Manual conversion alone would be very labor-intensive and time-consuming, and real-time transmission would be difficult to achieve. With the emergence of digital audio and video technology, especially the technology used by computer systems to process video and image data, traditional solutions achieve translation by hijacking the system audio driver, which often leads to abnormal function of the conference software microphone or echo interference. In addition, due to the differences in audio interfaces of different conference software, existing tools require customized plug-ins to be developed, and cross-platform compatibility cannot be achieved, such as the difference between Teams' NDIS protocol and Tencent Conference's customized audio stack. Moreover, the fixed-position subtitle window is prone to covering high-priority interactive areas such as shared content and voting buttons. In addition, sensitive information such as voiceprint features and meeting IDs are not desensitized during the translation process, posing a risk of privacy leakage.

[0004] Therefore, it is necessary to design a translation mechanism that is compatible with multiple platforms and can solve the problem of subtitle occlusion. Summary of the Invention

[0005] To solve the above problems, the present invention provides a real-time speech translation method and system for independently capturing audio in a conference software environment. Based on the MaxMind GeoLite2 database and process feature detection technology, a conference software adaptation rule library is constructed, and combined with a virtual sound card driver to achieve zero-interference audio diversion. Through dynamic interface rendering technology, DOM element selectors and WebGL overlay solve the subtitle occlusion problem, which is a substantial breakthrough.

[0006] The technical solution adopted in the present invention is:

[0007] A method for real-time speech translation of independently captured audio in a conference software environment comprises the following steps:

[0008] S1) Create an independent audio monitoring channel in the operating system kernel layer, capture the system mixing output signal through the audio virtualization driver, and keep the native microphone input channel of the conference software running independently;

[0009] S2) using an intelligent splitting algorithm to separate the audio stream captured by the audio virtualization driver, transmitting the separated main conference audio stream to the conference software, and transmitting the secondary audio stream to the multi-language commercial hybrid translation and recognition model;

[0010] S3) removing ambient noise and non-human voice interference from the secondary audio stream through adaptive noise suppression, and generating real-time subtitles using a multilingual commercial hybrid translation recognition model;

[0011] S4) Dynamically superimpose a real-time translated subtitle window on the display layer of the conference software interface, align it with the conference software interface elements through a coordinate matching algorithm, and ensure that the real-time subtitle display position does not block the core interactive area.

[0012] Preferably, in S1, a virtual audio device is created through an audio virtualization interface to receive the system's global mixing output; RTP stream mirroring technology is used to copy the audio output stream of the conference software to a multi-language commercial hybrid translation recognition model in real time; and a dynamic allocation strategy for the audio buffer is set. When the conference software occupies more than 80% of the bandwidth, the transmission quality of the conference audio stream is prioritized.

[0013] Preferably, in said S2, the intelligent diversion algorithm includes distinguishing between human voice and non-human voice segments based on Mel-frequency cepstral coefficients and short-time energy analysis; identifying the identity of the speaker according to the voiceprint characteristics, and allocating higher processing weight to the audio stream of the high-frequency speaker; when the network fluctuation exceeds the threshold, the sampling rate of the translation engine is automatically reduced to below 16kHz, giving priority to ensuring the fluency of the conference audio.

[0014] More preferably, the algorithm for assigning a higher processing weight to the audio stream of a high-frequency speaker specifically includes:

[0015] By comprehensively considering the multi-source data factors of high-frequency speakers, including speech frequency, number of keyword mentions, and high-frequency speech score of volume, the calculation equation is:

[0016]

[0017] Among them, Speak i represents the high-frequency speech score of speaker i, α1 represents the speech deviation coefficient, F i Indicates the number of words spoken by speaker i, F avg Indicates the average number of words spoken by all speakers in the meeting, s irepresents the number of times the speaker mentioned the conference keyword, β is a constant used to adjust the impact of keywords on the high-frequency speech score, α2 represents the mutual influence coefficient of the distance between the speaker and the microphone, and C i represents the speaking speed value of speaker i, D i Indicates the distance between speaker i and the microphone.

[0018] Preferably, in S3, the multilingual commercial hybrid translation recognition model includes:

[0019] Pre-trained multilingual acoustic models for mixed recognition of multiple languages;

[0020] An embedded keyword matching engine with a pre-defined multinational business-specific terminology library to extract keywords from input customer audio;

[0021] The privacy protection module is used to anonymize voice data in real time, deleting customer identity features before transmitting it to the cloud.

[0022] Preferably, in S4, the mechanism for ensuring that the real-time subtitle display position does not block the core interactive area is as follows:

[0023] The conference software interface is divided into grid areas, and the occlusion risk value of each area is calculated. The calculation formula of the occlusion risk value is as follows:

[0024]

[0025] Among them, Risk represents the occlusion risk value, ΔM q Represents the element weight of real-time subtitles, M represents the area occupied by real-time subtitles, and S represents the free area of the region;

[0026] Subtitles are placed preferentially in areas where the risk value is less than the preset threshold. If the risk value of all areas exceeds the standard, a dynamic compression mechanism is triggered, including reducing the font size and folding subtitles into multiple lines.

[0027] The present invention also provides an intelligent scene recognition system for real-time captioning and translation of an electronic whiteboard, comprising:

[0028] Kernel-level audio capture module, used to create audio virtualization drivers and capture system mixing output signals;

[0029] An intelligent splitter processor is used to separate the captured audio stream using an intelligent splitter algorithm, and transmit the separated main audio stream of the conference to the conference software, and transmit the secondary audio stream to the multi-language commercial hybrid translation and recognition model;

[0030] A multilingual translation engine is used to pre-process the captured secondary audio and input the pre-processed secondary audio into a multilingual commercial hybrid translation recognition model to generate real-time subtitles;

[0031] Adaptive rendering module, used to achieve pixel-level alignment of real-time subtitles and conference software interface.

[0032] The beneficial effects of the present invention are:

[0033] 1. The method of the present invention uses anonymization processing to make it impossible to associate voice data with customer identities, thereby reducing the risk of commercial espionage; it uses a kernel-layer virtual driver to achieve resource isolation between audio collection and translation, avoiding the process competition problem of traditional user-mode solutions; it automatically adjusts subtitles based on conference content keywords, such as "cross-border mergers and acquisitions" and "clinical trials" to increase speaker scores, thereby improving information transmission efficiency; it dynamically adjusts the trigger sensitivity according to the business activity patterns of the time zone, raising the threshold from 75% on weekdays to 90% on non-workdays, effectively reducing the misjudgment rate.

[0034] 2. This solution has been verified through cloud-based testing, achieving a 91% overseas scene recognition accuracy rate in cross-border telemedicine consultation scenarios, forming a complete technical closed loop covering data collection, processing, and output. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention is a flowchart of the steps of a real-time speech translation method for independently capturing audio in a conference software environment.

[0036] Figure 2 This is a flow chart of the steps of an intelligent diversion algorithm in a real-time speech translation method for independently capturing audio in a conference software environment of the present invention. DETAILED DESCRIPTION

[0037] The specific embodiments of the present invention will be further described below with reference to the examples.

[0038] See also Figure 1 As shown, in a first aspect of the present invention, a real-time speech translation method for independently capturing audio in a conference software environment is provided, comprising the following steps:

[0039] S1) Create an independent audio monitoring channel in the operating system kernel layer, capture the system mixing output signal through the audio virtualization driver, and keep the native microphone input channel of the conference software running independently;

[0040] Use the Windows WASAPI interface to create a virtual sound card device (model: VB-Audio Voicemeeter) and set up dual-channel audio output: the primary channel is transmitted to the Zoom client (bit rate 256kbps), and the secondary channel is input to the translation engine.

[0041] RTP mirroring technology (RFC 3550 protocol) is used to replicate audio streams in real time, with the mirroring delay controlled at 48±2ms.

[0042] This invention develops specialized audio drivers for different operating systems (Windows, macOS, Linux, etc.). This driver can penetrate the core of the system's audio processing and automatically detect and create an independent audio monitoring channel when the conferencing software starts. For example, in Windows systems, low-latency monitoring and capture of audio streams can be achieved by calling the Windows Audio Session API (WASAPI) related interfaces.

[0043] Integrate the developed audio driver into the conferencing software installation package, or provide a standalone installation tool, to facilitate the deployment of independent audio monitoring channels simultaneously with the installation or update of the conferencing software. During installation, the driver automatically configures the priority of the system audio device, ensuring that the microphone function of the conferencing software is not affected and that the independent monitoring channel can operate stably.

[0044] After establishing an audio monitoring channel, implement channel isolation technology to ensure its independence from the conference software's microphone input channel. By setting up independent audio buffers and signal processing processes at the driver level, audio signal interference between the two channels is prevented. For example, a dedicated memory area is allocated for audio data storage and processing for the independent monitoring channel, strictly isolated from the memory area of the conference software's microphone channel.

[0045] Optimize the performance of independent monitoring channels to reduce latency during audio signal capture, transmission, and processing. Use efficient audio codec algorithms, such as the Opus codec, to compress and decompress captured audio in real time, reducing data transmission while ensuring audio quality. Furthermore, optimize the communication interface between the driver and upper-layer applications to ensure fast and stable transmission of audio data to the speech translation module.

[0046] S2) using an intelligent splitting algorithm to separate the audio stream captured by the audio virtualization driver, transmitting the separated main conference audio stream to the conference software, and transmitting the secondary audio stream to the multi-language commercial hybrid translation and recognition model;

[0047] Deploy a voice activity detection algorithm in an independent audio monitoring channel. This algorithm monitors audio signals in real time and distinguishes between speech and non-speech signals (such as background noise and silence). A deep learning-based VAD model, such as one using a long short-term memory (LSTM) network, extracts and classifies audio signals to improve the accuracy and robustness of speech detection. For example, the model can learn characteristic patterns in speech signals, such as frequency, amplitude, and energy, to accurately determine whether valid speech is present in the audio.

[0048] When voice activity is detected, the system further classifies the speech. For example, it distinguishes the speaker's language (such as Chinese, English, or Japanese), as well as characteristics such as speech rate and volume. This classification information provides a basis for subsequent audio diversion and processing, ensuring that different types of voice signals receive targeted translation and processing.

[0049] Based on the voice activity detection and classification results, an intelligent splitting algorithm is formulated. The captured audio signal is divided into two parts:

[0050] The conference software's microphone channel transmits the original voice signal to the microphone channel according to the conference software's normal audio processing flow, ensuring that the conference software's audio recording function is not affected. This audio signal then passes through the conference software's audio processing modules (such as echo cancellation and noise suppression) before being transmitted to other participants.

[0051] The speech translation module performs in-depth processing on the other audio signal, including speech-to-text and translation. Multi-threading technology ensures that the two audio processing components execute in parallel without interfering with each other. For example, a high-priority thread is assigned to the conference software's microphone channel to ensure real-time and stable audio processing. A separate thread is also assigned to the speech translation module, leveraging the computing power of multi-core processors for efficient and accurate translation.

[0052] The intelligent traffic diversion algorithm includes Mel-frequency cepstral coefficients and short-time energy analysis to distinguish between human voice and non-human voice clips; identifies the speaker's identity based on voiceprint characteristics and assigns higher processing weight to the audio stream of high-frequency speakers; when network fluctuations exceed the threshold, it automatically reduces the sampling rate of the translation engine to below 16kHz, prioritizing the smoothness of the conference audio.

[0053] The present invention proposes an improved version of the high-frequency speaker evaluation equation. By comprehensively considering the multi-source data factors of high-frequency speakers, the multi-source data factors include speech frequency, number of keyword mentions, and high-frequency speech score of volume, the calculation equation is:

[0054]

[0055] Among them, Speak i represents the high-frequency speech score of speaker i, α1 represents the speech deviation coefficient, F i Indicates the number of words spoken by speaker i, F avg Indicates the average number of words spoken by all speakers in the meeting, s i represents the number of times the speaker mentioned the conference keyword, β is a constant used to adjust the impact of keywords on the high-frequency speech score, α2 represents the mutual influence coefficient of the distance between the speaker and the microphone, and C i represents the speaking speed value of speaker i, D i Indicates the distance between speaker i and the microphone.

[0056] Operation process:

[0057] Calculate the high-frequency speech score of speaker i Speak i :First, according to the real-time speech word count F of speaker i i and the average number of words spoken by all speakers in the meeting, F avg Calculate the difference in the number of words spoken (F i -F avg ), the greater the difference in the number of words in the speech, the higher the high-frequency speech score of speaker i Speak is by multiplying it with the speech deviation coefficient α1. i will be higher;

[0058] Considering that in a business conference environment, some business activities require explanation and introduction of specific technical keywords, etc., through communication between different speakers, the number of times the speaker mentions the conference keywords is s i Consider keywords to score high-frequency speeches i The positive effect is directly multiplied by the number of times the conference keywords are mentioned s i Normally, the network packet loss rate is greater than 1, so the number of times the speaker mentions the conference keyword is s i The more, the higher the frequency of speech score Speak i The higher;

[0059] Speak is a high-frequency speech score that takes into account the speaker's speaking speed, position, and distance from the microphone. i Impact: The impact of β-adjusted keywords on high-frequency speech scores, The larger the value, the faster the speaker speaks and the greater the probability of being a key speaker. i The higher;

[0060] There is an inversely proportional interaction between the speaker i’s current speaking position and the distance between the microphones: Item, consider the influence of position difference on sound, especially when the position difference between the two is small, Speak i There is a stronger driving force.

[0061] This comprehensive equation takes into account the interaction term between position and speaker keynote speaker, providing more refined prediction of high-frequency speech ratings.

[0062] By presetting the threshold, the high-frequency speech score Speak i When the score is greater than the preset threshold, it means that the speaker is the main speaker and all his speech content needs to be input into the multilingual commercial hybrid translation recognition model for translation; on the contrary, when the high-frequency speech score Speak i When the number is less than the preset threshold, it means that the speaker is a secondary speaker or other irrelevant noise, and his speech does not need to be translated.

[0063] During the audio diversion process, the diversion ratio and processing strategy are dynamically adjusted. For example, when a fast speech signal is detected, resources allocated to the speech translation module are appropriately increased to improve translation speed and accuracy. When the speech signal is weak, the gain control of the microphone channel is optimized to ensure conference audio quality. Through intelligent diversion algorithms, the conference software achieves coordinated optimization of audio capture and speech translation, improving overall performance.

[0064] S3) removing ambient noise and non-human voice interference from the secondary audio stream through adaptive noise suppression, and generating real-time subtitles using a multilingual commercial hybrid translation recognition model;

[0065] First, the speaker's position is determined. The speaker's position can be the set microphone's pickup position. In specific usage scenarios, the target voice can be limited to the speaker's voice. To better uniquely identify the speaker, the speaker's position, facial recognition information, and timbre feature information are used to identify the speaker. The timbre feature here is a feature extracted from the speaker's voice that can distinguish different people's timbre in the field of acoustics. The speaker is determined based on the correspondence between the speaker's position, facial recognition information, and timbre feature.

[0066] Based on the speaker's timbre characteristics, the speaker's voice is separated from the acquired sound source as the target voice. Before obtaining the target voice to be translated, the speaker's identity is identified based on the speaker's position, face recognition information and timbre characteristics.

[0067] This allows for better separation of the speech that needs to be translated from the target speech with noise that is subsequently acquired, reducing the difficulty of extraction.

[0068] Obtain the target speech to be recognized, input the target speech or audio into the local speech recognition model and perform feature extraction on the target speech to obtain the speech features of the target speech;

[0069] The acquired speech content is subjected to pre-processing operations such as noise reduction and enhancement to improve the quality of the speech signal, remove interference factors such as environmental noise, and ensure the accuracy of subsequent feature extraction.

[0070] Perform audio analysis on the target speech content using a local speech recognition model to extract speech features. Perform a preliminary comparison between the extracted speech features and a locally pre-set speech feature template to determine whether they meet the requirements for local recognition in overseas service scenarios.

[0071] Extract multiple speech features, including but not limited to:

[0072] Speech frequency characteristics: Analyze the frequency distribution of speech signals, such as fundamental frequency, spectrum, etc., to capture the pitch and timbre characteristics of speech.

[0073] Speech time domain features: Calculate the time domain features of speech signals such as zero-crossing rate and energy to reflect the time domain variation patterns of speech.

[0074] Speech prosodic features: Identify intonation and speech rate patterns in speech, such as whether the speaking rate is fast or whether there are significant changes in intonation. These features vary in different languages and scenarios.

[0075] Voice keyword detection: Extract specific keywords or phrases from the voice, such as common overseas service scenario-related words (such as "international business", "overseas service", etc.), as auxiliary judgment basis.

[0076] The extracted speech features are compared one by one with a pre-configured library of speech feature templates for various overseas service scenarios. This local template library contains typical speech feature combinations for common business communication, travel consultation, and other scenarios, such as specific speaking speed ranges, intonation pattern templates, and keyword templates.

[0077] Pre-trained multilingual acoustic models for mixed recognition of multiple languages;

[0078] The local speech recognition model includes:

[0079] An embedded keyword matching engine with a pre-defined multinational business-specific terminology library to extract keywords from input customer audio;

[0080] The privacy protection module is used to anonymize voice data in real time, deleting customer identity features before transmitting it to the cloud.

[0081] Enable deep interaction between the plug-in and the conference software. For example, in a Zoom meeting, when a user turns on the real-time subtitle and translation function, the plug-in can automatically obtain the audio parameters of the current meeting (such as sampling rate, number of channels, etc.) and adjust the settings of the independent audio monitoring channel and the voice translation module based on these parameters to ensure the best translation effect. At the same time, the plug-in can monitor the status changes of the conference software in real time (such as speaker switching, audio muting, etc.) and feed this information back to the voice translation module so that the translation strategy can be adjusted in time and the corresponding subtitle content can be displayed.

[0082] Extensive cross-platform compatibility testing was conducted to ensure that the independent audio monitoring channel and speech interpretation features could operate stably across various operating systems and conferencing software versions. During testing, adaptation and optimization were performed based on the audio architecture and communication protocols of different conferencing software. For example, for Microsoft Teams, in-depth research was conducted on its audio processing API and data transmission format, leading to the development of a dedicated adaptation layer to ensure that the independent audio monitoring channel could accurately capture audio signals and seamlessly integrate with Teams' audio system.

[0083] Establish a compatibility problem feedback and repair mechanism. Through user feedback channels and automated testing tools, collect compatibility issues that may occur in different conference software environments, such as audio delays and abnormal subtitle display. Analyze and repair these issues in a timely manner, and provide users with matching plug-in updates. At the same time, maintain communication and cooperation with conference software manufacturers to keep abreast of their software updates and make adaptive adjustments in advance to ensure the stability and reliability of this patented technology.

[0084] S4) Dynamically superimpose a real-time translated subtitle window on the display layer of the conference software interface, align it with the conference software interface elements through a coordinate matching algorithm, and ensure that the real-time subtitle display position does not block the core interactive area.

[0085] The mechanism for ensuring that the real-time subtitle display position does not block the core interactive area is as follows:

[0086] The conference software interface is divided into grid areas, and the occlusion risk value of each area is calculated. The calculation formula of the occlusion risk value is as follows:

[0087]

[0088] Among them, Risk represents the occlusion risk value, ΔM q Represents the element weight of real-time subtitles, M represents the area occupied by real-time subtitles, and S represents the free area of the region;

[0089] Subtitles are placed preferentially in areas where the risk value is less than the preset threshold. If the risk value of all areas exceeds the standard, a dynamic compression mechanism is triggered, including reducing the font size and folding subtitles into multiple lines.

[0090] Establish a mapping relationship between the screen coordinate system (Screen Space) and the local coordinate system of the conference software (such as the Zoom virtual desktop), achieve cross-platform alignment through affine transformation matrices, preset control features (such as process name and window class name) of mainstream software (Zoom, Teams, Tencent Meeting), and dynamically load the corresponding coordinate mapping rules; use Flexbox or Grid layout, combined with viewport units (vw / vh) to achieve adaptation under different resolutions, and the probability of subtitles blocking the core area can be reduced from 32% to 4.7%.

[0091] In a second aspect of the present invention, an intelligent scene recognition system for real-time captioning and translation of an electronic whiteboard is provided, comprising:

[0092] Kernel-level audio capture module, used to create audio virtualization drivers and capture system mixing output signals;

[0093] An intelligent splitter processor is used to separate the captured audio stream using an intelligent splitter algorithm, and transmit the separated main audio stream of the conference to the conference software, and transmit the secondary audio stream to the multi-language commercial hybrid translation and recognition model;

[0094] A multilingual translation engine is used to pre-process the captured secondary audio and input the pre-processed secondary audio into a multilingual commercial hybrid translation recognition model to generate real-time subtitles;

[0095] Adaptive rendering module, used to achieve pixel-level alignment of real-time subtitles and conference software interface.

[0096] In addition, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of any one of the methods described in the above-mentioned method for real-time speech translation of independently captured audio in a conference software environment are implemented.

[0097] A real-time speech translation terminal device for independently capturing audio in a conference software environment is also provided, comprising a memory, a processor, and a computer program stored in the memory. When the processor executes the program, the steps of any one of the methods for independently capturing audio in a conference software environment are implemented.

[0098] In real-time speech translation scenarios using real-time speech interpretation terminals that independently capture audio within conference software, the semantics of the same speech content in different application scenarios can be ambiguous due to differences in professional fields, context, and client intent. For example, in an educational setting, when a teacher mentions "cell," it should be translated as "cell" in a biology class, but might refer to "battery" in a physics class, requiring the use of subject-specific terminology models to distinguish between them. Similarly, in various technical conferences, the word "monitor" may be translated as "display" in hardware discussions and "monitoring system" in software monitoring scenarios, requiring the use of IT-specific translation models based on the attendee's context. Direct translation based on the meaning of the word in the first language can easily lead to translation ambiguity, necessitating the dynamic adaptation of domain-specific translation models to ensure subtitle accuracy.

[0099] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary engineering technicians in this field should fall within the scope of protection determined by the claims of the present invention.

Claims

1. A real-time speech translation method for independently capturing audio in a conference software environment, characterized in that: The following steps are involved: S1) Create an independent audio monitoring channel in the operating system kernel layer, capture the system mixing output signal through the audio virtualization driver, and keep the native microphone input channel of the conference software running independently; S2) using an intelligent splitting algorithm to separate the audio stream captured by the audio virtualization driver, transmitting the separated main conference audio stream to the conference software, and transmitting the secondary audio stream to the multi-language commercial hybrid translation and recognition model; S3) removing ambient noise and non-human voice interference from the secondary audio stream through adaptive noise suppression, and generating real-time subtitles using a multilingual commercial hybrid translation recognition model; S4) Dynamically superimpose a real-time translated subtitle window on the display layer of the conference software interface, align it with the conference software interface elements through a coordinate matching algorithm, and ensure that the real-time subtitle display position does not block the core interactive area.

2. The method for real-time speech translation of independently captured audio in a conference software environment according to claim 1, characterized in that: In step S1, a virtual audio device is created through the audio virtualization interface to receive the system's global mixing output; RTP stream mirroring technology is used to copy the audio output stream of the conference software to a multi-language commercial hybrid translation and recognition model in real time; and a dynamic allocation strategy for the audio buffer is set. When the conference software occupies more than 80% of the bandwidth, the transmission quality of the conference audio stream is prioritized.

3. The method for real-time speech translation of independently captured audio in a conference software environment according to claim 1, characterized in that: In step S2, the specific process of the intelligent traffic diversion algorithm includes: Distinguish between human and non-human voice segments based on Mel-frequency cepstral coefficients and short-time energy analysis; identify the speaker based on voiceprint characteristics, and assign higher processing weight to the audio streams of high-frequency speakers; When network fluctuations exceed the threshold, the sampling rate of the translation engine is automatically reduced to below 16kHz, prioritizing the smoothness of the conference audio.

4. The method for real-time speech translation of independently captured audio in a conference software environment according to claim 3, characterized in that: The specific process of allocating a higher processing weight to the audio stream of a high-frequency speaker includes: By comprehensively considering the multi-source data factors of high-frequency speakers, including speech frequency, number of keyword mentions, and high-frequency speech score of volume, the calculation equation is: In the formula, Speak i represents the high-frequency speech score of speaker i, α1 represents the speech deviation coefficient, F i Indicates the number of words spoken by speaker i, F avg Indicates the average number of words spoken by all speakers in the meeting, s i represents the number of times the speaker mentioned the conference keyword, β is a constant used to adjust the impact of keywords on the high-frequency speech score, α2 represents the mutual influence coefficient of the distance between the speaker and the microphone, and C i represents the speaking speed value of speaker i, D i Indicates the distance between speaker i and the microphone.

5. The method for real-time speech translation of independently captured audio in a conference software environment according to claim 1, characterized in that: In step S3, the specific process of the multilingual commercial hybrid translation recognition model includes: Pre-trained multilingual acoustic models for mixed recognition of multiple languages; An embedded keyword matching engine with a pre-defined multinational business-specific terminology library to extract keywords from input customer audio; The privacy protection module is used to anonymize voice data in real time, deleting customer identity features before transmitting it to the cloud.

6. The method for real-time speech translation of independently captured audio in a conference software environment according to claim 1, characterized in that: In step S4, the specific process of the mechanism for ensuring that the real-time subtitle display position does not block the core interactive area includes: The conference software interface is divided into grid areas, and the occlusion risk value of each area is calculated. The calculation formula of the occlusion risk value is as follows: Among them, Risk represents the occlusion risk value, ΔM q Represents the element weight of real-time subtitles, M represents the area occupied by real-time subtitles, and S represents the free area of the region; Subtitles are placed preferentially in areas where the risk value is less than the preset threshold. If the risk value of all areas exceeds the standard, a dynamic compression mechanism is triggered, including reducing the font size and folding subtitles into multiple lines.

7. A real-time speech translation system for independently capturing audio in a conference software environment, applied to the method according to any one of claims 1 to 6, characterized in that: Including kernel-level audio capture module, intelligent stream splitting processor, multi-language translation engine and adaptive rendering module: Kernel-level audio capture module, used to create audio virtualization drivers and capture system mixing output signals; An intelligent splitter processor is used to separate the captured audio stream using an intelligent splitter algorithm, and transmit the separated main audio stream of the conference to the conference software, and transmit the secondary audio stream to the multi-language commercial hybrid translation and recognition model; A multilingual translation engine is used to pre-process the captured secondary audio and input the pre-processed secondary audio into a multilingual commercial hybrid translation recognition model to generate real-time subtitles; Adaptive rendering module, used to achieve pixel-level alignment of real-time subtitles and conference software interface.

Images