Multifunctional audio intelligent optimization conference system

Abstract

The invention provides a multifunctional audio intelligent optimization conference system. The multifunctional audio intelligent optimization conference system comprises an audio acquisition module, an audio preprocessing module, an audio optimization module, a voice recognition module, a translation module, an audio transmission module, a conference control and management module, a display and interaction module and a security and privacy protection module. According to the invention, the audio signal is intelligently optimized by using the deep learning algorithm, and the audio optimization strategy can be adaptively adjusted according to different conference scenes and user requirements, so that the high-quality audio experience is ensured, and the flexibility and adaptability of the conference system are improved; meanwhile, through rich interaction functions, such as real-time translation and voice-to-text conversion, the interaction of the conference is enhanced, and the efficiency of the conference is improved; in addition, through the functions of data encryption, access control, privacy setting and the like, the conference data security and the privacy of the participants are ensured, the modern data protection requirement is met, the credibility of the participants is enhanced, and thus smooth proceeding of the conference is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of conference system and audio processing, and in particular to a multifunctional audio intelligent optimization conference system. Background Art

[0002] A conference system is a system that integrates multiple technologies such as sound, video, and data transmission. It is designed to achieve interaction between remote and local conferences through various terminal devices (such as video conferencing terminals, multimedia projectors, large-screen TVs, audio equipment, etc.), allowing people to exchange information and work together in different locations. Conference systems are widely used in government, public security, military, courts, science and technology, energy, medical care, education and other fields, providing convenience for all aspects of social life. Conference systems improve communication efficiency, reduce corporate travel expenses, and improve management effectiveness by realizing functions such as remote conferencing and remote training. With the rapid development of information technology, remote conferencing systems have become an indispensable part of modern work and life. However, in actual applications, there are still certain problems: 1. In different conference scenarios, audio signals are often interfered by various noises and echoes, resulting in reduced sound quality and affecting the auditory experience of participants. Especially in large conference rooms, open office areas and other environments, noise and echo problems are particularly prominent. Traditional audio processing technology can often only be optimized for specific situations and lacks adaptability and intelligence. This limits the flexibility and application scope of the conference system; Second, traditional teleconference systems usually only provide basic voice communication functions and lack rich interactive means. This makes it difficult for participants to communicate and collaborate effectively, reducing the efficiency of the meeting. In particular, in situations where real-time translation, voice-to-text, speech application, voting and other functions are required, traditional systems often cannot meet the needs; 3. Remote conference systems involve a large amount of sensitive information, such as conference content, participant information, etc. How to ensure the security and privacy of this data is an important challenge facing remote conference systems. Traditional data security protection methods can only provide basic encryption and access control functions, and cannot cope with increasingly complex network attacks and data leakage risks; Therefore, a multifunctional audio intelligent optimization conference system is proposed. Summary of the invention

[0003] In view of this, the embodiments of the present invention hope to provide a multifunctional audio intelligent optimization conference system to solve or alleviate the technical problems existing in the prior art and at least provide a beneficial choice.

[0004] In order to solve the above technical problems, the technical solution adopted in this application is: a multifunctional audio intelligent optimization conference system, including an audio acquisition module, an audio preprocessing module, an audio optimization module, a speech recognition module, a translation module, an audio transmission module, a conference control and management module, a display and interaction module and a security and privacy protection module; The audio acquisition module is used to collect audio signals at the conference site in real time and optimize the audio acquisition effect; The audio preprocessing module is used to receive the original audio signal collected by the audio collection module, and use digital signal processing technology to perform preliminary processing and optimization on the audio signal; The audio optimization module is used to use a deep learning algorithm to intelligently optimize the audio signal after preliminary processing and optimization, improve the sound quality, and adaptively adjust the audio optimization strategy according to the conference scenario and user needs; The speech recognition module is used to convert the conference audio signal into text records in real time using speech recognition technology; The translation module is used to use machine translation technology to provide real-time translation display and voice broadcast functions, and supports multiple translation modes and language selections; The audio transmission module is used to transmit the optimized audio signal or translated voice data to other devices or networks via wired or wireless means, supporting multi-channel real-time audio stream transmission and voice intercom; The conference control and management module is used to control and manage the entire conference system, including conference start and end, participant management, authority setting, and device control, so that the conference proceeds smoothly; The display and interaction module is used to display the conference content, participant information, and translation results to the participants in a visual manner, and provide interactive functions, including speech application and voting, to enhance the interactivity and participation of the meeting; The security and privacy protection module is used to protect the security of conference data and the privacy of participants, including data encryption, access control and privacy setting functions.

[0005] As a further preferred embodiment of the present technical solution, the audio acquisition module includes a microphone array unit, a preamplifier unit, an ADC unit and a signal conditioning unit; The microphone array unit is used to capture the audio signal at the conference site in real time; The preamplifier unit is used to receive the audio signal output by the microphone array unit and amplify it to a voltage or current level suitable for a subsequent processing circuit while maintaining the quality of the audio signal; The ADC unit is used to convert the analog audio signal output by the preamplifier unit into a digital audio signal; The signal conditioning unit is used to perform preliminary conditioning on the digital audio signal output by the ADC unit to remove noise and interference.

[0006] As a further preferred embodiment of the present technical solution, the audio optimization module includes a noise suppression unit, an echo cancellation unit and a dynamic range control unit; The noise suppression unit detects and suppresses non-steady-state and steady-state noise in the audio signal in real time using spectrum subtraction and adaptive filtering algorithms; The echo cancellation unit estimates and eliminates the echo component in the audio signal by using a double-ended echo cancellation technology through an adaptive filter and echo path estimation; The dynamic range control unit uses compression expansion technology and automatic gain control algorithm to automatically adjust the dynamic range of the audio signal and balance the high and low frequency signals.

[0007] As a further preferred embodiment of the present technical solution, the speech recognition module includes an acoustic model unit, a language model unit and a search decoding unit; The acoustic model unit uses a machine learning model to convert the audio signal into an acoustic feature vector to capture the acoustic characteristics of the speech signal; The language model unit is used to provide a probability distribution of vocabulary sequences, helping the recognition engine to infer vocabulary sequences based on the acoustic feature vectors and the language model; The search decoding unit combines the acoustic feature vector and the language model, and utilizes a search algorithm, such as the Viterbi algorithm or a weighted finite state converter, to search for a vocabulary sequence and perform an accurate conversion from audio to text.

[0008] As a further preferred embodiment of the present technical solution, the translation module supports a custom dictionary and a term base, allowing the user to add or modify professional terms and vocabulary according to specific meeting requirements.

[0009] As a further preferred embodiment of the present technical solution, the audio transmission module supports multiple audio coding standards and protocols, including AAC, MP3, and Opus.

[0010] As a further preferred embodiment of the present technical solution, in the spectrum subtraction algorithm used by the noise suppression unit, the audio signal is first converted from the time domain to the frequency domain by short-time Fourier transform, the PSD of the noise spectrum and the speech spectrum is calculated, and then the estimated value of the noise PSD is subtracted from the speech PSD to obtain the enhanced speech PSD, and finally the frequency domain signal is converted back to the time domain by inverse short-time Fourier transform to obtain the audio signal after noise suppression; the specific calculation formula is as follows: Short-time Fourier transform: ; in, is the short-time Fourier transform result of the m-th frame and the k-th frequency component, x(n) is the input audio signal, w(n) is the window function, R is the frame shift length, and N is the number of FFT points.

[0011] PSD estimation: ; in, is the noise PSD estimate of the kth frequency component in the mth frame, and α is the smoothing factor; Speech PSD estimation and noise suppression: ; in, is the enhanced speech PSD, β is the over-reduction factor; Inverse short-time Fourier transform: ; in, is the enhanced frequency domain signal, It is the time domain audio signal after noise suppression.

[0012] As a further preferred embodiment of the present technical solution, the echo cancellation unit uses an adaptive filter based on a normalized minimum mean square error algorithm, which updates the filter coefficients in an iterative manner to minimize the error between the echo signal and the expected signal; the specific calculation formula is as follows: Filter coefficient update:

[0013] Where h(n) is the filter coefficient vector for the nth iteration, x(n) is the input signal vector, e(n) is the error signal, μ is the step size factor, and δ is the regularization parameter; Error signal calculation:

[0014] Among them, is the expected signal, that is, the original signal without echo, and is the echo estimation signal output by the filter.

[0015] As a further preferred embodiment of the present technical solution, the deep learning model used by the acoustic model unit includes a recurrent neural network and a long short-term memory network.

[0016] As a further preferred embodiment of the present technical solution, the conference control and management module also includes a conference mode switching function, allowing users to select different conference modes according to conference requirements, such as discussion mode, demonstration mode or question-and-answer mode, to adapt to different conference scenarios.

[0017] The embodiment of the present invention has the following advantages due to the adoption of the above technical solution: 1. The present invention uses deep learning algorithms to intelligently optimize audio signals, and can adaptively adjust audio optimization strategies such as noise suppression and echo cancellation according to different conference scenarios and user needs, thereby ensuring high-quality audio experience in any environment. This intelligent processing capability not only improves the sound quality, but also enhances the flexibility and adaptability of the conference system; 2. The present invention provides rich interactive functions, such as real-time translation, voice-to-text, speech application, voting, etc. These functions can greatly enhance the interactivity of the meeting, allowing participants to participate in the meeting more actively. At the same time, these interactive functions can also effectively improve the efficiency of the meeting and reduce the time wasted due to poor communication or misunderstanding; 3. The present invention ensures the security of conference data and the privacy of participants from being leaked through functions such as data encryption, access control and privacy settings. This comprehensive protection and defense capability not only meets the requirements of modern data protection, but also enhances the trust of participants in the conference system and promotes the smooth progress of the meeting.

[0018] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 This is a functional module diagram of a multifunctional audio intelligent optimization conference system of the present invention; Figure 2 It is a structural schematic diagram of the audio acquisition module of the present invention; Figure 3 It is a structural schematic diagram of the audio optimization module of the present invention; Figure 4 It is a structural schematic diagram of the speech recognition module of the present invention. DETAILED DESCRIPTION

[0021] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0022] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0023] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0024] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0025] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0026] Figure 1 Schematic diagram of functional modules of a multifunctional audio intelligent optimization conference system according to an embodiment of the present application. Figure 1-Figure 4 As shown, a multifunctional audio intelligent optimization conference system includes an audio acquisition module, an audio preprocessing module, an audio optimization module, a speech recognition module, a translation module, an audio transmission module, a conference control and management module, a display and interaction module, and a security and privacy protection module; The audio acquisition module is used to collect audio signals at the conference site in real time and optimize the audio acquisition effect. The audio acquisition module is not only responsible for acquisition, but also optimizes the audio acquisition effect through built-in technical means, including reducing interference factors such as environmental noise and echo, and improving the signal-to-noise ratio and clarity of the audio signal; The audio preprocessing module is used to receive the original audio signal collected by the audio acquisition module, and use digital signal processing technology to perform preliminary processing and optimization on the audio signal, wherein the audio signal is preliminarily processed and optimized, including preliminary processing such as noise reduction and reverberation, to provide high-quality audio input for subsequent optimization; The audio optimization module is used to use deep learning algorithms to intelligently optimize the audio signals after preliminary processing and optimization, improve the sound quality, and adaptively adjust the audio optimization strategy according to the conference scenario and user needs. The audio optimization module can adaptively adjust the audio optimization strategy according to the conference scenario and user needs, such as improving the sound quality, balancing the volume, etc., to ensure that participants can hear clear and comfortable audio. By training a large amount of audio data, it can automatically identify and adjust the frequency, volume, timbre and other parameters of the audio signal to adapt to different conference scenarios and user needs. For example, in the lecture mode, the system can automatically increase the volume and clarity to ensure that the audience can clearly hear the speech content; in the question-and-answer mode, the system can automatically adjust the volume balance to ensure that the voices of the questioner and the answerer can be heard; The speech recognition module is used to convert conference audio signals into text records in real time using speech recognition technology. By adopting advanced speech recognition technology, the accuracy and real-time nature of text conversion are ensured, thereby providing participants with convenient conference record services for subsequent review and organization. The translation module uses machine translation technology to provide real-time translation display and voice broadcast functions, and supports multiple translation modes and language selections. Through machine translation technology, the translation module can translate the conference content into multiple languages ​​in real time and present it to participants through display or voice broadcast. The module also supports custom translation thesaurus and terminology libraries to ensure the accuracy and professionalism of the translation results. At the same time, the translation module also has automatic learning and optimization capabilities, which can continuously improve the quality and efficiency of translation; The audio transmission module is used to transmit the optimized audio signal or translated voice data to other devices or networks via wired or wireless means, supporting multi-channel real-time audio streaming transmission and voice intercom, thereby ensuring real-time sharing and interaction of conference information. At the same time, the module also has network adaptive capabilities and data compression technology, which can maintain stable transmission quality and low latency in different network environments; The conference control and management module is used to control and manage the entire conference system, including conference start, end, participant management, permission setting, and device control, so that the conference can proceed smoothly. Through the conference control and management module, the host can easily control the progress of the conference and the behavior of the participants to ensure the smooth progress and efficient management of the meeting. At the same time, the module also supports remote control and mobile device management, so that participants can participate in the meeting at any time; The display and interaction module is used to display the meeting content, participant information, and translation results to the participants in a visual way, and provides interactive functions such as speaking application and voting to enhance the interactivity and participation of the meeting. The display and interaction module displays meeting information to participants through high-definition display screens or mobile devices, and provides a convenient interactive interface for participants to use. Through this module, participants can easily understand the progress and content of the meeting and actively participate in the meeting; The security and privacy protection module is used to protect the security of conference data and the privacy of participants, including data encryption, access control and privacy setting functions. The security and privacy protection module uses data encryption technology to encrypt the transmission and storage of conference data to ensure the security of data during transmission and storage. At the same time, the module also has access control and privacy setting functions, and different permissions and privacy levels can be set as needed to protect the privacy of participants from being leaked. In addition, the module also supports security auditing and logging functions, which can easily monitor and trace the conference process.

[0027] In one embodiment, specifically, the audio acquisition module includes a microphone array unit, a preamplifier unit, an ADC unit and a signal conditioning unit; The microphone array unit is used to capture the audio signals at the conference site in real time. Through the collaborative work of multiple microphones, the precise positioning and collection of the sound source can be achieved. This technology can effectively reduce the interference of environmental noise and improve the purity of the audio signal. The preamplifier unit is used to receive the audio signal output by the microphone array unit and amplify it to a voltage or current level suitable for the subsequent processing circuit while maintaining the quality of the audio signal. The collected audio signal is amplified by the preamplifier unit to ensure that the amplitude of the signal meets the requirements of subsequent processing. A low-noise, high-linearity preamplifier is used to reduce the noise introduced by the amplifier itself and signal distortion; The ADC unit is used to convert the analog audio signal output by the preamplifier unit into a digital audio signal. Through the sampling and quantization process, the continuous-time analog signal is converted into a discrete-time digital signal for subsequent digital signal processing and analysis; The signal conditioning unit is used to perform preliminary conditioning on the digital audio signal output by the ADC unit to remove noise and interference. The signal conditioning unit performs preliminary noise suppression and filtering processing on the digital audio signal output by the ADC unit to remove noise and interference so as to improve the signal-to-noise ratio and audio quality.

[0028] In one embodiment, specifically, the audio optimization module includes a noise suppression unit, an echo cancellation unit, and a dynamic range control unit; The noise suppression unit uses spectrum subtraction and adaptive filtering algorithms to detect and suppress non-steady-state and steady-state noise in audio signals in real time. The noise suppression unit uses spectrum subtraction technology to subtract the noisy audio signal from the estimated noise spectrum to obtain a relatively pure voice signal. This method can effectively suppress non-steady-state noise, such as wind and rain. For steady-state noise, such as the hum of electrical equipment or the interference of AC power supply, the unit uses adaptive filtering algorithms to suppress it. By adjusting the parameters of the filter in real time, the filter can better match and eliminate steady-state noise. The echo cancellation unit uses a two-end echo cancellation technology to estimate and eliminate the echo component in the audio signal through an adaptive filter and echo path estimation. The echo cancellation unit uses a two-end echo cancellation technology, that is, the echo cancellation processing is performed at both the sending end and the receiving end of the audio signal, which can more effectively eliminate the echo caused by transmission delay and path difference. The audio signal is filtered through an adaptive filter, and the echo path estimation technology is used to estimate the path change of the audio signal during the transmission process. Based on these estimation results, the echo component in the audio signal can be accurately estimated and eliminated; The dynamic range control unit uses compression and expansion technology and automatic gain control algorithm to automatically adjust the dynamic range of the audio signal and balance the high and low frequency signals. The dynamic range control unit uses compression and expansion technology to adjust the dynamic range of the audio signal. When the amplitude of the audio signal is too large, the compression technology is used to reduce its amplitude; when the amplitude of the audio signal is too small, the expansion technology is used to increase its amplitude. This ensures that the dynamic range of the audio signal is within the appropriate range and avoids distortion or overload. In order to more accurately control the amplitude of the audio signal, the unit also uses an automatic gain control algorithm. By monitoring the amplitude changes of the audio signal in real time and automatically adjusting the gain coefficient, it ensures that the amplitude of the audio signal fluctuates within a predetermined range. This can further improve the clarity and audibility of the audio signal.

[0029] In one embodiment, specifically, the speech recognition module includes an acoustic model unit, a language model unit, and a search decoding unit; The acoustic model unit uses a machine learning model to convert the audio signal into an acoustic feature vector to capture the acoustic characteristics of the speech signal. The acoustic model unit usually uses a machine learning model to achieve the conversion of the audio signal to the acoustic feature vector. Commonly used machine learning models include hidden Markov models (HMM), deep neural networks (DNN), recurrent neural networks (RNN) or convolutional neural networks (CNN). These models are trained with a large amount of speech data and can learn the acoustic characteristics of the speech signal and convert it into a high-dimensional feature vector. The language model unit is used to provide the probability distribution of vocabulary sequences, helping the recognition engine to infer the vocabulary sequence based on the acoustic feature vector and the language model. The language model unit is usually constructed based on statistical methods or deep learning models. Statistical methods such as the n-gram model estimate the probability distribution of vocabulary sequences by counting the co-occurrence frequency of words in the corpus. Deep learning models such as recurrent neural networks (RNN) or Transformer models can learn more complex language structures and semantic information. The output of the language model unit is a probability distribution of a vocabulary sequence, which is used to guide the search decoding unit to search for vocabulary sequences; The search decoding unit combines the acoustic feature vector and the language model, and uses a search algorithm such as the Viterbi algorithm or the weighted finite state transducer to search for the vocabulary sequence and accurately convert the audio to text. The search decoding unit usually uses efficient search algorithms such as the Viterbi algorithm or the weighted finite state transducer (WFST). These algorithms can quickly search for the most likely vocabulary sequence based on the probability distribution of the acoustic feature vector and the language model. During the search process, the decoder will comprehensively consider the output of the acoustic model and the constraints of the language model to find the globally optimal vocabulary sequence. In addition, in order to improve the accuracy and efficiency of recognition, the search decoding unit can also adopt some optimization strategies, such as word graph construction and re-scoring. Word graph construction can convert the output of the acoustic model into a word graph structure to facilitate subsequent search and re-scoring operations. Re-scoring is to re-evaluate the initial recognition results using a more complex model or a more sophisticated language model to improve the accuracy of recognition.

[0030] In one embodiment, specifically, the translation module supports custom dictionaries and terminology libraries, allowing users to add or modify professional terms and vocabulary according to specific meeting requirements. Through custom dictionaries and terminology libraries, users can ensure that the translation results contain correct professional terms and vocabulary, thereby improving the accuracy of the translation; this function allows users to adjust the translation content according to different meeting or industry requirements, making the translation module more adaptable and flexible; this function is suitable for scenarios such as international conferences, professional lectures, legal translation, etc. that require a high degree of accuracy and professionalism.

[0031] In one embodiment, specifically, the audio transmission module supports multiple audio codec standards and protocols, including but not limited to AAC, MP3, Opus, among which, AAC is an efficient audio coding format that provides high-quality audio output while maintaining a small file size. It supports multiple audio sampling rates and bit rates, suitable for a variety of application scenarios. AAC has two main encapsulation formats: ADIF (Audio Data Interchange Format) and ADTS (Audio Data Transport Stream). ADIF is suitable for local file storage, while ADTS is more suitable for network transmission because it allows decoding at any frame. AAC encoding and decoding involves converting audio signals into digital data and then applying specific algorithms for data compression and decompression. The AAC encoder removes audio information that is imperceptible to the human ear, thereby reducing file size while maintaining audio quality.

[0032] MP3 is a widely used audio compression technology that uses its lossy compression algorithm to reduce file size by removing audio parts that are not easily perceived by the human ear. The MP3 format has a high compression ratio, acceptable sound quality, and wide compatibility. MP3 encoding includes steps such as audio sampling, quantization and encoding, frequency domain transformation, re-quantization and encoding, and data compression. These steps work together on the audio signal to reduce the file size while maintaining the sound quality. The emergence of MP3 has greatly changed the landscape of the music industry, making the distribution and acquisition of music works more convenient.

[0033] Opus is an open source audio codec designed to provide efficient, flexible, and scalable audio compression. It supports audio bandwidths from narrowband to full-band, supports voice and music, as well as mono and stereo. Opus supports bit rates from 6kb / s to 510kb / s, sampling rates from 8kHz to 48kHz, and frame sizes from 2.5ms to 60ms. It also supports constant bit rate (CBR) and variable bit rate (VBR), and has good loss robustness and packet loss concealment (PLC). Due to its high performance and flexibility, Opus is widely used in various audio application scenarios, such as network audio transmission, video conferencing, and streaming services.

[0034] In one embodiment, specifically, in the spectrum subtraction algorithm used by the noise suppression unit, the audio signal is first converted from the time domain to the frequency domain by short-time Fourier transform, the PSD of the noise spectrum and the speech spectrum is calculated, and then the estimated value of the noise PSD is subtracted from the speech PSD to obtain the enhanced speech PSD, and finally the frequency domain signal is converted back to the time domain by inverse short-time Fourier transform to obtain the audio signal after noise suppression; the specific calculation formula is as follows: Short-time Fourier transform: ; in, is the short-time Fourier transform result of the mth frame and the kth frequency component, x(n) is the input audio signal, w(n) is the window function, R is the frame shift length, and N is the number of FFT points; PSD estimation: ; in, is the noise PSD estimate of the kth frequency component in the mth frame, and α is the smoothing factor; Speech PSD estimation and noise suppression: ; in, is the enhanced speech PSD, β is the over-reduction factor; Inverse short-time Fourier transform: ; in, is the enhanced frequency domain signal, It is the time domain audio signal after noise suppression.

[0035] Due to the overlap of window functions, it may be necessary to superimpose the inverse short-time Fourier transform results of different frames (overlap-add method) to obtain a complete time domain signal. These steps together constitute the noise suppression processing flow based on the spectrum subtraction algorithm. By adjusting the smoothing factor α and the over-subtraction factor β, a balance can be found between reducing noise and maintaining speech quality.

[0036] In one embodiment, specifically, the echo cancellation unit uses an adaptive filter based on a normalized minimum mean square error algorithm, which updates the filter coefficients in an iterative manner to minimize the error between the echo signal and the desired signal; the specific calculation formula is as follows: Filter coefficient update:

[0037] Where h(n) is the filter coefficient vector for the nth iteration, x(n) is the input signal vector, e(n) is the error signal, μ is the step size factor, and δ is the regularization parameter; Error signal calculation:

[0038] Among them, is the expected signal, that is, the original signal without echo, and is the echo estimation signal output by the filter.

[0039] To process real-time audio streams, these calculations are usually performed in blocks (e.g., frames) and are often combined with the Fast Fourier Transform (FFT) or similar algorithms for more efficient calculations in the frequency domain. In addition, to further improve performance, more complex algorithm variants such as the variable-step NLMS algorithm or the leaky NLMS algorithm may be used.

[0040] In one embodiment, specifically, the deep learning model used by the acoustic model unit includes a recurrent neural network and a long short-term memory network.

[0041] Recurrent Neural Network (RNN) is a neural network model with memory function, which can process sequence data and capture contextual information in the sequence through memory units (such as long short-term memory unit, LSTM). RNN can be used to extract acoustic features of speech signals, such as Mel-frequency cepstral coefficients (MFCC). By training the RNN model, the abstract representation of speech signals can be learned, thereby improving the performance of the acoustic model. RNN can be used to model acoustic models, such as phoneme recognition and speech recognition. By combining RNN with Hidden Markov Model (HMM), the modeling and recognition of speech signals can be achieved. RNN can capture the temporal characteristics of speech signals and improve the accuracy of acoustic models. RNN can also be used to model language models. By learning the probability distribution of language models, the accuracy of speech recognition can be improved. RNN can capture the contextual information of language sequences, thereby improving the performance of speech recognition.

[0042] Long short-term memory network (LSTM) is a special type of RNN. It solves the problem of gradient vanishing or exploding in traditional RNN and the disadvantage of difficulty in handling long-term dependencies by introducing a gating mechanism. LSTM achieves effective modeling of sequence data through memory units and gating units. LSTM can be used as the basis of acoustic models to map speech signals to speech units such as phonemes, syllables, or words to achieve speech recognition tasks. LSTM can also be used as a component of a language model to help improve the speech recognition system's understanding of language context, thereby improving the accuracy and fluency of recognition results. In speech recognition, the implementation process of the LSTM model includes steps such as data preparation, acoustic feature extraction, model construction, model training, and model evaluation.

[0043] In one embodiment, specifically, the meeting control and management module also includes a meeting mode switching function, allowing users to select different meeting modes according to meeting needs, such as discussion mode, demonstration mode or question-and-answer mode, to adapt to different meeting scenarios. The discussion mode is suitable for meeting scenarios that require team members to have free discussions, exchange opinions and ideas; the demonstration mode is suitable for meeting scenarios that require displaying PPT, reports or videos; the question-and-answer mode is suitable for meeting scenarios that require questions and answers, such as Q&A sessions, press conferences, etc.

[0044] In summary, the multifunctional audio intelligent optimization conference system provided by the present invention realizes efficient collection, optimization processing and intelligent recognition and translation of conference audio by integrating modules such as audio collection, preprocessing, optimization, speech recognition, translation, transmission, control management, display interaction, and security and privacy protection. The system adopts deep learning algorithms and advanced digital signal processing technologies to significantly improve the sound quality and recognition accuracy of conference audio, and meets specific conference needs through custom dictionaries and terminology libraries. At the same time, it supports a variety of audio encoding and decoding standards and protocols to ensure the efficient transmission of audio data. In addition, the system also has powerful security and privacy protection functions to ensure the security of conference data and the privacy of participants. Overall, the present invention provides a new intelligent, efficient and secure solution for modern meetings, greatly improving the interactivity and participation of meetings.

[0045] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0046] In the present disclosure, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. The block diagrams of the devices, devices, equipment, and systems involved in the present disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The words "such as" used here refer to the phrase "such as but not limited to", and can be used interchangeably with them.

[0047] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0048] It should also be noted that in the system and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0049] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0050] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0051] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A multifunctional audio intelligent optimization conference system, characterized by: It includes audio acquisition module, audio preprocessing module, audio optimization module, speech recognition module, translation module, audio transmission module, conference control and management module, display and interaction module and security and privacy protection module; The audio acquisition module is used to collect audio signals at the conference site in real time and optimize the audio acquisition effect; The audio preprocessing module is used to receive the original audio signal collected by the audio collection module, and use digital signal processing technology to perform preliminary processing and optimization on the audio signal; The audio optimization module is used to use a deep learning algorithm to intelligently optimize the audio signal after preliminary processing and optimization, improve the sound quality, and adaptively adjust the audio optimization strategy according to the conference scenario and user needs; The speech recognition module is used to convert the conference audio signal into text records in real time using speech recognition technology; The translation module is used to utilize machine translation technology to provide real-time translation display and voice broadcast functions, and supports multiple translation modes and language selections; The audio transmission module is used to transmit the optimized audio signal or translated voice data to other devices or networks via wired or wireless means, supporting multi-channel real-time audio stream transmission and voice intercom; The conference control and management module is used to control and manage the entire conference system, including conference start and end, participant management, authority setting, and device control, so that the conference proceeds smoothly; The display and interaction module is used to display the conference content, participant information, and translation results to the participants in a visual manner, and provide interactive functions, including speech application and voting; The security and privacy protection module is used to protect the security of conference data and the privacy of participants, including data encryption, access control and privacy setting functions.

2. A multifunctional audio intelligent optimization conference system according to claim 1, characterized in that: The audio acquisition module includes a microphone array unit, a preamplifier unit, an ADC unit and a signal conditioning unit; The microphone array unit is used to capture the audio signal at the conference site in real time; The preamplifier unit is used to receive the audio signal output by the microphone array unit and amplify it to a voltage or current level suitable for a subsequent processing circuit while maintaining the quality of the audio signal; The ADC unit is used to convert the analog audio signal output by the preamplifier unit into a digital audio signal; The signal conditioning unit is used to perform preliminary conditioning on the digital audio signal output by the ADC unit to remove noise and interference.

3. The multifunctional audio intelligent optimization conference system according to claim 1 is characterized in that: The audio optimization module includes a noise suppression unit, an echo cancellation unit and a dynamic range control unit; The noise suppression unit detects and suppresses non-steady-state and steady-state noise in the audio signal in real time using spectrum subtraction and adaptive filtering algorithms; The echo cancellation unit estimates and eliminates the echo component in the audio signal by using a double-ended echo cancellation technology through an adaptive filter and echo path estimation; The dynamic range control unit uses compression expansion technology and automatic gain control algorithm to automatically adjust the dynamic range of the audio signal and balance the high and low frequency signals.

4. The multifunctional audio intelligent optimization conference system according to claim 1 is characterized in that: The speech recognition module includes an acoustic model unit, a language model unit and a search decoding unit; The acoustic model unit uses a machine learning model to convert the audio signal into an acoustic feature vector to capture the acoustic characteristics of the speech signal; The language model unit is used to provide a probability distribution of vocabulary sequences, helping the recognition engine to infer vocabulary sequences based on the acoustic feature vectors and the language model; The search decoding unit combines the acoustic feature vector and the language model, and uses a search algorithm to search for a vocabulary sequence to perform an accurate conversion from audio to text. The search algorithm includes a Viterbi algorithm and a weighted finite state converter.

5. The multifunctional audio intelligent optimization conference system according to claim 1 is characterized in that: The translation module supports custom dictionaries and term bases, allowing users to add or modify professional terms and vocabulary based on specific meeting needs.

6. The multifunctional audio intelligent optimization conference system according to claim 1 is characterized by: The audio transmission module supports multiple audio codec standards and protocols, including AAC, MP3 and Opus.

7. The multifunctional audio intelligent optimization conference system according to claim 3 is characterized by: In the spectrum subtraction algorithm used by the noise suppression unit, the audio signal is first converted from the time domain to the frequency domain by short-time Fourier transform, the PSD of the noise spectrum and the speech spectrum is calculated, and then the estimated value of the noise PSD is subtracted from the speech PSD to obtain the enhanced speech PSD, and finally the frequency domain signal is converted back to the time domain by inverse short-time Fourier transform to obtain the audio signal after noise suppression; the specific calculation formula is as follows: Short-time Fourier transform: ； in, is the short-time Fourier transform result of the mth frame and the kth frequency component, x(n) is the input audio signal, w(n) is the window function, R is the frame shift length, and N is the number of FFT points; PSD estimation: ； in, is the noise PSD estimate of the kth frequency component in the mth frame, and α is the smoothing factor; Speech PSD estimation and noise suppression: ； in, is the enhanced speech PSD, β is the over-reduction factor; Inverse short-time Fourier transform: ； in, is the enhanced frequency domain signal, It is the time domain audio signal after noise suppression.

8. The multifunctional audio intelligent optimization conference system according to claim 3 is characterized by: The echo cancellation unit uses an adaptive filter based on a normalized minimum mean square error algorithm, which updates the filter coefficients in an iterative manner to minimize the error between the echo signal and the desired signal; the specific calculation formula is as follows: Filter coefficient update: ； Where h(n) is the filter coefficient vector for the nth iteration, x(n) is the input signal vector, e(n) is the error signal, μ is the step size factor, and δ is the regularization parameter; Error signal calculation: Among them, is the expected signal, that is, the original signal without echo, and is the echo estimation signal output by the filter.

9. The multifunctional audio intelligent optimization conference system according to claim 4 is characterized in that: The deep learning models used by the acoustic model unit include recurrent neural networks and long short-term memory networks.

10. The multifunctional audio intelligent optimization conference system according to claim 1, characterized in that: The conference control and management module also includes a conference mode switching function, allowing users to select different conference modes according to conference requirements.

Images