Voice data noise reduction processing method and system for intelligent conference

Abstract

The invention discloses a voice data noise reduction processing method and system for an intelligent conference, and relates to the technical field of voice processing. Programmable gating is introduced, an FGCNN acceleration engine is built in for initialization, an FPGA board card is built, and a conference platform is established to be in butt joint with a data interface of the FPGA board card; according to the data interface, voice signals based on a conference platform are collected and imported into an FPGA board card to execute analog-to-digital conversion and digital filtering, an FGCNN acceleration engine receives time domain input of a noisy signal sequence, frequency domain noise reduction logic processing is executed, and de-noised voice signals are output and transmitted back to the conference platform based on the data interface. The method and the device are used for solving the technical problems that in the prior art, an integrated voice processing technology which can carry out real-time efficient noise reduction and can operate in an embedded system with low power consumption is lacked, and efficient processing and noise reduction quality under the same frequency of a conference process are difficult to meet, and low-power-consumption operation and efficient noise reduction processing under the same frequency of platform operation are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, in particular to a speech data noise reduction processing method and system for intelligent conference. BACKGROUND

[0002] In the current conference system application scenario, the quality of voice communication is directly related to the conference efficiency and the experience of participating in the conference. However, the traditional conference voice acquisition system is often affected by environmental background noise, multiple sound source superposition, device echo and other interference factors in actual use, resulting in a decrease in voice intelligibility and frequent misjudgments in communication, especially in the multi-person remote conference scenario.

[0003] Most of the existing voice noise reduction methods rely on digital signal processing (DSP) based filtering algorithms or lightweight neural network models deployed based on ARM kernel platforms. The former has limited noise reduction capability and cannot adapt to non-stationary noise in a complex sound field. The latter has weak parallelism, low memory bandwidth and large processing delay due to the limited performance of the processor, and cannot meet the high real-time and high-precision voice enhancement requirements.

[0004] In recent years, some solutions have attempted to deploy deep learning models on general-purpose GPU platforms, achieving better noise reduction performance. However, general-purpose GPUs are high in cost and power consumption, cannot be embedded in lightweight conference terminals, and have poor adaptability in low-power and edge deployment scenarios.

[0005] Therefore, there is currently a lack of an integrated voice processing technology that can both efficiently reduce noise in real time and run in an embedded system with low power consumption, making it difficult to meet the efficient processing and noise reduction quality under the same frequency of the conference process. SUMMARY

[0006] The present application provides a speech data noise reduction processing method and system for intelligent conference, which is used to solve the technical problem that there is a lack of an integrated voice processing technology that can both efficiently reduce noise in real time and run in an embedded system with low power consumption, making it difficult to meet the efficient processing and noise reduction quality under the same frequency of the conference process.

[0007] In view of the above problems, the present application provides a speech data noise reduction processing method and system for intelligent conference.

[0008] In a first aspect, the application provides a voice data noise reduction processing method for intelligent conference, the method comprising: introducing programmable gate and initializing built-in FGCNN acceleration engine, building FPGA board card, and establishing data interface docking between conference platform and FPGA board card; collecting voice signal based on conference platform according to the data interface, importing FPGA board card to perform analog-digital conversion and digital filtering, FGCNN acceleration engine receiving time domain input of noisy signal sequence, performing frequency domain noise reduction logic processing, outputting de-noised voice signal and returning to conference platform based on the data interface; wherein the frequency domain noise reduction logic processing comprises: FGCNN acceleration engine using forward and inverse short-time Fourier transform layer to perform frequency domain feature extraction and time domain feature reconstruction, using encoder and decoder with skip connection to perform frequency dimension compression and frequency dimension restoration under gating, and using residual connection time convolution module to capture long-time dependence.

[0009] In a second aspect, the application provides a voice data noise reduction processing system for intelligent conference, the system comprising: a building unit for introducing programmable gate and initializing built-in FGCNN acceleration engine, building FPGA board card, and establishing data interface docking between conference platform and FPGA board card; a noise reduction execution unit for collecting voice signal based on conference platform according to the data interface, importing FPGA board card to perform analog-digital conversion and digital filtering, FGCNN acceleration engine receiving time domain input of noisy signal sequence, performing frequency domain noise reduction logic processing, outputting de-noised voice signal and returning to conference platform based on the data interface; wherein the frequency domain noise reduction logic processing comprises: FGCNN acceleration engine using forward and inverse short-time Fourier transform layer to perform frequency domain feature extraction and time domain feature reconstruction, using encoder and decoder with skip connection to perform frequency dimension compression and frequency dimension restoration under gating, and using residual connection time convolution module to capture long-time dependence.

[0010] One or more technical solutions provided in the application have at least the following technical effects or advantages: The intelligent conference voice data noise reduction processing method provided by the embodiment of the application introduces programmable gating and internally builds an FG CNN acceleration engine for initialization, builds an FPGA board card, and establishes a data interface docking between the conference platform and the FPGA board card; according to the data interface, a voice signal based on the conference platform is collected, imported into the FPGA board card for analog-digital conversion and digital filtering, the FG CNN acceleration engine receives a time domain input containing a noise signal sequence, performs frequency domain noise reduction logic processing, outputs a noise reduction voice signal, and returns to the conference platform based on the data interface; wherein the frequency domain noise reduction logic processing includes that the FG CNN acceleration engine uses a forward and inverse short-time Fourier transform layer for frequency domain feature extraction and time domain feature reconstruction, uses a skip-connection encoder and decoder to perform frequency dimension compression and frequency dimension restoration under gating, and uses a residual connection time convolution module to capture long-time dependence, so as to solve the technical problems in the prior art that there is a lack of an integrated voice processing technology that can both realize real-time and efficient noise reduction and run in an embedded system with low power consumption, and it is difficult to meet the efficient processing and noise reduction quality under the same frequency of the conference process, and realize low-power operation and efficient noise reduction processing under the same frequency of the platform operation. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 An FPGA internal architecture block diagram in an intelligent conference voice data noise reduction processing method is provided for the application. Figure 2 A flowchart of an intelligent conference voice data noise reduction processing method is provided for the application. Figure 3 A system structure diagram of an intelligent conference voice data noise reduction processing system is provided for the application.

[0012] Explanation of reference signs: a building unit 11 and a noise reduction execution unit 12. DETAILED DESCRIPTION

[0013] The application provides an intelligent conference voice data noise reduction processing method and system, which are used to solve the technical problems in the prior art that there is a lack of an integrated voice processing technology that can both realize real-time and efficient noise reduction and run in an embedded system with low power consumption, and it is difficult to meet the efficient processing and noise reduction quality under the same frequency of the conference process.

[0014] Embodiment one: as shown in Figure 1 , Figure 2 The application provides an intelligent conference voice data noise reduction processing method, which includes: S1: introducing programmable gating and internally building an FG CNN acceleration engine for initialization, building an FPGA board card, and establishing a data interface docking between the conference platform and the FPGA board card.

[0015] In the embodiment of the present application, a programmable gate is introduced and an FG CNN acceleration engine is built in for initialization stage processing. The programmable gate refers to a gate control structure constructed by logic units under the premise of hardware resources, which can adjust the path of feature flow according to the dynamic changes of input signals, has the ability of dynamic information regulation and feature compression, and is beneficial to improve the selective expression ability of speech signals in deep neural networks.

[0016] In the present application, the programmable gate is the architecture and logic program writing area of the FG CNN acceleration engine, which is the built-in function area in the FPGA development board. By deploying the programming information of the FG CNN acceleration engine in the programmable gate of the FPGA development board, the initialization process is realized, and the deployed FPGA development board is used as an FPGA board card.

[0017] The FG CNN acceleration engine in the present application is a frequency domain gating convolutional neural network engine deployed on the FPGA board card, which internally includes a short-time Fourier transform layer (STFT layer), a multi-level gating convolutional encoder, a time convolution module, a mirrored decoder, and an inverse short-time Fourier transform layer, and is dedicated to performing frequency domain modeling and time domain reconstruction operations of speech signals. In the initialization process, the mapping and binding of the model logic and the FPGA resources are completed by loading and timing scheduling configuration of each layer structure of the FG CNN model, ensuring that the whole process of speech noise reduction processing can run in parallel inside the board card, and having low delay and high throughput processing capability.

[0018] For example, the Kintex-7 FPGA chip of the Xilinx series is selected, 256 DSP slices and 128 BRAM blocks can be configured in it for parallel convolution and spectrum processing tasks, and the logic description and wiring optimization of the FG CNN are realized through Verilog or Vivado HLS. This process ensures efficient coupling of the model structure of the neural network and the underlying resources of the FPGA, meeting the real-time processing requirements of high-concurrency input of conference speech signals.

[0019] Further, after the construction of the FPGA board card is completed, a data interface between the conference platform and the FPGA board card needs to be established for docking. To realize the automatic logic cycle of platform acquisition-speech noise reduction-noise reduction characterization.

[0020] Among them, the data interface refers to a data communication bridge for connecting the conference platform software and the FPGA board card hardware, for example, using a USB serial interface, an SPI high-speed interface, or a UDP interface based on Ethernet. The present application preferably uses a data interface based on a USB3.0 channel, which has a transmission rate of up to 5Gbps, which can meet the continuous input requirements of high-frame-rate speech data.

[0021] In a feasible embodiment, during the docking process, the collection control module and the voice data cache module in the conference platform establish a bidirectional communication link with the FPGA board card through the driving protocol, realizing real-time transmission and synchronization of voice data between the conference terminal and the FG CNN acceleration engine.

[0022] For example, when the conference platform detects a speaking instruction trigger, it immediately sends a collection instruction and an initial voice segment to the FPGA board card through the data interface, and the board card returns the noise-reduced voice for subsequent audio processing and playback. This step ensures data consistency and response timeliness between the platform and the hardware during voice collection and processing.

[0023] S2: According to the data interface, collect the voice signal based on the conference platform, import the FPGA board card to perform analog-to-digital conversion and digital filtering, the FG CNN acceleration engine receives the time domain input of the noise signal sequence, performs frequency domain noise reduction logic processing, and outputs the denoising voice signal and returns to the conference platform based on the data interface; Wherein, the frequency domain noise reduction logic processing includes: the FG CNN acceleration engine uses forward and inverse short-time Fourier transform layer for frequency domain feature extraction and time domain feature reconstruction, uses the encoder and decoder with skip connection to perform frequency dimension compression and frequency dimension restoration under the gating, and uses the time convolution module with residual connection to capture long-time dependence.

[0024] In the embodiment of the application, based on the aforementioned established data interface, the voice signal in the conference platform is first collected and processed. The conference platform triggers the collection operation when the conference starts or a speaking trigger event occurs. For example, the built-in audio collection module of the conference platform is called to start the microphone array to obtain real-time voice data, and the analog voice signal is continuously transmitted to the FPGA board card through the data interface. The data interface supports high-bandwidth and low-latency data transmission, ensuring that the voice sample is transmitted within milliseconds of delay, suitable for real-time voice communication environment.

[0025] Further, after the voice signal reaches the FPGA board card, it is first processed by the analog-to-digital conversion module to convert the input continuous analog signal into discrete digital signal.

[0026] For example, the analog-to-digital conversion module usually quantizes the sampling points with 12-bit or 16-bit resolution, and the sampling frequency is preferably set to 16 kHz to match the characteristics of the human voice frequency band. The conversion result is a single-channel digital voice sequence in 1xL format, where L is the number of sampling points per frame.

[0027] Subsequently, the digital voice sequence is input to the digital filter module to perform band-pass filtering to suppress background low-frequency noise and high-frequency electronic interference, ensuring that the signal energy is concentrated in the human voice main frequency, for example: 300Hz~3.4kHz, and improving the signal-to-noise ratio of subsequent feature extraction.

[0028] Further, the digital voice signal after the filtering process is input as a noisy signal sequence into the FG CNN acceleration engine in the FPGA board.

[0029] The FG CNN acceleration engine first calls a forward short-time Fourier transform layer (STFT) to perform window function weighting and Fourier transform on the signal to generate a 2×T×257 structure complex spectrum, where T is the number of frames, 257 is the number of frequency points, and 2 is the two channels of the real part and the imaginary part of the complex number. This frequency domain representation constitutes the input basis of frequency domain feature extraction, effectively revealing the frequency energy distribution and its time sequence evolution characteristics of the speech.

[0030] Next, the frequency domain features are input into the encoder module.

[0031] In an optional embodiment of the present application, the encoder is composed of multiple layers of gated convolution layers, which perform frequency dimension downsampling and feature compression, and use a skip connection mechanism to retain intermediate feature information. Each layer of gated convolution is designed based on a gating linear unit (GLU), which has the ability to select and control the feature flow channel, improving the robustness of feature expression. Finally, a 64×T×4 compressed feature map is output, which significantly reduces the feature dimension while retaining key information.

[0032] For example, one implementation of the encoder is as follows: feature extraction is performed through 6 levels of gated convolution layers, and each level of gated convolution layer performs the following operations: performs separable convolution operation on the input, with a convolution kernel size of 3×3; feature selection is achieved through a gating linear unit (GLU), and the activation function uses PReLU; batch normalization processing is performed; 2-fold downsampling is performed along the frequency dimension; and finally, a 64×T×4 compressed feature is output.

[0033] Subsequently, the compressed feature flow is transferred to the time convolution module, which is composed of multiple layers of residual connected dilated convolution, and is dedicated to extracting long-time dependent features in the speech.

[0034] For example, one implementation of the time convolution module is as follows: each residual block contains an input 1×1 convolution for expanding the channel number; a dilated depth convolution for expanding the time receptive field, i.e., processing the time sequence, with an exponentially increasing dilation coefficient; and an output 1×1 convolution for restoring the channel dimension.

[0035] Further, after the time sequence feature processing, it enters the decoder, and the decoder structure is symmetrical to the encoder and uses a skip connection to recover the intermediate features from each level of the encoder, layer by layer restores the structure information of the frequency dimension, and finally outputs a frequency spectrum with the same dimension as before encoding.

[0036] Then, time-domain reconstruction is performed through an inverse short-time Fourier transform layer (iSTFT), and the denoised speech signal sequence in the 1xL format consistent with the original time domain is output.

[0037] Optionally, the reshaped layer is used for integrating the upper layer output.

[0038] Finally, the denoised speech signal is returned to the conference platform through the data interface for real-time playing, speech recognition or archiving processing. The whole process from collection to output control is completed within extremely low delay, meeting the dual requirements of speech clarity and response speed of multi-party remote conference, and improving the conference interaction quality and system robustness.

[0039] Further, the step S1 includes: the encoder and the decoder are built by using the gated convolution principle and a skip connection mode, wherein the information transmission mode is dynamically adjusted by gating; and the time convolution module is stacked by a plurality of residual blocks, wherein each residual block includes an input convolution, an intermediate convolution and an output convolution, the input convolution is used to expand the number of input channels, and the output convolution is used to restore the number of channels.

[0040] In the embodiment of the application, the encoder and the decoder are both built based on the gated convolution principle. The gated convolution is an improved convolution structure, and the core is to introduce the dynamic adjustment ability of information flow by the gating mechanism, so that the network has stronger selectivity and expression ability when processing the speech sequence signal.

[0041] Specifically, each gated convolution unit in the application includes two groups of convolution kernels: one group is used to generate a main feature map, and the other group is used to generate a gating map, i.e. a control weight map; after convolution calculation, the two are fused by a gated linear unit, i.e. the main feature map and the gating map after sigmoid activation are multiplied element by element, so as to suppress redundant information and strengthen key features. This structure significantly improves the discrimination ability in the process of frequency dimension feature compression and reconstruction, and is suitable for complex scenes such as drastic channel change and local noise concentration in speech signals.

[0042] In order to further improve the feature propagation efficiency, the skip connection structure is used between the encoder and the decoder.

[0043] The skip connection structure retains the multi-level intermediate feature results in the encoding process, and corresponds to fuse them into the same layer structure of the decoder in the decoding process, realizes multi-scale expression and restoration of features, and solves the problems of information loss and gradient disappearance in deep network.

[0044] For example, when the encoder performs fourth layer gated convolution compression, the obtained feature map is directly connected to the corresponding up-sampling node in the decoder, and is combined with the features of the decoding path to improve the accuracy and stability of frequency restoration.

[0045] Further, in terms of time dimension modeling, the time convolution module is stacked with multiple residual blocks. Each residual block internally includes three main convolution units: input convolution, intermediate convolution, and output convolution.

[0046] In a preferred embodiment of the application, the input convolution adopts a 1x1 convolution kernel structure, which is used to expand the input feature map of the residual block in the channel dimension, increase the number of feature channels, and enable the subsequent convolution to have a larger representation space; the intermediate convolution adopts a depth separable convolution, and an expansion factor is introduced to expand the receptive field and enhance the perception ability of long-time dependence; the output convolution is also a 1x1 structure, which is used to restore the processed channel number to the input level, ensure the consistency of the dimensions with the residual input path, and realize the addition fusion operation of the residual connection. Preferably, a PReLU nonlinear activation function and a BatchNorm batch normalization process are combined after each convolution unit to improve the training stability and generalization ability.

[0047] Through the integration of the above structure, the FGCNN provided by the application has both the strong feature extraction capability of the deep learning model and the structural compression and restoration characteristics of the time and frequency dimensions in the speech sequence on the FPGA hardware, significantly improving the accuracy and real-time performance of the speech signal noise reduction processing.

[0048] Further, before initializing the built-in FGCNN acceleration engine, the step S1 of the application includes: building a hierarchical architecture of the FGCNN acceleration engine, wherein the hierarchical architecture is composed of a short-time Fourier transform layer, an encoder, a time convolution module, a decoder, and an inverse short-time Fourier transform layer, wherein channel pruning and expansion within the support level are supported; obtaining paired samples, wherein the paired samples are a sample pair set containing noisy speech and pure speech; and performing supervised training on the hierarchical architecture according to the paired samples until convergence to complete the construction of the FGCNN acceleration engine.

[0049] Wherein, the iterative convergence condition is used for supervised training constraint, wherein the optimization objective is to minimize the mean square error between the output signal and the target signal, wherein the output signal is the FGCNN output based on the noisy speech in the paired sample, and the target signal is the corresponding pure speech in the paired sample.

[0050] In the embodiment of the present application, a hierarchical architecture of the FG CNN acceleration engine is first constructed for deploying and implementing efficient speech signal noise reduction calculation on the FPGA hardware platform. The hierarchical architecture is distributed according to the designed speech processing flow and sequentially includes a short-time Fourier transform layer (STFT layer), an encoder, a time convolution module, a decoder, and an inverse short-time Fourier transform layer.

[0051] Among them, the short-time Fourier transform layer is used to convert the input time-domain speech signal into a frequency-domain complex spectrum representation, the encoder extracts frequency dimension features and realizes compression through multi-level gated convolution, the time convolution module models long-term sequential dependence through the dilated residual network, the decoder reconstructs and up-samples the frequency dimension through the symmetric structure, and the short-time Fourier transform layer restores the frequency-domain features after noise reduction to the time-domain speech signal, completing the entire end-to-end denoising process.

[0052] In a preferred embodiment provided by the present application, in the hierarchical architecture, a pruning and expansion mechanism supporting internal channels of each level is specially designed. Specifically, channel pruning refers to automatically removing redundant or low-contribution convolution channels according to the importance measure of the feature channels in the model optimization process, so as to reduce the parameter amount and the calculation complexity; and channel expansion increases the channel dimension through a lightweight method such as 1x1 convolution when the feature extraction demand is enhanced, so as to improve the feature expression capability. The pruning and expansion strategy facilitates subsequent mapping of the model to the FPGA board, and significantly improves the hardware resource utilization rate and the calculation efficiency.

[0053] Further, a paired sample set is obtained to complete the training of the FG CNN acceleration engine. The paired sample is composed of a noisy speech and its corresponding pure speech signal, and constitutes a one-to-one input-output relationship. The sample data covers different noise scenes, such as fan noise, crowd noise, current interference, etc., and different speakers, ensuring that the training result has strong generalization ability.

[0054] In the training stage, the above hierarchical structure is optimized in a supervised learning manner.

[0055] Specifically, the noisy speech sample is input into the FG CNN architecture, the model output signal is obtained, and the error comparison is performed between the output signal and the corresponding pure speech sample as the target signal; and an optimization target with the mean square error between the output signal and the target signal as the loss function is constructed.

[0056] In one feasible implementation manner, the loss function is as follows: ; wherein, is the total number of samples, is the corresponding pure speech sample, is the output denoised speech signal.

[0057] Preferably, an iterative convergence mechanism is adopted in the training process to continuously update the convolution kernel weights and bias parameters inside the FG CNN until the loss function reaches a stable minimum value or is lower than a set threshold, indicating that it has converged. Optionally, early stopping strategies and learning rate decay strategies are introduced to speed up convergence and avoid overfitting. After training is complete, the network structure and parameters are solidified, i.e., the construction of the FG CNN acceleration engine is completed, and then it is deployed in the FPGA development board for real-time speech noise reduction processing.

[0058] In summary, it ensures high speech restoration ability and noise reduction robustness in different noise scenarios.

[0059] Further, the step S1 includes introducing a digital-to-analog conversion interface, wherein the digital-to-analog conversion interface is deployed at both ends of the FG CNN acceleration engine, the digital-to-analog conversion interface at the front end performs analog-digital conversion, and the digital-to-analog conversion interface at the back end performs digital-analog conversion; a digital filter is deployed between the digital-to-analog conversion interface at the front end and the FG CNN acceleration engine to perform signal initialization filtering processing.

[0060] In the embodiment of the application, to realize the complete closed-loop processing flow of the speech signal on the FPGA platform, a digital-to-analog conversion interface is introduced and deployed at both ends of the FG CNN acceleration engine, i.e., at the input side and the output side respectively.

[0061] The digital-to-analog conversion interface at the front end is used to convert the analog speech signal collected from the microphone of the conference platform into a digital format, so that it can be processed by the subsequent processing, including the digital filter and the FG CNN acceleration engine in this application.

[0062] In an optional embodiment, the digital-to-analog conversion interface is preferably a high-fidelity converter with 16-bit sampling accuracy and a sampling rate of 16 kHz or 48 kHz, which has low delay and low distortion characteristics, and can suppress background electrical noise in the analog signal while preserving the key frequency band information of the human voice.

[0063] Similarly, after the FG CNN acceleration engine completes the frequency domain modeling, feature compression, time modeling, frequency domain reconstruction, and time domain restoration of the speech signal, the output result is still a digital format of the denoised speech signal. To make the result return to the audio output interface or the speaker playback module of the conference platform, it needs to be inversely converted by the digital-analog conversion interface at the back end.

[0064] Preferably, the digital-to-analog conversion interface cooperates with a high-quality low-pass filter to realize smooth reconstruction of the signal, and the output frequency response range covers the human audible bandwidth (20 Hz~20 kHz), ensuring natural and clear audio quality and meeting the requirements of the conference platform for voice output sound quality.

[0065] In order to further improve the noise reduction effect and suppress the high frequency noise interference that may exist in the input end, a digital filter is arranged between the digital-analog conversion interface of the pre-end and the FG CNN acceleration engine to perform initialization filtering processing on the signal.

[0066] Optionally, the digital filter preferably adopts an FIR (Finite Impulse Response) structure or an IIR (Infinite Impulse Response) structure, has a band-pass filtering capability, and can suppress noise components (such as fan whistling and high-frequency electronic noise) higher than the voice frequency band and environmental low-frequency disturbances lower than the lower limit of voice energy. The digital filter processes the sampled data in a streaming manner, and the output is a 1xL structure digital voice sequence in which the noise has been preliminarily reduced, which is used as the effective input of the FG CNN acceleration engine.

[0067] In summary, not only the information integrity and processing continuity in the analog-digital signal chain are ensured, but also the signal-to-noise ratio of the voice data before input is improved, a cleaner and more structured initial feature basis is provided for the FG CNN acceleration engine, and thus the voice noise reduction performance and the final sound quality output effect of the whole machine are significantly improved.

[0068] Further, before collecting the voice signal based on the conference platform, the step S2 of the present application comprises: According to the data interface, the running flow state of the conference platform is monitored to generate a monitoring requirement, wherein the monitoring requirement contains a collection trigger instruction, and the conference enters a preliminary collection stage is added, and the collection trigger instruction is generated at the start of the conference; according to the collection trigger instruction in the monitoring requirement, the conference voice collection of the target conference is driven to obtain real-time conference voice, and a temporary data channel of the target conference and the data interface is established, wherein the temporary data channel is marked with a conference code.

[0069] In the embodiment of the present application, in order to realize the automation and dynamic control of the conference voice collection process, first, the running flow state of the conference platform is monitored in real time based on the data interface. The running flow state refers to the conference business flow state of the conference platform in different stages, such as booking, entering, starting, speaking, pausing, etc. Optionally, the monitoring method includes periodically polling the system session management or listening to the state change event signal sent by the conference scheduler. The running parameters such as conference activity identifier, participant state, and voice channel activity information are continuously analyzed.

[0070] After the running flow state is identified, the corresponding monitoring requirement is generated according to the monitored state change. The monitoring requirement is a structured data control instruction set, and at least contains a collection trigger instruction for indicating that the audio data preparation collection process is entered.

[0071] For example, when the new conference state entry is detected, the preliminary collection type collection trigger instruction is automatically generated; when the conference state is converted to formal start or any speaker channel is activated, the trigger instruction of real-time collection type is generated. In the specific implementation process, the collection instruction is used to trigger the opening of the voice signal link and the activation of the FPGA data path.

[0072] Further, after receiving the collection trigger instruction, the conference voice collection operation of the target conference is driven. That is, the voice collection module in the conference platform responds to the collection instruction, starts the microphone array or the voice input channel, collects the speech data of the conference participants in real time, and establishes a temporary data channel between the voice collection module, the data interface and the FPGA board card of the conference platform through the data interface as the middleware.

[0073] Among them, the data channel is an audio data transmission path between the conference platform and the FG CNN acceleration engine. In an optional embodiment, the channel type can be a physical serial port link, a logical channel allocation or a temporary UDP connection based on IP packets. The preferred way is to enable an independent cache channel on the USB or PCIe bus.

[0074] Further, in order to distinguish the data of each conference in the case of multiple conference concurrent operation, the temporary data channel will be marked by a conference code, such as conference ID, UUID and other unique identifiers. In the specific implementation process, the conference code will be attached to the data packet header, or transmitted in the form of independent channel index, so that the corresponding session data matching and identification can be completed in the data aggregation, noise reduction processing and backhaul output stage.

[0075] In summary, the fine control strategy of controllable data collection process, unique data path identification and on-demand activation of audio processing module in the multi-conference environment is realized, which enhances the running flexibility and processing concurrency of the system.

[0076] Further, the voice signal based on the conference platform is collected and imported into the FPGA board card for analog-to-digital conversion and digital filtering. The step S2 of the present application comprises: According to the data interface, the real-time conference voice is imported into the FPGA board card; a digital-to-analog conversion interface is triggered to convert the real-time conference voice into a digital quantity to determine a digital conference voice; and the digital conference voice stream is converted to the digital filter to determine a noisy signal sequence, wherein the digital filter converts the digital conference voice into a sequence form with a dimension of 1xL, and L is the number of sampling points.

[0077] In the embodiment of the present application, first, the real-time conference voice is introduced into the FPGA board according to the data interface. Specifically, after the conference platform receives the collection trigger instruction and establishes the corresponding temporary data channel, the collected real-time voice signal in the form of analog voltage is transmitted to the data interface through a microphone array or other audio input device, and the interface is responsible for completing the physical signal transmission and protocol analysis between the conference platform and the FPGA board.

[0078] Preferably, the data interface can adopt a communication mode with low delay and high throughput capacity, such as high-speed USB, SPI or PCIe bus, to ensure that the voice signal is stably delivered to the FPGA processing platform within milliseconds.

[0079] Then, after the FPGA board receives the voice signal, the digital-analog conversion interface of the pre-stage is triggered to perform analog-digital conversion processing on the analog real-time voice signal.

[0080] In an optional embodiment, the digital-analog conversion interface continuously samples the input waveform at a preset sampling frequency (for example, 16 kHz or 48 kHz) and quantizes the analog voltage value into a discrete digital value to generate digital voice data in the form of a one-dimensional sample point array, that is, the digital conference voice. Preferably, the sampling bit width of the digital-analog conversion interface is 12 bits or 16 bits to ensure signal fidelity and improve the accuracy of subsequent feature extraction.

[0081] Subsequently, the obtained digital conference voice is introduced into a digital filter. The digital filter is disposed between the output of the digital-analog conversion interface and the input of the FG CNN acceleration engine and has a band-pass filtering characteristic. Its core function is to suppress non-voice frequency band interference components, such as low-frequency mechanical noise and high-frequency electromagnetic noise, to improve the signal-to-noise ratio of the voice frequency band signal.

[0082] In a preferred embodiment, the digital filter is preferably of FIR (Finite Impulse Response) structure and is designed according to the main frequency range of human voice. In the streaming processing process, the digital filter processes continuous input data in frames and formats each frame of digital voice data into a one-dimensional sequence with a dimension of 1xL, where L represents the number of sample points in the current frame, for example, L=512 or 1024.

[0083] Further, the output after filtering is a noisy signal sequence that retains the main frequency component of the voice information and removes part of the background interference, serving as the input basis for subsequent FG CNN frequency domain analysis and providing a stable and clean voice data source for high-precision noise reduction processing.

[0084] In summary, the above processing flow ensures the full-link digitization and signal quality optimization of the original analog signal before entering the FG CNN acceleration engine inference stage, laying a foundation for realizing high robustness and high real-time performance of the entire voice processing system.

[0085] Further, the frequency domain noise reduction logic processing is performed, and the step S2 includes: The noise-containing signal sequence is introduced into the FG CNN acceleration engine to trigger a short-time Fourier transform layer to perform frequency domain conversion and frequency domain feature extraction, wherein the data form is a complex spectrum of 2×T×257, 2 is the channel number identifying real and imaginary numbers, T is the frame number, and 257 is the frequency point number; the frequency domain feature stream is transferred to an encoder to expand the channel number and down-sample along the frame frequency dimension to determine a first encoding sequence, wherein the time dimension is quantitative, and the output data form is 64×T×4; the first encoding sequence is transferred to a time convolution module to determine a time encoding sequence by performing time dimension dependent relationship processing; the time encoding sequence is transferred to a decoder to perform mirror decoding processing to determine a first decoding sequence; and the first decoding sequence is subjected to time domain feature reconstruction by an inverse short-time Fourier transform layer to determine the denoising voice signal.

[0086] In the embodiment of the application, the noise-containing signal sequence subjected to digital filtering processing is first introduced into the FG CNN acceleration engine. The noise-containing signal sequence is time domain discrete voice data with a dimension of 1×L and has a complete time domain voice structure.

[0087] Further, after the FG CNN acceleration engine is started, the short-time Fourier transform layer integrated therein is first triggered to perform fast Fourier transform processing on the input time domain voice signal to realize frequency domain conversion and frequency domain feature extraction.

[0088] Specifically, the short-time Fourier transform layer divides the continuous time domain sequence into frames, applies Hamming window or Blackman window weighting information to each frame, and then performs fast Fourier transform on each frame to map the time information into a complex spectrum representation to form a three-dimensional feature tensor with a dimension of 2×T×257. Wherein, 2 represents the real and imaginary number channel of the complex spectrum, T is the total frame number, and 257 is the frequency point number reserved in each frame, which usually corresponds to the half frequency domain truncation processing after 512-point FFT. The spectrum tensor reflects the energy distribution characteristics of the voice signal in the time-frequency two-dimensional plane, and serves as the input of the subsequent deep learning noise reduction modeling.

[0089] Further, the frequency domain features are then passed to an encoder in the FG CNN acceleration engine. This module is composed of multiple gated convolutional layers in series, with the ability to downsample in the frequency dimension and compress features, while using a gating mechanism to enhance the response selectivity of the effective frequency channels. Each convolutional layer not only performs convolution operations in the frequency dimension, but also performs channel expansion and 2x downsampling, combined with PReLU activation and BatchNorm normalization processing. Finally, the output is a 64xTx4 encoded feature sequence, where 64 represents the expanded channel number, T is the time frame number which remains unchanged, and 4 is the number of frequency points retained after downsampling, constituting the first encoded sequence.

[0090] Subsequently, the first encoded sequence is input to a temporal convolution module. The temporal convolution module is composed of multiple dilated residual blocks, each of which is composed of an input 1x1 convolution, an intermediate dilated convolution, and an output 1x1 convolution. The convolution kernel is expanded along the time dimension, and the receptive field is expanded by an exponentially increasing dilation factor (such as 1, 2, 4, 8, etc.) to extract context features with long-term dependencies in speech. With the support of residual connection mechanism, the temporal convolution module can avoid gradient vanishing and maintain feature continuity. After processing by the temporal convolution module, the obtained temporal encoded sequence retains the original structure in the time dimension, while significantly enhancing the temporal context modeling capability.

[0091] Subsequently, the temporal encoded sequence is input to a decoder. The decoder is a mirror structure of the encoder, using gated deconvolution operations to gradually upsample the frequency dimension and restore the channels, and fusing with the output of the intermediate layers of the encoder through a skip connection mechanism to enhance the high-frequency information reconstruction capability. The decoder gradually recovers the frequency distribution structure and outputs the first decoded sequence, which has the same dimension as the output of the short-time Fourier transform layer, facilitating subsequent spectral reconstruction.

[0092] Finally, the first decoded sequence is input to an inverse short-time Fourier transform layer, which performs a frame-by-frame complex spectrum inverse transform to realize the restoration process from the frequency domain to the time domain. Specifically, this process combines the real and imaginary parts of the decoder output to restore the corresponding time-domain waveform, and finally outputs a denoised speech signal sequence with a dimension of 1xL. The reshaping layer proposed in this application is used to perform output integration processing of the upper layer structure, and is an optional trigger architecture.

[0093] In summary, the obtained denoised speech signal sequence is a high-fidelity, low-noise speech result, which can be directly used for speech playback, recognition, or storage processing on a conference platform, meeting the core needs of high real-time speech processing applications.

[0094] Further, the step S2 includes introducing a space-time code, wherein the space-time code is a timestamp of a speech collection time and a location code of a target conference port; and performing identification tracking under a noise reduction processing cycle on the real-time conference speech according to the space-time code.

[0095] In the embodiment of the application, in order to further improve the management accuracy and data processing matching capability of the voice signal in a multi-conference, multi-channel concurrent environment, a space-time code mechanism is introduced to uniquely identify and track the source of the collected voice data.

[0096] The space-time code is a composite identifier for describing a voice signal collection event, and specifically consists of two parts: a timestamp for recording the accurate time of the current voice data collection, usually represented in millisecond or microsecond level time format; and a target conference port location code, which is used to identify which conference terminal or device channel the audio signal comes from, for example, can be represented in the form of MAC address, IP port number, device number or logical channel index.

[0097] Specifically, in the voice signal collection process, each frame or each segment of collected data is attached with a corresponding space-time code, and before being transmitted to the FPGA board card, the data packet is sent to the processing link. While executing the FG CNN acceleration engine noise reduction processing, the space-time code in the input data is read and analyzed, and it is mapped to the corresponding processing thread or storage queue to realize accurate routing of voice data.

[0098] In the actual noise reduction processing cycle, the space-time code is used as the core reference identifier for identification tracking operation. That is, after the FG CNN processing result is output, the source attribute of the noise-reduced voice is re-bound according to the original space-time code, so that the noise-reduced voice stream can be accurately matched to the original conference session and speaking channel when it is returned to the conference platform, preventing multi-conference cross-misplacement or data flow error.

[0099] For example, when multiple conference terminals simultaneously initiate voice requests, the collection time and port number of each segment of voice are distinguished to ensure that each segment of data is processed and output independently according to the corresponding conference and speaking position.

[0100] In summary, not only the support capability for multi-task parallel processing is improved, but also a foundation is provided for subsequent function expansion, such as voice log tracing, data cache scheduling optimization, user speaking behavior analysis and other application scenarios based on space-time code, with high system scalability and data management reliability.

[0101] The voice data noise reduction processing method of the intelligent conference provided by the application has the following technical effects: 1. FG CNN acceleration engine construction and training: a layered architecture composed of short-time Fourier transform layer, encoder-decoder, and time convolution module is built, and supervised training is performed on noisy and clean speech samples to minimize mean square error for iterative optimization. The engine is given precise speech noise reduction capability, improving adaptability to different noise scenes and ensuring the reliability of the noise reduction model. FPGA hardware acceleration and data docking: programmable gate and FG CNN acceleration engine are introduced to initialize the FPGA board, establish the data interface between the conference platform and the board, and realize speech signal acquisition, analog-to-digital conversion and filtering processing. The parallel computing characteristics of FPGA are used to greatly improve the efficiency of speech processing and reduce the delay of real-time noise reduction, ensuring the smoothness of conference speech.

[0102] 2. Frequency domain noise reduction logic processing architecture: forward and inverse short-time Fourier transform is used to realize frequency domain and time domain conversion, and the encoder-decoder of gated convolution is used to compress and restore the frequency dimension, combined with the residual time convolution module to capture the long-time dependence of speech. Effectively separate noise and speech signals, enhance the ability to suppress complex environmental noise, and improve the clarity and integrity of the noise-reduced speech.

[0103] 3. Time-space code tracking and dynamic management: time stamp and conference port location code are introduced to form a time-space code to identify and track the real-time speech noise reduction process. It realizes accurate management and noise reduction matching of speech data in multiple conference scenarios, avoids data confusion, and improves the pertinence and accuracy of noise reduction processing. Full-process signal processing mechanism: from conference speech acquisition trigger, analog-to-digital conversion, digital filtering preprocessing, to FG CNN engine noise reduction, time domain reconstruction, forming a complete processing link. An end-to-end speech noise reduction scheme is constructed to ensure stable noise reduction of conference speech from acquisition to output, and to improve the quality of intelligent conference speech.

[0104] Embodiment two: based on the same inventive concept as the speech data noise reduction processing method of an intelligent conference in the foregoing embodiment, as shown in Figure 3 The present application provides an intelligent conference speech data noise reduction processing system, which comprises: The building unit 11 is used to introduce programmable gate and initialize the built-in FG CNN acceleration engine, build the FPGA board, and establish the data interface docking between the conference platform and the FPGA board; The noise reduction execution unit 12 is used to collect the speech signal based on the conference platform according to the data interface, import the FPGA board to perform analog-to-digital conversion and digital filtering, and the FG CNN acceleration engine receives the time domain input of the noisy signal sequence, performs frequency domain noise reduction logic processing, outputs the de-noised speech signal, and returns to the conference platform based on the data interface; The frequency domain noise reduction logic processing includes: The FG CNN acceleration engine adopts forward and inverse short-time Fourier transform layers for frequency domain feature extraction and time domain feature reconstruction, adopts an encoder and a decoder with skip connection to perform frequency dimension compression and frequency dimension restoration under gating, and adopts a time convolution module with residual connection to capture long-time dependence.

[0105] The building unit 11 performs the following steps: the encoder and the decoder are built using the gating convolution principle and the skip connection mode, wherein the information transmission mode is dynamically adjusted through gating; the time convolution module is stacked by a plurality of residual blocks, wherein each residual block includes an input convolution, an intermediate convolution and an output convolution, the input convolution is used to expand the number of input channels, and the output convolution is used to restore the number of channels.

[0106] The building unit 11 performs the following steps: a layered architecture of the FG CNN acceleration engine is constructed, wherein the layered architecture is composed of a short-time Fourier transform layer, an encoder, a time convolution module, a decoder and an inverse short-time Fourier transform layer, wherein pruning and expansion of intra-level channels are supported; paired samples are obtained, wherein the paired samples are a sample pair set of noisy speech and clean speech; the layered architecture is supervised trained to convergence according to the paired samples, and the construction of the FG CNN acceleration engine is completed.

[0107] The iterative convergence condition is used for supervised training constraint, the mean square error of the output signal and the target signal is minimized as the optimization objective, the output signal is the FG CNN output based on the noisy speech in the paired sample, and the target signal is the corresponding clean speech in the paired sample.

[0108] The building unit 11 performs the following steps: a digital-to-analog conversion interface is introduced, the digital-to-analog conversion interface is deployed at both ends of the FG CNN acceleration engine, the digital-to-analog conversion interface at the front end performs analog-digital conversion, and the digital-to-analog conversion interface at the rear end performs digital-analog conversion; a digital filter is disposed between the digital-to-analog conversion interface at the front end and the FG CNN acceleration engine to perform signal initialization filtering.

[0109] The noise reduction execution unit 12 performs the following steps: according to the data interface, the running flow state of the conference platform is monitored to generate a monitoring requirement, wherein the monitoring requirement includes a collection trigger instruction, the conference enters a preliminary collection stage, and the collection trigger instruction is generated at the start of the conference; according to the collection trigger instruction in the monitoring requirement, the conference voice collection of the target conference is driven to obtain real-time conference voice, and a temporary data channel of the target conference and the data interface is established, wherein the temporary data channel is marked with a conference code.

[0110] The noise reduction execution unit 12 performs the following steps: according to the data interface, the real-time conference voice is introduced into the FPGA board card; a digital-analog conversion interface is triggered to convert the real-time conference voice into a digital quantity to determine a digital conference voice; and the digital conference voice stream is converted to the digital filter to determine a noisy signal sequence, wherein the digital filter converts the digital conference voice into a sequence form with a dimension of 1xL, and L is the number of sampling points.

[0111] The noise reduction execution unit 12 performs the following steps: the noisy signal sequence is introduced into the FG CNN acceleration engine, a short-time Fourier transform layer is triggered to perform frequency domain conversion and frequency domain feature extraction, wherein the data form is a complex spectrum of 2xTx257, 2 is the channel number identifying real and imaginary numbers, T is the frame number, and 257 is the frequency point number; the frequency domain feature stream is converted to an encoder to expand the channel number and down-sample along the frame frequency dimension to determine a first encoding sequence, wherein the time dimension is quantitative, and the output data form is 64xTx4; the first encoding sequence is converted to a time convolution module to determine a time encoding sequence by performing time dimension dependency processing; the time encoding sequence is converted to a decoder to perform mirror decoding processing to determine a first decoding sequence; and the first decoding sequence is processed by an inverse short-time Fourier transform layer to perform time domain feature reconstruction to determine the de-noised voice signal.

[0112] The noise reduction execution unit 12 performs the following steps: a space-time code is introduced, wherein the space-time code is a timestamp of a voice collection time and a location code of a target conference port; and according to the space-time code, identification tracking under a noise reduction processing cycle is performed on the real-time conference voice.

[0113] The foregoing detailed description of the voice data noise reduction processing method of an intelligent conference enables those skilled in the art to clearly understand the voice data noise reduction processing method and system of an intelligent conference in the embodiment. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part is referred to the method part description.

[0114] The foregoing description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for noise reduction processing of voice data in intelligent conferencing, characterized in that, The method includes: Initialize the system by introducing a programmable gating system and building an FGCNN acceleration engine, build an FPGA board, and establish a data interface between the conference platform and the FPGA board. According to the data interface, the voice signal based on the conference platform is collected, imported into the FPGA board to perform analog-to-digital conversion and digital filtering, the FGCNN acceleration engine receives the time domain input of the noisy signal sequence, performs frequency domain noise reduction logic processing, outputs the denoised voice signal and transmits it back to the conference platform based on the data interface; The frequency domain noise reduction logic processing includes: The FGCNN acceleration engine uses forward and inverse short-time Fourier transform layers for frequency domain feature extraction and temporal feature reconstruction. It employs skip-connected encoders and decoders to perform gated frequency dimension compression and frequency dimension restoration, and uses residual-connected temporal convolutional modules to capture long-term dependencies.

2. The method for denoising voice data in intelligent conferencing as described in claim 1, characterized in that, The encoder and decoder are built using the gated convolution principle and employ a skip connection method, in which the information transmission method is dynamically adjusted through gating. The temporal convolution module is composed of multiple residual blocks stacked together. Each residual block contains an input convolution, an intermediate convolution, and an output convolution. The input convolution is used to increase the number of input channels, and the output convolution is used to restore the number of channels.

3. The method for denoising voice data in intelligent conferencing as described in claim 2, characterized in that, Before initializing the built-in FGCNN acceleration engine, the construction of the FGCNN acceleration engine includes: A hierarchical architecture for the FGCNN acceleration engine is constructed, which consists of a short-time Fourier transform layer, an encoder, a temporal convolution module, a decoder, and an inverse short-time Fourier transform layer, and supports pruning and expansion of channels within the layers. Obtain paired samples, wherein the paired samples are a set of sample pairs of noisy speech and clean speech; Based on the paired samples, the hierarchical architecture is trained under supervision until convergence, thus completing the construction of the FGCNN acceleration engine.

4. The method for denoising voice data in intelligent conferencing as described in claim 3, characterized in that, The training is supervised by iterative convergence conditions, wherein the optimization objective is to minimize the mean square error between the output signal and the target signal, wherein the output signal is the FGCNN output based on noisy speech in the paired samples, and the target signal is the clean speech in the paired samples.

5. The method for voice data noise reduction processing in intelligent conferencing as described in claim 4, characterized in that, A digital-to-analog conversion interface is introduced, wherein the digital-to-analog conversion interface is deployed at both ends of the FGCNN acceleration engine, wherein the front-end digital-to-analog conversion interface performs analog-to-digital conversion, and the back-end digital-to-analog conversion interface performs digital-to-analog conversion; A digital filter is deployed between the front-end digital-to-analog converter interface and the FGCNN acceleration engine to perform signal initialization filtering processing.

6. The method for denoising voice data in intelligent conferencing as described in claim 5, characterized in that, Before collecting voice signals from the conferencing platform, the following steps are required: Based on the data interface, the operation flow status of the conference platform is monitored, and monitoring requirements are generated. The monitoring requirements include collection trigger instructions, such as adding a new conference to enter the pre-collection stage and generating collection trigger instructions when the conference starts. According to the acquisition trigger instruction in the monitoring requirements, the system drives the acquisition of conference audio from the target conference, obtains real-time conference audio, and establishes a temporary data channel between the target conference and the data interface, wherein the temporary data channel is marked with a conference code.

7. The method for denoising voice data in intelligent conferencing as described in claim 6, characterized in that, The audio signal from the conference platform is acquired and imported into the FPGA board for analog-to-digital conversion and digital filtering, including: According to the data interface, the real-time conference audio is imported into the FPGA board; Trigger the digital-to-analog conversion interface to convert the real-time conference audio into a digital quantity, and determine the digital conference audio; The digital conference audio stream is transferred to the digital filter to determine the noisy signal sequence, wherein the digital filter converts the digital conference audio into a sequence with a dimension of 1×L, where L is the number of sampling points.

8. The method for denoising voice data in intelligent conferencing as described in claim 7, characterized in that, Perform frequency domain noise reduction logic processing, including: The noisy signal sequence is imported into the FGCNN acceleration engine, triggering the short-time Fourier transform layer to perform frequency domain transformation and frequency domain feature extraction. The data is in the form of a complex spectrum of 2×T×257, where 2 is the number of channels that identify real and imaginary numbers, T is the number of frames, and 257 is the number of frequency points. The frequency domain features are transferred to the encoder, the number of channels is amplified and downsampled along the frame frequency dimension to determine the first coding sequence, where the time dimension is quantitative and the output data format is 64×T×4. The first encoded sequence is transferred to the temporal convolution module, where temporal dependency processing is performed to determine the temporal encoded sequence. The time-encoded sequence is transferred to the decoder, where mirror inversion processing is performed to determine the first decoded sequence. The first decoded sequence is reconstructed using an inverse short-time Fourier transform layer to determine the denoised speech signal.

9. The method for denoising voice data in intelligent conferencing as described in claim 6, characterized in that, A spatiotemporal code is introduced, wherein the spatiotemporal code is the timestamp of the voice acquisition time and the location code of the target conference port; Based on the spatiotemporal code, perform noise reduction processing and identifier tracking on the real-time conference audio in a loop.

10. A voice data noise reduction processing system for intelligent conferencing, characterized in that, The system is used to perform a voice data noise reduction processing method for a smart conference according to any one of claims 1-9, the system comprising: The building unit is used to introduce programmable gating and initialize the built-in FGCNN acceleration engine, build the FPGA board, and establish a data interface between the conference platform and the FPGA board. The noise reduction execution unit is used to collect the voice signal based on the conference platform according to the data interface, import it into the FPGA board to perform analog-to-digital conversion and digital filtering, and the FGCNN acceleration engine receives the time domain input of the noisy signal sequence, performs frequency domain noise reduction logic processing, outputs the denoised voice signal and transmits it back to the conference platform based on the data interface. The frequency domain noise reduction logic processing includes: The FGCNN acceleration engine uses forward and inverse short-time Fourier transform layers for frequency domain feature extraction and temporal feature reconstruction. It employs skip-connected encoders and decoders to perform gated frequency dimension compression and frequency dimension restoration, and uses residual-connected temporal convolutional modules to capture long-term dependencies.

Images