Speech noise reduction method, system and storage medium based on pipeline architecture

Abstract

The present application discloses a speech noise reduction method, system and storage medium based on a pipeline architecture. The method includes: the upper computer sends the speech data to be processed to the lower computer, the lower computer receives the speech data to be processed sent by the upper computer, and performs noise reduction processing on the speech data to be processed according to a preset data processing accelerator to obtain the noise-reduced speech data. The data processing accelerator inputs the speech data to be processed into a preset neural network model; according to the characteristics of each layer of the neural network model, the number and computing tasks of each layer are dynamically allocated; each computing unit is controlled to perform parallel calculations on the data of each layer; if the data calculation of the current layer is completed, the calculation of the next neural network layer is controlled to obtain the noise-reduced speech data. This method performs neural network calculations on the lower computer in a pipeline architecture manner, which simplifies the design complexity, reduces resource utilization, reduces power consumption, and shortens the total time consumed in information processing.

Description

Technical Field

[0001] The present invention relates to the field of voice data processing technology, and in particular to a voice noise reduction method, system and storage medium based on a pipeline architecture. Background Art

[0002] With the continuous advancement of military modernization, the demand for intelligent control of weapons and equipment is increasing. Voice control technology can effectively improve the control efficiency of weapons and equipment. However, the battlefield environment is extremely complex, and noise interference in high-noise and harsh environments can seriously affect the accuracy of voice recognition, posing a challenge to voice control.

[0003] Furthermore, the speech noise reduction process requires high fidelity, real-time performance, and continuity, which cannot be met by CPUs alone. However, existing speech noise reduction recurrent neural network algorithms, when implemented on FPGAs (Field-Programmable Gate Arrays), are complex, resource-intensive, power-intensive, and time-consuming, severely impacting speech recognition efficiency. Summary of the Invention

[0004] The disclosed embodiments provide a pipeline-based speech noise reduction method, system, and storage medium. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is provided below. This summary is not intended to be a comprehensive review, identify key or important elements, or delineate the scope of protection for these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the detailed description that follows.

[0005] In a first aspect, an embodiment of the present application provides a speech noise reduction system based on a pipeline architecture, comprising:

[0006] The upper computer is used to send the voice data to be processed to the lower computer;

[0007] The lower computer is used to receive the voice data to be processed sent by the upper computer, and perform noise reduction processing on the voice data to be processed according to the preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture for calculation.

[0008] In an optional embodiment, the lower computer includes:

[0009] PCIE interface, AXI bus, AXI LITE bus, on-chip memory, on-chip cache, data processing accelerator;

[0010] The upper computer is connected to the lower computer via a PCIE interface, the PCIE interface is connected to a first AXI bus, the first AXI bus is connected to an on-chip memory, the on-chip memory is connected to an on-chip buffer via a second AXI bus, the on-chip buffer is connected to a data processing accelerator, the data processing accelerator is connected to an AXI LITE bus, and the AXI LITE bus is connected to the PCIE interface;

[0011] The host computer sends the voice data and parameter data to be processed to the data processing accelerator through the PCIE interface, the first AXI bus, the on-chip memory, the second AXI bus, and the on-chip buffer;

[0012] The host computer sends the instruction data to the data processing accelerator through the PCIE interface and AXI LITE bus.

[0013] In an optional embodiment, the data processing accelerator includes:

[0014] A control unit, used to control the computing unit to perform computing in a pipeline architecture;

[0015] A calculation unit, configured to calculate based on the speech data to be processed, the parameter data, and a preset neural network model to obtain the noise-reduced speech data;

[0016] The output unit is used to output the noise-reduced speech data.

[0017] In an optional embodiment, the control unit is configured to control the computing unit to perform computing in a pipeline architecture, including:

[0018] The control unit is specifically used to dynamically allocate the number and computing tasks of computing units in each layer according to the characteristics of each layer of the neural network model, and control each computing unit to perform parallel computing on the data of each layer according to the instruction data;

[0019] The control unit is also used to detect whether the data calculation of the current layer is completed. If the data calculation of the current layer is completed, the control unit will proceed to the calculation of the next neural network layer.

[0020] In an optional embodiment, it further includes:

[0021] The lower computer sends the noise-reduced voice data to the upper computer through the PCIE interface.

[0022] In an optional embodiment, the lower computer is a field programmable gate array.

[0023] In a second aspect, an embodiment of the present application provides a speech noise reduction method based on a pipeline architecture, comprising:

[0024] The upper computer sends the voice data to be processed to the lower computer;

[0025] The lower computer receives the voice data to be processed sent by the upper computer, and performs noise reduction processing on the voice data to be processed according to the preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture for calculation.

[0026] In an optional embodiment, performing noise reduction processing on the speech data to be processed according to a preset data processing accelerator includes:

[0027] The data processing accelerator inputs the speech data to be processed into a preset neural network model;

[0028] Dynamically allocate the number of computing units and computing tasks for each layer based on the characteristics of each layer of the neural network model;

[0029] Control each computing unit to perform parallel calculations on the data of each layer;

[0030] Check whether the data calculation of the current layer is completed. If the data calculation of the current layer is completed, control the calculation of the next neural network layer;

[0031] Get the noise-reduced speech data.

[0032] In an optional embodiment, the preset neural network model includes a speech activity detection module, which is composed of a first Dense layer, a first GRU layer and a second Dense layer connected in sequence, a noise spectrum estimation module, which is composed of a second GRU layer, and a spectrum subtraction module, which is composed of a third GRU layer and a third Dense layer.

[0033] In a third aspect, an embodiment of the present application provides a computer-readable medium on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement a speech noise reduction method based on a pipeline architecture provided in the above embodiment.

[0034] The technical solutions provided by the embodiments of the present application may have the following beneficial effects:

[0035] The speech noise reduction method based on pipeline architecture provided in the embodiment of the present application performs the calculation of the speech noise reduction neural network algorithm in the form of pipeline architecture on the lower computer, including dynamically allocating the number and calculation tasks of each layer of computing units according to the characteristics of each layer of the neural network model; controlling each computing unit to perform parallel calculations on the data of each layer, and if the data calculation of the current layer is completed, controlling the calculation of the next neural network layer to obtain the noise-reduced speech data. This method simplifies the design complexity, reduces resource utilization, reduces equipment power consumption, and greatly shortens the total time consumed for speech data processing, meeting the real-time requirements of speech data processing. This design can be applied to any algorithm architecture based on recurrent neural networks.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0038] Figure 1 is a structural diagram of a speech noise reduction system based on a pipeline architecture according to an exemplary embodiment;

[0039] Figure 2 is a schematic diagram of a pipeline architecture according to an exemplary embodiment;

[0040] Figure 3 This is a flow chart of a method for speech noise reduction based on a pipeline architecture according to an exemplary embodiment;

[0041] Figure 4 is a schematic diagram of a speech denoising neural network model according to an exemplary embodiment;

[0042] Figure 5 is a schematic diagram of a GRU network structure according to an exemplary embodiment;

[0043] Figure 6 It is a schematic diagram of a computer storage medium according to an exemplary embodiment. DETAILED DESCRIPTION

[0044] The following description and the drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

[0045] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0046] In the following description, unless otherwise indicated, identical numbers in different figures represent identical or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0047] Currently, the implementation process of speech noise reduction has high requirements for fidelity, real-time performance, and continuity. CPUs alone cannot meet these requirements. However, the existing speech noise reduction recurrent neural network algorithms, when implemented on FPGAs, are complex in design, consume excessive resources, consume high power, and are time-consuming, seriously impacting speech recognition efficiency. To address the problems existing in the existing technology, the present application provides a speech noise reduction system based on a pipeline architecture. This system simplifies design complexity, reduces resource utilization, reduces device power consumption, and significantly shortens the total time required for speech data processing, thus meeting the real-time requirements of speech data processing.

[0048] The following is a detailed introduction to the pipeline architecture-based speech noise reduction system provided in the embodiments of the present application in conjunction with the accompanying drawings.

[0049] See also Figure 1 ,The speech noise reduction system based on pipeline architecture specifically includes the ,following structure.

[0050] The upper computer is used to send the voice data to be processed to the lower computer.

[0051] In some embodiments of the present application, a speech noise reduction system based on a pipeline architecture includes a host computer, which may be, but is not limited to, a tablet computer, a laptop computer, a desktop computer, or a PC. A driver and control program runs on the host computer, which transmits control commands, speech data to be processed, and parameter data required for model calculation to a slave computer.

[0052] The lower computer is used to receive the voice data to be processed sent by the upper computer, and perform noise reduction processing on the voice data to be processed according to the preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture for calculation.

[0053] In some embodiments of the present application, the slave computer may be a field programmable gate array. Figure 1As shown, the lower computer includes: PCIE interface, AXI bus, AXI LITE bus, on-chip DDR (on-chip memory), on-chip cache, and data processing accelerator.

[0054] Among them, the upper computer is connected to the lower computer through the PCIE interface, the PCIE interface is connected to the first AXI bus, the first AXI bus is connected to the on-chip memory, the on-chip memory is connected to the on-chip cache through the second AXI bus, the on-chip cache is connected to the data processing accelerator, the data processing accelerator is connected to the AXI LITE bus, and the AXI LITE bus is connected to the PCIE interface.

[0055] Specifically, the driver and control program runs on the host computer, and the host computer sends the voice data to be processed and the parameter data required for model calculation to the data processing accelerator through the PCIE interface, the first AXI bus, the on-chip memory, the second AXI bus, and the on-chip buffer. The host computer sends the instruction data to the data processing accelerator through the PCIE interface and the AXI LITE bus. The host computer control commands mainly include: system reset command, audio bar reset command, and calculation module startup command, etc. The voice data to be processed enters the data processing accelerator in the FPGA from the host computer for processing, and the processed noise-reduced voice data is transmitted back to the host computer via the PCIE interface. From the perspective of the division of software and hardware tasks, the host computer program is mainly responsible for calling and controlling tasks, while the FPGA is mainly responsible for calculating large amounts of data.

[0056] In one possible implementation, a data processing accelerator performs noise reduction on the incoming speech data. A coarse-grained partitioning of computational tasks is employed, treating the computation of a neural network layer as a single task. The host computer then schedules and controls the FPGA to complete the computations. While coarse-grained task partitioning offers less flexibility, it provides higher computational performance.

[0057] Specifically, the data processing accelerator uses a pipeline architecture to perform calculations. Figure 2 is a schematic diagram of a pipeline architecture according to an exemplary embodiment. Figure 2 As shown, the data processing accelerator includes a control unit for controlling the computing unit to perform calculations using a pipeline architecture. It also includes multiple computing units for performing calculations based on the speech data to be processed, parameter data, and a preset neural network model to obtain noise-reduced speech data. It also includes an output unit for outputting the noise-reduced speech data.

[0058] Among them, the control unit is specifically used to dynamically allocate the number and computing tasks of computing units in each layer according to the characteristics of each layer of the neural network model, and control each computing unit to perform parallel calculations on the data of each layer according to the instruction data.

[0059] Specifically, the control unit can obtain the neural network model required for calculation, combine the common points of the network structure of different neural networks, modularize similar neural network sub-units, and perform hierarchical sequential execution in a reused form. Figure 2 As shown in the figure, the reused modules are computing unit 1, computing unit 2, and computing unit 3. These modules can be replaced by the memory gate module, forget gate module, update gate module, etc. in the recurrent neural network, and can also be replaced by the convolution module, activation module, pooling module, etc. in the convolutional neural network.

[0060] In one exemplary scenario, a neural network model includes a convolutional layer. When computing data for this layer, the control unit divides the data into three computational units: convolution calculations are performed in computational unit 1, activation function calculations are performed in computational unit 2, and pooling calculations are performed in computational unit 3. These three computational units are then controlled to perform computations in parallel, significantly improving data processing efficiency.

[0061] Furthermore, the control unit is also used to detect whether the data calculation of the current layer is completed. If the data calculation of the current layer is completed, the control unit controls the calculation of the next neural network layer.

[0062] Specifically, the control unit detects whether the calculation unit of the last step is completed. If the calculation is completed, it controls the distribution of the calculation data of the next network layer and dynamically adjusts the number of calculation units and calculation tasks according to the characteristics of the next network layer.

[0063] In this embodiment of the present application, the control unit can use a pipeline architecture to perform calculations. When calculating each layer of data, multiple computing units are divided into parallel calculations. After the calculation of the current layer of data is completed, the control unit controls the calculation of the next network layer until the data processing is completed. This approach greatly improves the efficiency of data processing and shortens the time required for the speech noise reduction system.

[0064] Optionally, the pipeline-based FPGA system provided in the embodiments of the present application can be applied to any algorithm architecture based on recurrent neural networks, thereby improving the operational efficiency of neural network models on FPGAs.

[0065] In the embodiment of the present application, multiple computing units input the speech data and parameter data to be processed into a preset neural network model for calculation to obtain the noise-reduced speech data. Figure 4 FIG is a schematic diagram of a preset neural network model according to an exemplary embodiment. Figure 4 As shown, the 42-dimensional audio feature data is input into the preset neural network model for calculation.

[0066] Specifically, the left half of the model includes a voice activity detection module, which includes the first Dense layer, the first GRU layer, and the second Dense layer connected in sequence, and is used to achieve a one-dimensional output of VAD. The right half includes a noise spectrum estimation module, which consists of the second GRU layer, and a spectrum subtraction module, which consists of the third GRU layer and the third Dense layer. The right half is used to achieve a 22-dimensional gain data output. Figure 4 As shown, the first Dense layer is a Dense tanh(24) layer, the first GRU layer is a GRU ReLU(24) layer, the second Dense layer is a Dense sig(1) layer, the second GRU layer is a GRU ReLU(48) layer, the third GRU layer is a GRU ReLU(96) layer, and the third Dense layer is a Dense sig(22) layer.

[0067] The left half is the network's VAD output module, which takes a 42-dimensional feature vector as input and outputs a 1-dimensional vector (vad). The VAD calculation is implemented by cascading the first Dense layer, the first GRU layer, and the second Dense layer.

[0068] In the first Dense layer calculation module, the input parameters include the first Dense layer weight parameter, the dimension is 24*42, the first Dense layer input feature vector dimension is 42*1, the two matrix multiplication and addition operations, the first Dense layer output feature vector dimension is 24*1.

[0069] First GRU layer: In the first GRU layer, the GRU module calculation is divided into four steps. Step 1: The input parameters include the first Dense layer output feature vector (dimension 24*1), the first intermediate state vector (dimension 24*1), and the first GRU layer weight parameters W1 (dimension 24*24), W2 (dimension 24*24). The first GRU layer weight parameters W1 and the first Dense layer output feature vector are matrix multiplied and added to obtain a vector of dimension 24*1. The first GRU layer weight parameters W2 and the first intermediate state vector (dimension 24*1) are matrix multiplied and added to obtain a vector of dimension 24*1. The two matrices are then added to obtain the first GRU layer reset gate vector r (dimension 24*1).

[0070] Step 2: The input parameters include the output feature vector of the first Dense layer 24*1, the first intermediate state vector state, dimension 24*1 and the first GRU layer weight parameter W3 dimension 24*24, W4 dimension 24*24, the first GRU layer weight parameter W3 and the first Dense layer output feature vector are matrix multiplied to obtain a vector with a dimension of 24*1; the first GRU layer weight parameter W4 and the first intermediate state vector state are matrix multiplied to obtain a vector with a dimension of 24*1; the two matrices are then added to obtain the first GRU layer update gating matrix Z dimension 24*1.

[0071] Step 3: The input parameters include the output feature vector 24*1 of the first Dense layer, the first intermediate state vector state24*1, and the weight parameter W5 of the first GRU layer with a dimension of 24*24 and a dimension of 24*24. The weight parameter W5 of the first GRU layer is matrix multiplied with the output feature vector of the first Dense layer to obtain a vector with a dimension of 24*1. The weight parameter W6 of the first GRU layer with a dimension of 24*24, the first intermediate state vector state with a dimension of 24*1, and the reset gating matrix r of the first GRU layer with a dimension of 24*1 are matrix multiplied to obtain a vector with a dimension of 24*1. The two matrices are then added together to obtain the result state variable h1 of the first GRU layer with a dimension of 24*1.

[0072] Step 4: Input parameters include the first GRU layer update gating matrix Z (24*1 dimension), the first intermediate state vector state (24*24 dimension), and the first GRU layer result state variable h1 (24*1 dimension). Matrix multiplication is performed on the first GRU layer update gating matrix Z and the first intermediate state vector state to obtain a 24*1 vector. Matrix multiplication is performed on the first GRU layer update gating matrix Z and the first GRU layer result state variable matrix h1 to obtain a 24*1 vector. These two matrices are then added together to obtain the first GRU layer result state matrix h (24*1 dimension). If the dimensions are insufficient during the matrix multiplication, they are automatically padded with 1 during the calculation.

[0073] Second Dense layer: In the second Dense layer calculation module, the input parameters include the first GRU layer result state matrix h with a dimension of 24*1 and the second Dense layer weight parameter with a dimension of 24*1. The two matrices are multiplied and added, and the dimension of the second Dense layer output feature vector is 1*1.

[0074] Right Half of the Network: The right half of the network consists of the 22-dimensional band-gain output module, which takes a 90-dimensional feature vector as input and outputs a 22-dimensional band-gain output. The band-gain output module is calculated by cascading the second GRU layer, the third GRU layer, and the third Dense layer.

[0075] Second GRU layer: In the second GRU layer, the GRU module calculation is divided into four steps. The first step: the input parameters include the second GRU layer input feature vector dimension of 90*1 (the first Dense layer input feature vector dimension of 42*1, the first Dense layer output feature vector dimension of 24*1, and the first GRU layer result state matrix h dimension of 24*1), the second intermediate state vector state dimension of 48*1 and the second GRU layer weight parameter W1 dimension of 48*90, W2 dimension of 48*90, the second GRU layer weight parameter W1 and the second GRU layer input feature vector perform matrix multiplication and addition operation to obtain a vector of dimension 48*1; the second GRU layer weight parameter W2 and the second intermediate state vector state perform matrix multiplication and addition operation to obtain a vector of dimension 48*1; the two matrices are then added to obtain the second GRU layer reset gate vector r dimension of 48*1.

[0076] Step 2: The input parameters include the second GRU layer input feature vector dimension of 90*1, the second intermediate state vector state dimension of 48*1, and the second GRU layer weight parameter W3 dimension of 48*90, W4 dimension of 48*90, the second GRU layer weight parameter W3 and the second GRU layer input feature vector perform matrix multiplication and addition operation to obtain a vector of dimension 48*1; the second GRU layer weight parameter W4 and the second intermediate state vector state perform matrix multiplication and addition operation to obtain a vector of dimension 48*1; the two matrices are then added together to obtain the second GRU layer update gating matrix Z dimension of 48*1.

[0077] Step 3: The input parameters include the second GRU layer input feature vector with a dimension of 90*1, the second intermediate state vector state with a dimension of 48*1, and the second GRU layer weight parameter W5 with a dimension of 48*90 and a dimension of 48*90. The second GRU layer weight parameter W5 is matrix multiplied with the second GRU layer input feature vector to obtain a vector with a dimension of 48*1; the second GRU layer weight parameter W6, the second intermediate state vector state with a dimension of 48*1, and the second GRU layer reset gate matrix r. The three matrices are matrix multiplied to obtain a vector with a dimension of 48*1; the two matrices are then added together to obtain the second GRU layer result state variable h1 with a dimension of 48*1.

[0078] Step 4: The input parameters include the second GRU layer update gating matrix Z with a dimension of 48*1, the second intermediate state vector state with a dimension of 48*48, and the second GRU layer result state variable h1 with a dimension of 48*1. The second GRU layer updates the gating matrix Z and the second intermediate state vector state for matrix multiplication to obtain a vector with a dimension of 48*1; the second GRU layer updates the gating matrix Z and the second GRU layer result state variable matrix h1 for matrix multiplication to obtain a vector with a dimension of 48*1; the two matrices are then added together to obtain the second GRU layer result state matrix h with a dimension of 48*1.

[0079] The third GRU layer: In the third GRU layer, the GRU module calculation is divided into four steps. The first step: the input parameters include the third GRU layer input feature vector dimension of 114*1 (the first Dense layer input feature vector dimension of 42*1, the first GRU layer result state matrix h dimension of 24*1, the second GRU layer result state matrix h dimension of 48*1), the third intermediate state vector state dimension of 96*1 and the third GRU layer weight parameter W1 dimension of 96*114, W2 dimension of 96*114, the third GRU layer weight parameter W1 and the third GRU layer input feature vector perform matrix multiplication and addition operation to obtain a vector of dimension 96*1; the third GRU layer weight parameter W2 and the second intermediate state vector state perform matrix multiplication and addition operation to obtain a vector of dimension 96*1; the two matrices are then added to obtain the third GRU layer reset gate vector r dimension of 96*1.

[0080] Step 2: The input parameters include the third GRU layer input feature vector dimension of 114*1, the third intermediate state vector state dimension of 96*1 and the third GRU layer weight parameter W3 dimension of 96*114, W4 dimension of 96*114. The second GRU layer weight parameter W3 and the third GRU layer input feature vector are matrix multiplied and added to obtain a vector with a dimension of 96*1; the second GRU layer weight parameter W4 and the third intermediate state vector state are matrix multiplied and added to obtain a vector with a dimension of 96*1; the two matrices are then added to obtain the third GRU layer update gating matrix Z dimension of 96*1.

[0081] Step 3: The input parameters include the third GRU layer input feature vector with a dimension of 114*1, the third intermediate state vector state with a dimension of 96*1 and the GRU ReLU (96) weight parameter W5 with a dimension of 96*114 and a dimension of 96*114. The third GRU layer weight parameter W5 is matrix multiplied with the third GRU layer input feature vector to obtain a vector with a dimension of 96*1; the third GRU layer weight parameter W6, the third intermediate state vector state and the third GRU layer reset gating matrix r are matrix multiplied to obtain a vector with a dimension of 96*1; the two matrices are then added together to obtain the third GRU layer result state variable h1 with a dimension of 96*1.

[0082] Step 4: The input parameters include the third GRU layer update gating matrix Z with a dimension of 96*1, the third intermediate state vector state with a dimension of 96*96, and the third GRU layer result state variable h1 with a dimension of 96*1. The third GRU layer updates the gating matrix Z and the third intermediate state vector state for matrix multiplication to obtain a vector with a dimension of 96*1; the third GRU layer updates the gating matrix Z and the third GRU layer result state variable matrix h1 for matrix multiplication to obtain a vector with a dimension of 96*1; the two matrices are then added together to obtain the third GRU layer result state matrix h with a dimension of 96*1.

[0083] Third Dense layer: In the third Dense layer calculation module, the input parameters include the third Dense layer weight parameter dimension of 22*96 and the third GRU layer result state matrix h dimension of 96*1. The two matrices are multiplied and added, and the dimension of the third Dense layer output feature vector is 22*1.

[0084] The computing unit calculates the speech data and parameter data to be processed by inputting them into a preset neural network model to obtain the denoised speech data. The preset neural network model in the embodiment of the present application mainly includes a GRU network. Compared with general sequence signal processing algorithms, such as the LSTM algorithm, the use of the GRU algorithm can achieve quite good results and is easier to train, which can greatly improve the training efficiency. Therefore, the embodiment of the present application intends to use the GRU network to achieve speech denoising in principle.

[0085] Figure 5 FIG. 1 is a schematic diagram of a GRU network structure according to an exemplary embodiment. Figure 5 As shown, x t Indicates the current input, h t-1 Represents the hidden state passed down from the previous node, combined with x t and h t-1 , GRU will get the output y of the current hidden node t and the hidden state h passed to the next nodet r represents the gate that controls reset, z represents the gate that controls update, and ⊙ represents the multiplication of corresponding elements in the matrix, so the two multiplication matrices are required to be of the same type. Represents the addition of two matrices.

[0086] The pipeline-based speech noise reduction system provided in the present embodiment utilizes DSP multiplexing technology, whereby one DSP performs two multiplication operations. The DSP currently integrated into the chip is the DSP48_E2, which supports a maximum multiplication bit width of 26 bits for port A and 18 bits for port B. This multiplier can perform two multiplication operations within a single cycle, conserving FPGA internal resources and reducing FPGA internal resource utilization, thereby reducing device power consumption.

[0087] Streaming data transmission technology allows multiple FPGA modules to work together simultaneously, increasing FPGA efficiency and avoiding frequent data reads and writes. This internal data streaming structure effectively increases data reusability, reduces resource usage, and increases operating speed.

[0088] The advantage of adopting a hierarchical structure is that the size of internal registers and distributed memory can be defined through upper-level configuration. For different neural network structures or different input sizes of the same neural network structure, the number of model parameters and the size of intermediate results are different. Through the hierarchical structure and scalable design, the internal processing power of the processor can be configured according to task requirements. In order to save resource consumption, it is necessary to perform detailed calculations on the parameter sizes in the network structure. Based on this, resources are allocated and just enough space is used to store and temporarily store data. This ensures that the neural network processor can operate normally while saving resources as much as possible to reduce power consumption, indirectly reducing resource usage.

[0089] The system uses a pipelined architecture to calculate the speech noise reduction neural network algorithm on the lower computer to obtain the noise-reduced speech data. This simplifies design complexity, reduces resource utilization, reduces device power consumption, and significantly shortens the total processing time of speech data, meeting the real-time requirements of speech data processing.

[0090] The present disclosure also provides a method for reducing speech noise based on a pipeline architecture. Figure 3 As shown, the method includes:

[0091] S301 The upper computer sends the voice data to be processed to the lower computer.

[0092] In step S302, the lower computer receives the voice data to be processed sent by the upper computer, and performs noise reduction processing on the voice data to be processed according to a preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture for calculation.

[0093] In an optional embodiment, noise reduction processing is performed on the voice data to be processed according to a preset data processing accelerator, including: the data processing accelerator inputs the voice data to be processed into a preset neural network model; dynamically allocates the number and computing tasks of computing units in each layer according to the characteristics of each layer of the neural network model; controls each computing unit to perform parallel computing on the data of each layer; detects whether the calculation of the data of the current layer is completed, and if the calculation of the data of the current layer is completed, controls the calculation of the next neural network layer; and obtains the noise-reduced voice data.

[0094] In an optional embodiment, the left half of the preset neural network model includes a voice activity detection module, which comprises a first Dense layer, a first GRU layer, and a second Dense layer connected in sequence, for implementing a one-dimensional VAD output. The right half includes a noise spectrum estimation module, which comprises a second GRU layer, and a spectral subtraction module, which comprises a third GRU layer and a third Dense layer. The right half is used to implement a 22-dimensional gain data output.

[0095] It should be noted that the speech noise reduction system based on the pipeline architecture provided in the above embodiment only uses the division of the above-mentioned functional modules as an example when executing the speech noise reduction method based on the pipeline architecture. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the speech noise reduction method based on the pipeline architecture provided in the above embodiment and the speech noise reduction system embodiment based on the pipeline architecture belong to the same concept. The implementation process thereof is detailed in the system embodiment and will not be repeated here.

[0096] The present application also provides a computer-readable storage medium corresponding to the pipeline-based speech noise reduction method provided in the above embodiment. Figure 6 The computer-readable storage medium shown is a CD 600 on which a computer program (i.e., a program product) is stored. When the computer program is run by the processor, it will execute the speech noise reduction method based on the pipeline architecture provided by any of the aforementioned embodiments.

[0097] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0098] The computer-readable storage medium provided in the above-mentioned embodiment of the present application and the speech noise reduction method based on the pipeline architecture provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0099] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A speech noise reduction system based on a pipeline architecture, characterized in that: include: The upper computer is used to send the voice data to be processed to the lower computer; The lower computer is used to receive the voice data to be processed sent by the upper computer, and perform noise reduction processing on the voice data to be processed according to a preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture to perform calculations; The data processing accelerator includes a control unit for controlling the computing unit to perform calculations in a pipeline architecture; includes multiple computing units for performing calculations based on the voice data to be processed, parameter data and a preset neural network model to obtain noise-reduced voice data; includes an output unit for outputting the noise-reduced voice data; wherein the control unit is specifically used to dynamically allocate the number and computing tasks of computing units in each layer according to the characteristics of each layer of the neural network model, and control each computing unit to perform parallel calculations on the data of each layer according to the instruction data; the control unit detects whether the calculation of the computing unit in the last step is completed. If the calculation is completed, it controls the distribution of the calculation data of the next network layer, and dynamically adjusts the number and computing tasks of the computing units according to the characteristics of the next network layer.

2. The system according to claim 1, wherein: The lower computer includes: PCIE interface, AXI bus, AXI LITE bus, on-chip memory, on-chip cache, data processing accelerator; The upper computer is connected to the lower computer via a PCIE interface, the PCIE interface is connected to a first AXI bus, the first AXI bus is connected to an on-chip memory, the on-chip memory is connected to an on-chip buffer via a second AXI bus, the on-chip buffer is connected to a data processing accelerator, the data processing accelerator is connected to an AXI LITE bus, and the AXI LITE bus is connected to the PCIE interface; The host computer sends the voice data and parameter data to be processed to the data processing accelerator through the PCIE interface, the first AXI bus, the on-chip memory, the second AXI bus, and the on-chip buffer; The host computer sends the instruction data to the data processing accelerator through the PCIE interface and the AXI LITE bus.

3. The system according to claim 1, wherein: Also includes: The lower computer sends the noise-reduced voice data to the upper computer through the PCIE interface.

4. The system according to any one of claims 1 to 3, characterized in that: The lower computer is a field programmable gate array.

5. A speech noise reduction method based on a pipeline architecture, characterized in that: include: The upper computer sends the voice data to be processed to the lower computer; The lower computer receives the voice data to be processed sent by the upper computer, and performs noise reduction processing on the voice data to be processed according to a preset data processing accelerator to obtain the noise-reduced voice data, wherein the data processing accelerator adopts a pipeline architecture for calculation; Among them, noise reduction processing is performed on the voice data to be processed according to a preset data processing accelerator, including: the data processing accelerator inputs the voice data to be processed into a preset neural network model; according to the characteristics of each layer of the neural network model, the number and computing tasks of each layer are dynamically allocated; each computing unit is controlled to perform parallel computing on the data of each layer; it is detected whether the data of the current layer is calculated. If the data of the current layer is calculated, the calculation of the next neural network layer is controlled; and according to the characteristics of the next network layer, the number and computing tasks of the computing units are dynamically adjusted to obtain the noise-reduced voice data.

6. The method according to claim 5, characterized in that The preset neural network model includes a speech activity detection module, which is composed of a first Dense layer, a first GRU layer and a second Dense layer connected in sequence, a noise spectrum estimation module, which is composed of a second GRU layer, and a spectrum subtraction module, which is composed of a third GRU layer and a third Dense layer.

7. A computer-readable medium, characterized in that Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement a speech noise reduction method based on a pipeline architecture as described in any one of claims 5 to 6.

Images