Audio processing method and device, equipment, medium and product

Abstract

The embodiment of the invention discloses an audio processing method and device, equipment, a medium and a product, which can be applied to the technical field of data processing. The method comprises the following steps: performing feature extraction on time-frequency spectrum data of first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix comprises channel features, time features and frequency band features; obtaining a first channel weight matrix and a first frequency band weight matrix based on the channel characteristics, the time characteristics and the frequency band characteristics; performing matrix product on the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix; performing Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix; and denoising the first audio data based on the second feature matrix to obtain second audio data corresponding to the first audio data. By adopting the embodiment of the invention, the audio denoising quality of the audio data can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to audio processing methods, devices, equipment, media and products. Background Art

[0002] At present, in the audio data processing scenario, the noise in the audio data can be denoised through methods such as adaptive filtering, spectral subtraction, Wiener filtering, etc., so as to find the audio of the audio data.

[0003] However, the inventors have found in practice that the denoising effect of the existing audio processing method is affected by the signal-to-noise ratio of the noisy audio data. For example, for a long segment of audio data (for example, language data X), once the signal-to-noise ratio of the audio data X is small, a large amount of noise will exist in this segment of audio data. Therefore, in the process of performing audio denoising on this segment of audio data using the existing audio processing method, it will be difficult to effectively denoise the large amount of noise appearing in the audio data, so that the audio data after denoising will still carry some noise, thereby reducing the audio denoising quality of the audio data. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, apparatus, device, medium and product, which can extract a channel weight matrix and a frequency band weight matrix during the processing of audio data, so as to determine the weights corresponding to the channels and frequency bands in the time characteristics, thereby improving the audio denoising quality of the audio data.

[0005] On the one hand, an embodiment of the present application provides an audio processing method, the method comprising:

[0006] Extracting features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features, and frequency band features;

[0007] Based on the channel characteristics, the time characteristics and the frequency band characteristics, a first channel weight matrix and a first frequency band weight matrix are obtained;

[0008] Perform matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix;

[0009] Perform Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix;

[0010] The first audio data is denoised based on the second characteristic matrix to obtain second audio data corresponding to the first audio data.

[0011] On the one hand, an embodiment of the present application provides an audio processing device, the device comprising:

[0012] A first feature extraction module is used to extract features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features and frequency band features;

[0013] A weight extraction module, used to obtain a first channel weight matrix and a first frequency band weight matrix based on channel features, time features and frequency band features;

[0014] A weight fusion module, used for performing matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix;

[0015] A feature fusion module, used for performing Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix;

[0016] The audio denoising module is used to denoise the first audio data based on the second feature matrix to obtain second audio data corresponding to the first audio data.

[0017] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program is suitable for being loaded and executed by a processor so that a computer device having the processor executes the method provided by the embodiment of the present application.

[0018] In one aspect, the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the present application.

[0019] In an embodiment of the present application, when processing audio data, feature extraction can be performed on the time-frequency spectrum data of the audio data to obtain a first feature matrix, and then based on the channel features, time features and frequency band features in the first feature matrix, a first channel weight matrix and a first frequency band weight matrix can be determined. The first channel weight matrix and the first frequency band weight matrix can focus on the feature differences between channels and frequency bands in time features, and then matrix product can be performed based on the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix, which can characterize the weight matrix obtained by fusion of frequency band feature weights and channel feature weights in time features, and then the first feature matrix and the time channel weight matrix can be Hadamard-multiplied to add the time channel weight matrix to the first feature matrix to obtain a second feature matrix, and then the second feature matrix can focus on the feature weight differences in time features, thereby characterizing the potential noise laws in time features, and audio data of various time lengths can be unaffected by the signal-to-noise ratio on the denoising of the audio data, and then denoising can be performed based on the second feature matrix, and noise information in the audio data can be removed, thereby improving the audio denoising quality of the audio data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 It is a structural diagram of a data processing system provided in an embodiment of the present application;

[0022] Figure 2 is a scene schematic diagram of an audio processing method provided in an embodiment of the present application;

[0023] Figure 3 It is a flowchart of an audio processing method provided in an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of a feature extraction process provided in an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of a channel weight extraction process provided by an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of a frequency band weight extraction process provided by an embodiment of the present application;

[0027] Figure 7It is a schematic diagram of the effect of a method for determining a second characteristic matrix provided in an embodiment of the present application;

[0028] Figure 8 is a flowchart of another audio processing method provided in an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of a convolutional modulation network provided in an embodiment of the present application;

[0030] Fig.10 is a schematic diagram of an audio feature extraction provided in an embodiment of the present application;

[0031] Fig.11 is a schematic diagram of an audio processing process provided by an embodiment of the present application;

[0032] Fig.12 is a schematic diagram of a model structure of an audio processing model provided in an embodiment of the present application;

[0033] Fig.13 is a structural schematic diagram of an audio processing device provided in an embodiment of the present application;

[0034] Fig.14 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] See also Figure 1 , Figure 1 Schematic diagram of a data processing system provided in an embodiment of the present application. Figure 1 As shown, the data processing system may include terminal devices (such as device 11a, device 12a, device 13a) and server 100a. It can be understood that Figure 1 The number of terminal devices and servers in the embodiment is only for illustration, and any number of terminal devices and servers may be provided according to the implementation requirements. The terminal devices (such as the device 11a, the device 12a, and the device 13a) may communicate with the server through a network (i.e., a medium providing a communication link through a wired or wireless communication link or an optical fiber cable, etc.) to transmit data.

[0037] It is understandable that a client may be running on the terminal device (such as device 12a), and the client may be a program that provides local services to users (also called business objects, operation objects). Server 100a may be a server corresponding to the client, and a program for providing resources, service data and other services to the client may be running in the server 100a. It is understandable that the client running on the terminal device may also be called an application client, a business client, and the like. For example, the client running on the terminal device may be a client for providing audio denoising services and audio processing services, and then user A may determine the audio data X1 to be subjected to audio denoising through the business client on the terminal device (such as device 11a), and then perform audio denoising on the audio data X1 in the terminal device, or send the audio data X1 to the server, and the server performs audio denoising on the audio data X1.

[0038] It is understandable that the terminal device (such as device 11a) may include but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, smart speakers, smart home appliances, etc., and there is no limitation here. The server 100a may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, and there is no limitation here.

[0039] See also Figure 2 , Figure 2 Schematic diagram of a scenario of an audio processing method provided in an embodiment of the present application. Figure 2 As shown, firstly, the first audio data to be processed can be obtained (such as Figure 2 The first audio data may be collected in real time or stored in advance in a computer device, which is not limited here. Further, the time-frequency spectrum data of the first audio data may be obtained (such as Figure 2 As shown in 202a in the figure), the time-frequency spectrum data may be obtained by performing a short-time Fourier transform on the first audio data. Specifically, the first audio data may be framed, windowed, and Fourier transformed to obtain the time-frequency spectrum data of the first audio data. The time-frequency spectrum data may include real data and imaginary data. The real data may include data on the real part of the frequency domain signal of T voice frames, and the imaginary data may include data on the imaginary part of the frequency domain signal of T voice frames. The frequency domain signal of each voice frame in the T voice frames may include D frequency band points. In other words, the characteristic dimension of the time-frequency spectrum data is 2×T×D, where 2 represents the real channel corresponding to the real data and the imaginary channel corresponding to the imaginary data.

[0040] Furthermore, feature extraction can be performed on the time-frequency spectrum data to obtain a first feature matrix (such as Figure 2 203a). The first feature matrix includes time features, channel features and frequency band features, the time features are features on the time dimension, the channel features are features on the channel dimension, and the frequency band features can be features on the frequency band dimension. Feature extraction here can be convolution processing, and the dimension of the obtained first feature matrix can be C×H×W. C is the size of the channel dimension, H is the size of the time dimension, and W is the size of the frequency band dimension. The time dimension can be a dimension associated with the T speech frames of the time-frequency spectrum data, which can be the row dimension of the first feature matrix. The channel dimension represents the dimension associated with the real channel and the imaginary channel of the time-frequency spectrum data, which can be the depth dimension of the first feature matrix. W can be a dimension in the first feature matrix associated with the D frequency band points in the time-frequency spectrum, which can be the column dimension of the first feature matrix.

[0041] Furthermore, a first channel weight matrix (such as Figure 2 204a) and the first frequency band weight matrix (as shown in Figure 2 205a). The first channel weight matrix may be a matrix composed of the weight values ​​of each channel feature weight in the time dimension, that is, the first channel weight matrix may include the weight values ​​of the channel feature weights for each channel dimension in H time dimensions, and the dimension of the first channel weight matrix is ​​H×C×1. The first frequency band weight matrix may be a matrix composed of the weight values ​​corresponding to each frequency band feature weight in the time feature, that is, the first frequency band weight matrix may include the weight values ​​of the frequency band feature weights for each frequency band dimension in H time dimensions, and the dimension of the first frequency band weight matrix is ​​H×W×1.

[0042] Furthermore, the first channel weight matrix (such as Figure 2 204a) and the first frequency band weight matrix (as shown in Figure 2 205a) to perform matrix product to obtain the time channel weight matrix (as shown in Figure 2 206a). The dimension of the time channel weight matrix is ​​H×C×W.

[0043] Furthermore, the time channel weight matrix (such as Figure 2 206a) and the first characteristic matrix to perform Hadamard product to obtain the second characteristic matrix (as shown in Figure 2207a). It should be understood that performing Hadamard product on two matrices requires that the dimensions of the two matrices are consistent. Therefore, the time channel weight matrix can be transposed so that the dimension of the time channel weight matrix is ​​consistent with the dimension of the first feature matrix, thereby performing Hadamard product on the transposed time channel weight matrix and the first feature matrix to obtain the second feature matrix.

[0044] Furthermore, the first audio data can be denoised based on the second feature matrix to obtain second audio data corresponding to the first audio data (eg Figure 2 It should be understood that the second audio data may be audio data obtained after audio processing of the first audio data.

[0045] In an embodiment of the present application, when processing audio data, feature extraction can be performed on the time-frequency spectrum data of the audio data to obtain a first feature matrix, and then based on the channel features, time features and frequency band features in the first feature matrix, a first channel weight matrix and a first frequency band weight matrix can be determined. The first channel weight matrix and the first frequency band weight matrix can focus on the feature differences of channels and frequency bands in time features, and then matrix product can be performed based on the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix, which can characterize the weights of the fusion of frequency bands and channels in time features, and then the first feature matrix and the time channel weight matrix can be Hadamard-multiplied to add the time channel weight matrix to the first feature matrix to obtain a second feature matrix. The second feature matrix can be a feature with a potential noise law focusing on time features, which is not affected by the signal-to-noise ratio on the denoising of the audio data, and then denoising can be performed based on the second feature matrix, which can remove noise information in the audio data and improve the audio denoising quality of the audio data.

[0046] It should be understood that the embodiments of the present application can be applied to a variety of application scenarios, for example, it can be used in a speech-to-text scenario, and then the audio data can be denoised by the embodiments of the present application, and then the corresponding text can be converted based on the denoised audio data. For another example, the embodiments of the present application can be applied to voice call scenarios, intelligent voice question-and-answer scenarios, etc., which are not limited here.

[0047] It is understandable that the embodiments of the present application can be applied to a computer device, which can be the above-mentioned Figure 1The server 100a in the example, or other data processing devices for data processing, can obtain the first audio data from the terminal device, and then perform audio processing on the first audio data through the audio processing model in the server to obtain the corresponding second audio data, and then return the second audio data to the terminal device, or, if the second audio data can be subsequently processed (such as extracting text from the second audio data), the server can return the subsequent processing result based on the second audio data to the terminal device. Alternatively, the computer device can be the above Figure 1 If a terminal device (such as device 12a) is used in the process, the terminal device can obtain the first audio data, and then the terminal device performs audio denoising on the first audio data based on the above-mentioned audio processing scheme. If the second audio data can be subsequently processed (such as extracting text from the second audio data), the second audio data can be subsequently processed by the terminal device or the server, which is not limited here.

[0048] It should be noted that before collecting relevant data of users and during the process of collecting relevant data of users, this application can display a prompt interface, pop-up window or output voice prompt information, and the prompt interface, pop-up window or voice prompt information is used to prompt the user that its relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are collected with the consent and authorization of the user, and the collection, use and processing of relevant user data need to comply with the relevant laws, regulations and standards of the relevant regions.

[0049] It is understandable that the above cloud server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The above terminal devices include but are not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, smart speakers, smart home appliances, etc.

[0050] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0051] For further information, see Figure 3 , Figure 3 1 is a flowchart of an audio processing method provided in an embodiment of the present application. The method can be executed by a computer device, for example, the computer device can be the server 100a mentioned above. The method can at least include the following steps S101 to S105.

[0052] S101. Extract features from time-frequency spectrum data of first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features, and frequency band features.

[0053] Among them, the first audio data can be audio data to be processed, and can also be called a first audio signal. The first audio data can be an audio signal collected in real time by an audio acquisition device, such as audio data collected in real time by some intelligent dialogue robots, so that after performing audio denoising on the audio data collected in real time, the denoised audio is processed (such as making an intelligent reply); or, the first audio data can also be audio data stored in a terminal device, such as recording data stored in a terminal device. This application does not limit the method for obtaining the first audio data.

[0054] Among them, the time-frequency spectrum data of the first audio data can be data used to characterize the characteristics of the first audio data in the time domain and the frequency domain. The time-frequency spectrum data can be specifically obtained based on the time-frequency transformation of T voice frames in the first audio data; T is a positive integer. It can be understood that since the audio signal (such as the first audio data) is a random signal and has periodicity and stability in a short time (10-30ms), it is possible to obtain T voice frames after framing and windowing the audio data, and then perform a discrete Fourier transform (Discrete Fourier Transform, DFT) on each voice frame to obtain the frequency domain information corresponding to each voice frame, and then determine the time-frequency spectrum of the first audio data (also referred to as the time-frequency spectrum) based on the frequency domain information of each voice frame. The above process is the process of performing a short-time Fourier transform (STFT) on the first audio data. It should be understood that the time-frequency spectrum can be restored to obtain the audio data in the time domain through the inverse transform of the short-time Fourier transform (i.e., ISTFT, inverse short-time Fourier transform).

[0055] It should be understood that the T voice frames are obtained by framing and windowing the first audio data. Wherein, when the first audio data is framed, the frame processing can be performed based on the preset frame length and frame shift, so as to obtain T frame time domain signals corresponding to the first audio data (the frame time domain signal is the sub-time domain signal obtained by framing the time domain signal indicated by the first audio data), wherein the frame length is the time length of each frame time domain signal (the frame lengths of each frame time domain signal are the same), and the frame shift is the time length of the overlap of two adjacent frame time domain signals, for example, the time length of the frame time domain signal of the kth frame is t1-t2, and the time length of the k+1th frame time domain signal is t3-t4, wherein t1 is earlier than t3, t3 is earlier than t2, t4 is later than t2, and the time lengths of t1-t2 and t3-t4 are the same, that is, both are equal to the frame length, and t3-t2 can be the frame shift. Furthermore, after determining T frame time domain signals, each frame time domain signal can be windowed. For example, each frame time domain signal can be windowed through a window function to obtain T windowed signals corresponding to the T frame time domain signals, so that the T windowed signals can be determined as T speech frames, wherein one frame time domain signal is used to determine one windowed signal, and one windowed signal is used as a speech frame. The length of the window function should be consistent with the length of the frame. The window function can use a Hanning window, a rectangular window, a triangular window, a Hamming window, a Gaussian window, etc., which is not limited here.

[0056] It should be understood that the time-frequency transformation processing of the T voice frames in the first audio data can be used to indicate that the T voice frames are converted from the time domain to the frequency domain, that is, each voice frame can be converted to obtain a corresponding frequency domain signal (also referred to as frequency domain information). For example, the time-frequency transformation processing can be a discrete Fourier transform processing (Fourier transform can be used to convert an audio signal from the time domain to the frequency domain), and then each voice frame can be converted into a corresponding frame frequency domain signal (frame frequency domain signal is a frequency domain signal converted by the windowed signal of a voice frame) through discrete Fourier transform processing, and the frame frequency domain signal of each voice frame can correspond to data on D frequency band points (also referred to as sampling frequency, frequency point), D is a positive integer, and the frequency band point here can be a discrete frequency sampling point on any frame frequency domain signal, or it can be a dimension called discrete Fourier transform, for example, a frame frequency domain signal can include 256 frequency band points, that is, D=256.

[0057] Furthermore, after determining the frame frequency domain signal corresponding to each speech frame, the frame frequency domain signals of the T speech frames may be arranged in the time sequence of each speech frame to obtain the time-frequency spectrum data corresponding to the first audio data.

[0058] It can be understood that each frame frequency domain signal obtained by Fourier transform of each speech frame is a complex number, which is composed of a real part and an imaginary part. Then, arranging the frame frequency domain signals of T speech frames according to the time sequence of each speech frame to obtain the time-frequency spectrum data corresponding to the first audio data can include: arranging the real part information of the frame frequency domain signals corresponding to the T speech frames according to the time sequence of each speech frame to obtain the real part data; further, arranging the imaginary part information of the frame frequency domain signals corresponding to the T speech frames according to the time sequence of each speech frame to obtain the imaginary part data; and then determining the time-frequency spectrum data of the first audio data based on the real part data and the imaginary part data. It should be understood that the real part of any frame frequency domain signal represents the amplitude of the corresponding frame timing signal, and the imaginary part represents the phase information of the corresponding frame timing signal.

[0059] It should be understood that the feature dimensions of the real features corresponding to the real data and the imaginary features corresponding to the imaginary data are both T×D, where D is a positive integer and is used to characterize the number of frequency band points corresponding to the time-frequency spectrum. Here, D is determined according to the dimension of the above-mentioned discrete Fourier transform. Then, the dimension of the time-frequency spectrum data is 2×T×D, where 2 represents the real channel corresponding to the real data and the imaginary channel corresponding to the imaginary data. In other words, the real data can represent the real part information of T speech frames at D frequency band points, and the imaginary data can represent the imaginary part information of T speech frames at D frequency band points.

[0060] The first feature matrix may be a matrix obtained by extracting features from the time-frequency spectrum data. It is understandable that the feature extraction here may be performed by convolution processing on the time-frequency spectrum data through a convolution layer, and the first feature matrix may also be called a convolution feature of the time-frequency spectrum data. The convolution layer may be composed of a two-dimensional convolution.

[0061] It should be understood that the first feature matrix may include channel features, time features and frequency band features. Among them, channel features may refer to features on the channel dimension associated with the real channel and imaginary channel of the time-spectral data. Time features may refer to features on the time dimension associated with the speech frame of the time-spectral data. Frequency band features may refer to features on the frequency band dimension associated with the frequency band points of the time-spectral data.

[0062] It can be understood that the first feature matrix can be a three-dimensional matrix, and the three dimensions of the three-dimensional matrix can be the row dimension (i.e., the first dimension), the column dimension (i.e., the second dimension) and the depth dimension (i.e., the third dimension). Then, in the first feature matrix, the row dimension (i.e., the first dimension) can be the dimension corresponding to the time feature (i.e., the time dimension), the column dimension (i.e., the second dimension) can be the dimension corresponding to the frequency band feature (i.e., the frequency band dimension), and the depth dimension (i.e., the third dimension) can be the dimension corresponding to the channel feature (i.e., the channel dimension).

[0063] For example, the dimension of the first feature matrix may be C×H×W, where C is the size of the channel dimension, i.e., the number of depth dimensions of the first feature matrix; H is the size of the time dimension, i.e., the number of row dimensions of the first feature matrix; and W is the size of the frequency band dimension, i.e., the number of column dimensions of the first feature matrix. It is understandable that the first feature matrix may be a three-dimensional matrix obtained by combining C two-dimensional matrices, each of which has H rows and W columns. It is understandable that the sizes of the three dimensions of the first feature matrix may be determined based on the size of the convolution kernel in the feature extraction (i.e., convolution processing) process.

[0064] For example, see Figure 4 , Figure 4 Schematic diagram of a feature extraction process provided by an embodiment of the present application. Figure 4 As shown, the time-frequency spectrum data may include real data and imaginary data, and both the real data and the imaginary data may be expressed as a two-dimensional matrix of dimension T×D, where T represents the number of speech frames, and D represents the dimension of Fourier transform, that is, the number of frequency band points corresponding to the time-frequency spectrum data, which will not be described here, that is, the dimension of the time-frequency spectrum data is 2×T×D. After feature extraction (i.e., convolution processing) is performed on the time-frequency spectrum data, a first feature matrix may be obtained, for example, the dimension of the first feature matrix is ​​32×T / 2×D / 2, that is, in the first feature matrix of C×H×W, the channel dimension C=32, the time dimension H=T / 2, and the frequency band dimension W=D / 2.

[0065] S102: Obtain a first channel weight matrix and a first frequency band weight matrix based on channel characteristics, time characteristics and frequency band characteristics.

[0066] Among them, the first channel weight matrix can be a matrix composed of the weight values ​​of each channel feature weight in the time dimension, that is, each numerical value in the first channel weight matrix represents the weight on the corresponding time feature and channel feature. The first channel weight matrix can also be a three-dimensional matrix, and the depth dimension (that is, the third dimension) of the first channel weight matrix is ​​the time dimension (size is H), the row dimension (that is, the first dimension) is the channel dimension (size is C), and the column dimension (second dimension) is the dimension corresponding to the channel feature weight, and the size of the column dimension is 1, that is, the dimension of the first channel weight matrix can be H×C×1.

[0067] Among them, the first frequency band weight matrix can be a matrix composed of weight values ​​corresponding to each frequency band feature weight on the time feature, that is, each numerical value in the first frequency band weight matrix represents the weight on the corresponding time feature and frequency band feature. The first frequency band weight matrix can be a three-dimensional matrix, and the depth dimension (that is, the third dimension) of the first frequency band weight matrix is ​​the time dimension (size is H), the row dimension (that is, the second dimension) is the frequency band dimension (size is W), and the column dimension is the dimension corresponding to the frequency band feature weight (the size of the column dimension is 1), that is, the dimension of the first frequency band weight matrix can be H×W×1.

[0068] It can be understood that the first channel weight matrix and the first frequency band weight matrix of the present application can produce an effect similar to the attention mechanism, extracting the weights of each speech frame for each channel and each frequency band, which helps to extract and pay attention to the feature differences in each speech frame channel and frequency band, thereby improving the audio denoising effect.

[0069] It can be understood that in the embodiment of the present application, the first channel weight matrix can be determined by the channel weight extraction component, that is, the channel weight extraction component refers to a component for extracting the first channel weight matrix, and then the first feature matrix can be input into the channel weight extraction component to obtain the first channel weight matrix. The first frequency band weight matrix can be determined by the frequency band weight extraction component, that is, the frequency band weight extraction component refers to a component for extracting the first frequency band weight matrix, and then the first feature matrix can be input into the frequency band weight extraction component to obtain the first frequency band weight matrix.

[0070] In one embodiment, based on channel features, time features and frequency band features, obtaining a first channel weight matrix includes the following steps: performing matrix transposition on the channel features and time features in the first feature matrix to obtain a third feature matrix; further, performing a linear transformation on the third feature matrix to obtain a second channel weight matrix; further, normalizing the second channel weight matrix to obtain a first channel weight matrix.

[0071] In one embodiment, obtaining a first frequency band weight matrix based on channel features, time features and frequency band features may include the following steps: performing matrix transposition on the channel features, time features and frequency band features in the first feature matrix to obtain a fourth feature matrix; further, performing a linear transformation on the fourth feature matrix to obtain a second frequency band weight matrix; further, normalizing the second frequency band weight matrix to obtain a first frequency band weight matrix.

[0072] Among them, the third feature matrix can be a feature matrix obtained by performing matrix transposition on the channel features and time features in the first feature matrix. It should be understood that the purpose of performing matrix transposition on the first feature matrix to obtain the third feature matrix is ​​to transform the depth dimension of the first feature matrix from the dimension corresponding to the channel features to the dimension corresponding to the time features, and to transform the row dimension from the dimension corresponding to the time features to the dimension corresponding to the channel features, so as to more conveniently extract the channel feature weights. For example, the dimension of the first feature matrix is ​​C×H×W, where H is the size of the time dimension, C is the size of the channel dimension, and W is the size of the frequency band dimension. Then, the time features and channel features of the first feature matrix are matrix transposed, and the feature dimension of the third feature matrix obtained is H×C×W. It can be understood that the matrix transposition of the first feature matrix here to obtain the third feature matrix can be implemented by the matrix transposition layer (reshape layer) in the channel weight extraction component, and the matrix transposition layer here can be called the first matrix transposition layer. It can be understood that through matrix transposition, a feature matrix that is more suitable for obtaining the channel weight matrix can be obtained, so that subsequent simple linear calculations can be performed to obtain the matrix corresponding to the channel weights of each speech frame on each channel (i.e., the first channel weight matrix), thereby reducing the complexity of calculations and helping to improve the computational efficiency of the model.

[0073] Among them, the fourth characteristic matrix can be a characteristic matrix obtained by performing matrix transposition on the channel features, time features, and frequency band features in the first characteristic matrix. It should be understood that the purpose of performing matrix transposition on the first characteristic matrix to obtain the fourth characteristic matrix is ​​to transform the depth dimension of the first characteristic matrix from the dimension corresponding to the channel features to the dimension corresponding to the time features, transform the row dimension from the dimension corresponding to the time features to the dimension corresponding to the frequency band features, and transform the column dimension from the dimension corresponding to the frequency band features to the dimension corresponding to the time features. For example, the characteristic dimension of the first characteristic matrix is ​​C×H×W, where H is the size of the time dimension, C is the size of the channel dimension, and W is the size of the frequency band dimension. Then, the channel features, time features, and frequency band features of the first characteristic matrix are matrix transposed, and the dimension of the fourth characteristic matrix obtained is H×W×C. It can be understood that the matrix transposition of the first characteristic matrix here to obtain the fourth characteristic matrix can be realized by the matrix transposition layer (reshape layer) in the frequency band weight extraction component, and the matrix transposition layer here can be called the second matrix transposition layer. It can be understood that through matrix transposition, a feature matrix that is more suitable for obtaining the frequency band weight matrix can be obtained, so that subsequent simple linear calculations can be performed to obtain the matrix corresponding to the frequency band weights of each speech frame at each frequency band point (i.e., the first frequency band weight matrix), thereby reducing the complexity of calculations and helping to improve the computational efficiency of the model.

[0074] In one embodiment, performing a linear transformation on the third feature matrix to obtain a second channel weight matrix may include the following steps: performing a linear transformation on the time features, channel features and frequency band features in the third feature matrix to obtain a second channel weight matrix, wherein the second channel weight matrix includes time features, channel features and weight values ​​representing channel feature weights.

[0075] Among them, the second channel weight matrix can be a matrix obtained by linearly transforming the time features, channel features and frequency band features in the third feature matrix. The dimension of the second channel weight matrix here is consistent with the dimension of the first channel weight matrix mentioned above, and will not be repeated here. It can be understood that the linear transformation of the third feature matrix to obtain the second channel weight matrix is ​​processed by the linear layer (also called the fully connected layer) in the channel weight extraction component, and the linear layer here can be called the first linear layer.

[0076] It can be understood that the second channel weight matrix includes time features, channel features and weight values ​​representing the channel feature weights. Among them, the dimension of the second channel weight matrix can be H×C×1, that is, in the second channel weight matrix, the depth dimension is the dimension corresponding to the time feature, the row dimension is the dimension corresponding to the channel feature, and the number of column dimensions is 1. The value on this column can be the weight value of the channel feature weight.

[0077] It can be understood that the dimension of the third feature matrix is ​​H×C×W. Then, when the third feature matrix is ​​linearly transformed, the data on each row dimension (i.e., each channel dimension) can be linearly calculated on each depth dimension (i.e., each time dimension) to obtain the linear value corresponding to each row dimension. The data on each row dimension can be the numerical value corresponding to the W frequency band points corresponding to each channel feature. Then, based on the linear values ​​corresponding to each row dimension on each depth dimension, the second channel weight matrix can be determined. The linear value corresponding to each row dimension is the weight value of the channel feature weight. The second channel weight matrix is ​​H×C×1. For example, the third feature matrix can be understood as H C×W two-dimensional matrices. Then, in each C×W two-dimensional matrix, the weight value of the channel feature weight corresponding to the c-th channel dimension can be linearly calculated based on the W numerical values ​​corresponding to the c-th channel dimension to obtain the linear value corresponding to the c-th channel dimension, so that the linear value corresponding to each channel dimension, i.e., the weight value of the channel feature weight, can be determined. In this way, the weights corresponding to each channel dimension on each time dimension can be determined, so that the features subsequently determined based on the weights of the channel features can characterize the potential rules on each channel dimension corresponding to each speech frame, so as to extract more accurate feature representations, thereby improving the audio denoising effect.

[0078] Among them, normalizing the second channel weight matrix refers to an operation for transforming each numerical value in the second channel weight matrix into a specified range, for example, the specified range can be 0-1. Normalizing the second channel weight matrix can be obtained by processing a normalization layer (layer norm), and the normalization layer here can be called a first normalization layer. It can be understood that the normalization process will not change the dimensional size of the second channel weight matrix, that is, the dimension of the second channel weight matrix is ​​consistent with the dimension of the first channel weight matrix.

[0079] In one embodiment, the fourth feature matrix is ​​linearly transformed, and the second frequency band weight matrix may include the following steps: the time features, frequency band features and channel features in the fourth feature matrix are linearly transformed to obtain the second frequency band weight matrix, and the second frequency band weight matrix includes time features, frequency band features and weight values ​​representing the weights of the frequency band features.

[0080] Among them, the second frequency band weight matrix can be a matrix obtained by linearly transforming the time features, frequency band features and channel features in the fourth feature matrix. The dimension of the second frequency band weight matrix here is consistent with the dimension of the first frequency band weight matrix mentioned above, and will not be repeated here. It can be understood that the fourth feature matrix is ​​linearly transformed to obtain the second frequency band weight matrix, which is processed by the linear layer (also called the fully connected layer) in the channel weight extraction component, and the linear layer here can be called the second linear layer.

[0081] It can be understood that the second band weight matrix includes time features, band features and weight values ​​representing the weights of the band features. Among them, the dimension of the second band weight matrix can be H×W×1, that is, in the second band weight matrix, the depth dimension is the dimension corresponding to the time feature, the row dimension is the dimension corresponding to the band feature, the number of column dimensions is 1, and the value on this column can be the weight value of the band feature weight.

[0082] It can be understood that the dimension of the fourth feature matrix is ​​H×W×C. Then, when the fourth feature matrix is ​​linearly transformed, the data on each row dimension (i.e., each band feature) can be linearly calculated on each depth dimension (i.e., each time feature) to obtain the linear value corresponding to each row dimension. The data on each row dimension can be the numerical value corresponding to the C channels corresponding to each band feature. Then, based on the linear values ​​corresponding to each row dimension on each depth dimension, the second band weight matrix can be determined. The linear value corresponding to each row dimension is the weight value of the band feature weight. The second band weight matrix is ​​H×W×1. For example, the fourth feature matrix can be understood as H W×C two-dimensional matrices. Then, in each W×C two-dimensional matrix, the weight value of the band feature weight corresponding to the wth band dimension can be linearly calculated based on the C numerical values ​​corresponding to the wth band dimension to obtain the linear value corresponding to the wth band dimension, so that the linear value corresponding to each band dimension, i.e., the weight value of the band feature weight, can be determined. In this way, the weights corresponding to each frequency band dimension on each time dimension can be determined, so that the features determined subsequently based on the weights of the frequency band features can characterize the potential rules on each frequency band dimension corresponding to each speech frame, so as to extract more accurate feature representations, thereby improving the audio denoising effect.

[0083] Among them, normalizing the second band weight matrix refers to an operation for transforming each numerical value in the second band weight matrix into a specified range, for example, the specified range can be 0-1. Normalizing the second band weight matrix can be obtained by processing through a normalization layer (layer norm), and the normalization layer here can be referred to as a second normalization layer. It can be understood that the normalization process will not change the dimensional size of the second band weight matrix, that is, the dimension of the second band weight matrix is ​​consistent with the dimension of the first band weight matrix.

[0084] For example, see Figure 5 , Figure 5 It is a schematic diagram of a channel weight extraction process provided by an embodiment of the present application. First, the first feature matrix R1 is obtained, and the dimension of the first feature matrix R1 is C×H×W. Then, the first feature matrix R1 can be transposed to obtain the third feature matrix G1, and the dimension of the third feature matrix is ​​H×C×W. Then, the third feature matrix G1 can be linearly transformed to obtain the second channel weight matrix S01, and the dimension of the second channel weight matrix is ​​H×C×1. Furthermore, the second channel weight matrix can be normalized to obtain the first channel weight matrix S11, and the dimension of the first channel weight matrix S11 is H×C×1, but the values ​​of each element in the first channel weight matrix (that is, each numerical value in the matrix) all belong to the value range of 0-1.

[0085] For example, see Figure 6 , Figure 6 It is a schematic diagram of a frequency band weight extraction process provided by an embodiment of the present application. First, the first characteristic matrix R1 is obtained, and the dimension of the first characteristic matrix R1 is C×H×W. Then, the first characteristic matrix R1 can be transposed to obtain the fourth characteristic matrix G2, and the dimension of the fourth characteristic matrix is ​​H×W×C. Then, the fourth characteristic matrix G2 can be linearly transformed to obtain the second frequency band weight matrix S02, and the dimension of the second frequency band weight matrix is ​​H×W×1. Furthermore, the second frequency band weight matrix can be normalized to obtain the first frequency band weight matrix S12, and the dimension of the first frequency band weight matrix S12 is H×W×1, but the values ​​of each element in the first frequency band weight matrix (that is, each numerical value in the matrix) all belong to the value range of 0-1.

[0086] It can be understood that in the embodiment of the present application, the length of the audio data for different speech data may be different, and the number of speech frames that can be extracted from audio data of different lengths may be different. Furthermore, the number of time dimensions in the extracted first feature matrix is ​​different. Then, when the audio data is processed by an audio processing model trained by sample audio data of different lengths, the channel weights and frequency band weights of the audio data on the time dimension corresponding to each speech frame can be extracted, rather than directly extracting the channel weights or frequency band weights corresponding to the audio data as a whole. This allows attention to be paid to the underlying rules of the features corresponding to each speech frame, thereby improving the feature expression effect of the feature matrix determined for the audio data, thereby improving the denoising effect of the audio data. It can be understood that in the embodiments of the present application, when extracting frequency band weights and channel weights, the time dimension can be kept unchanged. Because for the audio processing model, the length of the input audio data can be different, and the size of the time dimension in the first feature matrix corresponding to the time-frequency spectrum data that can be determined can be different. Therefore, the present application can perform a matrix transposition on the first feature matrix so that when performing a linear transformation, the time dimension is not processed, but the channel feature weights and frequency band feature weights on each time dimension are determined.

[0087] S103 . Perform matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix.

[0088] The time channel weight matrix may be a matrix obtained by performing matrix product of the first channel weight matrix and the first frequency band weight matrix. The dimension of the time channel weight matrix may be H×C×W, that is, the depth dimension of the time channel matrix is ​​the dimension corresponding to the time feature (i.e., the time dimension), the row dimension is the dimension corresponding to the channel feature (i.e., the channel dimension), and the column dimension is the dimension corresponding to the frequency band feature (i.e., the frequency band dimension).

[0089] It should be understood that when performing matrix multiplication of the first channel weight matrix and the first frequency band weight matrix to obtain the time channel weight matrix, the row dimension and column dimension of the first frequency band weight matrix can be transposed, so that the first channel weight matrix can be multiplied by the transposed first frequency band weight matrix to obtain the time channel weight matrix. For example, if the dimension of the first channel weight matrix is ​​H×C×1 and the dimension of the first frequency band weight matrix is ​​H×W×1, the row dimension and column dimension of the first frequency band weight matrix can be transposed, and the dimension of the transposed first frequency band weight matrix is ​​H×1×W, and then the first channel weight matrix with the dimension of H×C×1 can be multiplied by the transposed first frequency band weight matrix with the dimension of H×1×W to obtain the time channel weight matrix with the dimension of H×C×W.

[0090] S104, performing Hadamard product on the first characteristic matrix and the time channel weight matrix to obtain a second characteristic matrix.

[0091] It should be understood that after determining the time channel weight matrix, the first feature matrix and the time channel weight matrix can be matrix fused to obtain a feature matrix, i.e., the second feature matrix. Matrix fusion can be performed in a variety of ways, for example, the first feature matrix and the time channel weight matrix can be Hadamard product, or the first feature matrix and the time channel weight matrix can be matrix added, etc., which are not limited here.

[0092] Among them, Hadamard product is a special matrix multiplication operation, also known as element-by-element multiplication or element-by-element multiplication. This multiplication requires that the shapes (i.e., sizes) of the two matrices A and B must be the same in order to perform element-by-element multiplication. The definition of Hadamard product is: for two matrices A and B of the same type, their Hadamard product C = A⊙B, where each element C_{i,j} of C is equal to the product of the element in the i-th row and j-th column of A and the element in the i-th row and j-th column of B, that is, C_{i,j} = A_{i,j}×B_{i,j}.

[0093] Then, when performing the Hadamard product on the first feature matrix and the time channel weight matrix, it is necessary to transform the time channel weight matrix into a matrix with the same dimension as the first feature matrix, and then perform a matrix product on the transformed time channel weight matrix and the first feature matrix.

[0094] Specifically, after performing matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain the time channel weight matrix, the method further includes: performing matrix transposition on the time channel weight matrix to obtain the transposed time channel weight matrix.

[0095] It should be understood that the purpose of performing matrix transposition on the time channel weight matrix is ​​to transform the time channel weight matrix into a matrix consistent with the dimension of the first feature matrix. The purpose of performing matrix transposition on the time channel weight matrix is ​​to transform the depth dimension of the time channel weight matrix from the dimension corresponding to the time feature (i.e., the time dimension) to the dimension corresponding to the channel feature (i.e., the channel dimension), and to transform the row dimension from the dimension corresponding to the channel feature (i.e., the channel dimension) to the dimension corresponding to the time feature (i.e., the time dimension). In other words, the dimension of the time channel weight matrix after transposition is C×H×W. It can be understood that the matrix transposition of the time channel weight matrix can be processed by a matrix transposition layer, and the matrix transposition layer here can be called a third matrix transposition layer.

[0096] Furthermore, the first feature matrix and the transposed time channel weight matrix can be matrix fused, such as performing Hadamard product on the first feature matrix and the transposed time channel weight matrix to obtain a second feature matrix. Thus, by fusing the time channel weight matrix into the first feature matrix, the fused second feature matrix can characterize the potential rules of the audio data in each time dimension, thereby improving the audio denoising effect when denoising is performed based on the second feature matrix.

[0097] For example, see Figure 7 , Figure 7 Schematic diagram of the effect of a method for determining a second characteristic matrix provided in an embodiment of the present application. Figure 7As shown, first, the first feature matrix R1 can be obtained, and then the first feature matrix can be processed by the channel weight extraction component to obtain the first channel weight matrix S11, and the dimension of the first channel weight matrix S11 is H×C×1. The first feature matrix is ​​processed by the band weight extraction component to obtain the first band weight matrix S12, and the dimension of the first band weight matrix S12 is H×W×1. Further, the first channel weight matrix S11 and the first band weight matrix S12 can be matrix-producted to obtain the time channel weight matrix W1, and the dimension of the time channel weight matrix W1 is H×C×W. Further, the time channel weight matrix W1 can be matrix-transposed to obtain the transposed time channel weight matrix W', and the dimension of the transposed time channel weight matrix W' is C×H×W. Then, the transposed time channel weight matrix W' can be Hadamard-producted with the first feature matrix R1 to obtain the second feature matrix R2, and the dimension of the second feature matrix R2 is C×H×W.

[0098] S105 . De-noise the first audio data based on the second characteristic matrix to obtain second audio data corresponding to the first audio data.

[0099] It can be understood that the second audio data may be audio data obtained after denoising the first audio data.

[0100] It is understandable that denoising based on the second characteristic matrix may include determining the denoised time-frequency spectrum data (i.e. denoised time-frequency spectrum data) based on the second characteristic matrix, and then performing audio restoration based on the denoised time-frequency spectrum data to obtain the first audio data. The audio restoration may be obtained by processing based on the above-mentioned inverse Fourier transform, which will not be described in detail here.

[0101] It is understandable that the second audio data can be obtained by processing the first audio data through the audio processing model. Before the first audio data is processed by the audio processing model, the first audio data can be trained with a large amount of sample data, such as can be obtained by an end-to-end training method. While adopting a convolutional neural network, the present application models the real and imaginary parts of the time-frequency spectrum data of the audio data, and uses an end-to-end training method based on a convolutional neural network to help improve the training effect of the audio processing model.

[0102] Specifically, the embodiment of the present application may also include the following steps: obtaining a sample audio data pair; the sample audio data pair includes sample noisy audio data and sample clean audio data associated with the sample noisy audio data; extracting features from the time-frequency spectrum data of the sample noisy audio data through an initial audio processing model to obtain a first sample feature matrix; the first sample feature matrix includes channel features, time features and frequency band features; further, obtaining a first sample channel weight matrix and a first sample frequency band weight matrix based on the channel features, time features and frequency band features in the first sample feature matrix through the initial audio processing model; further, performing matrix product of the first sample channel weight matrix and the first sample frequency band weight matrix through the initial audio processing model to obtain a sample time channel weight matrix; performing Hadamard product of the first sample feature matrix and the sample time channel weight matrix through the initial audio processing model to obtain a second sample feature matrix; denoising the sample noisy audio data based on the second sample feature matrix through the initial audio processing model to obtain sample denoised audio data corresponding to the sample noisy audio data; further, adjusting parameters of the initial audio processing model based on the sample clean audio data and the sample denoised audio data to obtain an audio processing model.

[0103] The sample audio data pair (also referred to as the sample data pair) may be an audio data pair used to train an audio processing model. An audio data pair may include a sample noisy audio data and a sample clean audio data associated with the sample noisy audio data. The sample clean audio data is equivalent to a training target of supervised training for training an audio processing model. The sample noisy audio data may be noise data used to train an audio processing model, and the sample clean audio data may be clean audio data corresponding to the sample noisy audio data. For example, in some scenarios, some clean audio data can be obtained, and noise can be added to the clean audio data, so that the noisy audio data corresponding to the clean audio data can be obtained, and then the clean audio data can be determined as sample clean audio data, and the noisy audio data corresponding to the clean audio data can be determined as sample noisy data; or, the noisy audio data can be directly obtained, and then denoised by a combination of machine and manual methods to obtain denoised audio data, and then the denoised audio data can be determined as sample clean audio data, and the noisy audio data directly obtained can be determined as sample noisy data; or the embodiment of the present application can also obtain sample audio data pairs by other methods, which are not limited here.

[0104] It should be understood that the initial audio processing model can be a model that has not completed training. The model structure of the initial audio processing model is consistent with the model structure of the above-mentioned audio processing model, which will not be described in detail here. Among them, the method for determining the first sample feature matrix can refer to the relevant description of the above-mentioned first feature matrix, which will not be described in detail here. The method for determining the first sample channel weight matrix and the first sample frequency band weight matrix can refer to the relevant description of the above-mentioned first channel weight matrix and the first frequency band weight matrix, the method for determining the second sample feature matrix can refer to the relevant description of the above-mentioned second feature matrix, and the method for determining the sample denoised audio data can refer to the relevant description of the above-mentioned second audio data, which will not be described in detail here.

[0105] The sample denoised audio data may be audio data obtained by processing the sample noisy audio data using an initial audio processing model.

[0106] Among them, adjusting the parameters of the initial audio processing model based on the sample clean audio data and the sample denoised audio data can be used to indicate that the sample denoised audio data gradually approaches the sample clean audio data by adjusting the parameters of the initial audio processing model. For example, the loss information can be determined by the time-frequency spectrum data of the sample clean audio data (or it can be a frequency domain signal) and the time-frequency spectrum data of the sample denoised audio data (or it can be a frequency domain signal), and then the model parameters of the initial audio processing model are adjusted based on the loss information until the loss information converges, and then the initial audio processing model can be determined as the final audio processing model. It should be understood that when training the initial audio processing model, a batch of training can include multiple sample data pairs (such as the number of sample data pairs input in a batch can be set by Batch size), and then the loss information can be determined based on the sample clean audio data and sample denoised audio data of multiple sample data pairs to train the initial audio processing model.

[0107] In an embodiment of the present application, when processing audio data, feature extraction can be performed on the time-frequency spectrum data of the audio data to obtain a first feature matrix, and then based on the channel features, time features and frequency band features in the first feature matrix, a first channel weight matrix and a first frequency band weight matrix can be determined. The first channel weight matrix and the first frequency band weight matrix can focus on the feature differences of channels and frequency bands in time features, and then matrix product can be performed based on the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix, which can characterize the weights of the fusion of frequency bands and channels in time features, and then the first feature matrix and the time channel weight matrix can be Hadamard-multiplied to add the time channel weight matrix to the first feature matrix to obtain a second feature matrix. The second feature matrix can be a feature with a potential noise law focusing on time features, which is not affected by the signal-to-noise ratio on the denoising of the audio data, and then denoising can be performed based on the second feature matrix, and noise information in the audio data can be removed, thereby improving the audio denoising quality of the audio data.

[0108] For further information, see Figure 8 , Figure 8 1 is a flowchart of another audio processing method provided in an embodiment of the present application. The method can be executed by a computer device, for example, the computer device can be the server 100a mentioned above. The method can at least include the following steps S201 to S208.

[0109] S201. Extract features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features, and frequency band features.

[0110] S202: Obtain a first channel weight matrix and a first frequency band weight matrix based on channel characteristics, time characteristics and frequency band characteristics.

[0111] S203 . Perform matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix.

[0112] S204 , performing Hadamard product on the first characteristic matrix and the time channel weight matrix to obtain a second characteristic matrix.

[0113] The processing process of steps S201-S204 may refer to the relevant description of the above steps S101-S104, which will not be repeated here.

[0114] S205 , denoising the second characteristic matrix to obtain a denoised second characteristic matrix.

[0115] The second characteristic matrix is ​​denoised to obtain a denoised second characteristic matrix, which can be determined based on a modulation denoising component in a convolutional modulation network. The denoised second characteristic matrix can be called a denoised characteristic matrix.

[0116] It should be understood that the convolution modulation network can be a network for performing convolution modulation on the first feature matrix. The function of the convolution modulation network is to better extract the feature information of the audio data without noise, or to further remove the noise. The convolution modulation network can also include the above-mentioned channel weight extraction component (refer to the above Figure 5 ) and the band weight extraction component (see above Figure 6 It can be understood that the convolutional modulation network provided in the embodiment of the present application can be used in various audio processing models to perform convolutional modulation processing on the feature matrix of audio data, thereby improving the denoising effect of the audio data.

[0117] The modulation denoising component may be a component for denoising the second characteristic matrix. The input data of the modulation denoising component is the second characteristic matrix, and the output data may be the second characteristic matrix after denoising. The modulation denoising component may be composed of M modulation denoising subcomponents of the same structure, where M is a positive integer, and the M modulation denoising subcomponents are connected in series, and the output data of the previous modulation denoising subcomponent may be used as the input data of the next modulation denoising subcomponent. The number of modulation denoising subcomponents may be set based on actual needs, and this application does not limit this.

[0118] In any modulation denoising subcomponent, a modulation path and a modulated path may be included, the modulation path includes a one-dimensional convolution layer (i.e., Conv1d) and a two-dimensional deformable convolution layer (i.e., D-Conv2d), wherein the two-dimensional deformable convolution layer is a two-dimensional convolution network structure with a dilation rate (dilate rate) that is not 1, and its function is to increase the receptive field and obtain richer modulation information; the modulated path includes a one-dimensional convolution layer (i.e., Conv1d), which plays a convolution smoothing role; the modulation denoising subcomponent, after receiving input data of dimension C×H×W (the input data may be a second feature matrix or the output data of a previous modulation denoising subcomponent), may input the input data into the modulation path and the modulated path respectively, and obtain data of the same dimension as the input respectively, and then calculate the Hadamard product of the two, and then further calculate the eigenvalue through a one-dimensional convolution layer, and finally implement modulation denoising through a residual connection to obtain a denoised component.

[0119] For example, see Fig. 9 , Fig. 9 Schematic diagram of a convolutional modulation network provided by an embodiment of the present application. Fig. 9 As shown, the convolutional modulation network can refer to Fig. 9 As shown in 900a in FIG. 1 , the feature dimension of the input data (i.e., the first feature matrix) for inputting the convolutional modulation network is (C, H, W), where C represents the number of channels (Channel), H represents the H time dimensions associated with the T speech frames, and W represents W frequency band points. The input data can then be input into the channel weight extraction component (e.g., Fig. 9 901a in the channel weight extraction component). In the channel weight extraction component, a matrix transposition layer (Reshape layer, such as Fig. 9 911a in FIG. 9A ) performs matrix transposition on the input data, thereby transforming the input data into features (i.e., the third feature matrix) with dimensions (H, C, W), and then passes through a linear layer (e.g., Fig. 9 912a in FIG. 9 ) performs a linear transformation on the third feature matrix to obtain a feature with a dimension of (H, C, 1) (i.e., a second channel weight matrix). Optionally, after the linear layer 912a, a normalization layer (layer norm, such as Fig. 9 The second channel weight matrix is ​​normalized (as shown in 913a in FIG) to obtain a normalized second channel weight matrix (i.e., the first channel weight matrix). The dimension of the normalized second channel weight matrix remains unchanged, i.e., (H, C, 1).

[0120] In addition, the input data (i.e., the first feature matrix) can be input into the band weight extraction component (e.g., Fig. 9 In the frequency band weight extraction component, a matrix transposition layer (Reshape, such as Fig. 9 921a in FIG. 9) performs a matrix transposition on the first feature matrix, thereby transforming the input data into a feature with features (H, W, C) (i.e., the fourth feature matrix), and then passes through a linear layer (e.g., Fig. 9 922a in FIG. 9 ) performs a linear transformation on the fourth feature matrix to obtain a feature with a dimension of (H, W, 1) (i.e., a second frequency band weight matrix). Optionally, after the linear layer, a normalization layer (layer norm, such as Fig. 9 The second frequency band weight matrix is ​​normalized (as shown in 923a in FIG) to obtain a normalized second frequency band weight matrix (i.e., the first frequency band weight matrix). The dimension of the normalized second frequency band weight matrix remains unchanged, i.e., (H, W, 1).

[0121] Furthermore, the output of the channel weight extraction component (i.e., the first channel weight matrix) and the output of the band weight component (i.e., the first band weight matrix) can be matrix multiplied to obtain the time channel weight feature, that is, through Dot[(H, C, 1), (H, W, 1)], the time channel weight feature with a feature dimension of (H, C, W) can be determined. Then, the time channel weight feature can be matrix transposed through the matrix transposition layer (Reshape), and the feature dimension of the transposed time channel weight feature is (C, H, W), so that the dimension of the transposed time channel weight feature is consistent with the dimension of the input data. Finally, the time channel weight feature after the matrix transposition is added to the input data through the Hadamard product (i.e., Hadamard Production), that is, the input data (i.e., the first feature matrix) and the transposed time channel weight feature are feature fused to obtain a second feature matrix, and the dimension of the second feature matrix is ​​(C, H, W). This is conducive to improving the feature extraction of each time dimension in the audio denoising process and improving the performance of audio denoising.

[0122] Further, the obtained second characteristic matrix can be input into the modulation denoising component, and the modulation denoising component can be obtained by connecting M modulation denoising subcomponents in series, as shown in Fig. 9 Taking a modulation denoising subcomponent shown in 903a as an example, the second feature matrix can be input into the modulation denoising subcomponent, and then the data with the same dimension as the input data can be obtained through the two paths (Conv1d, D-Conv2d) (i.e., modulation path) and Conv1d (modulated path), that is, the features of (C, H, W) are obtained respectively. Then the Hadamard product of the two is calculated, and then the eigenvalue is further calculated through a Conv1d, and finally the convolution modulation is realized through the residual connection to obtain the output data of the entire convolution modulation network (i.e., the intermediate processing feature matrix), so that the feature extraction capability in the audio denoising process can be further improved through the modulation denoising component.

[0123] S206 , performing feature decoding processing on the denoised second feature matrix to obtain a decoded second feature matrix.

[0124] Among them, the feature decoding processing of the denoised second feature matrix can be obtained by processing the denoised second feature matrix through a decoding network. The feature dimension of the decoded second feature matrix can be a feature consistent with the dimension of the time-frequency spectrum data. For example, if the dimension of the first audio data is 2×T×D, then the dimension of the decoded second feature matrix is ​​2×T×D, and the decoded second feature matrix includes data on the real channel and the imaginary channel, that is, it can include a T×D feature matrix on the real channel and a T×D feature matrix on the imaginary channel.

[0125] Each feature decoding network includes a decoding layer (also called a deconvolution layer) and a decoding convolution modulation network. The decoding layer is used to decode and restore the denoised second feature matrix to obtain a first decoding feature matrix with the same dimension as the time-frequency spectrum, and then the first decoding feature matrix can be processed by the decoding convolution modulation network to obtain a decoded second feature matrix, such as a second decoding feature matrix.

[0126] Among them, the decoding layer can be a two-dimensional deconvolution layer (i.e., ConvTranspose 2d). The network structure of the decoding convolutional modulation network can be consistent with the network structure of the above-mentioned convolutional modulation network, so as to reduce the redundant information in the decoding features. The method of processing the first decoding feature matrix through the decoding convolutional modulation network can refer to the relevant description of processing the first feature matrix through the convolutional modulation network, which will not be repeated here.

[0127] S207 . Obtain denoised time-frequency spectrum data of the first audio data through the decoded second feature matrix.

[0128] It should be understood that the denoised time-frequency spectrum data of the first audio data is constructed by constructing the decoded second feature matrix, and the decoded second feature matrix can be directly determined as the denoised time-frequency spectrum data, or it can be obtained by processing the decoded second feature matrix through the attention mechanism layer, and performing convolution modulation processing on the output data of the attention mechanism layer through the convolution modulation network, etc., which is not limited here.

[0129] S208: Perform audio restoration on the denoised time-frequency spectrum data to obtain second audio data corresponding to the first audio data.

[0130] The processing method for performing audio restoration on the denoised time-frequency spectrum data may refer to the above-mentioned related description, which will not be described in detail here.

[0131] It can be understood that in an embodiment of the present application, the second audio data can be obtained by inputting the first audio data into an audio processing model, and the audio processing model can be a network model for performing speech enhancement (also known as audio denoising) on ​​the audio data. The audio processing model can include the above-mentioned convolution layer for extracting features from the time-frequency spectrum data, a convolution modulation network for determining the second feature matrix after denoising, and a feature decoding network. Among them, the above-mentioned convolution layer for extracting features from the time-frequency spectrum data and the convolution modulation network for determining the second feature matrix after denoising can be combined into a feature extraction network, that is, the time-frequency spectrum data is input into the feature extraction network to obtain the second feature matrix after denoising.

[0132] Optionally, in an embodiment of the present application, the audio processing model may include a feature extraction network and a feature decoding network corresponding to the feature extraction network.

[0133] Alternatively, in the audio processing model, multiple cascaded feature extraction networks may also be included, and each feature extraction network may be connected to a corresponding feature decoding network. For multiple cascaded feature extraction networks in the audio processing model, the network structures of the feature extraction networks are the same, but the input data of different feature extraction networks are different, wherein the input data of the first feature extraction network in the audio processing model is the time-frequency spectrum data, and the output data is the second feature matrix after denoising, and the input data of the next feature extraction network of the first feature extraction network (i.e., the next feature extraction network cascaded after the first feature extraction network, recorded as the second feature extraction network) is the output data of the first feature extraction network (i.e., the second feature matrix after denoising), and the input data of the next feature extraction network of the next feature extraction network of the first feature extraction network (i.e., the third feature extraction network) is the output data of the second feature extraction network, that is, in the audio processing model, the output data of the previous feature extraction network can be used as the input data of the next feature extraction network. It is understandable that, for ease of description, the output data of each feature extraction network can be referred to as an intermediate processing feature matrix. For example, the intermediate processing feature matrix output by the first feature extraction network can be the second feature matrix after denoising. It is understandable that the audio processing model of the present invention adopts a fully convolutional network for modeling, and the computational complexity is low, so that the real-time rate of audio processing by the audio processing model can be lower, which is more suitable for practical application scenarios.

[0134] See also Fig.10 , Fig.10 FIG. 1 is a schematic diagram of an audio feature extraction method provided in an embodiment of the present application. Fig.10As shown, here, the four cascaded feature extraction networks included in the audio processing model are taken as an example. For example, the four feature extraction networks can be, from high to low in terms of hierarchy, feature extraction network F1 (also called the first feature extraction network, i.e., the first feature extraction network), feature extraction network F2 (also called the second feature extraction network, i.e., the second feature extraction network), feature extraction network F3 (also called the third feature extraction network, i.e., the third feature extraction network), and feature extraction network F4 (also called the fourth feature extraction network, i.e., the fourth feature extraction network). First, the time-frequency spectrum data can be obtained, and the feature dimension of the time-frequency spectrum data is 2×T×D. Then, the time-frequency spectrum data can be input into the feature extraction network F1 to obtain the intermediate processing feature matrix E1, which can be the above-mentioned second feature matrix after denoising. The dimension of the intermediate processing feature matrix E1 is C1×H1×W1. For example, the feature dimension of the intermediate processing feature matrix E1 is 32×T / 2×D / 2, i.e., C1=32, H1=T / 2, and W1=D / 2.

[0135] Furthermore, the intermediate processing feature matrix E1 can be used as input data of the feature extraction network F2, so as to input the intermediate processing feature matrix E1 into the feature extraction network F2 to obtain the intermediate processing feature matrix E2. The dimension of the intermediate processing feature matrix E2 is C2×H2×W2. For example, the feature dimension of the intermediate processing feature matrix E2 is 64×T / 4×D / 4, that is, C2=64, H2=T / 4, and W2=D / 4.

[0136] Furthermore, the intermediate processing feature matrix E2 can be used as input data of the feature extraction network F3, so as to input the intermediate processing feature matrix E2 into the feature extraction network F3 to obtain the intermediate processing feature matrix E3. The dimension of the intermediate processing feature matrix E3 is C3×H3×W3. For example, the feature dimension of the intermediate processing feature matrix E3 is 128×T / 8×D / 8, that is, C3=128, H3=T / 8, and W3=D / 8.

[0137] Furthermore, the intermediate processing feature matrix E3 can be used as input data of the feature extraction network F4, so as to input the intermediate processing feature matrix E3 into the feature extraction network F4 to obtain the intermediate processing feature matrix E4, the dimension of which is C4×H4×W4. For example, the feature dimension of the intermediate processing feature matrix E4 is 256×T / 16×D / 16, i.e., C4=256, H4=T / 16, and W4=D / 16. Based on this, the intermediate processing feature matrices corresponding to the feature extraction networks at multiple levels can be obtained.

[0138] It can be understood that a corresponding feature decoding network can be connected after each feature extraction network, and each feature decoding network can be used to perform feature decoding processing on the intermediate processing feature matrix output by the corresponding feature extraction network to obtain corresponding decoding features. For example, the output data of the feature decoding network connected after the first feature extraction network can be the above-mentioned second feature matrix after decoding. For the convenience of description, the output data of the feature decoding network connected after each feature extraction network can be called decoding features.

[0139] It should be understood that the dimensions of each decoded feature are consistent with the dimensions of the time-frequency spectrum data of the first audio data. For example, here, taking the audio processing model including two feature extraction networks as an example, the feature decoding network corresponding to the first feature extraction network (i.e., the first feature decoding network) can be used to perform feature decoding processing on the intermediate processing feature matrix (i.e., the first intermediate processing feature matrix) output by the first feature extraction network to obtain two decoding feature maps with a resolution of (T, D), that is, decoding features with a dimension of 2×T×D are also obtained. Similarly, the feature decoding network corresponding to the second feature extraction network (i.e., the second feature decoding network) can be used to perform feature decoding processing on the intermediate processing feature matrix (i.e., the first intermediate processing feature matrix) output by the second feature extraction network to obtain two decoding feature maps with a resolution of (T, D), that is, decoding features with a dimension of 2×T×D are also obtained.

[0140] Furthermore, feature splicing processing can be performed based on the decoding features output by each feature decoding network to obtain decoding splicing features, and denoised time-frequency spectrum data of the first audio data can be constructed based on the decoding splicing features.

[0141] Among them, the decoding splicing feature can be a feature obtained by performing feature splicing processing on the decoding features determined by each feature decoding network. Among them, the feature splicing processing here can be used to indicate the feature of splicing multiple decoding features into a whole. It should be understood that each decoding feature can include a T×D feature matrix on the real channel and a T×D feature matrix on the imaginary channel, and then the T×D feature matrix on the real channel and the T×D feature matrix on the imaginary channel in each decoding feature can be spliced ​​to obtain the splicing feature corresponding to each decoding feature, and the dimension of the splicing feature is 1×T×2D, and then the splicing feature corresponding to each decoding feature is used to determine the decoding splicing feature, and the splicing feature corresponding to each decoding feature is used as data on a depth dimension of the matrix. For example, if the number of feature extraction networks in the audio processing model is 4, the feature dimension of the obtained decoding splicing feature is 4×T×2D.

[0142] See also Fig.11 , Fig.11is a schematic diagram of an audio processing process provided by an embodiment of the present application. Fig.11 As shown, when the time-frequency spectrum data of the first audio data is obtained, the time-frequency spectrum data can be input into the feature extraction network F1 (i.e., the feature extraction network deployed on the first level, such as Fig.11 1101a in FIG), so that the intermediate processing feature matrix E1 can be extracted through the feature extraction network, and the feature decoding network J1 (as shown in FIG) corresponding to the feature extraction network F1 can be used. Fig.11 1102a in the figure), the intermediate processing feature matrix E1 is subjected to feature decoding processing to obtain the decoded feature V1. Further, the intermediate processing feature matrix E1 can be used as a feature extraction network F2 (i.e., a feature extraction network at the second level, such as Fig.11 1103a in the figure), thereby inputting the intermediate processing feature matrix E1 into the feature extraction network F2, through which the intermediate processing feature matrix E2 can be extracted, and the feature decoding network J2 (as shown in FIG. 1103a) corresponding to the feature extraction network F2 can be used. Fig.11 ), perform feature decoding processing on the intermediate processed feature matrix E2 to obtain decoded features V2.

[0143] Similarly, the intermediate processing feature matrix E2 can be used as the feature extraction network F3 (i.e., the feature extraction network at the third level, such as Fig.11 1105a in FIG), thereby inputting the intermediate processing feature matrix E2 into the feature extraction network F3, through which the intermediate processing feature matrix E3 can be extracted, and the feature decoding network J3 (as shown in FIG. Fig.11 1106a in FIG), perform feature decoding on the intermediate feature matrix E3 to obtain decoded features V3. The intermediate feature matrix E3 is used as a feature extraction network F4 (i.e., a feature extraction network at the fourth level, as shown in FIG. Fig.11 1107a in FIG), thereby inputting the intermediate processing feature matrix E3 into the feature extraction network F4, so that the intermediate processing feature matrix E4 can be extracted through the feature extraction network F4, and the feature decoding network J4 (as shown in FIG. Fig.11 1108a in the figure), the intermediate processing feature matrix E4 is subjected to feature decoding processing to obtain the decoding feature V4. Based on this, the multi-level feature extraction network can be obtained to extract the multi-granularity intermediate processing feature matrix, and the decoding features corresponding to the multi-granularity intermediate processing feature matrix are determined. Further, the decoding features can be feature spliced ​​(as shown in FIG. Fig.111109a in FIG), that is, the decoding feature V1, the decoding feature V2, the decoding feature V3, and the decoding feature V4 are concatenated to obtain the decoding concatenated feature (as shown in FIG. Fig.11 Further, the denoised time-frequency spectrum data may be determined based on the decoded splicing feature, and an inverse short-time Fourier transform may be performed based on the denoised time-frequency spectrum data to obtain the second audio data corresponding to the first audio data.

[0144] It should be understood that the denoised time-frequency spectrum data of the first audio data is constructed based on the decoded splicing features, which may include: extracting weights of the decoded splicing features through the attention mechanism layer to obtain weights corresponding to each decoded feature in the decoded splicing features, such as when the number of feature extraction networks is 4, the dimension of the determined weights is 4×1×1; using the weights corresponding to each decoded feature to perform matrix product on the splicing features corresponding to the corresponding decoded features in the decoded splicing features to obtain a decoding attention matrix (i.e., a decoding splicing feature with added weights, with a feature dimension of 4×T×2D); restoring the dimension of the decoding attention matrix through a dimension reduction layer (such as a two-dimensional convolution layer) to obtain a decoding attention feature matrix, which is a feature matrix with a dimension of 2×T×D. Furthermore, the decoding attention feature matrix can also be convolutionally modulated through a convolutional modulation network to obtain denoised time-frequency spectrum data. The network structure of the convolutional modulation network here can be consistent with the network structure of the convolutional modulation network in the above-mentioned feature extraction network. The specific processing process can refer to the relevant description of the above-mentioned convolutional modulation network, which will not be repeated here.

[0145] Here, combined with diagrams, we introduce the model structure of an audio processing model with four feature extraction networks. Fig.12 , Fig.12 Schematic diagram of the model structure of an audio processing model provided in an embodiment of the present application. Fig.12 As shown, the time-frequency spectrum data of the first audio data can be determined by short-time Fourier transform, and its dimension is (2, T, D), where 2 represents the real part and the imaginary part, T represents the number of speech frames, and D represents the dimension of Fourier transform.

[0146] Furthermore, the time-frequency spectrum data can be input into the first feature extraction network (i.e., the first feature extraction network), which may include a convolution layer Conv2d_1 and a convolution modulation network Conv_Modulation, wherein Conv2d_1 is a convolution layer composed of two-dimensional convolutions, and its main function is to extract 32 feature maps (i.e., the first feature matrix) with a resolution of (T / 2, D / 2) from the input data. In other words, the features (i.e., the first feature matrix) obtained by the convolution layer processing based on the first feature extraction network have a dimension of (32, T / 2, D / 2), and the intermediate processing feature matrix (i.e., the second feature matrix after denoising) output by the first feature extraction network has a dimension of (32, T / 2, D / 2), and the processing process of the convolution modulation network Conv_Modulation can refer to the above. Fig. 9 The relevant description is not repeated here. Furthermore, the output data of the first feature extraction network can be input into the next feature extraction network (i.e., the second feature extraction network), and the second feature extraction network can also include a convolutional layer Conv2d_2 and a convolutional modulation network Conv_Modulation, wherein Conv2d_2 can extract the input data to obtain 64 feature maps with a resolution of (T / 4, D / 4), that is, the dimension of the features obtained by the convolutional layer processing of the second feature extraction network is (64, T / 4, D / 4), and the processing process of the convolutional modulation network Conv_Modulation can refer to the above Fig. 9 The relevant description is not repeated here. The dimension of the intermediate processing feature matrix of the output of the second feature extraction network is (64, T / 4, D / 4). Similarly, the output data of the second feature extraction network can be input into the third feature extraction network. The third feature extraction network can include a convolution layer Conv2d_3 and a convolution modulation network Conv_Modulation, wherein Conv2d_3 can extract the input data to obtain 128 feature maps with a resolution of (T / 8, D / 8). That is to say, the dimension of the features obtained by the convolution layer processing of the third feature extraction network is (128, T / 8, D / 8), and the processing process of the convolution modulation network Conv_Modulation can refer to the above. Fig. 9The relevant description is not repeated here. The dimension of the intermediate processing feature matrix output by the third feature extraction network is (128, T / 8, D / 8). Furthermore, the output data of the third feature extraction network can be input into the fourth feature extraction network. The third feature extraction network can include a convolution layer Conv2d_4 and a convolution modulation network Conv_Modulation, wherein Conv2d_4 can extract 256 feature maps with a resolution of (T / 16, D / 16) from the input data. That is to say, the dimension of the features obtained by the convolution layer processing based on the fourth feature extraction network is (256, T / 16, D / 16), and the processing process of the convolution modulation network Conv_Modulation can refer to the above. Fig. 9 The relevant description is not repeated here. The dimension of the intermediate processing feature matrix output by the fourth feature extraction network is (256, T / 16, D / 16). It should be understood that adding Conv_Modulation after each convolutional layer in Conv2d_1-4 is to better extract feature information without noise, or further remove noise.

[0147] Furthermore, the feature decoding network corresponding to each feature extraction network can be used to decode the intermediate feature matrix. In any feature decoding network, a deconvolution layer (i.e., a decoding layer) and a convolution modulation network connected after the deconvolution layer can be included. The deconvolution layer is composed of two-dimensional deconvolution. The convolution modulation network can refer to the above Fig. 9Specifically, when the intermediate processing feature matrix output by the first feature extraction network is feature decoded, the corresponding feature decoding network may include a deconvolution layer ConvTranspose2d_1 that can restore the intermediate processing feature matrix obtained by the first feature extraction network into data with 2 resolutions (T, D), and process it through a convolution modulation network Conv_Modulation to obtain a decoding feature with a dimension of (2, T, D). Similarly, in the feature decoding network corresponding to the second feature extraction network, the intermediate processing feature matrix obtained by the second feature extraction network can be restored to 2 data with resolutions of (T, D) through the deconvolution layer ConvTranspose2d_2, and processed through the convolution modulation network Conv_Modulation to obtain decoding features with a dimension of (2, T, D); in the feature decoding network corresponding to the third feature extraction network, the intermediate processing feature matrix obtained by the third feature extraction network can be restored to 2 data with resolutions of (T, D) through the deconvolution layer ConvTranspose2d_3, and processed through the convolution modulation network Conv_Modulation to obtain decoding features with a dimension of (2, T, D); in the feature decoding network corresponding to the fourth feature extraction network, the intermediate processing feature matrix obtained by the fourth feature extraction network can be restored to 2 data with resolutions of (T, D) through the deconvolution layer ConvTranspose2d_4, and processed through the convolution modulation network Conv_Modulation to obtain decoding features with a dimension of (2, T, D).

[0148] Furthermore, after obtaining the four decoding features, the attention extraction (i.e., Conv2d_5) can be performed on the four decoding features. The purpose is to more efficiently utilize the feature information during post-processing, which helps to effectively improve the audio denoising ability of the audio processing model. In practice, the decoding features (2, T, D) determined by the feature decoding network corresponding to ConvTranspose2d_1-4 are first modified to (1, T, 2D). Here, 2 is used to indicate the real channel and the imaginary channel, so that the feature map indicated by the real data and the feature data indicated by the imaginary data can be merged into a two-dimensional matrix. Then, the two-dimensional matrices (i.e., splicing features) corresponding to each decoding feature can be spliced ​​in the depth dimension to obtain a decoding splicing feature. The dimension of the decoding splicing feature is (4, T, 2D), that is, in the decoding splicing feature, 4 two-dimensional matrices (i.e., splicing matrices) with dimensions of (T, 2D) are included, and one decoding feature corresponds to one two-dimensional matrix. Then, Conv2d_5 (attention mechanism layer) is used to obtain the weight of each decoding feature, and the dimension of the weight is (4, 1, 1). The weight is multiplied by the decoding concatenation feature (4, T, 2D) to obtain the decoding feature information with the weights of each decoding feature added. Then, Conv2d_6 is used to restore the decoding attention feature matrix (2, T, D), and the convolution modulation network Conv_Modulation is used to perform one-step convolution modulation processing to obtain the time-frequency spectrum data after audio denoising (i.e., denoised time-frequency spectrum data). Finally, the denoised time-frequency spectrum data is transformed into a video through the inverse short-time Fourier transform (ISTFT) to restore the audio data after audio denoising of the first audio data, i.e., the second audio data.

[0149] In an embodiment of the present application, when processing audio data, feature extraction can be performed on the time-frequency spectrum data of the audio data to obtain a first feature matrix, and then based on the channel features, time features and frequency band features in the first feature matrix, a first channel weight matrix and a first frequency band weight matrix can be determined. The first channel weight matrix and the first frequency band weight matrix can focus on the feature differences of channels and frequency bands in time features, and then matrix product can be performed based on the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix, which can characterize the weight matrix obtained by fusion of frequency band feature weights and channel feature weights in time features, and then the first feature matrix and the time channel weight matrix can be Hadamard-multiplied to add the time channel weight matrix to the first feature matrix to obtain a second feature matrix, and then the second feature matrix can focus on the feature weight differences in time features, thereby characterizing the potential noise laws in time features, and audio data of various time lengths can be unaffected by the signal-to-noise ratio on the denoising of the audio data, and then denoising can be performed based on the second feature matrix, and noise information in the audio data can be removed, thereby improving the audio denoising quality of the audio data.

[0150] See also Fig.13 , Fig.13 Schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. Fig.13 As shown, the audio processing device 1 can be a computer program (including program code) running on a computer device (such as the server 100a mentioned above), for example, the audio processing device 1 is an application software; it can be understood that the audio processing device 1 can be used to execute the corresponding steps in the data processing method provided in the embodiment of the present application. Fig.13 As shown, the audio processing device 1 may include: a first feature extraction module 11, a weight extraction module 12, a weight fusion module 13, a feature fusion module 14, and an audio denoising module 15;

[0151] A first feature extraction module 11 is used to extract features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features and frequency band features;

[0152] A weight extraction module 12, used to obtain a first channel weight matrix and a first frequency band weight matrix based on channel characteristics, time characteristics and frequency band characteristics;

[0153] A weight fusion module 13, used for performing matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix;

[0154] A feature fusion module 14 is used to perform a Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix;

[0155] The audio denoising module 15 is used to denoise the first audio data based on the second characteristic matrix to obtain second audio data corresponding to the first audio data.

[0156] The weight extraction module 12 is specifically used for:

[0157] Performing matrix transposition on the channel features and time features in the first feature matrix to obtain a third feature matrix;

[0158] Performing a linear transformation on the third characteristic matrix to obtain a second channel weight matrix;

[0159] The second channel weight matrix is ​​normalized to obtain the first channel weight matrix.

[0160] The weight extraction module 12 is specifically used for:

[0161] The time features, channel features and frequency band features in the third feature matrix are linearly transformed to obtain a second channel weight matrix, wherein the second channel weight matrix includes time features, channel features and weight values ​​representing the weights of the channel features.

[0162] The weight extraction module 12 is specifically used for:

[0163] Performing matrix transposition on the channel features, time features, and frequency band features in the first feature matrix to obtain a fourth feature matrix;

[0164] Performing a linear transformation on the fourth characteristic matrix to obtain a second frequency band weight matrix;

[0165] The second frequency band weight matrix is ​​normalized to obtain the first frequency band weight matrix.

[0166] The weight extraction module 12 is specifically used for:

[0167] A linear transformation is performed on the time features, frequency band features and channel features in the fourth feature matrix to obtain a second frequency band weight matrix, wherein the second frequency band weight matrix includes time features, frequency band features and weight values ​​representing weights of the frequency band features.

[0168] The audio processing device 1 further includes: a weight matrix transposition module 16;

[0169] The weight matrix transposition module 16 is specifically used for:

[0170] Perform matrix transposition on the time channel weight matrix to obtain the transposed time channel weight matrix.

[0171] The audio denoising module 15 is specifically used for:

[0172] De-noising the second characteristic matrix to obtain a de-noised second characteristic matrix;

[0173] Performing feature decoding processing on the denoised second feature matrix to obtain a decoded second feature matrix;

[0174] The denoised time-frequency spectrum data of the first audio data is obtained by constructing the decoded second feature matrix;

[0175] Audio restoration is performed on the denoised time-frequency spectrum data to obtain second audio data corresponding to the first audio data.

[0176] See also Fig.14 , Fig.14 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.14 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the user interface 1003 may optionally include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 1005 may optionally also be at least one storage device located away from the aforementioned processor 1001. As Fig.14 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0177] In such Fig.14 In the computer device 1000 shown in the figure, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to execute the above Figure 3 or Figure 8 The description of the data processing method in the embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0178] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the audio processing device 1 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the description of the data processing method in the above embodiment, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0179] The above-mentioned computer-readable storage medium can be the internal storage unit of the audio processing device or the above-mentioned computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media Card, SMC), a secure digital (sSecure digital, SD) card, a flash card (flash Card), etc. equipped on the computer device. Further, the computer-readable storage medium can also include both the internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0180] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 3 or Figure 8 The method provided by the corresponding embodiment. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0181] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0182] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0183] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0184] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Extracting features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features, and frequency band features; Based on the channel characteristics, the time characteristics and the frequency band characteristics, a first channel weight matrix and a first frequency band weight matrix are obtained; Performing a matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix; Performing a Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix; The first audio data is denoised based on the second feature matrix to obtain second audio data corresponding to the first audio data.

2. The method according to claim 1, characterized in that The obtaining a first channel weight matrix based on the channel feature, the time feature and the frequency band feature comprises: Performing matrix transposition on the channel features and time features in the first feature matrix to obtain a third feature matrix; Performing a linear transformation on the third characteristic matrix to obtain a second channel weight matrix; The second channel weight matrix is ​​normalized to obtain the first channel weight matrix.

3. The method according to claim 2, characterized in that The linear transformation of the third feature matrix to obtain a second channel weight matrix includes: The time features, channel features and frequency band features in the third feature matrix are linearly transformed to obtain a second channel weight matrix, wherein the second channel weight matrix includes time features, channel features and weight values ​​representing the weights of the channel features.

4. The method according to claim 1, characterized in that The obtaining a first frequency band weight matrix based on the channel feature, the time feature and the frequency band feature comprises: Performing matrix transposition on the channel features, time features, and frequency band features in the first feature matrix to obtain a fourth feature matrix; Performing a linear transformation on the fourth characteristic matrix to obtain a second frequency band weight matrix; The second frequency band weight matrix is ​​normalized to obtain the first frequency band weight matrix.

5. The method according to claim 4, characterized in that The fourth characteristic matrix is ​​linearly transformed to obtain a second frequency band weight matrix, comprising: A linear transformation is performed on the time features, frequency band features and channel features in the fourth feature matrix to obtain a second frequency band weight matrix, wherein the second frequency band weight matrix includes time features, frequency band features and weight values ​​representing weights of the frequency band features.

6. The method according to claim 1, characterized in that After performing matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix, the method further includes: Perform matrix transposition on the time channel weight matrix to obtain a transposed time channel weight matrix.

7. The method according to claim 1, characterized in that The denoising the first audio data based on the second feature matrix to obtain second audio data corresponding to the first audio data includes: De-noising the second characteristic matrix to obtain a de-noised second characteristic matrix; Performing feature decoding processing on the denoised second feature matrix to obtain a decoded second feature matrix; Obtaining denoised time-frequency spectrum data of the first audio data by constructing the decoded second feature matrix; Audio restoration is performed on the denoised time-frequency spectrum data to obtain second audio data corresponding to the first audio data.

8. An audio processing device, characterized in that: The device comprises: A first feature extraction module, configured to extract features from the time-frequency spectrum data of the first audio data to obtain a first feature matrix of the time-frequency spectrum data; the first feature matrix includes channel features, time features, and frequency band features; A weight extraction module, used for obtaining a first channel weight matrix and a first frequency band weight matrix based on the channel feature, the time feature and the frequency band feature; A weight fusion module, used for performing matrix product of the first channel weight matrix and the first frequency band weight matrix to obtain a time channel weight matrix; A feature fusion module, used for performing a Hadamard product on the first feature matrix and the time channel weight matrix to obtain a second feature matrix; An audio denoising module is used to denoise the first audio data based on the second feature matrix to obtain second audio data corresponding to the first audio data.

9. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded and executed by a processor, so that a computer device having the processor executes the method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises a computer program / instruction, which implements the method according to any one of claims 1 to 7 when executed by a processor.

Images