Audio data processing method and apparatus, device, and medium

Abstract

The embodiment of the application provides an audio data processing method, device and equipment and medium, the method comprises: the target audio data and reference audio data are obtained Fourier transform to obtain relevant time-frequency spectrum data, to carry out feature extraction to obtain relevant amplitude data and phase data, then, after speech optimization processing is carried out to the target original input feature data constructed, the target optimization feature data after reference time-frequency spectrum data as noise data is separated and removed from target time-frequency spectrum data can be obtained, to carry out filtering processing to the fusion time-frequency spectrum data obtained based on the multi-path audio data to which the target audio data belongs, then, Fourier inverse transform is carried out to the filter time-frequency spectrum data corresponding to the target time-frequency spectrum data obtained, that is, the audio optimization data of the target audio data can be obtained and is used to carry out speech control. By adopting the embodiment of the application, noise in the target audio data can be effectively removed, so that the accuracy of speech control is improved.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to an audio data processing method, apparatus, device, and medium. Background Technology

[0002] Currently, for in-vehicle voice systems, after receiving a voice signal (such as a voice control command like "Please turn on the air conditioning") input by the user through a microphone, the in-vehicle device can turn on the air conditioning equipment associated with the in-vehicle device according to the voice control command carried in the voice signal, so as to realize intelligent control of the in-vehicle device to turn on the air conditioning equipment located in the vehicle where the in-vehicle device is located.

[0003] However, the inventors discovered in practice that during vehicle operation, the audio data received by the in-vehicle device can include not only the voice signal of the in-vehicle user transmitted from the voice input device associated with the in-vehicle device (e.g., at least one microphone integrated into the in-vehicle device or at least one microphone independently deployed on the in-vehicle device), but also other noise signals. These include echo signals caused by audio data played in the vehicle by the voice output device associated with the in-vehicle device (e.g., audio system), tire noise caused by tire friction against the ground, and wind noise caused by friction between the vehicle and the air. Therefore, in an in-vehicle voice system, directly using noisy audio data to execute voice control commands will make it difficult to ensure the accuracy of voice control. Summary of the Invention

[0004] This application provides an audio data processing method, apparatus, device, and medium that can effectively separate and optimize received multi-channel voice signals to obtain a noise-removed target voice audio signal. When voice control is performed using the noise-removed target voice audio signal, the accuracy of voice control can also be effectively improved.

[0005] One embodiment of this application provides an audio data processing method, including:

[0006] Obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data; the reference audio data is the noise data in the target audio data, and the target audio data is the audio data in the multi-channel audio data. The target time-spectrum data includes the target real part data and the target imaginary part data obtained after performing a Fourier transform on the target audio data; the reference time-spectrum data includes the reference real part data and the reference imaginary part data obtained after performing a Fourier transform on the reference audio data.

[0007] The target amplitude data and target phase data of the target audio data are determined by using the target real part data and the target imaginary part data, and the reference amplitude data of the reference audio data is determined by using the reference real part data and the reference imaginary part data.

[0008] Based on target amplitude data, target phase data, and reference amplitude data, target original input feature data is constructed. Speech optimization processing is performed on the target original input feature data to obtain target optimized feature data after separating the target time spectrum data and removing the reference time spectrum data.

[0009] Data fusion processing is performed on the time-spectrum data corresponding to the multiple audio data to obtain fused time-spectrum data. The fused time-spectrum data is then filtered using target optimization feature data to obtain filtered time-frequency data corresponding to the target time-spectrum data separated from the fused time-spectrum data. An inverse Fourier transform is performed on the filtered time-spectrum data to obtain target audio optimization data corresponding to the target audio data. Voice control is then performed using the voice control commands carried in the target audio optimization data.

[0010] One embodiment of this application provides an audio data processing apparatus, including:

[0011] The time-spectrum data acquisition module is used to acquire the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data. The reference audio data is the noise data in the target audio data, and the target audio data is the audio data in the multi-channel audio data. The target time-spectrum data includes the target real part data and the target imaginary part data obtained after performing a Fourier transform on the target audio data. The reference time-spectrum data includes the reference real part data and the reference imaginary part data obtained after performing a Fourier transform on the reference audio data.

[0012] The amplitude and phase calculation module is used to determine the target amplitude data and target phase data of the target audio data through the target real part data and the target imaginary part data, and to determine the reference amplitude data of the reference audio data through the reference real part data and the reference imaginary part data.

[0013] The speech optimization processing module is used to construct the original target input feature data based on the target amplitude data, target phase data and reference amplitude data, and perform speech optimization processing on the original target input feature data to obtain the target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data.

[0014] The optimized audio acquisition module is used to perform data fusion processing on the time-spectrum data corresponding to multiple audio data to obtain fused time-spectrum data. The fused time-spectrum data is then filtered using target optimization feature data to obtain filtered time-frequency data corresponding to the target time-spectrum data separated from the fused time-spectrum data. An inverse Fourier transform is performed on the filtered time-spectrum data to obtain target audio optimization data corresponding to the target audio data. Voice control is then performed using the voice control commands carried in the target audio optimization data.

[0015] The target audio data includes first audio data and second audio data; the target time-spectrum data includes the first time-spectrum data corresponding to the first audio data and the second time-spectrum data corresponding to the second audio data; the target real part data includes the first real part data and the second real part data; the target imaginary part data includes the first imaginary part data and the second imaginary part data.

[0016] The amplitude and phase calculation module includes:

[0017] The first amplitude and phase calculation unit is used to obtain an amplitude calculation function for amplitude calculation and a phase calculation function for phase calculation. The amplitude calculation function is used to calculate the amplitude of the first real part data and the first imaginary part data to obtain the first amplitude data of the first audio data. The phase calculation function is used to calculate the phase of the first real part data and the first imaginary part data to obtain the first phase data of the first audio data.

[0018] The second amplitude and phase calculation unit is used to perform amplitude calculation on the second real part data and the second imaginary part data through the amplitude calculation function to obtain the second amplitude data of the second audio data, and to perform phase calculation on the second real part data and the second imaginary part data through the phase calculation function to obtain the second phase data of the second audio data.

[0019] The target amplitude and phase determination unit is used to determine the target amplitude data of the target audio data in the time dimension based on the first amplitude data and the second amplitude data, and to determine the target phase data of the target audio data in the frequency dimension based on the first phase data and the second phase data.

[0020] The reference amplitude calculation unit is used to perform amplitude calculations on the reference real part data and the reference imaginary part data using an amplitude calculation function to obtain the reference amplitude data of the reference audio data.

[0021] The voice optimization processing module includes:

[0022] A phase data acquisition unit is used to acquire first phase data and second phase data from target phase data;

[0023] A phase difference calculation unit is used to calculate the phase difference between the first phase data and the second phase data to obtain the first phase difference.

[0024] The first original input construction unit is used to construct the target original input feature data based on the first amplitude data, the second amplitude data, the reference amplitude data, and the first phase difference data.

[0025] The number of audio data streams is N, where N is a positive integer greater than 2. The target phase data includes N phase data. The N phase data are obtained by performing phase calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams. Each audio data stream corresponds to one time-frequency data. The target amplitude data includes N amplitude data. The N amplitude data are obtained by performing amplitude calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams.

[0026] The voice optimization processing module also includes:

[0027] A phase data pair construction unit is used to construct M phase data pairs associated with N phase data based on any two phase data obtained from N phase data; M is a positive integer greater than or equal to N.

[0028] A phase data determination unit is used to determine a target phase data pair from M phase data pairs, and to determine the two phase data in the target phase data pair as the first phase data to be processed and the second-generation processed phase data, respectively.

[0029] The phase difference calculation unit is used to calculate the phase difference between the first phase data to be processed and the second phase data to be processed, and to obtain the target phase difference of the target phase data pair. The target phase difference of the M phase data pairs is obtained when each of the M phase data pairs is determined to be the target phase data pair.

[0030] The second original input construction unit is used to construct the target original input feature data based on the target phase difference of N amplitude data, reference amplitude data and M phase data pairs.

[0031] The voice optimization processing module also includes:

[0032] The data encoding unit is used to acquire the target speech model for speech optimization processing, and to perform data encoding processing on the target original input feature data through the target speech model to obtain the encoded feature data corresponding to the target original input feature data.

[0033] The feature concatenation unit is used to concatenate the target's original input feature data and encoded feature data using the target speech model to obtain concatenated feature data.

[0034] The data decoding unit is used to perform data decoding processing on the spliced ​​feature data through the target speech model to obtain the decoded feature data corresponding to the spliced ​​feature data.

[0035] The sequence extraction unit is used to extract sequence information from the decoded feature data through the target speech model to obtain the sequence extracted feature data corresponding to the decoded feature data. The sequence extracted feature data is used as the target optimized feature data after separating and removing the reference time-spectrum data from the target time-spectrum data.

[0036] The target speech model includes an encoder consisting of multiple unit coding layers;

[0037] The data encoding unit can also be used to acquire the target speech model for speech optimization processing, and input the target raw input feature data into the encoder in the target speech model;

[0038] The data encoding unit is also used to obtain the first unit encoding layer and the second unit encoding layer from the multiple unit encoding layers contained in the encoder.

[0039] The data encoding unit is also used to perform a first encoding process on the original input feature data through the first unit encoding layer to obtain first encoded feature data, and to perform a second encoding process on the first encoded feature data through the second unit encoding layer to obtain second encoded feature data; the feature dimension of the second encoded feature data is different from the feature dimension of the first encoded feature data.

[0040] The data encoding unit is also used to determine the encoded feature data corresponding to the target original input feature data based on the second encoded feature data.

[0041] In this system, each unit coding layer corresponds to a coding sequence number. Each unit coding layer contains a convolutional layer, and the number of filters in a convolutional layer increases with the coding sequence number. When the coding sequence number of the first unit coding layer is less than that of the second unit coding layer, the number of filters in the first convolutional layer is less than that in the second convolutional layer.

[0042] The data encoding unit is also used to determine the convolutional layer in the first unit encoding layer as the first convolutional layer, and to determine the convolutional layer in the second unit encoding layer as the second convolutional layer;

[0043] The data encoding unit is further configured to perform a first convolution process on the original input feature data through the filters in the first convolutional layer to obtain first convolutionally encoded feature data, perform a first normalization process on the first convolutionally encoded feature data to obtain first normalized feature data, and perform a first encoding activation process on the first normalized feature data to obtain first encoded feature data; the number of channels in the channel feature dimension of the first encoded feature data is consistent with the number of filters in the first convolutional layer;

[0044] The data encoding unit is further configured to perform a second convolution process on the first encoded feature data through filters in the second convolutional layer to obtain second convolutionally encoded feature data; perform a second normalization process on the second convolutionally encoded feature data to obtain second normalized feature data; and perform a second encoding activation process on the second normalized feature data to obtain second encoded feature data. The number of channels in the second encoded feature data in the channel feature dimension is consistent with the number of filters in the second convolutional layer. The number of channels in the second encoded feature data in the channel feature dimension is greater than the number of channels in the first encoded feature data in the channel feature dimension.

[0045] The encoder contains a linear layer for dimensional transformation of the data;

[0046] The data encoding unit is also used to determine the third encoded feature data corresponding to the target original input feature data based on the second encoded feature data;

[0047] The data encoding unit is also used to perform a linear transformation on the feature map corresponding to the third encoded feature data in the channel dimension through the linear layer of the encoder, and use the linearly transformed third encoded feature data as the encoded feature data corresponding to the target original input feature data; the size of the feature map corresponding to the linearly transformed third encoded feature data is consistent with the size of the feature map corresponding to the third encoded feature data.

[0048] The feature dimensions of both the encoded feature data and the target raw input feature data include channel dimension, time dimension, and frequency dimension. The encoded feature data is determined by K1 encoded feature maps, where K1 represents the number of channels in the encoded feature data in the channel dimension. The encoded feature maps are composed of feature extraction data in the time dimension and feature extraction data in the frequency dimension. The target raw input feature data is determined by K2 input feature maps, where K2 represents the number of channels in the target raw input feature data in the channel dimension. The input feature maps are composed of feature input data in the time dimension and feature input data in the frequency dimension. K1 and K2 are both positive integers.

[0049] The feature splicing unit is also used to copy K1 encoded feature maps based on K2 channels to obtain K2 encoded feature maps;

[0050] The feature concatenation unit is also used to concatenate the feature extraction data from the K2 encoded feature maps and the feature input data from the K2 input feature maps in the frequency dimension to obtain K2 frequency concatenated feature maps.

[0051] The feature splicing unit is also used to determine the K2 frequency spliced ​​feature maps as spliced ​​feature data obtained by splicing the original input feature data and encoded feature data of the target.

[0052] The target speech model includes a decoder consisting of multiple unit decoding layers;

[0053] The data decoding unit is also used to obtain the first unit decoding layer and the second unit decoding layer from the multiple unit decoding layers contained in the decoder;

[0054] The data decoding unit is also used to perform a first decoding process on the spliced ​​feature data through the first unit decoding layer to obtain the first decoded feature data, and to perform a second decoding process on the first decoded feature data through the second unit decoding layer to obtain the second decoded feature data;

[0055] The data decoding unit is also used to determine the decoded feature data corresponding to the spliced ​​feature data based on the second decoded feature data.

[0056] Among the multiple unit decoding layers, each unit decoding layer contains a deconvolution layer;

[0057] The data decoding unit is also used to determine the deconvolution layer in the first unit decoding layer as the first deconvolution layer, and to determine the deconvolution layer in the second unit decoding layer as the second deconvolution layer;

[0058] The data decoding unit is also used to perform a first deconvolution process on the spliced ​​feature data through a first deconvolution layer to obtain first deconvolution decoded feature data, perform a third normalization process on the first deconvolution decoded feature data to obtain third normalized feature data, and perform a first decoding activation process on the third normalized feature data to obtain first decoded feature data.

[0059] The data decoding unit is also used to perform a second deconvolution process on the first decoded feature data through the second deconvolution layer to obtain the second deconvolution decoded feature data, perform a fourth normalization process on the second deconvolution decoded feature data to obtain the fourth normalized feature data, and perform a second decoding activation process on the fourth normalized feature data to obtain the second decoded feature data.

[0060] The target speech model includes a sequence information extractor consisting of multiple sequence modeling layers; a sequence modeling layer is used to extract sequence information from an input sequence data under a historical state data to obtain an output sequence data under a target state data; a target state data is used to characterize the latent state feature data obtained in the process of extracting sequence information from an input sequence data in a sequence modeling layer.

[0061] The sequence extraction unit is further configured to obtain a first sequence modeling layer and a second sequence modeling layer from the sequence modeling layer contained in the sequence information extractor; the historical state data corresponding to the first sequence modeling layer is the first historical state data, the historical state data corresponding to the second sequence modeling layer is the second historical state data, and the second historical state data is determined based on the first target state data obtained in the first sequence modeling layer;

[0062] The sequence extraction unit is also used to take the decoded feature data as the first input sequence data, input the first input sequence data and the first historical state data into the first sequence modeling layer, and perform first sequence feature extraction processing on the first input sequence data under the first historical state data to obtain the first hidden state feature data. When the first hidden state feature data is used as the first target state data, the first hidden state feature data is subjected to second sequence feature extraction processing to obtain the first output sequence data under the first target state data.

[0063] The sequence extraction unit is also used to use the first output sequence data as the second input sequence data corresponding to the second sequence modeling layer, and to use the first target state data as the second historical state data corresponding to the second sequence modeling layer.

[0064] The sequence extraction unit is also used to input the second input sequence data and the second historical state data into the second sequence modeling layer, and the second sequence modeling layer performs a third sequence feature extraction process on the second input sequence data under the second historical state data to obtain the second hidden state feature data. When the second hidden state feature data is used as the second target state data, a fourth sequence feature extraction process is performed on the second hidden state feature data to obtain the second output sequence data under the second target state data.

[0065] The sequence extraction unit is also used to determine the sequence extraction feature data corresponding to the decoded feature data based on the second output sequence data.

[0066] The first sequence modeling layer includes a first linear layer, a first sequence extraction convolutional layer, a second linear layer, and a normalization layer.

[0067] The sequence extraction unit is also used to take the decoded feature data as the first input sequence data, input the first input sequence data and the first historical state data into the first sequence modeling layer, and under the first historical state data, the first linear layer performs a first linear transformation on the first input sequence data to obtain the first linear transformation feature data;

[0068] The sequence extraction unit is also used to perform convolution processing on the first linear transformation feature data through the first sequence extraction convolution layer to obtain the first intermediate convolution feature data;

[0069] The sequence extraction unit is also used to perform feature summation processing on the first intermediate convolutional feature data and the first historical state data to obtain the first hidden state feature data. When the first hidden state feature data is used as the first target state data, the first hidden state feature data is subjected to a second linear transformation through the second linear layer to obtain the second linear transformation feature data.

[0070] The sequence extraction unit is also used to normalize the second linear transformation feature data through a normalization layer, and use the normalized second linear transformation feature data as the first output sequence data output by the first sequence modeling layer under the first target state data.

[0071] The voice optimization processing module also includes:

[0072] The sample audio data acquisition unit is used to acquire a sample audio data set for training the initial speech model; the sample audio data set includes first sample audio data, sample reference audio data, and second sample audio data obtained by adding noise to the first sample audio data; the second sample audio data is audio data from multiple sample audio data.

[0073] The sample time-spectrum acquisition unit is used to acquire the sample time-spectrum data corresponding to the second sample audio data and the sample reference time-spectrum data corresponding to the sample reference audio data; the sample reference audio data is the noise data in the second sample audio data; the sample time-spectrum data includes the sample real part data and sample imaginary part data obtained after performing a Fourier transform on the second sample audio data; the sample reference time-spectrum data includes the sample reference real part data and sample reference imaginary part data obtained after performing a Fourier transform on the sample reference audio data.

[0074] The sample data feature extraction unit is used to determine the sample amplitude data and sample phase data of the second sample audio data through the sample real part data and sample imaginary part data, and to determine the sample reference amplitude data of the sample reference audio data through the sample reference real part data and sample reference imaginary part data.

[0075] The sample audio optimization unit is used to construct the original input feature data of the sample based on the sample amplitude data, sample phase data and sample reference amplitude data, obtain the original speech model to be trained, and perform speech optimization processing on the original input feature data of the sample through the original speech model to obtain the optimized feature data of the sample.

[0076] The sample audio filtering unit performs data fusion processing on the time-spectrum data corresponding to the multiple sample audio data to obtain sample fused time-spectrum data. It then filters the sample fused time-spectrum data using sample optimization feature data to obtain sample filtered time-spectrum data corresponding to the sample time-spectrum data separated from the sample fused time-spectrum data. Finally, it performs an inverse Fourier transform on the sample filtered time-spectrum data to obtain sample audio optimization data corresponding to the second sample audio data.

[0077] The original speech model update unit is used to calculate the data loss value through the first sample audio data and the sample audio optimization data, and to iteratively update the original speech model through the data loss value, so that the updated original speech model can be used as the target speech model.

[0078] Among them, the sample audio optimization unit is also used to perform a logarithmic operation on the sample amplitude data to obtain the sample amplitude logarithmic data, and to perform a logarithmic operation on the sample reference amplitude data to obtain the sample reference amplitude logarithmic data.

[0079] The sample audio optimization unit is also used to normalize the logarithmic data of sample amplitude to obtain sample normalized amplitude data, and to normalize the logarithmic data of reference amplitude to obtain reference normalized amplitude data. Based on the sample normalized amplitude data, sample phase data and reference normalized amplitude data, the original input feature data of the sample is constructed.

[0080] One aspect of this application provides a computer device, including a memory and a processor. The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to call the computer programs so that the computer device executes the method provided in one aspect of this application.

[0081] One aspect of this application provides a computer-readable storage medium storing a computer program adapted to be loaded and executed by a processor, so that a computer device having a processor performs the method provided in one aspect of this application.

[0082] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above aspect.

[0083] This application embodiment can perform short-time Fourier transform on the target audio data (e.g., audio signals from various microphones) and reference audio data (e.g., echo signals from speakers) upon acquisition, to obtain target time-spectrum data corresponding to the target audio data and reference time-spectrum data corresponding to the reference audio data. It is understood that both the target time-spectrum data and the reference time-spectrum data are essentially time-spectrum data composed of real and imaginary data. Furthermore, the target audio data can be audio data extracted from any one of multiple audio signals. Based on this, this application embodiment can convert audio data represented in the time domain into time-spectrum data represented in both the time and frequency domains using short-time Fourier transform. Then, the real and imaginary data in the converted time-spectrum data can be used to achieve intelligent extraction of relevant feature data. For example, this application embodiment can use the target real and target imaginary data present in the target time-spectrum data to calculate the target amplitude data used to characterize amplitude features and the target phase data used to characterize phase features in the target audio data. Similarly, in embodiments of this application, when obtaining reference time-spectrum data, reference amplitude data used to characterize another amplitude feature in the reference audio data can be calculated using the reference real part data and reference imaginary part data present in the reference time-spectrum data. Furthermore, embodiments of this application can construct target original input feature data based on the target amplitude data, the target phase data, and the reference amplitude data. By performing speech optimization on the target original input feature data, reference time-spectrum data, which serves as noise data, can be separated and removed from the target time-spectrum data to finally obtain noise-removed target optimized feature data. Furthermore, in this embodiment, when data fusion is performed on the time-spectrum data corresponding to multiple audio data (e.g., audio data from multiple microphones) to obtain fused time-spectrum data, the fused time-spectrum data can be further filtered using the denoised target optimization feature data. This yields filtered time-spectrum data corresponding to the target time-spectrum data separated from the fused time-spectrum data. Thus, even if other audio data besides the target audio data exists in the fused time-spectrum data, filtering can be used to filter and suppress these other audio data. This intelligently selects the filtered time-spectrum data (essentially a type of time-spectrum data obtained after filtering) corresponding to the target audio data. By performing an inverse Fourier transform on the filtered time-spectrum data, the target audio optimization data corresponding to the target audio data can be obtained. In this way, when voice control is performed using the voice control commands carried in the target audio optimization data, the accuracy of voice control can be effectively improved. Attached Figure Description

[0084] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0086] Figure 2 This is a schematic diagram of a data interaction scenario provided in an embodiment of this application;

[0087] Figure 3 This is a schematic diagram of an audio data processing method provided in an embodiment of this application;

[0088] Figure 4 This is a schematic diagram of a process for extracting time-spectrum data features according to an embodiment of this application;

[0089] Figure 5 This is a schematic diagram illustrating a scenario where a target speech model encoder performs encoding processing, as provided in an embodiment of this application.

[0090] Figure 6 This is a schematic diagram illustrating a scenario where the convolutional layer of an encoder performs a convolution operation, as provided in an embodiment of this application.

[0091] Figure 7 This is a schematic diagram illustrating a scenario where a target speech model decoder performs decoding processing according to an embodiment of this application;

[0092] Figure 8 This is a schematic diagram illustrating a sequence extraction scenario provided by an embodiment of this application using a sequence information extractor.

[0093] Figure 9 This is a schematic diagram of another audio data processing method provided in an embodiment of this application;

[0094] Figure 10 This is a schematic diagram of a feature splicing scenario provided in an embodiment of this application;

[0095] Figure 11 This is a schematic diagram of the training process of an original speech model provided in an embodiment of this application;

[0096] Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application;

[0097] Figure 13This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0098] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0099] Please see Figure 1 , Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application. Figure 1 As shown, this network architecture may include a server 10d and a user terminal cluster. The user terminal cluster may include one or more user terminals; the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster can specifically include user terminal 10a, user terminal 10b, and user terminal 10c. Server 10d can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. User terminals 10a, 10b, and 10c can all include: smartphones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices (such as smartwatches and smart bracelets), smart voice interaction devices, smart home appliances (such as smart TVs), and in-vehicle devices—electronic devices with audio and video playback functions. Figure 1 As shown, user terminals 10a, 10b, and 10c can each connect to server 10d via a network, so that each user terminal can interact with server 10d through the network connection.

[0100] In this embodiment, by extracting features from the obtained target audio data and reference audio data, the target original input feature data suitable for speech optimization processing can be obtained. It is understood that the device performing feature extraction here can be... Figure 1User terminals 10a, 10b, and 10c are included. Taking user terminal 10a as an example, after user terminal 10a receives target audio data (e.g., a voice signal input by the user through a microphone), it transforms the target audio data and reference audio data (e.g., the echo signal collected by user terminal 10a at the speaker) into corresponding time-spectrum data (i.e., target time-spectrum data and reference time-spectrum data) through Fourier transform. Then, it calculates the relevant amplitude data and phase data through the real and imaginary data in the time-spectrum data, thereby constructing the target original input feature data that can be used for speech optimization processing.

[0101] Understandably, the speech optimization processing of the original target input feature data here can use a pre-trained target speech model. This target speech model is constructed using a neural network and performs a series of steps on the original target input feature data, including global feature extraction, encoding, decoding, and sequence information extraction. Finally, it outputs the optimized target feature data after separating and removing the reference time-spectral data from the target time-spectral data. Understandably, this target speech model can exist within... Figure 1 In server 10d, the target speech model is trained. After user terminals 10a, 10b, and 10c extract features from the target and reference audio data, the obtained raw target input feature data can be input into the trained target speech model on server 10d for speech optimization. After obtaining the optimized target feature data, server 10d can return the optimized target feature data to the user terminals that sent the raw target input feature data (e.g., user terminals 10a, 10b, and 10c) for subsequent filtering. Alternatively, taking user terminal 10a for feature extraction as an example, server 10d can directly send the trained target speech model to user terminal 10a, allowing user terminal 10a to directly use the target speech model to complete the relevant speech optimization. After obtaining the target optimization feature data, the user terminal 10a can use the target optimization feature data to perform relevant filtering processing on the fused time-spectrum data corresponding to the multi-channel audio data to which the target audio data belongs, so as to separate the filtered time-spectrum data corresponding to the target time-spectrum data. Then, the filtered time-spectrum data is subjected to an inverse Fourier transform to obtain the target audio optimization data corresponding to the target audio data. Finally, the voice control commands carried in the target audio optimization data can be used for voice control.

[0102] For further details, please see Figure 2 , Figure 2 This is a schematic diagram illustrating a data interaction scenario provided in an embodiment of this application. For example... Figure 2As shown, in this embodiment, after receiving multiple target audio data (i.e., audio data 28a, audio data 28b, ..., audio data 28n), the multiple target audio data and the reference audio data 21a are subjected to Fourier transforms respectively to obtain multiple target time-spectrum data (i.e., time-spectrum data 29a, time-spectrum data 29b, ..., time-spectrum data 29n) corresponding to the multiple target audio data and the reference time-spectrum data 22a corresponding to the reference audio data 21a. Here, "multiple target audio data" can be understood as multiple target audio data. Taking an in-vehicle system as an example, when a certain in-vehicle microphone receives a voice signal, it will send the received voice signal as one target audio data to the in-vehicle device for subsequent voice optimization processing. However, if multiple in-vehicle microphones receive voice signals and send them to the in-vehicle device, the in-vehicle device will receive multiple target audio data, or in other words, the in-vehicle device will receive multiple target audio data sent to it by multiple microphones. Therefore, in this embodiment, "multiple target audio data" can also be referred to as "multiple target audio data". It is understandable that the reference audio data 21a here refers to the noise data carried in these multiple target audio data. For example, when an in-vehicle device with an in-vehicle voice system receives a voice signal input by a user in the vehicle through a microphone, the voice signal may carry a sound signal emitted by the speaker. The audio signal of the speaker at this time is the reference audio data 21a. When the user terminal obtains the target time-spectrum data (time-spectrum data 29a, time-spectrum data 29b, ..., time-spectrum data 29n) and reference time-spectrum data 22a corresponding to the multiple audio data, they can be input into the feature extraction unit for relevant feature extraction. It is understandable that the feature extraction unit mainly uses the real and imaginary data in each of the target time-spectrum data and the reference time-spectrum data 22a to calculate the relevant amplitude and phase data, so as to obtain the amplitude and phase features corresponding to these multiple target audio data and the amplitude features corresponding to the reference audio data 21a. Furthermore, the target original input feature data 24a is constructed. Each target time-spectrum data here can be transformed into a corresponding target time-spectrum data after Fourier transform. For example, audio data 28a can be transformed into corresponding time-spectrum data 29a.

[0103] After obtaining the target's original input feature data 24a, the target's original input feature data 24a can be input into, for example... Figure 2The target speech model shown is used for related speech optimization processing. This target speech model can exist on a user terminal or on a server; this is not limited here. After the target speech model obtains the original target input feature data 24a, its global feature extraction unit (encoder) performs related encoding processing (i.e., global feature extraction) to obtain the encoded feature data corresponding to the original target input feature data 24a. The target speech model then concatenates the encoded feature data and the original target input feature data 24a to obtain concatenated feature data, which is then input into the decoder for decoding to obtain decoded feature data. Finally, a sequence information extractor performs sequence information extraction processing on the decoded feature data to remove noise features and obtain the target optimized feature data 25a. The target optimized feature data 25a is the optimized feature data obtained after separating these multiple target time-spectral data and removing the reference time-spectral data 22a.

[0104] The target optimized feature data 25a output by the target speech model can also be regarded as a filter that can be filtered. At this time, the fused time-spectrum data 23a can be filtered by the target optimized feature data 25a to obtain multiple filtered time-spectrum data (i.e., filtered time-spectrum data 26a, filtered time-spectrum data 26b, ..., filtered time-spectrum data 26n). It can be understood that the fused time-spectrum data 23a is generated by the time-spectrum data (i.e., time-spectrum data 29a, time-spectrum data 29b, ..., time-spectrum data 29n) corresponding to these multiple target audio data. By fusing the elements at the same position of these multiple time-spectrum data, the fused time-spectrum data 23a can be obtained. At this time, the fused time-spectrum data 23a can be understood as concatenating the real and imaginary parts of these multiple time-spectrum data according to their vector dimensions to form real and imaginary vector data about these multiple time-spectrum data. It can be understood that the fused time-spectrum data 23a obtained at this time also contains the relevant information of the time-spectrum data corresponding to the multiple target audio data. Furthermore, the target optimization feature data 25a can filter the fused time-spectrum data 23a, which contains time-spectrum data corresponding to multiple target audio data, to separate the filtered time-spectrum data corresponding to each of the multiple target spectral data. For example, the filtered time-spectrum data 26a can be understood as the filtered time-spectrum data corresponding to the time-spectrum data 29a filtered from the fused time-spectrum data 23a. At this time, the filtered time-spectrum data of the multiple target audio data are subjected to inverse Fourier transform, transforming them from time-spectrum data into audio data to obtain multiple target audio optimization data (i.e., target audio optimization data 27a, target audio optimization data 27b, ..., target audio optimization data 27n). It can be understood that each of the multiple target audio optimization data is the audio optimization data of a target audio data. For example, the target audio optimization data 27a is obtained by performing an inverse Fourier transform on the filtered time-spectrum data 26a, therefore, the target audio optimization data 27a is the audio optimization data corresponding to the audio data 28a in the target audio data. The multiple audio optimization data obtained here can be used to perform related voice control by using the voice commands they carry.

[0105] In this embodiment, by performing relevant Fourier transforms on the input multi-channel audio data containing target audio data and reference audio data, and extracting relevant data features, target original input feature data that is convenient for the target speech model to process can be obtained. The target speech model can then use a global feature extraction unit (i.e., encoder) to perform global feature extraction (i.e., encoding) on ​​the target original input feature data, obtaining more holistic feature data (i.e., encoded feature data) about the multi-channel audio data. This encoded feature data can better represent the features of the target audio data over a larger range, thus improving the denoising effect of the target speech model. The target speech model then concatenates the encoded feature data and the target original input feature data, and fuses them using a decoder (i.e., encoding) to obtain decoded feature data that simultaneously contains local feature data (i.e., target original input feature data) and global feature data (i.e., encoded feature data). This allows the sequence information extractor to better extract sequence information from the decoded feature data and remove noise (including relevant feature data from the reference audio data) to obtain the target optimized feature data. The fused time-spectrum data formed by the time-spectrum data of multiple audio data is further filtered using target optimization feature data to separate the filtered time-spectrum data corresponding to the target time-spectrum data from the time-spectrum data of the multiple audio data, thereby achieving the effect of separating the input multiple audio data. Finally, the target audio optimization data obtained by performing an inverse Fourier transform on the filtered time-spectrum data is the audio optimization data corresponding to the target time-frequency data obtained in this embodiment of the application, which can then be used for related voice control.

[0106] It is understood that in the specific implementation of this application, user business data (e.g., user voice and audio data) may be involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0107] For further details, please see Figure 3 , Figure 3 This is a schematic diagram illustrating an audio data processing method provided in an embodiment of this application. It can be understood that... Figure 3 The audio data processing method shown can be performed by a user terminal (the user terminal can be, for example, ...). Figure 1 The user terminal 10a shown can be used to execute the command, or a user terminal federation server can be used (the server here can be...). Figure 1 The server 10d shown is used in conjunction with the computer device executing the audio data processing method, and no specific limitations will be imposed on it here. For ease of understanding, this embodiment of the application takes the computer device executing the audio data processing method as the user terminal as an example. Figure 3 As shown, the audio data processing method may include at least the following: Figure 3 Steps S101 to S104 in the process.

[0108] Step S101: Obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data;

[0109] Specifically, after a user terminal (e.g., a computer device integrated with an in-vehicle voice system, such as an in-vehicle terminal or in-vehicle device) receives the target audio data, it can perform a Fourier transform on the target audio data along with the reference audio data to obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data. Here, the target audio data refers to one or more audio data streams from a multi-channel audio dataset. Taking an in-vehicle device as an example, a typical home in-vehicle device can simultaneously receive four channels of target audio data (i.e., microphone signal data). (Generally, there is one microphone per seat; when passengers in all four seats speak into the microphones simultaneously, the in-vehicle device can simultaneously receive the corresponding audio data from all four microphones.) This can also be understood as the in-vehicle device simultaneously receiving four target audio data streams. However, the four received target audio data streams may interfere with each other. For example, the voice received by the microphone of seat A in a car may simultaneously include the voices of passengers in seat A and other seats (e.g., seat B, seat C, and seat D). Therefore, the in-vehicle equipment needs to analyze all received microphone signals to separate the voice signal of the user in seat A for subsequent use. In this case, the voice signal received by the microphone of seat A and sent to the in-vehicle equipment can be considered as one target audio data stream. The in-vehicle equipment needs to separate the target audio data corresponding to the microphone signal of seat A (i.e., the optimized audio data corresponding to the target audio data) from the microphone signals from seats A, B, C, and D (i.e., multiple target audio data streams). Of course, the microphone signals of seats B, C, and D can also be considered as the microphone signal of seat A, and they can be separated using the embodiments of this application. Therefore, this invention can ultimately achieve the effect of partitioning the input in-vehicle microphone signal. Therefore, the embodiments of this application perform Fourier transform on multiple audio data streams to obtain the time-spectrum data corresponding to the multiple audio data streams for subsequent separation steps.

[0110] The reference audio data here can be understood as noise data within the target audio data. Taking an in-vehicle device as an example, when the user in seat A speaks into the microphone, the car's audio system may be playing audio, such as music. This music audio, along with the speech from the user in seat A, is input to the microphone in seat A. The microphone in seat A then sends the speech signal to the in-vehicle device. This audio emitted by the audio system can be considered reference audio data, as it interferes with the user's speech. The speech signal collected by the microphone in seat A (i.e., the target audio signal) contains noise generated by the audio emitted by the audio system. Therefore, it is necessary to collect the audio data emitted by the audio system at this time as reference audio data. The reference audio data, along with the target audio data, undergoes a Fourier transform for subsequent feature extraction, enabling the target speech model provided in this embodiment to better remove noise data (i.e., reference audio data) from the target audio data.

[0111] The Fourier transform performed after receiving the target and reference audio data is a short-time Fourier transform (SFT). This divides the audio data into multiple small windows based on time, and performs a Fourier transform on each window within that time period. Understandably, a typical Fourier transform divides the audio data into real and imaginary parts in the frequency dimension. The SFT divides the audio data into real and imaginary data in both the time and frequency dimensions. Each unit in the time dimension represents one of the initially divided small windows. If the real and imaginary data obtained from the target and reference audio data have a time dimension value of T, it means there are T small windows in the SFT. If the real and imaginary data obtained from the target and reference audio data have a frequency dimension value of W, it means the Fourier transform results in data at W frequency points, where T and W are positive integers. At this point, the real and imaginary data in the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data can be regarded as matrix data with row length T and column length W. However, each cell in the matrix stores the real and imaginary parts of a certain imaginary number. The imaginary number here is the imaginary number obtained at a certain frequency point after the audio data undergoes Fourier transform.

[0112] Step S102: Determine the target amplitude data and target phase data of the target audio data using the target real part data and the target imaginary part data, and determine the reference amplitude data of the reference audio data using the reference real part data and the reference imaginary part data.

[0113] Specifically, after obtaining the target real part and target imaginary part data from the target time-spectrum data corresponding to the target audio data, the target amplitude data and target phase data corresponding to the target audio data can be calculated using these target real part and target imaginary part data. Similarly, after obtaining the reference real part and reference imaginary part data from the reference time-spectrum data corresponding to the reference audio data, the reference amplitude data of the reference audio data can also be calculated using these reference real part and reference imaginary part data. Regarding the calculation of amplitude and phase data using the real and imaginary part data, specific amplitude calculation functions and phase calculation functions can be used.

[0114] The amplitude calculation function is shown below:

[0115] mag = (y real 2 +y imag 2 ) 0.5 Formula (1)

[0116] In formula (1), the parameter y real The parameter y is used to characterize the real part of a certain time-frequency spectrum. imag That is, the imaginary part of the time-spectrum data is used to characterize the real part of the time-spectrum data, y. real And imaginary part data y imag The amplitude data mag corresponding to the time spectrum data can then be calculated.

[0117] The phase calculation function here is as follows:

[0118] phase = atan2(y imag ,y real ) Formula (2)

[0119] In formula (2), the parameter y real and parameter y imag Similar to formula (1), it is used to characterize the real and imaginary parts of the time-frequency spectrum data. Here, the phase is also represented by the real part y of the time-frequency spectrum data. real And imaginary part data y imag The phase data corresponding to the calculated time-frequency spectrum data.

[0120] In this embodiment, after obtaining the target time-spectrum data corresponding to the target audio data, the target amplitude data and target phase data corresponding to the target audio data can be calculated using the above formula (1) (i.e., amplitude calculation function) and formula (2) (i.e., phase calculation function). It should be noted that the target real part data and target imaginary part data in the target time-spectrum data can both be considered as a matrix, where each element stores the real or imaginary value of a certain imaginary number. When calculating the amplitude and phase using the above formula (1) and formula (2), the real and imaginary values ​​at each position of the real and imaginary matrices are extracted and denoted as yi. real and y imag The amplitude and phase at that position are calculated. Therefore, the target amplitude data and target phase data obtained by calculating the target real part data and target imaginary part data in the target time spectrum data are also two matrix data with the same size as the target real part data and target imaginary part data. For example, if the matrix dimension of the target real part data and the target imaginary part data are both T×W, then the matrix dimension of the target amplitude data and the target phase data obtained at this time are also T×W. Here, T can also be understood as the value of the data in the time dimension, and W can also be understood as the value of the data in the frequency dimension. As for the calculation of the reference amplitude data of the reference audio data, it can be understood as substituting the reference real part data and the reference imaginary part data in the reference time spectrum data into the above formula (1), at this time y real and y imag These are the reference real data and the reference imaginary data, respectively. The specific calculation process can be found in the description of the target amplitude data calculation process above, which will not be repeated here. It can be understood that the reference amplitude data obtained here can also be regarded as a matrix data, and its size is the same as that of the target amplitude data and the target phase data.

[0121] Of course, since the target audio data here is audio data from multiple audio data streams, the calculated target amplitude data and target phase data may contain multiple streams. If the target audio data contains two audio data streams, let these two audio data streams be the first audio data and the second audio data, then the target time spectrum data includes the first time spectrum data corresponding to the first audio data and the second time spectrum data corresponding to the second audio data. When calculating the target amplitude data and target phase data of the target time spectrum data, the amplitude of the first real part data and the first imaginary part data contained in the first time spectrum data is calculated using the above formula (1) to obtain the first amplitude data of the first audio data, and the phase of the first real part data and the first imaginary part data is calculated using the above formula (2) to obtain the first phase data of the first audio data. The calculation of the second amplitude data and the second phase data of the second audio data can also refer to the above process of calculating the first amplitude data and the first phase data, which will not be repeated here. Since the target audio data contains the first audio data and the second audio data, the target audio data obtained here includes the first amplitude data and the second amplitude data, and the target phase data also includes the first phase data and the second phase data. It is understood that the above example of target audio data containing two audio data channels may include more channels (e.g., 3 channels, 4 channels, etc.) during the operation of this application embodiment, and will not be exemplified here.

[0122] Step S103: Based on the target amplitude data, target phase data and reference amplitude data, the original target input feature data is constructed. The original target input feature data is then subjected to speech optimization processing to obtain the target optimized feature data after separating the target time spectrum data and removing the reference time spectrum data.

[0123] Specifically, after obtaining the target amplitude data, target phase data, and reference amplitude data through step S102, the target original input feature data can be constructed based on them. Here, the obtained target amplitude data and reference amplitude data are logarithmized and mean normalized. It is understood that, in this embodiment, after obtaining the target original input feature data, the original input feature data is input into the target speech model for speech optimization processing. This target speech model needs to be trained using sample audio data and sample reference audio data before it can be used. Only after the target speech model has been trained can it be used to optimize the target original input feature data for speech. During the training of the target speech model, the sample amplitude data corresponding to the sample audio data and the sample reference amplitude data corresponding to the sample reference audio data are logarithmically processed and mean normalized. This is to amplify the less pronounced nuances in the training speech (i.e., the sample audio data) to enhance the nuance processing capability of the target speech model. This mean normalization calculates the mean and variance for different batches of sample amplitude data and sample reference amplitude data to normalize the sample amplitude data and sample reference amplitude data for each batch. Since the processing of target amplitude data and reference amplitude data should be the same as that used during speech model training when using the target speech model, logarithmic processing and mean normalization are also required here. However, in reality, there is only one batch of target amplitude data and reference amplitude data. Therefore, the mean normalization operation here still returns the target amplitude data and reference amplitude data themselves. Thus, it is essentially just a logarithmic operation on the target amplitude data and reference amplitude data.

[0124] After obtaining the target phase data, the phase difference needs to be calculated to obtain the target phase difference. As described above, the target audio data is audio data from multiple audio data streams. Here, it is assumed that the multiple audio data streams are N audio data streams, and the target audio data is also N audio data streams. Then, the target phase data calculated from the target audio data includes N phase data streams. It can be understood that these N phase data streams are the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams in the target audio data, respectively, obtained by performing phase calculations using the formula (2) (phase calculation function) mentioned above. The calculation of the target phase difference is obtained by calculating the difference between each pair of the multiple phase data streams present in the target phase data. If the target phase data containing N phase data streams is used to calculate the target phase difference, then the obtained target phase difference can be considered to be M, where the M target phase differences are obtained by calculating the difference between any two of these N phase data streams. For example, if N is 2, then M will be 1, meaning that the difference between two phase data points equals one phase difference; if N is 3, then M will be 3, meaning that the difference between each pair of three phase data points equals three phase differences; and if N is 4, then M will be 6, meaning that the difference between each pair of four phase data points equals six phase differences. There are many possible values ​​for N, and we will not provide further examples here.

[0125] After calculating M target phase differences from N phase data, these M target phase differences can be concatenated with the previously logarithmized and mean-normalized target amplitude data and reference amplitude data along the channel dimension to form the target input feature data. It can be understood that the logarithmized and mean-normalized target amplitude data, reference amplitude data, and the M target phase differences can each be viewed as a feature matrix or a feature map. These feature maps have the same size in both the time and frequency dimensions. Therefore, the final target input feature data can be understood as multiple feature maps of the same size in both the time and frequency dimensions. These feature maps are the logarithmized and mean-normalized target amplitude data, reference amplitude data, and multiple target phase differences. It can be understood that if there are M target phase differences, then the target phase data contains N phase data, and the target amplitude data contains N amplitude data. The reference amplitude data contains 1 amplitude data. Therefore, the final target input feature data consists of these N phase data and N+1 amplitude data, including 2N+1 feature maps.

[0126] For further details, please see Figure 4 , Figure 4 This is a schematic diagram of a process for extracting time-spectrum data features according to an embodiment of this application. Figure 4As shown, after obtaining the target time-spectrum data and reference time-spectrum data by performing Fourier transform on the target audio data and reference audio data, step S11 can be executed to calculate the target phase data using the target real part data and target imaginary part data in the target time-spectrum data. Figure 4 The target phase data obtained includes first phase data, second phase data, and third phase data. This is because the target time-spectrum data at this time is the time-spectrum data corresponding to the three audio data streams (i.e., first time-spectrum data, second time-spectrum data, and third time-spectrum data). The first phase data is calculated from the first time-spectrum data, the second phase data from the second time-spectrum data, and the third phase data from the third time-spectrum data. The target phase difference is obtained by calculating the phase difference between each pair of the first, second, and third phase data. Figure 4 As shown, the target phase difference at this time includes the first phase difference data, the second phase difference data, and the third phase difference data. The first phase difference data is obtained by calculating the difference between the first phase data and the second phase data. The second phase difference data is obtained by calculating the difference between the first phase data and the third phase data. The third phase difference data is obtained by calculating the difference between the second phase data and the third phase data.

[0127] At this point, step S12 can be executed to calculate the target amplitude data using the target real part and target imaginary part data from the target time-spectrum data. It can be understood that the target time-spectrum data at this time corresponds to the time-spectrum data of the three audio data streams. Therefore, the obtained target amplitude data also includes three amplitude data (i.e., the first amplitude data, the second amplitude data, and the third amplitude data). The first amplitude data is obtained by calculating the amplitude from the first time-spectrum data, the second amplitude data is obtained by calculating the amplitude from the second time-spectrum data, and the third amplitude data is obtained by calculating the amplitude from the third time-spectrum data. After obtaining the reference time-spectrum data, step S13 can be executed to calculate the reference amplitude data using the reference real part and reference imaginary part data from the reference time-spectrum data. At this point, the reference amplitude data is a single amplitude data. Furthermore, after calculating the target phase difference, target amplitude data, and reference amplitude data, step S14 can be executed. By concatenating the three amplitude data in the target amplitude data, the first phase difference data, the second phase difference data, and the third phase difference data in the target phase difference, and one amplitude data in the reference amplitude data in the channel dimension, the target input feature data is obtained. It can be understood that the target input feature data at this time is the feature data composed of seven feature maps (feature maps corresponding to the three amplitude data in the target amplitude data, feature maps corresponding to the first phase difference data, the second phase difference data, and the third phase difference data, and feature map corresponding to one amplitude data in the amplitude data).

[0128] After obtaining the target input feature data, it can be input into the target speech model for corresponding speech optimization processing, resulting in optimized target feature data after separating and removing reference time-spectral data from the target time-spectral data. Upon receiving the target input feature data, the speech model first inputs it into the encoder for encoding, obtaining the encoded feature data corresponding to the original target input feature data. The encoder can be understood as a global feature extraction unit, capable of extracting features from the target input feature data (which initially contains only local features) over a larger range (including time and frequency ranges). This ensures that the encoded feature data contains features from a wider range of audio data, facilitating subsequent processing by the target speech model. Specifically, the encoder contains multiple unit coding layers, each encoding the obtained feature data and transmitting the encoded feature data to subsequent unit coding layers. Each unit coding layer extracts features over a larger range, and through the superposition of multiple unit coding layers, the final encoder output feature data contains more historical information from the target audio data. During the training of the target speech model, the more unit coding layers are set, the wider the historical audio range captured. The more the model (i.e., the original speech model) learns the statistical characteristics of the audio, thus enhancing its ability to separate the input speech (i.e., the target audio data). Here, by passing the original target input feature data through more unit coding layers for feature extraction (i.e., encoding), the encoded feature data output by the encoder can contain more historical information from the target audio data. Of course, in actual operation, due to the limitations of the device's computing power, the number of unit coding layers needs to be flexibly adjusted according to actual needs or the device's computing power; no specific limit is placed on the number of unit coding layers here.

[0129] In this embodiment, since the target speech model may include an encoder composed of multiple unit coding layers, this means that the encoder in the target speech model can contain multiple unit coding layers, and each unit coding layer corresponds to a coding sequence number. For ease of understanding, this embodiment can take two unit coding layers of the multiple unit layers as an example to illustrate the specific process of encoding the target raw input feature data input to the encoder through two unit coding layers. In these two unit coding layers, the unit coding layer corresponding to the first coding sequence number can be called the first unit coding layer, and the unit coding layer corresponding to the next coding sequence number of the first coding sequence number can be called the second unit coding layer. That is, the second unit coding layer here is the next unit coding layer cascaded after the first unit coding layer. Based on this, this embodiment can refer to the method of encoding the input target raw input feature data through the first unit coding layer as the first coding process. Specifically, the first coding process refers to the use of filters in the convolutional layers of the first unit coding layer (e.g., filters that match the first coding sequence number (e.g., i=1), where the number of filters matches the coding sequence number; for example, the number of filters can be 2). i =2 1 =2), linear layers, and activation functions are used for convolution, linear transformation, and function activation. After the first unit encoder performs the first encoding process on the original input feature data to obtain the first encoded feature data, the first encoded feature data can be input into the second unit encoding layer for encoding processing. In this embodiment, the method of encoding the input first encoded feature data through the second unit encoding layer is collectively referred to as the second encoding process. The second encoding process is similar to the first encoding process described above, also using filters, linear layers, and activation functions in the convolutional layers of the second unit encoding layer to perform convolution, linear transformation, and function activation operations. The difference is that the number of filters in the convolutional layers of the second unit encoding layer is increased compared to the number of filters in the convolutional layers of the first unit encoding layer (for example, when the encoding sequence number of the first unit encoder is i=1, the number of filters in the convolutional layers of the first unit encoder can be 2). i =2 1 = 2, and at this time, the encoding sequence number of the second unit encoder is i+1=2, and the corresponding number of filters in the convolutional layer of the second unit encoder can be 2. i+1 =2 2 =4, which means that the number of filters increases with the coding sequence number of the unit coding layer.

[0130] Specifically, each unit coding layer in the encoder of the target speech model contains a convolutional layer for processing the input data. Taking the first unit coding layer in the encoder as an example, after receiving the original input feature data, the first unit coding layer can perform the first encoding process. Specifically, the first convolutional layer in the first unit coding layer performs a convolution operation on the original input feature data, so that each element in the original input feature data contains feature data over a larger range in the time and frequency dimensions. After obtaining the convolutional original input feature data (i.e., the first convolutionally encoded feature data), subsequent normalization and encoding activation processing can be performed to obtain the first encoded feature data. Regarding the normalization processing of data in the target speech model, its main purpose is to make the input distribution of each layer of the original speech model more stable when training the target speech model (i.e., the original speech model), so as to speed up the training speed of the original speech model. The function activation operation in the original speech model mainly aims to use a non-linear function to perform non-linear activation on the feature data, so that the original speech model can learn the non-linear features in the data during training. The encoding activation here uses a parametric rectified linear unit (PReLU). Compared to the rectified linear unit (ReLU), it adds learnable parameters in the negative region of the function. Its negative slope reduces sensitivity to noise and outliers, allowing the target speech model (i.e., the original speech model) to better learn the nonlinear features of noise during training. This improves the trained original speech model's ability to remove uncertainty and noise from the input audio data. The first encoding layer of the encoder completes the encoding process of the original input feature data, obtaining the first encoded feature data. This first encoded feature data can then be passed to the second encoding layer for a second encoding process. The second encoding process is similar to the first encoding process performed in the first encoding layer, and will not be elaborated further here. The original input feature data passes through each layer of the encoder sequentially as described above, finally yielding the encoded feature data. It's important to note that the encoded feature data passes through a linear layer before the final encoder output. This linear transformation of the encoded feature data in the channel dimension changes its size, making it easier for the target speech model to process. Typically, the channel dimension is removed, meaning the number of channels in the encoded feature data is set to 1. This allows for better replication in the channel dimension for subsequent feature concatenation. However, this linear layer only performs a linear transformation on the channel dimension and does not change the size of the feature map.

[0131] For details, please see Figure 5 , Figure 5 This is a schematic diagram illustrating a scenario where a target speech model encoder performs encoding processing, as provided in an embodiment of this application. Figure 5 As shown, after the encoder in the target speech model receives the original target input data 31a, it sequentially processes the original input data 31a through unit coding layers 36a, 36a, ..., 36n to obtain coded feature data 35a. The specific coding process of the unit coding layers is as follows... Figure 5 As shown, after the unit coding layer 36a receives the target raw input data 31a, it will use the convolutional layer 37a to perform convolution processing to obtain convolutionally encoded feature data 32a. This convolution processing will enable the features in the convolutionally encoded feature data 32a to include a larger range of the target audio data. Batch normalization of the obtained convolutionally encoded feature data 32a will yield normalized feature data 33a. Then, activation function 38a will be used to encode and activate the normalized feature data 33a to obtain the first encoded feature data 34a.

[0132] The obtained first encoded feature data 34a can be input into the second unit coding layer in the target speech model (i.e. Figure 5 The unit coding layer 36b performs a second encoding process. The encoding process of the first encoded feature data 34a by unit coding layer 36b is similar to the encoding process of the original target input data 31a by unit coding layer 36a. It also uses a convolutional layer 37b to convolve the first encoded feature data 34a to obtain convolutionally encoded feature data 32b. The features in the convolutionally encoded feature data 32b represent a wider range than those in the first encoded feature data 34a. Furthermore, unit coding layer 36b can batch normalize the obtained convolutionally encoded feature data 32b to obtain normalized feature data 33b, and then use activation function 38b to encode and activate the normalized feature data 33b to obtain the second encoded feature data 34b. The obtained second encoded feature data 34b can then be input into subsequent unit coding layers of the encoder to perform similar encoding steps, which will not be elaborated further here. After the encoding of the feature data is completed in the last unit encoding layer 36n of the encoder, the encoded feature data 35a can be obtained. The encoder will also use a linear layer to perform linear transformation on the channel dimension of the encoded feature data 35a to obtain the final output feature data of the encoder, namely the encoded feature data 35b.

[0133] It's important to note that when the convolutional layers in the aforementioned unit coding layers convolve the obtained data, the convolutional kernels have a stride of 2 along the frequency dimension and a stride of 1 along the time dimension. This allows the convolutional kernels to utilize a larger frequency dimension of context during feature extraction (i.e., convolution processing), resulting in features in the convolutional feature data containing a larger frequency dimension. However, this also halves the size of the corresponding feature map in the frequency dimension. Since the stride along the time dimension is 1, the size of the corresponding feature map in the time dimension remains unchanged. Therefore, after multiple unit coding layers of convolution processing, the size of the final encoded feature map in the frequency dimension is halved multiple times. To improve the feature extraction range of the network, the number of filters in the convolutional layers of each unit coding layer gradually increases with the number of unit coding layers. Since the number of channels in the feature data output by the convolutional layer is consistent with the number of filters in the convolutional layer, the size of the encoded feature data output by each unit coding layer in the channel dimension also increases with the number of layers. The number of filters added per layer can be set according to the computing power of the device, such as adding a fixed value layer by layer, or doubling layer by layer, etc., without limitation. In this embodiment, by gradually increasing the number of filters, the feature extraction range of the unit coding layer is made larger and larger, thereby enhancing the feature extraction range of the encoder. Since the size of the encoded feature data output by the last unit coding layer in the channel dimension may grow significantly, a linear layer is needed at the end to change the size of the encoded feature data output by the encoder in the channel dimension.

[0134] For details, please see Figure 6 , Figure 6 This is a schematic diagram illustrating a scenario where the convolutional layer of an encoder performs a convolution operation, as provided in an embodiment of this application. Figure 6As shown, after receiving the original target input data, the first unit encoding layer in the encoder can perform relevant encoding processing. When encoding the original target input data, the first unit encoding layer performs convolution operations through convolutional layer 71a. Since convolutional layer 71a has two filters (filters 72a and 72b) convolving the data, the output data (i.e., convolutionally encoded feature data 75a) will have a size of 2 in channel dimension 74a. This can be understood as the data being composed of two feature maps (i.e., feature map data 73a and feature map data 73b). This is because the convolutional layer changes the channel dimension of the data during convolution, ensuring the number of channels matches the number of filters in the convolutional layer. That is, each filter convolves the received data to obtain a corresponding feature map. Furthermore, the first unit coding layer performs a first normalization and a first coding activation on the obtained convolutional coding feature data 75a to obtain the output data of the first unit coding layer (i.e., the first coding feature data). It should be noted that the first normalization and the first coding activation do not change the dimension of the data. Therefore, the size of the channel dimension 74b of the final first coding feature data is the same as the size of the channel dimension 74a of the convolutional coding feature data 75a (both are 2, i.e., the number of channels is 2). At this time, the first coding feature data is also composed of two feature maps (i.e., feature map data 73c and feature map data 73d).

[0135] After receiving the first encoded feature data output by the first unit coding layer, the second unit coding layer also uses a convolutional layer (i.e., convolutional layer 71b) to perform a convolution operation on the first encoded feature data. In this application, the encoder increases the number of filters in the convolutional layers of each unit coding layer as the number of layers increases. For example... Figure 6 As shown, the number of filters in convolutional layer 71b at this point has increased by two compared to convolutional layer 71a in the first unit coding layer, becoming four filters (i.e., filters 72c, 72d, 72e, and 72f) capable of convolving the input data. Therefore, after convolving the first encoded feature data, the size of the generated convolutional encoded feature data 75b in the channel dimension 74c will also become 4, and there will be four corresponding feature maps (i.e., feature map data 73e, feature map data 73f, feature map data 73g, and feature map data 73h). It can be understood that each feature map here is generated by the filters in the convolutional layer through convolution processing of the first encoded feature data. Finally, the second unit coding layer can perform second normalization and second coding activation processing on the convolutional encoded feature data 75b obtained from the convolution to obtain the second encoded feature data.

[0136] After the target speech model encodes the original input feature data using an encoder to obtain coded feature data, the original input feature data and the coded feature data need to be concatenated to obtain concatenated feature data. Subsequent processing of the concatenated feature data allows the model to utilize both local features from the original input feature data and global features from the coded feature data, which improves the model's ability to remove noise features from the audio data. Regarding the feature concatenation between the original input feature data and the coded feature data, this concatenation is dimensional. Therefore, the coded feature data is first copied along the channel dimension (since the coded feature data has undergone a linear layer dimensionality transformation, its size along the channel dimension is 1), ensuring that the copied coded feature data and the original input feature data have the same size along the channel dimension. This allows the feature maps of the copied coded feature data and the original input feature data in each channel dimension to be concatenated along the frequency dimension, finally yielding the concatenated feature data. It is understandable that the size of the feature map of the concatenated feature data in the channel dimension and time dimension is the same as that of the original input feature data in the channel dimension and time dimension. However, since the concatenation is performed in the frequency dimension, the size of the feature map of the concatenated feature data in the frequency dimension is the sum of the size of the original input feature data in the frequency dimension and the size of the encoded feature data in the frequency dimension.

[0137] After obtaining the concatenated feature data through feature concatenation, the concatenated feature data needs to be input into the decoder of the target speech model for feature fusion (i.e., decoding processing). In this embodiment, since the target speech model may include a decoder composed of multiple unit decoding layers, this means that the decoder in the target speech model can contain multiple unit decoding layers, and each unit decoding layer corresponds to a decoding sequence number. For ease of understanding, this embodiment can take two unit decoding layers of these multiple unit layers as an example to illustrate the specific process of decoding the target original input feature data input to the decoder through two unit decoding layers. Among these two unit decoding layers, the unit decoding layer corresponding to the first decoding sequence number can be called the first unit decoding layer, and the unit decoding layer corresponding to the next decoding sequence number of the first decoding sequence number can be called the second unit decoding layer. That is, the second unit decoding layer here is the next unit decoding layer cascaded after the first unit decoding layer. Based on this, the embodiments of this application can refer to the method of decoding the input target original input feature data through the first unit decoding layer as the first decoding process. Specifically, the first decoding process refers to the filtering in the deconvolution layer of the first unit decoding layer (for example, the filter that matches the first decoding sequence number (e.g., j=1), where the number of filters matches the decoding sequence number, for example, the number of filters can be (0.5).j-1 a = (0.5) 1-1 a = a_those, where a is a positive integer, determined by the number of unit decoding layers in the decoder. The value of a must satisfy the condition that the number of filters in the deconvolution layer of each unit decoding layer is greater than or equal to 1 (no specific limit is placed on the value of a here). Linear layers and activation functions undergo convolution, linear transformation, and function activation operations. After the first unit decoder performs the first decoding process on the original input feature data to obtain the first decoded feature data, this first decoded feature data can be input into the second unit decoding layer for further decoding. In this application embodiment, the method of decoding the input first decoded feature data through the second unit decoding layer can be collectively referred to as the second decoding process. The second decoding process is similar to the first decoding process described above, also using filters, linear layers, and activation functions in the deconvolution layer of the second unit decoding layer to perform convolution processing, linear transformation, and function activation. In one possible scenario of this application embodiment, the number of filters in the convolutional layers of the second unit decoding layer is reduced compared to the number of filters in the convolutional layers of the first unit decoding layer (for example, when the decoding sequence number of the first unit decoder is j=1, the number of filters in the convolutional layers of the first unit decoder can be (0.5)). j-1 a = (0.5) 1-1 a = a, and at this time, the decoding sequence number of the second unit decoder is j+1 = 2, and the number of filters in the convolutional layer of the corresponding second unit decoder can be (0.5). j+1- 1 a = (0.5) 2-1 a = 0.5a, which means that the number of filters here decreases as the decoding sequence number of the unit decoding layer increases.

[0138] Understandably, the decoder here is similar to the encoder described above, also containing multiple unit decoding layers. The concatenated feature data, after being input into the decoder, is processed layer by layer through these unit decoding layers, with the output of one unit decoding layer being input into the next. Each of these unit decoding layers contains a deconvolution layer for deconvolutional operations on the data. Taking the first unit decoding layer as an example, after receiving the concatenated feature data, it performs deconvolution processing on the concatenated feature data through a deconvolution layer (i.e., the first deconvolution layer) to obtain the first deconvolutional decoded feature data. This deconvolution process fuses the features in the concatenated feature data. After obtaining the first deconvolutional decoded feature data through the first deconvolution layer, the first unit decoding layer normalizes this first deconvolutional decoded feature data to obtain the corresponding normalized feature data. Then, it performs decoding activation on this normalized feature data to obtain the first decoded feature data. This decoding activation is similar to the encoding activation in the encoder described above, using the PReLU activation function to activate the obtained feature data. The concatenated feature data is decoded in the first unit decoding layer to obtain the first decoded feature data. This first decoded feature data can then be input into subsequent unit decoding layers of the decoder for a similar decoding process. The decoded feature data output by the last unit decoding layer is the decoded feature data obtained by the decoder. The application stride of the deconvolution kernels and the number of related filters in the deconvolution layers of the decoder can be set according to the specific application scenario (for example, in one possible design, the number of filters in the deconvolution can decrease as the number of unit decoding layers increases), and no specific limitations are imposed here.

[0139] For further details, please see Figure 7 , Figure 7 This is a schematic diagram illustrating a scenario where a target speech model decoder performs decoding processing according to an embodiment of this application. Figure 7 As shown, after the decoder of the target speech model receives the concatenated feature data 51a, it can decode through multiple unit decoding layers (i.e., Figure 7 The unit decoding layers 56a, 56b, ..., 56n in the decoder process the concatenated feature data 51a layer by layer, finally outputting the decoded feature data 55a. The decoding process of these multiple unit decoding layers is as follows... Figure 7As shown, after the unit decoding layer 56a receives the spliced ​​feature data 51a, it performs deconvolution processing on the spliced ​​feature data 51a through the deconvolution layer 57a (i.e., merging the local and global features present in the spliced ​​feature data 51a) to obtain deconvolutional decoded feature data 52a. Then, the unit decoding layer 56a performs batch normalization on the deconvolutional decoded feature data 52a to obtain normalized data 53a. Finally, the normalized data 53a is activated by the activation function 58a to obtain the first decoded feature data 54a. It can be understood that the first decoded feature data 54a here is also the decoded feature data obtained by the unit decoding layer 56a from the decoding processing of the spliced ​​feature data 51a.

[0140] After the first decoded feature data 54a is output from the unit decoding layer 56a, the first decoded feature data 54a can be input into the unit decoding layer 56b for decoding processing similar to that of the unit decoding layer 56a. Figure 7 As shown, after the unit decoding layer 56b receives the first decoded feature data 54a, it uses a deconvolution layer 57a to perform deconvolution processing to further fuse the local and global features present in the first decoded feature data 54a, thus obtaining deconvolutional decoded feature data 52b. Then, the unit decoding layer 56b performs batch normalization on the deconvolutional decoded feature data 52b to obtain normalized data 53b, and finally activates the normalized data 53a through an activation function 58b to obtain the second decoded feature data 54b. This second decoded feature data 54b is the decoded feature data obtained by the unit decoding layer 56b from the first decoded feature data 54a. By inputting the second decoded feature data 54b into the remaining unit decoding layers of the decoder for similar decoding processing, the last unit decoding layer of the decoder (i.e., unit decoding layer 56n) outputs decoded feature data 55a. This decoded feature data 55a is the final decoded feature data obtained by the decoder from the input concatenated feature data 51a.

[0141] After the target speech model obtains decoded feature data through the decoder output, a sequence information extractor can be used to extract sequence information from the decoded feature data. This extracted sequence information is then used to further utilize the contextual information about the target audio data contained within the sequence information to identify and remove relevant noise. Specifically, the sequence information extractor of the target speech model includes multiple sequence modeling layers. These layers sequentially extract sequence information from the decoded feature data input to the extractor, resulting in the final output of sequence-extracted feature data corresponding to the decoded feature data. This extracted feature data is the target optimized feature data obtained by separating and removing the reference time-spectral data from the initial target time-spectral data. This target optimized feature data can also be understood as clean feature data with noise removed. Each sequence modeling layer (e.g., the first sequence modeling layer) can be used to extract sequence information from an input sequence (i.e., the first input sequence) under a historical state (i.e., the first historical state data), resulting in an output sequence (the first output sequence) under a target state (i.e., the first target state data). The target state can be understood as the latent state feature data obtained during the sequence information extraction process of the input sequence data in a sequence modeling layer. The target state data obtained by the current sequence modeling layer (the first sequence modeling layer) can serve as the historical state data (the second historical state data) corresponding to the next sequence modeling layer (the second sequence modeling layer), enabling the second sequence modeling layer to perform sequence feature extraction on the first output sequence data under the second historical state data, similar to the first sequence modeling layer. It can be understood that data is passed down and processed layer by layer in the sequence information extractor until the final extracted sequence feature data is output.

[0142] Specifically, after the sequence information extractor of the target speech model obtains the decoded feature data, it uses this data as the first input sequence data and inputs it into the first sequence modeling layer (i.e., the first sequence modeling layer) in the sequence information extractor. This allows the first sequence modeling layer to perform a linear transformation on the first input sequence data through its first linear layer (i.e., the first linear layer), resulting in first linearly transformed feature data. The main purpose of this linear transformation is to change the feature dimension of the decoded feature data so that the first sequence modeling layer can adapt to the feature dimension size of the data for subsequent processing. After obtaining the linearly transformed first input sequence data (i.e., the first linearly transformed feature data), it can be input into the first sequence extraction convolutional layer in the first sequence modeling layer for sequence extraction convolution processing, resulting in first intermediate convolutional feature data. This convolutional operation extracts the sequence information present in the decoded feature data and uses this sequence information to remove noise from the decoded feature data.

[0143] The obtained first intermediate convolutional feature data is then summed with the first historical state data. This first historical state data is input into the first sequence modeling layer along with the first input sequence data, but it is only used at this point. The feature summation here mainly involves adding elements with the same position in the feature maps. Each sequence modeling layer in the sequence information extractor involves the use of relevant historical state data. Because the structure of each sequence modeling layer is similar, the historical state data used by each sequence modeling layer can be understood as feature data that preserves the sequence information extracted by the previous sequence modeling layers. Since the first sequence modeling layer is the first sequence modeling layer in the sequence information extractor, the historical state data it uses (i.e., the first historical state data) uses an initialized value. The feature size of the first historical state data is the same as the feature size of the first intermediate convolutional feature data. However, the specific value in the first historical state data can be initialized to zero. Of course, the specific initialized value can be set as needed, and no specific restrictions will be placed on it here. By summing the first historical state data and the first intermediate convolutional feature data, the first hidden state feature data can be obtained. It can be understood that by adding the state information of each sequence modeling layer to the output of the convolutional layer, a skip connection technique is used, ensuring that the extraction does not disappear quickly. The obtained first hidden state feature data can also be understood as the hidden state information generated by the first sequence modeling layer during sequence extraction. This first hidden state feature data is used as the first target state data, and this first target state data is also used as the second historical state data for the second sequence modeling layer. It is added to the output of the convolutional layer in the second sequence modeling layer to obtain the target state data corresponding to the second sequence modeling layer. In this way, the state information is passed down unit by unit. The state information obtained by each sequence modeling layer is stored in a state storage unit. Each time a new sequence modeling layer obtains new state information, the state value stored in the state storage unit is updated synchronously.

[0144] When the first sequence modeling layer obtains the first hidden state data by adding the first historical state data and the first intermediate convolutional feature data and uses it as the first target state data, the second linear layer can be used to perform a linear transformation on the first hidden state data. The first hidden state data after linear transformation (i.e., the second linear transformation feature data) is then normalized by a normalization layer. Finally, the normalized second linear transformation feature data is used as the first output sequence data output by the first sequence modeling layer under the first target state data. Here, the first output sequence data can be understood as the final data obtained by the first sequence modeling layer from the first input sequence data (i.e., the decoded feature data) through sequence extraction. The first input sequence data will be input together with the second historical state data (i.e., the first target state data) into the subsequent sequence modeling layer of the sequence information extractor for processing similar to that described above for the first historical state data and the first input data. The output sequence data of the last sequence modeling layer of the sequence information extractor is the sequence extraction feature data that the sequence information extractor finally outputs. This sequence extraction feature data can be used as the target optimization feature data of the final output of the target speech model. It can be understood that the more sequence modeling layers there are in the sequence information extractor, the stronger the overall ability of the sequence information extractor to model sequence information. However, in actual use, due to limitations such as device computing power, the number of sequence modeling layers cannot be too large. Therefore, no specific limit will be placed on the number of sequence modeling layers here.

[0145] For details, please see Figure 8 , Figure 8 This is a schematic diagram illustrating a sequence extraction scenario using a sequence information extractor provided in an embodiment of this application. For example... Figure 8As shown, after the sequence information extractor in the target speech model obtains the decoded feature data output from the decoder, it inputs the decoded feature data into the sequence information extractor. Sequence modeling layers 62a, 62b, ..., 62n are then used for sequence information extraction. Historical state data generated during this process is also stored in the state storage unit, ensuring that the sequence information extracted by each sequence modeling layer is saved for use by subsequent sequence modeling layers. Finally, the encoder obtains the final sequence extracted feature data. Specifically, after the sequence information extractor obtains the decoded feature data, it uses it as the first input sequence data 61a. The sequence modeling layer 62a (i.e., the first sequence modeling layer of the sequence information extractor) then performs sequence information extraction on the first input sequence data 61a under the first historical state data 63a. The sequence modeling layer 62a first uses a linear layer 64a to perform a linear transformation on the first input sequence data 61a to obtain the first linearly transformed feature data. Furthermore, the first input sequence data 61a after linear transformation (i.e., the first linearly transformed feature data) is convolved through a convolutional layer 65a. This process extracts the sequence information present in the data and uses this sequence information to determine the location of noise features in the data for noise removal. After the convolutional layer 65a performs convolution on the first input sequence data 61a after linear transformation, the output of the convolutional layer 65a (i.e., the first intermediate convolutional feature data) can be summed with the first historical state data 63a. Since the sequence modeling layer 62a is the first sequence modeling layer of the sequence information extractor, the first historical state data 63a is the value set during initialization. Its feature dimension size is the same as that of the output of the convolutional layer 65a, so feature summation can be performed to obtain the first hidden state data. This first hidden state data is the first target state data, and this first target state data can also be used as the second historical state data 63b for the second sequence modeling unit. Next, the sequence modeling layer 62a will perform a linear transformation on the first hidden state data through the linear layer 66a, and then normalize the first hidden state data after the linear transformation to finally obtain the first output sequence data 67a.

[0146] After performing sequence information extraction processing on the first input sequence data 61a, the sequence modeling layer 62a can use the resulting first output sequence data 67a as the second input sequence data to the second sequence modeling layer (i.e., sequence modeling layer 62b) in the sequence information extractor to continue performing sequence information extraction processing similar to that of sequence modeling layer 62a. For example... Figure 8As shown, the sequence modeling layer 62b first performs a linear transformation on the second input sequence data (i.e., the first output sequence data 67a) through the linear layer 64b, and then inputs the linearly transformed second input sequence data into the convolutional layer 65b for convolution processing. This convolution also extracts sequence information from the input data and removes relevant noise features. However, compared to the sequence modeling layer 62a, the sequence information extracted here can represent a larger range and utilize greater contextual information from the audio data to remove noise. After the linearly transformed second input sequence data is processed by convolution in the convolutional layer 65b, the output of the convolutional layer 65b and the second historical state data 63b are summed to obtain the second hidden state data, which can be used as the second target state data. This second target state data stores the relevant sequence information extracted by the sequence modeling layer 62a. The use of feature summation allows the sequence information to be passed down continuously, enabling the sequence information extractor to better model the overall sequence information. The second target state data here stores the sequence information extracted by the current sequence modeling layer (i.e., sequence modeling layer 62b), and can be used as the third historical state data 63c to input into subsequent sequence modeling layers for further processing. Sequence modeling layer 62b performs a linear transformation on the obtained second hidden state data through a linear layer 66b, and then normalizes the linearly transformed second hidden state data to output the second output sequence data 67b. The second output sequence data 67b is the final output of sequence modeling layer 62b. This second output sequence data 67b obtained through sequence modeling layer 62b will also be used as the third input sequence data, along with the third historical state data 63c, and input into subsequent sequence modeling layers in the sequence information extractor for similar sequence information extraction processing. Finally, the sequence information extractor outputs the final output sequence extracted feature data through the last sequence modeling layer (i.e., sequence modeling layer 62n).

[0147] Step S104: Perform data fusion processing on the time spectrum data corresponding to the multi-channel audio data to obtain fused time spectrum data. Filter the fused time spectrum data using target optimization feature data to obtain filtered time spectrum data corresponding to the target time spectrum data separated from the fused time spectrum data. Perform inverse Fourier transform on the filtered time spectrum data to obtain target audio optimization data corresponding to the target audio data. Perform voice control using the voice control commands carried in the target audio optimization data.

[0148] Specifically, in this embodiment, after performing speech optimization processing on the original target input feature data to obtain target optimized feature data, the target optimized feature data can be used as a filter to filter the fused time-spectrum data corresponding to the multiple audio data to which the target audio data belongs, thus obtaining the filtered time-spectrum data corresponding to the target audio data. It can be understood that the target audio data is the audio data from N audio data (multi-channel audio data), where N is a positive integer. The target optimized feature data can be understood as N complex vector matrices obtained through the sequence information extractor in the target speech model, where each vector in the N complex vector matrices corresponding to the target optimized feature data is a column vector of dimension N. It can also be understood that the fused time-spectrum data is obtained by fusing the time-spectrum data corresponding to the N audio data (multi-channel audio data) to which the target audio data belongs. Specifically, the time-spectrum data corresponding to each of the N audio data streams can be understood as a complex matrix (where each element is a complex number). The time-spectrum data corresponding to each of the N audio streams corresponds to N time-spectrum data sets (i.e., N complex matrices). The data fusion process involves concatenating the elements at the same position in these N complex matrices (the time-spectrum data corresponding to each audio stream) into a vector. Because each element in these N complex matrices is a complex number, the concatenation results in a complex vector matrix (where each element is a complex column vector of dimension N, obtained by concatenating elements at the same position from the N complex matrices). This complex vector matrix is ​​the fused time-spectrum data that requires filtering. Since there are N complex vector matrices corresponding to the target optimization feature data here, it can be understood that there are N filters. The fused time spectrum data obtained here is filtered through these N filters, and finally N filtered time spectrum data can be obtained.

[0149] Specifically, when filtering the fused spectral data using the target optimization feature data, the filter corresponding to a specific audio data point (e.g., the first audio data) is used (i.e., the complex vector matrix corresponding to the first audio data, which can be considered as the first complex vector matrix). This filters the fused spectral data to obtain the first filtered spectral data corresponding to the first audio data. Finally, by performing an inverse Fourier transform on the first filtered spectral data, the first target optimized audio data corresponding to the first audio data can be obtained. The specific filtering process involves multiplying the conjugate transpose of a vector in the complex vector matrix (i.e., the first complex vector matrix) with a vector in the complex vector matrix corresponding to the fused spectral data. This results in a new complex matrix, which represents the first filtered spectral data corresponding to the first audio data, since a complex matrix can be decomposed into a real and an imaginary part. Finally, performing an inverse Fourier transform on the obtained first filtered spectral data yields the target optimized audio data corresponding to the first audio data. The above example uses only the first audio data. In reality, the target audio data may contain multiple audio data streams, and the target optimization feature data also contains filters corresponding to the multiple audio data streams. By filtering the fused temporal spectrum data using these multiple filters, the filtered temporal spectrum data corresponding to the multiple audio data streams can be obtained. Then, by performing inverse Fourier transforms on these multiple filtered temporal spectrum data streams, the target audio optimization data corresponding to the multiple audio data streams can be obtained. Finally, the target audio optimization data can be used for related voice control.

[0150] It is understood that in this embodiment, by performing Fourier transform on the target audio data and its noise data (reference audio data) from multiple audio data streams to obtain relevant time-spectrum data, and then performing feature extraction and speech optimization on the obtained time-spectrum data, clean feature data (target optimized feature data) after removing noise features can be obtained. After fusing the time-spectrum data corresponding to the multiple audio data streams to which the target audio data belongs, using the obtained target optimized feature data to obtain fused time-spectrum data, the fused time-spectrum data can be filtered using the target audio optimized data to separate the filtered time-spectrum data corresponding to the target time-spectrum data. Performing an inverse Fourier transform on the obtained filtered time-spectrum data yields the target audio optimized data corresponding to the target audio data. Finally, the target audio optimized data is used for relevant speech control. It is understood that the target audio optimized data is used instead of the original target audio data for relevant speech control. Since the target audio optimized data is denoised audio data, the speech control commands it carries are clearer, making speech control using the target audio optimized data more effective and accurate.

[0151] For further details, please see Figure 9 , Figure 9 This is a schematic diagram illustrating another audio data processing method provided in an embodiment of this application. It is understood that... Figure 9 The audio data processing method shown can be performed by a user terminal (the user terminal can be, for example, ...). Figure 1 The user terminal 10a shown can be used to execute the command, or a user terminal federation server can be used (the server here can be...). Figure 1 The server 10d shown is used in conjunction with the computer device executing the audio data processing method, and no specific limitations will be imposed on it here. For ease of understanding, this embodiment of the application takes the computer device executing the audio data processing method as the user terminal as an example. Figure 9 As shown, the audio data processing method may include at least the following: Figure 9 Steps S201 to S211 in the process.

[0152] Step S201: Obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data;

[0153] Specifically, after acquiring the target audio data and reference audio data, Fourier transforms are performed on them respectively to obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data. The obtained time-spectrum data consists of real and imaginary parts. The target time-spectrum data includes the target real and target imaginary parts, while the reference time-spectrum data includes the reference real and reference imaginary parts. It can be understood that the reference audio data here represents noise data within the target audio data. Converting it from audio data to time-spectrum data facilitates the subsequent extraction of features from the time-spectrum data and the subsequent speech optimization of the target audio data's target time-spectrum data to remove noise (including the reference audio data) from the target audio data. The specific process of step S201 can be found in the description of step S101 above, and will not be repeated here.

[0154] Step S202: Obtain an amplitude calculation function for amplitude calculation and a phase calculation function for phase calculation. Perform amplitude calculation on the first real part data and the first imaginary part data using the amplitude calculation function to obtain the first amplitude data of the first audio data. Perform phase calculation on the first real part data and the first imaginary part data using the phase calculation function to obtain the first phase data of the first audio data.

[0155] Step S203: The amplitude of the second real part data and the second imaginary part data is calculated by the amplitude calculation function to obtain the second amplitude data of the second audio data; and the phase of the second real part data and the second imaginary part data is calculated by the phase calculation function to obtain the second phase data of the second audio data.

[0156] Step S204: Based on the first amplitude data and the second amplitude data, determine the target amplitude data of the target audio data in the time dimension, and based on the first phase data and the second phase data, determine the target phase data of the target audio data in the frequency dimension.

[0157] Specifically, after obtaining the target time-spectrum data and reference time-spectrum data through step S201, feature extraction is performed on them to extract the amplitude and phase information of the corresponding audio data. Since the target audio data is audio data from multiple audio streams, two audio streams are used as an example: the first audio data and the second audio data. When extracting the phase of the first audio data, amplitude and phase calculations are performed using the real part (i.e., the first real part data) and imaginary part (i.e., the first imaginary part data) of the time-spectrum data corresponding to the first audio data to obtain the first amplitude data and the first phase data corresponding to the first audio data. The amplitude and phase calculations are performed by substituting the first real part data and the first imaginary part data into the amplitude calculation function and the phase calculation function, respectively, to obtain the final first amplitude data and the first phase data. The specific content of the amplitude calculation function and the phase calculation function can be found in the description of step S102 above, and will not be repeated here. Regarding the second amplitude data and second phase data corresponding to the second audio data, the process is similar to the feature extraction process for the first audio data described above. The real part (i.e., the second real part data) and imaginary part (i.e., the second imaginary part data) of the time-spectrum data corresponding to the second audio data are then substituted into the amplitude calculation function and the phase calculation function. The target amplitude data corresponding to the target audio data obtained here includes the first amplitude data and the second amplitude data, and the target phase data corresponding to the target audio data includes the first phase data and the second phase data obtained here.

[0158] Step S205: Perform amplitude calculation on the reference real part data and the reference imaginary part data using the amplitude calculation function to obtain the reference amplitude data of the reference audio data;

[0159] For feature extraction of the reference audio data, it is only necessary to calculate the amplitude data of the reference audio data (i.e., the reference amplitude data). The calculation of the reference amplitude data is similar to that of the target amplitude data; simply substitute the reference real part and reference imaginary part data from the time-spectrum data (i.e., the reference time-spectrum data) corresponding to the reference audio data into the amplitude calculation formula.

[0160] For details on steps S202 to S205, please refer to the description of step S102 above, which will not be repeated here.

[0161] Step S206: Based on the target amplitude data, target phase data, and reference amplitude data, construct the target original input feature data;

[0162] After obtaining the target phase data, the phase difference can be calculated pairwise based on the phase data corresponding to the multiple audio data in the target phase data to obtain the corresponding phase difference data. Here, it is assumed that there are N phase data points in the target phase data. When calculating the phase difference, any two phase data points are obtained and their differences are calculated to obtain a target phase difference. This process continues until all N phase data points have been paired and their phase differences calculated. The final target phase differences can then be considered as the target phase difference data. The original target input feature data is constructed by concatenating the target phase difference data, target amplitude data, and reference amplitude data along the channel dimension. The resulting concatenated data is the original target input feature data.

[0163] Step S207: Obtain the target speech model for speech optimization processing, and perform data encoding processing on the target original input feature data through the target speech model to obtain the encoded feature data corresponding to the target original input feature data;

[0164] After obtaining the original target input feature data, it can be optimized using a target speech model. When optimizing the data, the target speech model first performs data encoding processing through an encoder. Here, the encoder encodes the original target input feature data one by one through multiple unit encoding layers. This data encoding process involves performing convolution operations on the obtained data through convolutional layers. During the convolution operation, global features (which can represent a wider range of features in the audio data) are gradually extracted. After encoding the original target input feature data through the encoder, the encoded feature data is obtained.

[0165] Step S208: The target original input feature data and encoded feature data are concatenated using the target speech model to obtain concatenated feature data;

[0166] After the encoder in the target speech model encodes the original input feature data, the target speech model concatenates the resulting encoded feature data with the original input feature data to obtain concatenated feature data. This concatenated feature data includes both local and global features of the audio data input to the speech model, allowing the subsequent speech model to use more comprehensive feature data for noise removal. However, the encoder in the target speech model encodes the original input feature data, resulting in a change in both channel and frequency dimensions compared to the original input feature data. To concatenate the encoded feature data along the frequency dimension, the number of channels in the encoded feature data needs to be adjusted to match the number of channels in the original input feature data. The number of channels in the encoder's output encoded feature data will become 1. At this point, it's sufficient to copy the encoded feature data multiple times along the channel dimension. Both the encoded feature data and the target original input feature data can be understood as feature data composed of feature maps. Copying the encoded feature data multiple times along the channel dimension copies the encoded feature maps within the encoded feature data, ensuring the number of encoded feature maps in the encoded feature data is the same as the number of input feature maps in the target original input feature data. After copying the encoded feature data, it can be concatenated with the target original input feature data along the frequency dimension. Specifically, the encoded feature maps in the copied encoded feature data and the input feature maps in the target original input feature data are concatenated along the frequency dimension to obtain frequency-concatenated feature maps. The number of frequency-concatenated feature maps obtained here is the same as the number of encoded feature maps in the copied encoded feature data and the number of input feature maps in the target original input feature data. These multiple frequency-concatenated feature maps are the final concatenated feature data.

[0167] For further details, please see Figure 10 , Figure 10 This is a schematic diagram illustrating a feature splicing scenario provided in an embodiment of this application. For example... Figure 10 As shown, after the target speech model encodes the original target input feature data to obtain encoded feature data, the original target input feature data and the encoded feature data are then concatenated. At this point, the encoded feature data contains only one encoded feature map (i.e., Figure 10The encoded feature map data 41a is given. At this point, the time dimension (43a) of encoded feature map data 41a has a size of T, and the frequency dimension (44a) has a size of W1. Since there is only one encoded feature map, the size (number of channels) of the encoded feature data in the channel dimension (46a) is considered to be 1. The target original input feature data at this point contains three input feature maps: input feature map data 42a, input feature map data 42b, and input feature map data 42c. The time dimension (43c) of input feature map data 42a has a size of T, and the frequency dimension (44c) has a size of W2. The sizes of input feature map data 42b and input feature map data 42c are the same as those of input feature map data 42a in the time and frequency dimensions. Since the target original input feature data at this point contains three input feature maps, this means that the size of the target original input feature data in the channel dimension (46c) is 3.

[0168] To facilitate subsequent concatenation, the encoded feature data needs to be copied along the channel dimension, ensuring that the number of channels in the encoded feature data is also 3. Specifically, as follows... Figure 10 As shown, after copying the encoded feature data along the channel dimension based on the size of channel 46c of the original target input feature data, three encoded feature maps (i.e., encoded feature map data 41b, encoded feature map data 41c, and encoded feature map data 41d) are obtained. Encoded feature map data 41b has a size of T in the time dimension 43b and a size of W1 in the frequency dimension 44b. The sizes of encoded feature map data 41c and encoded feature map data 41d in the time and frequency dimensions are the same as those of encoded feature map data 41b. This is because encoded feature map data 41b, encoded feature map data 41c, and encoded feature map data 41d are obtained by copying encoded feature map data 41a, and their sizes in the time and frequency dimensions will also be the same as those of encoded feature map data 41a. Furthermore, the content of encoded feature map data 41b, encoded feature map data 41c, and encoded feature map data 41d is also consistent with the content of encoded feature map data 41a. At this point, the copying of the encoded feature data is complete, and the size of the copied encoded feature data in the channel dimension 46b (i.e., the number of channels) has also become 3.

[0169] After copying the encoded feature data, it can be concatenated with the target original input data. Specifically, the feature maps of the encoded feature data and the target original input data located in the same channel dimension are concatenated along the frequency dimension, resulting in a concatenated frequency-concatenated feature map in each channel dimension. For example... Figure 10As shown, the feature maps in the copied encoded feature data and the original target input data are concatenated. Specifically, encoded feature map data 41b and input feature map data 42a are in the same channel dimension, and they are concatenated along the frequency dimension to obtain frequency concatenated feature map data 45a; encoded feature map data 41c and input feature map data 42b are in the same channel dimension, and they are concatenated along the frequency dimension to obtain frequency concatenated feature map data 45b; encoded feature map data 41d and input feature map data 42c are in the same channel dimension, and they are concatenated along the frequency dimension to obtain frequency concatenated feature map data 45c; the final concatenated feature map data 45a has a size of T in the time dimension 43d and a size of W1+W2 in the frequency dimension 44d, while the feature dimension dimensions of concatenated feature map data 45b and concatenated feature map data 45c are consistent with those of concatenated feature map data 45a. The temporal dimension of the three concatenated feature maps is the same as the temporal dimension of the encoded feature map and the input feature map for each channel. The frequency dimension of the three concatenated feature maps is the sum of the frequency dimensions of the encoded feature map and the input feature map for each channel. The final concatenated feature map data 45a, 45b, and 45c constitute the final concatenated feature data. The number of channels in the concatenated feature data (i.e., the channel dimension 46d) should also be the same as the number of channels in the copied encoded feature data and the number of channels in the original target input data, which is 3.

[0170] Step S209: The spliced ​​feature data is decoded using the target speech model to obtain the decoded feature data corresponding to the spliced ​​feature data.

[0171] After the target speech model obtains concatenated feature data by concatenating encoded feature data and the original target input feature data, a decoder containing multiple unit decoding layers decodes the input concatenated feature data layer by layer, finally obtaining the decoded feature data corresponding to the concatenated feature data. The decoding process performed by the unit decoding layers can fuse the features in the concatenated feature data, allowing the target speech model to use the decoded feature data, which integrates global and local features, for subsequent speech optimization. The target speech model can then perform deconvolution processing on the concatenated feature data through the deconvolution layers contained in the unit decoding layers to achieve feature fusion.

[0172] Step S210: Sequence information extraction processing is performed on the decoded feature data through the target speech model to obtain the sequence extracted feature data corresponding to the decoded feature data. The sequence extracted feature data is used as the target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data.

[0173] After the target speech model obtains decoded feature data through the decoder, multiple sequence modeling layers in the sequence information extractor can perform relevant sequence information extraction processing on the decoded feature data. The extracted sequence information is then used to remove noise features from the decoded feature data. The sequence modeling layers in the sequence information extractor process the obtained data through a series of linear transformations and convolutions to obtain sequence information extraction features. During this process, the current state information of the sequence modeling layer is generated. The sequence information extraction features and state information obtained from each sequence modeling layer are passed to the next sequence modeling layer to continue the relevant sequence information extraction operations, thus modeling the overall sequence information of the audio data. Finally, through the sequence information extractor's processing of sequence information extraction and noise removal from the decoded feature data, the final sequence extracted feature data is the target optimized feature data after separating and removing noise data (including reference time-spectral data) from the target time-spectral data.

[0174] For details on steps S206 to S210, please refer to the description of step S103 above, which will not be repeated here.

[0175] Step S211: Perform data fusion processing on the time spectrum data corresponding to the multiple audio data to obtain fused time spectrum data. Filter the fused time spectrum data using target optimization feature data to obtain filtered time spectrum data corresponding to the target time spectrum data separated from the fused time spectrum data. Perform inverse Fourier transform on the filtered time spectrum data to obtain target audio optimization data corresponding to the target audio data. Perform voice control using the voice control commands carried in the target audio optimization data.

[0176] Specifically, the target audio data belongs to the audio data within the multi-channel audio data. Here, the time-spectrum graphs corresponding to the multi-channel audio data are concatenated along the vector dimension to obtain a complex vector matrix corresponding to the multi-channel audio data. This complex vector matrix is ​​the fused time-spectrum data. The target optimization feature data output by the target speech model can be understood as multiple complex vector matrices, because the target audio data has multiple channels. By taking the conjugate transpose of each vector in the complex vector matrix of the target optimization feature data and then performing a vector product with the vector in the complex vector matrix corresponding to the fused time-spectrum data, multiple filtered time-spectrum data corresponding to the multi-channel target audio data can be separated from the fused time-spectrum data. Finally, an inverse Fourier transform is performed on the obtained filtered time-spectrum data to obtain the optimized audio data corresponding to the target audio data (i.e., the target optimized audio data). Finally, the relevant speech control can be completed using the target optimized audio data. For the specific process of step S211, please refer to the description of step S104 above, which will not be repeated here.

[0177] It should be noted that the target speech model used in this embodiment for speech optimization of the target original input data is a speech model composed of neural networks. Therefore, it needs to be trained before using the target speech model for speech optimization. The training process of the target speech model to be trained (i.e., the original speech model) is the same as the process of using the target speech model. However, when training the original speech model, the data in the input model will not be the target audio data, but rather a group of sample audio data used for training.

[0178] The sample audio data set here includes first sample audio data, sample reference audio data, and second sample audio data obtained by adding noise to the first sample audio data. It can be understood that the first sample audio data is relevant audio data collected in advance for training the original speech model. Adding noise to the first sample audio data includes adding noise to the first sample audio data and interfering with it using sample reference audio data, finally obtaining the second sample audio data. Subsequently, Fourier transforms will be performed on the second sample audio data and the sample reference audio data to obtain their corresponding time-spectral data. Feature extraction will then be performed using their corresponding time-spectral data to obtain the sample amplitude data and sample phase data corresponding to the second sample audio data, as well as the sample reference amplitude data corresponding to the sample reference audio data. The feature extraction process here can be referred to in steps S101 to S102 above, which describes the feature extraction process for the target audio data and the reference audio data. The only difference is that the target audio data and the reference audio data are replaced with the second sample audio data and the sample reference audio data, respectively; the processing procedure remains the same. After obtaining sample amplitude data, sample phase data, and sample reference amplitude data through feature extraction, the original input feature data of the samples can be constructed from them. This original input feature data is then input into the original speech model for speech optimization processing to output optimized sample feature data. The process of the original speech model optimizing the original input feature data is similar to the process of the target speech model optimizing the target original input feature data in step S103, except that the parameters of the target speech model and the original speech model are different; therefore, it will not be elaborated further. Finally, the optimized sample feature data output by the original speech model is used to filter the fusion spectrogram data associated with the second sample audio data to obtain the filtered fusion spectrogram data corresponding to the second sample audio data. The filtering process here can be found in step S104 above, where the target optimized feature data is used to filter the fusion spectrogram data; therefore, it will not be elaborated further. Finally, an inverse Fourier transform is performed on the filtered fusion spectrogram data to obtain the optimized sample audio data corresponding to the second sample audio data. Since the original speech model used for speech optimization is not fully trained, the final sample audio optimized data and the truly noise-free data of the second sample audio data (i.e., the first sample audio data) are very different. Therefore, it is necessary to calculate the loss value between the sample audio optimized data and the first sample audio data, and then update the parameters in the original speech model using the loss value. Through multiple update processes, the parameters in the original speech model converge, and finally the target speech model (i.e., the updated original speech model) can be obtained.

[0179] For further details, please see Figure 11 , Figure 11This is a schematic diagram illustrating the training process of a raw speech model provided in an embodiment of this application. For example... Figure 11As shown, after obtaining the first sample audio data and the sample reference audio data, step S21 is executed to add noise to the first sample audio data to obtain the second sample audio data. This second sample audio data now includes noisy audio (the sample reference audio data). Then, step S22 is executed to perform a Fourier transform on the second sample audio data and the sample reference audio data to obtain the sample time-spectrum data corresponding to the second sample audio data and the sample reference time-spectrum data corresponding to the sample reference audio data. Next, feature extraction can be performed on the obtained sample time-spectrum data and sample reference time-spectrum data. Step S23 is executed here, where the sample real part and sample imaginary part of the sample time-spectrum data are input into the amplitude calculation function and the phase calculation function to obtain the sample amplitude data and sample phase data of the second sample audio data. Simultaneously, the sample reference real part and sample reference imaginary part of the sample reference time-spectrum data are input into the amplitude calculation function to calculate the sample reference amplitude data of the sample reference audio data. Since this is for model training, steps S24 and S25 are executed here. The logarithms of the obtained sample amplitude data and sample reference amplitude data are taken and normalized to obtain normalized sample amplitude data and sample reference amplitude data. This amplifies the less pronounced nuances in the speech audio, enhancing the nuance processing capability of the final trained target speech model. Because the sample audio data is from multiple audio streams, step S26 is executed here to calculate the phase difference between each pair of sample phase data obtained from the sample phase data. Finally, using the obtained normalized sample amplitude data, sample reference amplitude data, and phase difference data, the original sample input feature data is constructed. This original sample input feature data is then input into the original speech model to be trained, and step S27 is executed to use the original speech model to perform speech optimization processing on the original sample input feature data, obtaining optimized sample feature data. Using the optimized feature data from these samples, step S28 can be executed to filter the fused temporal spectrum data of the multi-channel audio data related to the first sample audio data. This fused temporal spectrum data is obtained by vector concatenation of the temporal spectrum data corresponding to the multi-channel audio data to which the first sample audio data belongs. After filtering the fused temporal spectrum data, the filtered temporal spectrum data corresponding to the sample temporal spectrum data can be separated. Step S29 is then executed on the separated filtered temporal spectrum data, i.e., an inverse Fourier transform is performed to obtain the optimized sample audio data corresponding to the second sample audio data. Since the optimized audio data is obtained based on the original speech model to be trained, the loss value needs to be calculated using the optimized sample audio data and the first sample audio data from the sample audio data, i.e., step S30 is executed. This first sample audio data is the clean, noise-free data of the second sample audio data.Based on the obtained loss value, step S31 can be executed to update the original speech model. The above training process is performed multiple times using different sample data to iteratively update the original speech model until the parameters in the original speech model converge. This indicates that the training of the original speech model is complete, and the original speech model can then be used as the target speech model to complete the relevant speech optimization processing tasks.

[0180] It is understood that the target audio data processed in this embodiment is audio data from multiple audio streams. The phase features obtained by feature extraction of the time-spectrum data corresponding to the target audio data include the phase data (e.g., first phase data, second phase data) corresponding to multiple audio streams. The phase difference between each pair of these phase data is then calculated to obtain the target phase data. The obtained target phase data, target amplitude data, and reference amplitude data are then used together to construct the target original input feature data for speech optimization processing. Target optimized feature data is obtained by performing speech optimization processing on the target original input feature data. After filtering the fused time-spectrum data obtained by data fusion processing of the time-spectrum data corresponding to each of the multiple audio streams using the target optimized feature data, audio optimized data corresponding to each of the multiple audio streams can be obtained. This is because the fused time-spectrum data is obtained from the time-spectrum data corresponding to each of the multiple audio streams, and the filter (target optimized feature data) has a separation effect, separating the target audio optimized data corresponding to the target audio data from the multiple audio streams. Finally, the target audio optimized data is used for voice control to effectively improve the accuracy of voice control.

[0181] For further details, please see Figure 12 , Figure 12 This is a schematic diagram of the structure of an audio data processing device provided in an embodiment of this application. The audio data processing device 1 can be a computer program (including program code) running on a computer device; for example, the audio data processing device 1 can be an application software. The audio data processing device 1 can be used to execute the functions provided in the embodiment of this application. Figure 3 and Figure 9 The corresponding steps in the method. The audio data processing device 1 may include: a time-spectrum data acquisition module 11, an amplitude-phase calculation module 12, a speech optimization processing module 13, and an optimized audio acquisition module 14;

[0182] The time-spectrum data acquisition module 11 is used to acquire the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data; the reference audio data is the noise data in the target audio data, the target audio data is the audio data in the multi-channel audio data, the target time-spectrum data includes the target real part data and the target imaginary part data obtained after performing a Fourier transform on the target audio data; the reference time-spectrum data includes the reference real part data and the reference imaginary part data obtained after performing a Fourier transform on the reference audio data;

[0183] The amplitude and phase calculation module 12 is used to determine the target amplitude data and target phase data of the target audio data through the target real part data and the target imaginary part data, and to determine the reference amplitude data of the reference audio data through the reference real part data and the reference imaginary part data.

[0184] The speech optimization processing module 13 is used to construct the original target input feature data based on the target amplitude data, target phase data and reference amplitude data, and perform speech optimization processing on the original target input feature data to obtain the target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data.

[0185] The optimized audio acquisition module 14 is used to perform data fusion processing on the time spectrum data corresponding to multiple audio data to obtain fused time spectrum data. The fused time spectrum data is then filtered using target optimization feature data to obtain filtered time spectrum data corresponding to the target time spectrum data separated from the fused time spectrum data. The filtered time spectrum data is then subjected to inverse Fourier transform to obtain target audio optimization data corresponding to the target audio data. Finally, voice control is performed using the voice control commands carried in the target audio optimization data.

[0186] The target audio data includes first audio data and second audio data; the target time-spectrum data includes the first time-spectrum data corresponding to the first audio data and the second time-spectrum data corresponding to the second audio data; the target real part data includes the first real part data and the second real part data; the target imaginary part data includes the first imaginary part data and the second imaginary part data.

[0187] The amplitude and phase calculation module 12 includes: a first amplitude and phase calculation unit 121, a second amplitude and phase calculation unit 122, a target amplitude and phase determination unit 123, and a reference amplitude calculation unit 124;

[0188] The first amplitude and phase calculation unit 121 is used to obtain an amplitude calculation function for amplitude calculation and a phase calculation function for phase calculation, perform amplitude calculation on the first real part data and the first imaginary part data through the amplitude calculation function to obtain the first amplitude data of the first audio data, and perform phase calculation on the first real part data and the first imaginary part data through the phase calculation function to obtain the first phase data of the first audio data.

[0189] The second amplitude and phase calculation unit 122 is used to perform amplitude calculation on the second real part data and the second imaginary part data through the amplitude calculation function to obtain the second amplitude data of the second audio data, and to perform phase calculation on the second real part data and the second imaginary part data through the phase calculation function to obtain the second phase data of the second audio data.

[0190] The target amplitude and phase determination unit 123 is used to determine the target amplitude data of the target audio data in the time dimension based on the first amplitude data and the second amplitude data, and to determine the target phase data of the target audio data in the frequency dimension based on the first phase data and the second phase data.

[0191] The reference amplitude calculation unit 124 is used to perform amplitude calculation on the reference real part data and the reference imaginary part data through the amplitude calculation function to obtain the reference amplitude data of the reference audio data.

[0192] The specific implementation methods of the first amplitude and phase calculation unit 121, the second amplitude and phase calculation unit 122, the target amplitude and phase determination unit 123, and the reference amplitude calculation unit 124 can be found in the following description. Figure 3 The description of step S102 in the corresponding embodiments will not be repeated here.

[0193] The speech optimization processing module 13 includes: a phase data acquisition unit 131, a phase difference calculation unit 132, and a first raw input construction unit 133;

[0194] Phase data acquisition unit 131 is used to acquire first phase data and second phase data from target phase data;

[0195] The phase difference calculation unit 132 is used to calculate the phase difference between the first phase data and the second phase data to obtain the first phase difference.

[0196] The first original input construction unit 133 is used to construct target original input feature data based on the first amplitude data, the second amplitude data, the reference amplitude data and the first phase difference data.

[0197] The specific implementation methods of the phase data acquisition unit 131, the phase difference calculation unit 132, and the first raw input construction unit 133 can be found in the above description. Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0198] The number of audio data streams is N, where N is a positive integer greater than 2. The target phase data includes N phase data. The N phase data are obtained by performing phase calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams. Each audio data stream corresponds to one time-frequency data. The target amplitude data includes N amplitude data. The N amplitude data are obtained by performing amplitude calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams.

[0199] The speech optimization processing module 13 also includes: a phase data pair construction unit 134, a phase data determination unit 135, a phase difference calculation unit 136, and a second raw input construction unit 137;

[0200] The phase data pair construction unit 134 is used to construct M phase data pairs associated with the N phase data based on any two phase data obtained from the N phase data; M is a positive integer greater than or equal to N.

[0201] Phase data determination unit 135 is used to determine a target phase data pair from M phase data pairs, and to determine two phase data in the target phase data pair as the first phase data to be processed and the second-generation processed phase data, respectively.

[0202] The phase difference calculation unit 136 is used to calculate the phase difference between the first phase data to be processed and the second phase data to be processed, and to obtain the target phase difference of the target phase data pair until each of the M phase data pairs is determined to be the target phase data pair, and the target phase difference of the M phase data pairs is obtained.

[0203] The second original input construction unit 137 is used to construct the target original input feature data based on the target phase difference of N amplitude data, reference amplitude data and M phase data pairs.

[0204] For details regarding the specific implementation of the phase data pair construction unit 134, the phase data determination unit 135, the phase difference calculation unit 136, and the second original input construction unit 137, please refer to the above. Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0205] The speech optimization processing module 13 further includes: a data encoding unit 138, a feature splicing unit 139, a data decoding unit 140, and a sequence extraction unit 141;

[0206] The data encoding unit 138 is used to acquire the target speech model for speech optimization processing, and to perform data encoding processing on the target original input feature data through the target speech model to obtain the encoded feature data corresponding to the target original input feature data.

[0207] The feature splicing unit 139 is used to splice the target original input feature data and encoded feature data through the target speech model to obtain spliced ​​feature data;

[0208] The data decoding unit 140 is used to perform data decoding processing on the spliced ​​feature data through the target speech model to obtain the decoded feature data corresponding to the spliced ​​feature data.

[0209] The sequence extraction unit 141 is used to perform sequence information extraction processing on the decoded feature data through the target speech model to obtain the sequence extracted feature data corresponding to the decoded feature data, and use the sequence extracted feature data as the target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data.

[0210] The specific implementation methods of the data encoding unit 138, feature splicing unit 139, data decoding unit 140, and sequence extraction unit 141 can be found in the above description. Figure 3 The description of step S103 in the corresponding embodiment will not be repeated here.

[0211] The target speech model includes an encoder consisting of multiple unit coding layers;

[0212] The data encoding unit 138 can also be used to acquire the target speech model for speech optimization processing and input the target raw input feature data into the encoder in the target speech model.

[0213] The data encoding unit 138 is further configured to obtain the first unit encoding layer and the second unit encoding layer from the multiple unit encoding layers contained in the encoder.

[0214] The data encoding unit 138 is further configured to perform a first encoding process on the original input feature data through a first unit encoding layer to obtain first encoded feature data, and to perform a second encoding process on the first encoded feature data through a second unit encoding layer to obtain second encoded feature data; the feature dimension of the second encoded feature data is different from the feature dimension of the first encoded feature data.

[0215] The data encoding unit 138 is also used to determine the encoded feature data corresponding to the target original input feature data based on the second encoded feature data.

[0216] In this system, each unit coding layer corresponds to a coding sequence number. Each unit coding layer contains a convolutional layer, and the number of filters in a convolutional layer increases with the coding sequence number. When the coding sequence number of the first unit coding layer is less than that of the second unit coding layer, the number of filters in the first convolutional layer is less than that in the second convolutional layer.

[0217] The data encoding unit 138 is further configured to determine the convolutional layer in the first unit encoding layer as the first convolutional layer, and to determine the convolutional layer in the second unit encoding layer as the second convolutional layer;

[0218] The data encoding unit 138 is further configured to perform a first convolution process on the original input feature data through the filters in the first convolutional layer to obtain first convolutionally encoded feature data, perform a first normalization process on the first convolutionally encoded feature data to obtain first normalized feature data, and perform a first encoding activation process on the first normalized feature data to obtain first encoded feature data; the number of channels in the channel feature dimension of the first encoded feature data is consistent with the number of filters in the first convolutional layer;

[0219] The data encoding unit 138 is further configured to perform a second convolution process on the first encoded feature data through filters in the second convolutional layer to obtain second convolutionally encoded feature data; perform a second normalization process on the second convolutionally encoded feature data to obtain second normalized feature data; and perform a second encoding activation process on the second normalized feature data to obtain second encoded feature data. The number of channels in the second encoded feature data in the channel feature dimension is consistent with the number of filters in the second convolutional layer. The number of channels in the second encoded feature data in the channel feature dimension is greater than the number of channels in the first encoded feature data in the channel feature dimension.

[0220] The encoder contains a linear layer for dimensional transformation of the data;

[0221] The data encoding unit 138 is also used to determine the third encoded feature data corresponding to the target original input feature data based on the second encoded feature data;

[0222] The data encoding unit 138 is also used to perform a linear transformation on the feature map corresponding to the third encoded feature data in the channel dimension through the linear layer of the encoder, and use the linearly transformed third encoded feature data as the encoded feature data corresponding to the target original input feature data; the size of the feature map corresponding to the linearly transformed third encoded feature data is consistent with the size of the feature map corresponding to the third encoded feature data.

[0223] The feature dimensions of both the encoded feature data and the target raw input feature data include channel dimension, time dimension, and frequency dimension. The encoded feature data is determined by K1 encoded feature maps, where K1 represents the number of channels in the encoded feature data in the channel dimension. The encoded feature maps are composed of feature extraction data in the time dimension and feature extraction data in the frequency dimension. The target raw input feature data is determined by K2 input feature maps, where K2 represents the number of channels in the target raw input feature data in the channel dimension. The input feature maps are composed of feature input data in the time dimension and feature input data in the frequency dimension. K1 and K2 are both positive integers.

[0224] The feature splicing unit 139 is also used to copy K1 encoded feature maps based on K2 channels to obtain K2 encoded feature maps;

[0225] The feature splicing unit 139 is also used to splice the feature extraction data from the K2 encoded feature maps and the feature input data from the K2 input feature maps in the frequency dimension to obtain K2 frequency spliced ​​feature maps.

[0226] The feature splicing unit 139 is also used to determine the K2 frequency splicing feature maps as splicing feature data obtained by splicing the target original input feature data and encoded feature data.

[0227] The target speech model includes a decoder consisting of multiple unit decoding layers;

[0228] The data decoding unit 140 is also used to obtain the first unit decoding layer and the second unit decoding layer from the multiple unit decoding layers contained in the decoder;

[0229] The data decoding unit 140 is also used to perform a first decoding process on the spliced ​​feature data through the first unit decoding layer to obtain the first decoded feature data, and to perform a second decoding process on the first decoded feature data through the second unit decoding layer to obtain the second decoded feature data.

[0230] The data decoding unit 140 is also used to determine the decoding feature data corresponding to the spliced ​​feature data based on the second decoding feature data.

[0231] Among the multiple unit decoding layers, each unit decoding layer contains a deconvolution layer;

[0232] The data decoding unit 140 is also used to determine the deconvolution layer in the first unit decoding layer as the first deconvolution layer, and to determine the deconvolution layer in the second unit decoding layer as the second deconvolution layer;

[0233] The data decoding unit 140 is further configured to perform a first deconvolution process on the spliced ​​feature data through a first deconvolution layer to obtain first deconvolution decoded feature data, perform a third normalization process on the first deconvolution decoded feature data to obtain third normalized feature data, and perform a first decoding activation process on the third normalized feature data to obtain first decoded feature data.

[0234] The data decoding unit 140 is further configured to perform a second deconvolution process on the first decoded feature data through the second deconvolution layer to obtain the second deconvolution decoded feature data, perform a fourth normalization process on the second deconvolution decoded feature data to obtain the fourth normalized feature data, and perform a second decoding activation process on the fourth normalized feature data to obtain the second decoded feature data.

[0235] The target speech model includes a sequence information extractor consisting of multiple sequence modeling layers; a sequence modeling layer is used to extract sequence information from an input sequence data under a historical state data to obtain an output sequence data under a target state data; a target state data is used to characterize the latent state feature data obtained in the process of extracting sequence information from an input sequence data in a sequence modeling layer.

[0236] The sequence extraction unit 141 is further configured to obtain a first sequence modeling layer and a second sequence modeling layer from the sequence modeling layer included in the sequence information extractor; the historical state data corresponding to the first sequence modeling layer is the first historical state data, the historical state data corresponding to the second sequence modeling layer is the second historical state data, and the second historical state data is determined based on the first target state data obtained in the first sequence modeling layer;

[0237] The sequence extraction unit 141 is further configured to use the decoded feature data as the first input sequence data, input the first input sequence data and the first historical state data into the first sequence modeling layer, and perform first sequence feature extraction processing on the first input sequence data under the first historical state data to obtain first hidden state feature data. When the first hidden state feature data is used as the first target state data, the first hidden state feature data is subjected to second sequence feature extraction processing to obtain first output sequence data under the first target state data.

[0238] The sequence extraction unit 141 is also used to use the first output sequence data as the second input sequence data corresponding to the second sequence modeling layer, and to use the first target state data as the second historical state data corresponding to the second sequence modeling layer.

[0239] The sequence extraction unit 141 is also used to input the second input sequence data and the second historical state data into the second sequence modeling layer, and the second sequence modeling layer performs a third sequence feature extraction process on the second input sequence data under the second historical state data to obtain the second hidden state feature data. When the second hidden state feature data is used as the second target state data, a fourth sequence feature extraction process is performed on the second hidden state feature data to obtain the second output sequence data under the second target state data.

[0240] The sequence extraction unit 141 is also used to determine the sequence extraction feature data corresponding to the decoded feature data based on the second output sequence data.

[0241] The first sequence modeling layer includes a first linear layer, a first sequence extraction convolutional layer, a second linear layer, and a normalization layer.

[0242] The sequence extraction unit 141 is also used to take the decoded feature data as the first input sequence data, input the first input sequence data and the first historical state data into the first sequence modeling layer, and under the first historical state data, the first linear layer performs a first linear transformation on the first input sequence data to obtain the first linear transformation feature data;

[0243] The sequence extraction unit 141 is also used to perform convolution processing on the first linear transformation feature data through the first sequence extraction convolution layer to obtain the first intermediate convolution feature data.

[0244] The sequence extraction unit 141 is also used to perform feature summation processing on the first intermediate convolutional feature data and the first historical state data to obtain the first hidden state feature data. When the first hidden state feature data is used as the first target state data, the first hidden state feature data is subjected to a second linear transformation through the second linear layer to obtain the second linear transformation feature data.

[0245] The sequence extraction unit 141 is also used to normalize the second linear transformation feature data through a normalization layer, and use the normalized second linear transformation feature data as the first output sequence data output by the first sequence modeling layer under the first target state data.

[0246] The speech optimization processing module 13 further includes: a sample audio data acquisition unit 142, a sample time spectrum acquisition unit 143, a sample data feature extraction unit 144, a sample audio optimization unit 145, a sample audio filtering unit 146, and an original speech model update unit 147.

[0247] The sample audio data acquisition unit 142 is used to acquire a sample audio data set for training the initial speech model; the sample audio data set includes first sample audio data, sample reference audio data, and second sample audio data obtained by adding noise to the first sample audio data; the second sample audio data is audio data from multiple sample audio data.

[0248] The sample time-spectrum acquisition unit 143 is used to acquire the sample time-spectrum data corresponding to the second sample audio data and the sample reference time-spectrum data corresponding to the sample reference audio data; the sample reference audio data is the noise data in the second sample audio data; the sample time-spectrum data includes the sample real part data and sample imaginary part data obtained after performing a Fourier transform on the second sample audio data; the sample reference time-spectrum data includes the sample reference real part data and sample reference imaginary part data obtained after performing a Fourier transform on the sample reference audio data;

[0249] The sample data feature extraction unit 144 is used to determine the sample amplitude data and sample phase data of the second sample audio data through the sample real part data and sample imaginary part data, and to determine the sample reference amplitude data of the sample reference audio data through the sample reference real part data and sample reference imaginary part data.

[0250] The sample audio optimization unit 145 is used to construct the original input feature data of the sample based on the sample amplitude data, sample phase data and sample reference amplitude data, obtain the original speech model to be trained, and perform speech optimization processing on the original input feature data of the sample through the original speech model to obtain the optimized feature data of the sample.

[0251] The sample audio filtering unit 146 performs data fusion processing on the time-spectrum data corresponding to the multiple sample audio data to obtain sample fused time-spectrum data. It then filters the sample fused time-spectrum data using sample optimization feature data to obtain sample filtered time-spectrum data corresponding to the sample time-spectrum data separated from the sample fused time-spectrum data. Finally, it performs an inverse Fourier transform on the sample filtered time-spectrum data to obtain sample audio optimization data corresponding to the second sample audio data.

[0252] The original speech model update unit 147 is used to calculate the data loss value through the first sample audio data and the sample audio optimization data, and to iteratively update the original speech model through the data loss value, so that the updated original speech model can be used as the target speech model.

[0253] The specific implementation methods of the sample audio data acquisition unit 142, the sample time-spectrum acquisition unit 143, the sample data feature extraction unit 144, the sample audio optimization unit 145, the sample audio filtering unit 146, and the original speech model update unit 147 can be found in the above-mentioned... Figure 11The corresponding implementation steps will not be described in detail here.

[0254] Among them, the sample audio optimization unit 145 is also used to perform a logarithmic operation on the sample amplitude data to obtain the sample amplitude logarithmic data, and to perform a logarithmic operation on the sample reference amplitude data to obtain the sample reference amplitude logarithmic data.

[0255] The sample audio optimization unit 145 is also used to normalize the logarithmic data of the sample amplitude to obtain sample normalized amplitude data, normalize the logarithmic data of the reference amplitude to obtain reference normalized amplitude data, and construct the original input feature data of the sample based on the sample normalized amplitude data, sample phase data and reference normalized amplitude data.

[0256] The specific implementation methods of the time-spectrum data acquisition module 11, amplitude-phase calculation module 12, speech optimization processing module 13, and optimized audio acquisition module 14 can be found in the above description. Figure 3 The descriptions in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0257] Further, please see Figure 13 , Figure 13 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 13 As shown, the computer device 1000 can be a user terminal, for example, the one described above. Figure 1 The user terminal 10a in the corresponding embodiment can also be a server, for example, as described above. Figure 1 The server 10d in the corresponding embodiment will not be limited here. For ease of understanding, this application takes a computer device as a user terminal as an example. The computer device 1000 may include: a processor 1001, a network interface 1004, and a memory 1005. In addition, the computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located away from the aforementioned processor 1001. Figure 13 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application.

[0258] The network interface 1004 in the computer device 1000 can also provide network communication functionality. Figure 13 In the computer device 1000 shown, the network interface 1004 provides network communication functionality; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to implement the aforementioned... Figure 3 or Figure 9 The description of the audio data processing method in the corresponding embodiments can also be performed as described above. Figure 12 The description of the audio data processing device 1 in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated here.

[0259] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned audio data processing device 1. The computer program includes computer instructions, and when the processor executes the computer instructions, it can execute the aforementioned... Figure 3 or Figure 9 The description of the audio data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application.

[0260] Furthermore, it should be noted that this application also provides a computer program product or computer program, which may include computer instructions, which may be stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, causing the computer device to perform the aforementioned actions. Figure 3 or Figure 9 The description of the audio data processing method in the corresponding embodiments is already provided and will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer program products or computer program embodiments related to this application, please refer to the description of the method embodiments of this application.

[0261] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0262] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0263] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0264] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0265] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0266] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. An audio data processing method, characterized in that, include: Obtain the target time-spectrum data corresponding to the target audio data and the reference time-spectrum data corresponding to the reference audio data; The reference audio data is noise data in the target audio data, the target audio data is audio data in multi-channel audio data, and the target time-spectrum data includes the target real part data and target imaginary part data obtained after performing a Fourier transform on the target audio data; the reference time-spectrum data includes the reference real part data and reference imaginary part data obtained after performing a Fourier transform on the reference audio data. The target amplitude data and target phase data of the target audio data are determined by the target real part data and the target imaginary part data, and the reference amplitude data of the reference audio data is determined by the reference real part data and the reference imaginary part data. Based on the target amplitude data, the target phase data, and the reference amplitude data, target original input feature data is constructed. Speech optimization processing is performed on the target original input feature data to obtain target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data. The time-spectrum data corresponding to the multiple audio data are fused to obtain fused time-spectrum data. The fused time-spectrum data is then filtered using the target optimization feature data to obtain filtered time-spectrum data corresponding to the target time-spectrum data separated from the fused time-spectrum data. The filtered time-spectrum data is then subjected to an inverse Fourier transform to obtain target audio optimization data corresponding to the target audio data. Voice control is then performed using the voice control commands carried in the target audio optimization data.

2. The method according to claim 1, characterized in that, The target audio data includes first audio data and second audio data; the target time-spectrum data includes the first time-spectrum data corresponding to the first audio data and the second time-spectrum data corresponding to the second audio data; the target real part data includes the first real part data and the second real part data; the target imaginary part data includes the first imaginary part data and the second imaginary part data. The step of determining the target amplitude data and target phase data of the target audio data using the target real part data and the target imaginary part data, and determining the reference amplitude data of the reference audio data using the reference real part data and the reference imaginary part data, includes: Obtain an amplitude calculation function for amplitude calculation and a phase calculation function for phase calculation. Perform amplitude calculation on the first real part data and the first imaginary part data using the amplitude calculation function to obtain the first amplitude data of the first audio data. Perform phase calculation on the first real part data and the first imaginary part data using the phase calculation function to obtain the first phase data of the first audio data. The amplitude of the second real part data and the second imaginary part data are calculated by the amplitude calculation function to obtain the second amplitude data of the second audio data, and the phase of the second real part data and the second imaginary part data are calculated by the phase calculation function to obtain the second phase data of the second audio data. Based on the first amplitude data and the second amplitude data, the target amplitude data of the target audio data is determined, and based on the first phase data and the second phase data, the target phase data of the target audio data is determined. The reference amplitude data of the reference audio data is obtained by performing amplitude calculation on the reference real part data and the reference imaginary part data using the amplitude calculation function.

3. The method according to claim 2, characterized in that, The process of constructing the original input feature data of the target based on the target amplitude data, the target phase data, and the reference amplitude data includes: Obtain the first phase data and the second phase data from the target phase data; The phase difference between the first phase data and the second phase data is calculated to obtain the first phase difference; Based on the first amplitude data, the second amplitude data, the reference amplitude data, and the first phase difference data, the target original input feature data is constructed.

4. The method according to claim 1, characterized in that, The number of multiple audio data streams is N, where N is a positive integer greater than 2. The target phase data includes N phase data; the N phase data are obtained by performing phase calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams respectively; one audio data stream corresponds to one time-spectrum data stream; the target amplitude data includes N amplitude data, which are obtained by performing amplitude calculations on the real and imaginary parts of the N time-spectrum data corresponding to the N audio data streams respectively. The process of constructing the original input feature data of the target based on the target amplitude data, the target phase data, and the reference amplitude data includes: Based on any two phase data obtained from the N phase data, M phase data pairs associated with the N phase data are constructed; M is a positive integer greater than or equal to N. From the M phase data pairs, a target phase data pair is determined, and the two phase data in the target phase data pair are respectively determined as the first phase data to be processed and the second generation of processed phase data. The phase difference is calculated on the first phase data to be processed and the second phase data to be processed to obtain the target phase difference of the target phase data pair. The target phase difference of the M phase data pairs is obtained when each of the M phase data pairs is determined to be the target phase data pair. Based on the target phase difference of the N amplitude data, the reference amplitude data, and the M phase data pairs, the target original input feature data is constructed.

5. The method according to claim 1, characterized in that, The step of performing speech optimization processing on the original target input feature data to obtain target optimized feature data after separating and removing the reference time-spectrum data from the target time-spectrum data includes: A target speech model for speech optimization processing is obtained, and the target original input feature data is encoded using the target speech model to obtain the encoded feature data corresponding to the target original input feature data. The target speech model is used to concatenate the original input feature data and the encoded feature data to obtain concatenated feature data. The spliced ​​feature data is decoded using the target speech model to obtain the decoded feature data corresponding to the spliced ​​feature data. The decoded feature data is processed by extracting sequence information using the target speech model to obtain the sequence extracted feature data corresponding to the decoded feature data. The sequence extracted feature data is then used as the target optimized feature data after separating and removing the reference time-spectrum data from the target time-spectrum data.

6. The method according to claim 5, characterized in that, The target speech model includes an encoder composed of multiple unit coding layers; The step of obtaining a target speech model for speech optimization processing, and then using the target speech model to encode the target original input feature data to obtain the encoded feature data corresponding to the target original input feature data, includes: Obtain a target speech model for speech optimization processing, and input the original input feature data of the target speech model into the encoder; From the plurality of unit coding layers included in the encoder, obtain the first unit coding layer and the second unit coding layer. The target original input feature data is processed by the first unit encoding layer to obtain first encoded feature data, and the first encoded feature data is processed by the second unit encoding layer to obtain second encoded feature data; the feature dimension of the second encoded feature data is different from the feature dimension of the first encoded feature data. Based on the second encoded feature data, the encoded feature data corresponding to the target original input feature data is determined.

7. The method according to claim 6, characterized in that, Each of the multiple unit coding layers corresponds to a coding sequence number. Each unit coding layer contains a convolutional layer, and the number of filters in a convolutional layer increases with the coding sequence number. When the coding sequence number of the first unit coding layer is less than the coding sequence number of the second unit coding layer, the number of filters in the first convolutional layer is less than the number of filters in the second convolutional layer. The process of performing a first encoding process on the original input feature data through the first unit encoding layer to obtain first encoded feature data, and then performing a second encoding process on the first encoded feature data through the second unit encoding layer to obtain second encoded feature data, includes: The convolutional layer in the first unit coding layer is determined as the first convolutional layer, and the convolutional layer in the second unit coding layer is determined as the second convolutional layer; The original input feature data is subjected to a first convolutional process by the filters in the first convolutional layer to obtain first convolutionally encoded feature data. The first convolutionally encoded feature data is then subjected to a first normalization process to obtain first normalized feature data. Finally, the first normalized feature data is subjected to a first encoding activation process to obtain first encoded feature data. The number of channels in the channel feature dimension of the first encoded feature data is consistent with the number of filters in the first convolutional layer. The first encoded feature data is subjected to a second convolutional process by filters in the second convolutional layer to obtain second convolutional encoded feature data. The second convolutional encoded feature data is then subjected to a second normalization process to obtain second normalized feature data. The second normalized feature data is then subjected to a second encoding activation process to obtain second encoded feature data. The number of channels in the second encoded feature data in the channel feature dimension is consistent with the number of filters in the second convolutional layer. The number of channels in the second encoded feature data in the channel feature dimension is greater than the number of channels in the first encoded feature data in the channel feature dimension.

8. The method according to claim 6, characterized in that, The encoder includes a linear layer for dimensional transformation of the data; The step of determining the encoded feature data corresponding to the target original input feature data based on the second encoded feature data includes: Based on the second encoded feature data, the target original input feature data is determined to correspond to the third encoded feature data; The encoder performs a linear transformation on the feature map corresponding to the third encoded feature data in the channel dimension through a linear layer, and uses the linearly transformed third encoded feature data as the encoded feature data corresponding to the original target input feature data; the size of the feature map corresponding to the linearly transformed third encoded feature data is consistent with the size of the feature map corresponding to the third encoded feature data.

9. The method according to claim 5, characterized in that, Both the encoded feature data and the target original input feature data include channel dimension, time dimension, and frequency dimension in their feature dimensions. The encoded feature data is determined by K1 encoded feature maps, where K1 represents the number of channels in the channel dimension. The encoded feature map is composed of feature extraction data in the time dimension and feature extraction data in the frequency dimension. The target original input feature data is determined by K2 input feature maps, where K2 represents the number of channels in the channel dimension. The input feature map is composed of feature input data in the time dimension and feature input data in the frequency dimension. K1 and K2 are both positive integers. The step of concatenating the target original input feature data and the encoded feature data using the target speech model to obtain concatenated feature data includes: Based on the K2 channels, the K1 encoded feature maps are copied to obtain the K2 encoded feature maps; In the frequency dimension, the feature extraction data from the K2 encoded feature maps and the feature input data from the K2 input feature maps are concatenated to obtain K2 frequency concatenated feature maps; The K2 frequency spliced ​​feature maps are determined as spliced ​​feature data obtained by splicing the target original input feature data and the encoded feature data.

10. The method according to claim 5, characterized in that, The target speech model includes a decoder composed of multiple unit decoding layers; The step of decoding the spliced ​​feature data using the target speech model to obtain the decoded feature data corresponding to the spliced ​​feature data includes: From the plurality of unit decoding layers contained in the decoder, obtain the first unit decoding layer and the second unit decoding layer; The first unit decoding layer performs a first decoding process on the spliced ​​feature data to obtain first decoded feature data, and the second unit decoding layer performs a second decoding process on the first decoded feature data to obtain second decoded feature data. Based on the second decoding feature data, the decoding feature data corresponding to the spliced ​​feature data is determined.

11. The method according to claim 10, characterized in that, In the plurality of unit decoding layers, each unit decoding layer contains a deconvolution layer; The process of performing a first decoding process on the concatenated feature data through the first unit decoding layer to obtain first decoded feature data, and then performing a second decoding process on the first decoded feature data through the second unit decoding layer to obtain second decoded feature data, includes: The deconvolution layer in the first unit decoding layer is determined as the first deconvolution layer, and the deconvolution layer in the second unit decoding layer is determined as the second deconvolution layer; The first deconvolution layer performs a first deconvolution process on the spliced ​​feature data to obtain first deconvolution decoded feature data. The first deconvolution decoded feature data is then subjected to a third normalization process to obtain third normalized feature data. Finally, the third normalized feature data is subjected to a first decoding activation process to obtain first decoded feature data. The first decoded feature data is subjected to a second deconvolution process by the second deconvolution layer to obtain second deconvolution decoded feature data. The second deconvolution decoded feature data is then subjected to a fourth normalization process to obtain fourth normalized feature data. Finally, the fourth normalized feature data is subjected to a second decoding activation process to obtain second decoded feature data.

12. An audio data processing device, characterized in that, The device includes: The time-spectrum data acquisition module is used to acquire target time-spectrum data corresponding to target audio data and reference time-spectrum data corresponding to reference audio data; the reference audio data is noise data in the target audio data, the target audio data is audio data in multiple audio data, the target time-spectrum data includes target real part data and target imaginary part data obtained after performing Fourier transform on the target audio data; the reference time-spectrum data includes reference real part data and reference imaginary part data obtained after performing Fourier transform on the reference audio data. An amplitude and phase calculation module is used to determine the target amplitude data and target phase data of the target audio data using the target real part data and the target imaginary part data, and to determine the reference amplitude data of the reference audio data using the reference real part data and the reference imaginary part data; The speech optimization processing module is used to construct target original input feature data based on the target amplitude data, the target phase data and the reference amplitude data, and to perform speech optimization processing on the target original input feature data to obtain target optimized feature data after separating and removing the reference time spectrum data from the target time spectrum data. An optimized audio acquisition module is used to perform data fusion processing on the time-spectrum data corresponding to multiple audio data to obtain fused time-spectrum data. The fused time-spectrum data is then filtered using the target optimization feature data to obtain filtered time-frequency data corresponding to the target time-spectrum data separated from the fused time-spectrum data. Finally, an inverse Fourier transform is performed on the filtered time-spectrum data to obtain the target audio optimization data corresponding to the target audio data.

13. A computer device, characterized in that, Including memory and processor; The memory is connected to the processor, the memory is used to store computer programs, and the processor is used to invoke the computer programs so that the computer device performs the method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded and executed by a processor to cause a computer device having the processor to perform the method of any one of claims 1-11.

15. A computer program product, characterized in that, Includes a computer program / instruction that, when executed by a processor, implements the method according to any one of claims 1-11.

Images