Method executed by electronic device, electronic device, storage medium and program product

Abstract

The embodiment of the invention provides a method executed by electronic equipment, the electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: dividing a first audio feature of a target audio to obtain at least two groups of sub-audio features; for each sub-audio feature, performing inter-block and / or inter-frame feature processing through a corresponding neural network to obtain a second audio feature corresponding to the sub-audio feature; and obtaining a target audio feature based on the second audio feature. Optionally, the method performed by the electronic device may be performed using an artificial intelligence model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method executed by an electronic device, an electronic device, a storage medium, and a program product. Background Art

[0002] In the existing audio processing technology, the audio sequence is generally processed through the DPRNN (Dual-Path Recurrent Neural Network) module. The DPRNN module connects multiple DPRNN blocks in series, and uses these blocks to model the input feature vectors locally (Intra-chunk) and globally (Inter-chunk) respectively, and outputs new feature vectors.

[0003] However, existing methods obtain poor audio processing results. Summary of the invention

[0004] The present disclosure provides a method executed by an electronic device, an electronic device, a storage medium, and a program product. The technical solution is as follows:

[0005] According to one aspect of an embodiment of the present disclosure, a method performed by an electronic device is provided, the method including:

[0006] Dividing the first audio feature of the target audio to obtain at least two groups of sub-audio features;

[0007] For each group of sub-audio features, perform inter-block and / or inter-frame feature processing through a corresponding neural network to obtain a second audio feature corresponding to the group of sub-audio features;

[0008] A target audio feature is obtained based on the second audio feature.

[0009] In a feasible embodiment, before the step of dividing the first audio feature of the target audio, the method further includes:

[0010] Performing full-band coding and sub-band coding on the frequency domain features of the target audio to obtain full-band coding features and sub-band coding features;

[0011] Based on the full-band coding feature and the sub-band coding feature, a frequency domain coding feature is obtained;

[0012] Obtaining the first audio feature based on the frequency domain coding feature;

[0013] The method further comprises:

[0014] Performing full-band decoding and sub-band decoding on the target audio features respectively to obtain full-band decoding features and sub-band decoding features;

[0015] A processed target audio is obtained based on the full-band decoding features and the sub-band decoding features.

[0016] In a feasible embodiment, the frequency domain features of the target audio are respectively subjected to full-band coding and sub-band coding to obtain full-band coding features and sub-band coding features, including:

[0017] Performing feature extraction on the acquired target audio to obtain frequency domain features of the target audio;

[0018] Performing full-band coding on the frequency domain features to obtain full-band coding features;

[0019] The frequency domain features are divided to obtain multiple groups of sub-frequency domain features, and sub-band coding is performed on each group of sub-frequency domain features to obtain corresponding sub-band coding features.

[0020] In a feasible embodiment, performing full-band encoding on the frequency domain feature to obtain the full-band encoding feature includes:

[0021] Performing feature processing on the frequency domain features to obtain amplitude features;

[0022] Performing full-band coding on the amplitude feature to obtain a full-band coding feature;

[0023] Or, the frequency domain features are divided to obtain at least two groups of sub-frequency domain features, including:

[0024] Performing feature processing on the frequency domain features to obtain amplitude features;

[0025] The amplitude feature is divided to obtain at least two groups of sub-frequency domain features.

[0026] In a feasible embodiment, performing full-band decoding and sub-band decoding on the target audio feature to obtain full-band decoding features and sub-band decoding features includes:

[0027] Splitting the target audio feature to obtain a full-band audio feature and at least two groups of sub-band audio features;

[0028] Perform full-band decoding based on the full-band coding feature and the full-band audio feature to obtain a full-band decoding feature;

[0029] Sub-band decoding is performed based on the sub-band coding feature and the sub-band audio feature to obtain sub-band decoding features.

[0030] In a feasible embodiment, performing full-band decoding based on the full-band coding feature and the full-band audio feature to obtain the full-band decoding feature includes:

[0031] By using a full-band decoder, full-band decoding is performed based on the full-band coding features output by each level of the full-band encoder and the full-band audio features to obtain a full-band mask feature;

[0032] A full-band decoding feature is obtained based on the full-band mask feature and the full-band encoding feature, or a full-band decoding feature is obtained based on the full-band mask feature and the frequency domain feature.

[0033] In a feasible embodiment, performing sub-band decoding based on the sub-band coding feature and the sub-band audio feature to obtain the sub-band decoding feature includes:

[0034] By each sub-band decoder, sub-band decoding is performed based on the corresponding sub-band coding feature and the sub-band audio feature to obtain a sub-band mask feature;

[0035] A subband decoding feature is obtained based on the subband mask feature and the subband coding feature, or a subband decoding feature is obtained based on the subband mask feature and the frequency domain feature.

[0036] In a feasible embodiment, obtaining the processed target audio based on the full-band decoding feature and the sub-band decoding feature includes:

[0037] Based on the full-band decoding feature and the sub-band decoding feature, a frequency domain decoding feature is obtained by fusing;

[0038] A time-frequency transform is performed on the frequency domain decoding features to obtain processed audio information.

[0039] In a feasible embodiment, obtaining the frequency domain coding features for feature modeling in the audio features based on the full-band coding features and the sub-band coding features includes:

[0040] The full-band coding feature and the sub-band coding feature are fused to obtain a fused coding feature;

[0041] Performing dimensionality reduction on the fused coding features to obtain frequency domain coding features in the audio features for feature modeling;

[0042] The performing full-band decoding and sub-band decoding on the target audio feature to obtain full-band decoding features and sub-band decoding features includes:

[0043] The target audio features are dimensionally upgraded, and full-band decoding and sub-band decoding are performed based on the dimensionally upgraded target audio features to obtain full-band decoding features and sub-band decoding features.

[0044] In a feasible embodiment, the first audio feature includes feature information obtained by performing intra-block and / or intra-frame feature processing on audio features of the target audio;

[0045] The method further comprises: iteratively performing a preset number of intra-block and / or intra-frame, inter-block and / or inter-frame feature processing operations, and using the target audio feature obtained in the last iteration as the target audio feature finally outputted;

[0046] The audio features of the target audio in the current iteration are the target audio features obtained in the previous iteration.

[0047] In a feasible embodiment, the full-band coding feature and the sub-band coding feature are fused to obtain a fused coding feature, including:

[0048] Perform feature processing on all-band coding features to obtain global coding features;

[0049] The global coding feature and the sub-band coding feature are connected, and the connected features are fused to obtain a fused coding feature.

[0050] In a feasible embodiment, the target audio includes audio data input in an instant call;

[0051] The method further comprises:

[0052] Based on the target audio features, the audio data in the instant call is enhanced and the enhanced audio data is output.

[0053] According to another aspect of an embodiment of the present disclosure, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method performed by the electronic device.

[0054] According to another aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method performed by the electronic device are implemented.

[0055] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the steps of the method performed by the electronic device when the computer program is executed by a processor.

[0056] The technical solution provided by the embodiments of the present disclosure has the following beneficial effects:

[0057] The present disclosure provides a method, an electronic device, a storage medium and a program product executed by an electronic device. Specifically, when performing feature modeling between blocks and / or frames, the first audio feature of the target audio is first divided to obtain at least two groups of sub-audio features; on this basis, for each group of sub-audio features, feature processing between blocks and / or frames is performed through a corresponding neural network to obtain a second audio feature corresponding to the group of sub-audio features, and then the target audio feature is obtained based on the second audio feature. The implementation of the present disclosure uses independent network nodes (neural networks corresponding to each sub-audio feature) to model the same segment of features between blocks and / or frames, expands the capacity of the network by increasing the number of network nodes, and effectively improves network performance while keeping the network calculation complexity unchanged. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for describing the embodiments of the present disclosure are briefly introduced below.

[0059] Figure 1 A flowchart of a method performed by an electronic device provided in an embodiment of the present disclosure;

[0060] Figure 2a A schematic diagram of a network structure of a neural network provided in an embodiment of the present disclosure;

[0061] Figure 2b A schematic diagram of a network structure of a DPE module provided in an embodiment of the present disclosure;

[0062] Figure 3 A schematic diagram of a feature vector representation of intra-frame processing provided by an embodiment of the present disclosure;

[0063] Figure 4 A schematic diagram of a feature vector representation of inter-frame processing provided by an embodiment of the present disclosure;

[0064] Figure 5a A feature processing flow chart provided for an embodiment of the present disclosure;

[0065] Figure 5b A schematic diagram of intra-frame and inter-frame characteristics provided by an embodiment of the present disclosure;

[0066] Figure 5c Another schematic diagram of intra-frame and inter-frame characteristics provided by an embodiment of the present disclosure;

[0067] Figure 6 A schematic diagram of an audio processing flow provided by an embodiment of the present disclosure;

[0068] Figure 7 A schematic diagram of a network structure provided in an embodiment of the present disclosure;

[0069] Figure 8 A schematic diagram of a network structure of a full-band-sub-band encoder module provided in an embodiment of the present disclosure;

[0070] Fig. 9 A processing flow chart of a full-band-sub-band encoder module provided in an embodiment of the present disclosure;

[0071] Fig.10 A schematic diagram of a network structure of a full-band-sub-band decoder module provided in an embodiment of the present disclosure;

[0072] Fig.11 A framework diagram related to a feature processing module provided in an embodiment of the present disclosure;

[0073] Fig.12 A processing flow chart related to a feature processing module provided in an embodiment of the present disclosure;

[0074] Fig.13a A schematic diagram of an audio processing flow provided by an embodiment of the present disclosure;

[0075] Fig.13b A schematic diagram of the structure of a FSPEN network provided in an embodiment of the present disclosure;

[0076] Fig.14a A signal diagram of a target audio provided by an embodiment of the present disclosure;

[0077] Fig.14b A signal schematic diagram of a processed target audio provided by an embodiment of the present disclosure;

[0078] Fig.15 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0079] The following description with reference to the accompanying drawings is provided to facilitate a comprehensive understanding of the various embodiments of the present disclosure as defined by the claims and their equivalents. This description includes various specific details to facilitate understanding but should be considered as exemplary only. Therefore, one of ordinary skill in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of the present disclosure. In addition, for the sake of clarity and conciseness, descriptions of well-known functions and structures may be omitted.

[0080] The terms and expressions used in the following specification and claims are not limited to their dictionary meanings, but are merely used by the inventor to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustration purposes only and not for the purpose of limiting the present disclosure as defined in the appended claims and their equivalents.

[0081] It should be understood that the singular forms "a", "an", and "the" may also include plural references unless the context clearly indicates otherwise. Thus, for example, reference to "a component surface" includes reference to one or more such surfaces. When we refer to an element as being "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or the one element and the other element may establish a connection relationship through an intermediate element. In addition, "connected" or "coupled" as used herein may include wireless connection or wireless coupling.

[0082] The term "include" or "may include" refers to the presence of the corresponding disclosed functions, operations or components that can be used in various embodiments of the present disclosure, rather than limiting the presence of one or more additional functions, operations or features. In addition, the term "include" or "have" may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components or combinations thereof, but should not be interpreted as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components or combinations thereof.

[0083] The term "or" used in various embodiments of the present disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B. When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple, or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" may be implemented as parameter A including A1 or A2 or A3, or may be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0084] Unless defined differently, all terms (including technical terms or scientific terms) used in the present disclosure have the same meanings as understood by those skilled in the art described in the present disclosure. Common terms as defined in dictionaries are interpreted as having meanings consistent with the context in the relevant technical field, and should not be interpreted ideally or overly formally unless clearly defined in the present disclosure.

[0085] At least some functions of the device or electronic device provided in the embodiments of the present disclosure may be implemented by an AI (Artificial Intelligence) model, such as at least one module among multiple modules of the device or electronic device may be implemented by an AI model. Functions associated with AI may be performed by a non-volatile memory, a volatile memory, and a processor.

[0086] The processor may include one or more processors. In this case, the one or more processors may be general-purpose processors, such as a central processing unit (CPU), an application processor (AP), etc., or pure graphics processing units, such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an AI-specific processor, such as a neural processing unit (NPU).

[0087] The one or more processors control the processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. The predefined operating rules or artificial intelligence models are provided by training or learning.

[0088] Here, providing by learning means obtaining a predefined operating rule or an AI model with desired characteristics by applying a learning algorithm to a plurality of learning data. The learning can be performed in the device or electronic device itself in which the AI ​​according to the embodiment is executed, and / or can be implemented by a separate server / system.

[0089] The AI ​​model may include multiple neural network layers. Each layer has multiple weight values, and each layer performs neural network calculations by calculating between the input data of the layer (such as the calculation results of the previous layer and / or the input data of the AI ​​model) and the multiple weight values ​​of the current layer. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks.

[0090] A learning algorithm is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0091] According to the present disclosure, at least one step performed in an electronic device, such as convolving the quantized representation of the first top-level feature to obtain the semantic group index of the first bottom-level feature, can be implemented using an artificial intelligence model. The processor of the electronic device can perform preprocessing operations on the data to convert it into a form suitable for use as an input to the artificial intelligence model. The artificial intelligence model can be obtained through training. Here, "obtained through training" means training a basic artificial intelligence model with multiple training data through a training algorithm to obtain a predefined operation rule or artificial intelligence model configured to perform the desired feature (or purpose).

[0092] Speech Enhancement (SE) or noise suppression is an important front-end module for voice calls, automatic speech recognition (ASR) and hearing aids, which aims to improve the quality and intelligibility of noisy speech. The application of deep neural networks (DNNs) in speech enhancement has received extensive attention. Many DNN-based methods have achieved impressive performance in speech enhancement tasks, but the performance improvement also increases the model overhead.

[0093] In the related art, a network structure of a convolution recurrent network (CRN) is proposed. Taking advantage of the advantages of CNN and RNN, CRN can not only capture the local pattern of the spectrogram, but also simulate the dependency between consecutive frames. In this technology, a dual-path recurrent neural network (DPRNN) is proposed to solve the problem of long sequence modeling, in which the long sequence features are divided into smaller blocks and processed by RNNs within and between blocks, thereby reducing the length of the sequence that each RNN needs to process. The dual-path structure solves the problem of long sequence modeling well and improves network performance. In the DPRNN network structure of the related art, multiple DPRNN blocks are connected in series, which can use these blocks to process the input feature vector locally (Intra-chunk) and globally (Inter-chunk) respectively, and output a new feature vector. However, the network nodes of the DPRNN module are few, the network cannot extract good features, and the network performance is poor.

[0094] As can be seen from the above methods, audio processing methods are increasingly implemented using neural networks. Although they have achieved good speech noise reduction effects, in practical applications, the size and computational complexity of the model are important factors that limit whether it can be used in products. Therefore, in speech noise reduction products that need to run in real time, such as TWS headphones, hearing aids, etc., ultra-lightweight models must be used. The computational complexity of neural networks is generally very high. To apply the model on devices such as headphones (on device), the complexity needs to be reduced, and the real-time nature of the system needs to be met.

[0095] In view of the problems existing in the related art or the areas that need improvement, an ultra-lightweight network is proposed in the disclosure, which can be used for real-time speech enhancement tasks. Specifically, a novel ultra-lightweight real-time speech enhancement network (FSPEN) is proposed. The network structure includes a DEP module (Dual Path enhancer with path Expending). When the constructed feature vector is modeled between blocks and / or frames through the DEP module, a separate neural network is used to model the same segment of features between blocks and / or frames, and the network capacity is expanded by increasing the number of network nodes, thereby improving the performance of the network and keeping the computational complexity of the network unchanged.

[0096] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present disclosure and the technical effects produced by the technical solutions of the present disclosure. It should be noted that the following embodiments can refer to, draw on or combine with each other, and the same terms, similar features and similar implementation steps in different embodiments will not be described repeatedly.

[0097] Figure 1 A flow chart of a method performed by an electronic device provided by an embodiment of the present disclosure is shown, and the method can be performed by a terminal or a server. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted device, a smart headset, etc.; the server can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to these.

[0098] The following describes a method performed by an electronic device according to an embodiment of the present disclosure.

[0099] First, a lightweight network structure provided by an embodiment of the present disclosure is described: Figure 2a As shown, in the artificial neural network (ANN), a neural network ANN-1 for intra-block and / or intra-frame feature modeling is included, and N neural networks ANN-21 to ANN-2N for inter-block and / or inter-frame feature modeling are respectively connected to the ANN-1, wherein n∈(1,N). The value of N is related to the model size, computational complexity, and the properties of the audio features (such as the number of frequency points per frame), and can be adjusted according to actual needs, which is not limited in the present disclosure. The multiply-accumulate operation (MAC) in the network provided in the embodiment of the present disclosure is only 98M, and the model size is 80k, which is a lightweight network.

[0100] Specifically, Figure 1 As shown, the method performed by the electronic device provided in the embodiment of the present disclosure includes the following steps S101 to S103:

[0101] Step S101: dividing the first audio feature of the target audio to obtain at least two groups of sub-audio features.

[0102] The first audio feature of the target audio may include a time domain feature, a frequency domain feature, or a time-frequency domain feature. Optionally, the first audio feature includes feature information obtained by performing feature processing within a block and / or within a frame on the audio feature of the target audio. Figure 2a As shown, when performing feature processing within a block and / or within a frame, the input audio feature can be processed, and when obtaining a first audio feature, the first audio feature can be divided to obtain multiple groups of sub-audio features for feature processing between blocks and / or between frames.

[0103] Among them, the feature processing within the block includes processing several features divided into the same block in the audio sequence, and the feature processing between blocks includes processing the same feature in different blocks in the audio sequence. For example, assuming that the first 1-200 features in the target audio are divided into block 1 and the first 201-400 features are divided into block 2, when performing feature processing within the block, the features included in each block are processed; when performing feature processing between blocks, the features at the same position between different blocks are processed, such as processing the first feature in block 1 (the first feature in the audio sequence) and the first feature in block 2 (the 201st feature in the audio sequence).

[0104] The intra-frame feature processing includes processing several features belonging to the same frame in the audio sequence, and the inter-frame feature processing includes processing the same feature belonging to different frames in the audio sequence. Figure 3 and Figure 4 As shown, assuming that the target audio includes 3 frames, when performing feature processing within a frame, the frequency points (a, b, c) of the 1st frame, the frequency points (a, b, c) of the 2nd frame, and the frequency points (a, b, c) of the 3rd frame are processed respectively in units of frames; when performing feature processing between frames, the frequency points a belonging to the 1st frame, the 2nd frame, and the 3rd frame, the frequency points b belonging to the 1st frame, the 2nd frame, and the 3rd frame, and the frequency points c belonging to the 1st frame, the 2nd frame, and the 3rd frame are processed respectively in units of features.

[0105] Optionally, when the first audio feature is divided, it can be divided evenly or unevenly. For example, when the first audio feature includes 20 features, it can be evenly divided into 4 groups of sub-audio features, each group including 5 features; it can also be unevenly divided into 2 groups, one group including at least one feature and the other group including the remaining features.

[0106] Step S102: For each group of sub-audio features, inter-block and / or inter-frame feature processing is performed through a corresponding neural network to obtain a second audio feature corresponding to the group of sub-audio features.

[0107] Among them, Figure 2a and Figure 5a As shown, when multiple groups of sub-audio features are obtained through division, each group of sub-audio features can be subjected to inter-block and / or inter-frame feature processing through independent ANNs to obtain the second audio features output by each ANN. Figure 5a In the example of, when the first audio feature is divided into three groups of sub-audio features, three independent ANNs (ANN-a, ANN-b, ANN-c) are used to perform inter-block and / or inter-frame feature processing to obtain three groups of second audio features; when processing the first audio feature belonging to the frequency domain, it is equivalent to each ANN processing the frequency domain features of one or several sub-bands.

[0108] Optionally, when the number of sub-audio feature groups obtained by division is greater than the number of independent neural networks arranged in the network architecture, an ANN can be used to perform inter-block and / or inter-frame feature processing on two or more adjacent groups of sub-audio features.

[0109] Optionally, any one of the multiple ANNs connected in parallel may process a group of sub-audio features obtained through division, or may process multiple groups of sub-audio features obtained through division.

[0110] Optionally, ANN can be a recurrent neural network (RNN), a sequence model based on an attention mechanism (Transformer), a long short-term memory network (LSTM), etc., which can be adjusted according to needs, and the present disclosure does not limit this. Accordingly, the network used for feature modeling can be DPRNN, DPTNet (dual-path transformer network, a dual-path sequence network based on an attention mechanism), etc.

[0111] Step S103: obtaining a target audio feature based on the second audio feature.

[0112] Alternatively, if Figure 1 As shown, all second audio features can be combined to obtain the target audio features output by the network for feature modeling.

[0113] Optionally, to better improve the performance of the network, multiple models such as Figure 2a The feature processing module shown can perform multiple feature processing iterations based on the audio features of the target audio.

[0114] Among them, Fig.12 As shown, the feature processing operations within the block and / or within the frame, between the blocks and / or between the frames may be iteratively performed a preset number of times, and the target audio features obtained in the last iteration are used as the target audio features finally outputted. In the current iteration, the audio features of the target audio are the target audio features obtained in the previous iteration.

[0115] It is understandable that in one iterative process, the following is first used: Figure 1 The ANN-1 shown can be a Bi-LSTM (Bidirectional Long Short-Term Memory), and can also be expressed as a Bi-ANN, which performs intra-block and / or intra-frame feature processing on the input features to obtain the first audio feature, and divides the first audio feature to obtain at least two groups of sub-audio features, and respectively inputs the at least two groups of sub-audio features into different ANN-2n (LSTM can be used) for inter-block and / or inter-frame feature processing to obtain the second audio features output by each ANN, and then fuses all the second audio features to obtain the target audio features of the current iteration. If this iteration is the first time, the features of the input feature processing module are the output of the previous module in the model; if this iteration is the second or subsequent iterations, the features of the input feature processing module are the target audio features obtained in the previous iteration.

[0116] When the feature processing module provided in the embodiment of the present disclosure processes the feature vector within a block (when the input is in the frequency domain, it can be within a frame), an ANN is reused for processing, which can effectively maintain the temporal nature of the features within the block and / or frame. When processing the feature vector between blocks (when the input is in the frequency domain, it can be between frames), a separate ANN network is used to process the same segment of features between blocks. If the input is in the frequency domain, it is equivalent to each ANN network processing one or more sub-bands.

[0117] In the embodiment of the present disclosure, a frame path extension method is proposed (applied to Fig.13b For N groups of subbands, N neural networks are used to perform inter-block and / or inter-frame processing. The implementation of this method improves network capacity while maintaining extremely low complexity. The difference between DPE and the dual-path network in the related art is as follows: Figure 5b and Figure 5c As shown (the left side is an example of intra-frame processing, and the right side is an example of inter-frame processing). Figure 5b In , all inter-frame features use the same set of network coefficients. Figure 5c In the example, the 2 subbands of group 1 use the same set of network coefficients, while the 2 subbands of group 2 use another set of network coefficients. When DPE is applied, the size of the model increases, but the complexity remains unchanged. In the corresponding FSPEN model, after applying path expansion, the model size increases from 10k to 20k, for example.

[0118] The DEP module can be composed of three cascaded DEP modules (that is, iterative processing is performed three times). The detailed structure of the DEP module is as follows: Figure 2b As shown. Exemplarily, the processing of frequency domain features is taken as an example: for the features input to the DEP module, intra-frame processing is performed through the Bi GRU (Gate Recurrent Unit, a recurrent neural network) layer, and then the features corresponding to each frame obtained by processing are spliced ​​through the fully connected (FC) layer, and after processing through the normalization layer, the normalized features are fused with the input features to obtain the first audio feature. On this basis, the first audio feature is divided to obtain at least two groups of sub-audio features, and each group of sub-audio features is input into the GRU layer for inter-frame processing, and then the features corresponding to each group obtained by processing are spliced ​​through the FC layer, and finally the spliced ​​features are fused with the first audio feature to obtain the final output of the DEP module.

[0119] In a feasible embodiment, for audio processing, considering that different frequency bands in an audio signal contain different information, low frequency bands may contain higher energy components, such as tones and long-lasting sounds, while high frequency bands may have low energy components, such as noise and rapidly decaying sounds. In the real-time noise reduction scheme involved in audio processing in the related art, for input frequency domain features, only the frequency domain of the full band is considered for processing. In small model processing, due to the small number of network nodes and the small number of network layers, the network cannot extract good features, so the network performance needs to be improved.

[0120] In order to achieve high performance on a small model, the disclosed embodiment also provides a full-band-sub-band (full-band-sub-band) network structure. Figure 6 As shown, the feature processing module provided in the above embodiment is adapted, and a full-band-sub-band encoder is arranged before it (the sub-band encoder is used to assist the full-band encoder in extracting audio features), and a full-band-sub-band decoder is arranged after it, so that the frequency domain features input to the feature processing module include full-band coding features and sub-band coding features. Therefore, the full-band features and frequency band features can be processed, and then in the decoding stage, full-band decoding and sub-band decoding can be performed respectively for the target audio features output by the feature processing module, so that the network can learn the relationship between the frequency domain features in the full band and sub-band, smoothly transition the boundary effects brought by the sub-band features, and effectively improve the network performance.

[0121] Optionally, before executing step S101 to divide the first audio feature of the target audio, the method further includes steps S001 to S003:

[0122] Step S001: performing full-band coding and sub-band coding on the frequency domain features of the target audio to obtain full-band coding features and sub-band coding features.

[0123] Among them, when processing the frequency domain features, full-band encoding and sub-band encoding can be performed respectively through the full-band-sub-band encoder to obtain full-band encoding features and sub-band encoding features. It can be understood that full-band encoding is equivalent to encoding the frequency domain features of the input encoder as a whole, and sub-band encoding is equivalent to encoding a certain section of the frequency domain features of the input encoder (each section of the features is encoded separately).

[0124] Step S002: Based on the full-band coding features and the sub-band coding features, frequency domain coding features for feature modeling in the audio features are obtained.

[0125] Among them, Figure 6As shown, the full-band coding features and sub-band coding features obtained by the full-band-sub-band encoder can be input into the feature processing module for feature modeling. It can be understood that since the full-band-sub-band encoder processes frequency domain features, when its output is used as the input of the feature processing module, the frequency domain coding features are input into the feature processing module.

[0126] Optionally, when the feature processing module processes the time domain features, the time domain coding features input to the feature processing module can be obtained by extracting the time domain features from the target audio through a feature extractor provided by the relevant technology and then encoding them through an encoder.

[0127] Step S003: Obtain a first audio feature based on the frequency domain coding feature.

[0128] Among them, in steps S101 to S103 provided in the above embodiment, when adapted to the processing of frequency domain features, the frequency domain coding features can be used as the input of the feature processing module, and the feature processing module performs intra-frame feature processing on the frequency domain coding features to obtain the first audio features.

[0129] Accordingly, after executing step S103 to obtain the target audio features, the method further includes steps S104 and S105:

[0130] Step S104: performing full-band decoding and sub-band decoding on the target audio features to obtain full-band decoding features and sub-band decoding features.

[0131] Among them, Figure 6 As shown, the target audio features output by the feature processing module can be input into the full-band-sub-band decoder for full-band decoding and sub-band decoding respectively. It can be understood that full-band decoding is equivalent to decoding the input frequency domain features as a whole, and sub-band decoding is equivalent to decoding a certain section of the input frequency domain features (each section feature is decoded separately).

[0132] Optionally, there is also a connection relationship between the full-band-subband encoder and the full-band-subband decoder, that is, the output of the full-band-subband encoder can also be used as the input of the full-band-subband decoder, and the full-band-subband decoder combines the full-band coding features, sub-band coding features and target audio features to perform decoding processing.

[0133] Step S105: Obtaining processed target audio based on the full-band decoding features and the sub-band decoding features.

[0134] Among them, Figure 6 As shown, the full-band decoding features and the sub-band decoding features output by the full-band-sub-band decoder can be processed to obtain the processed target audio that is finally output.

[0135] In a possible embodiment, if Figure 7 , 8 As shown in 9, the full-band-subband encoder includes a full-band encoder and at least two sub-band encoders arranged in parallel. Through the full-band encoder, the network can learn global feature information, and the feature information is evenly distributed over the entire frequency band. Through the sub-band encoder, sub-band division can be performed first in the frequency domain, so that the network can allocate more nodes to the low-frequency band among the limited network nodes, so that the network can learn more low-frequency band information, and also learn certain high-frequency band information; combining the full-band encoder and the sub-band encoder, the network can better learn global and local information in the frequency domain under the condition of limited nodes. Optionally, before encoding, the network provided in the embodiment of the present disclosure is also provided with a feature extraction module, such as an STFT (short-time Fourier transform) module.

[0136] Accordingly, the following description is made by taking the target audio as an audio signal with a sampling rate of 16k and a length of 4s as an example. In step S001, full-band coding and sub-band coding are performed on the frequency domain features of the target audio to obtain full-band coding features and sub-band coding features, including steps A1 to A3:

[0137] Step A1: extracting features from the acquired target audio to obtain frequency domain features of the target audio.

[0138] Among them, Figure 7 , 8 As shown in Figure 9, for the target audio of the input network, feature extraction can be first performed through STFT to obtain the frequency domain features of the input full-band-subband encoder.

[0139] Optionally, the signal can be processed in the time-frequency (TF) domain using STFT, where the input signal x∈R of sample T can be 1×T Divided into J w L segments (frames) (with some overlap); the process of converting the input signal x to the TF domain is shown in the following formula (1):

[0140] X=STFT(x) ......Formula (1)

[0142] Where X∈R L×K×2 , K is the number of frequency bands. The last dimension is the real and imaginary parts of the frequency bands.

[0143] like Fig.13b As shown, the model provided by the embodiment of the present disclosure includes three stages: full-band and sub-band encoders, DPE module, and full-band and sub-band decoders.

[0144] Optionally, during the feature extraction process, the input audio signal s1 is framed, windowed, and short-time Fourier transformed. For a signal with a sampling rate of 16k and a duration of n seconds, there are sampling point data L = n*16000, and an FFT with a scale of s_n can be performed, that is, the number of sampling points per frame is s_n, and the overlapping area between frames is s_n / 2 (50% overlap, which can be set according to requirements); after STFT, the number of frames k = L / (s_n / 2)-1; the number of frequency points per frame is: f = s_n / 2. On this basis, the real part and the imaginary part of the frequency domain are taken out respectively to output the feature vector f k The dimension is [k, 2*f].

[0145] For calculating the STFT of 512 points of 4s signal, the number of points per frame is s_n / 2=512 / 2=256, and the eigenvector f is obtained. k The dimension is [249, 256], that is, there are 249 frames, each frame has 256 frequency points, and each frequency point is represented by a real part and an imaginary part, where k = {0, 1, 2, ..., 249}, and k represents the frame number.

[0146] Step A2: Perform full-band encoding on the frequency domain features to obtain full-band encoding features.

[0147] For the full-band encoder, its input features are the frequency domain features of the full band. The dimension of the feature vector is [1, 1, 249, 512], where each dimension represents the batch size, the number of channels, the number of frames, and the number of frequency points (including the real and imaginary parts). The full-band encoder can use a 3-layer convolutional network (such as Fig. 9 As shown), a higher-dimensional feature vector is obtained by encoding, and the feature vector dimension is [B, C, T, F], where B is the batch size, C is the number of channels, T is the number of frames, and F is the number of points in the frequency direction after encoding. In this example, the output feature vector dimension is [1, 32, 249, 64].

[0148] Among them, the setting of the number of convolutional layers included in the full-band encoder is related to the computational complexity and network performance, which can be adjusted according to needs, and the present disclosure does not limit this.

[0149] Step A3: dividing the frequency domain features to obtain at least two groups of sub-frequency domain features, and performing sub-band coding on each group of sub-frequency domain features to obtain corresponding sub-band coding features, wherein the dimensions of the full-band coding features and the sub-band coding features are consistent.

[0150] Among them, Figure 7 , 8As shown in Figure 9, the sub-band feature encoding method is adopted. First, the frequency band is divided into N sub-bands (the more sub-bands it is divided into, the finer the feature processing is, but more parallel-connected encoders will be introduced, which will increase the model complexity. Considering the performance and model complexity, 4-6 encoders can be used).

[0151] This embodiment uses a 4-subband division method as an example for explanation: Corresponding to the frequency domain characteristics obtained in the previous step, the data of 256 frequency points in each frame can be divided into 4 subbands f1 k , f2 k , f3 k , f4 k The frequency points contained in each sub-band are {1~32}, {33~64}, {65~128}, {129~256}, and the corresponding frequencies are 0~2k, 2k~4k, 4k-8k, 8k~16k.

[0152] In the subband encoder, the frequency domain features of the full band are divided into multiple groups of sub-frequency domain features. Each group of sub-frequency domain features needs to be encoded using a different subband encoder, thereby achieving parallel encoding and reducing complexity. If the number of subband divisions is N, there are N subband encoders corresponding to each group of sub-frequency domain features. This embodiment has 4 subband encoders, each of which processes the corresponding sub-frequency domain features and encodes them into higher-dimensional feature vectors. The following is an example of the processing of each subband encoder:

[0153] Calculate the eigenvector x1 k :Pair band f1 k The dimension [249,64] is expanded to [1, 1, 249,64], and a 2D convolution operation is performed. The output channel is 32, the convolution kernel is 5*5, and the step size is 1*1, and the output vector x1 is obtained. k [1, 32, 249, 64].

[0154] Calculate the eigenvector x2 k :Pair band f2 k The dimension [249,64] is expanded to [1, 1, 249,64], and a 2D convolution operation is performed. The output channel is 32, the convolution kernel is 5*5, and the step size is 1*1, and the output vector x2 is obtained. k [1, 32, 249, 64].

[0155] Calculate the eigenvector x3 k :Pair with f3 k The dimension [249, 128] is expanded to [1, 1, 249, 128], and a 2D convolution operation is performed. The output channel is 32, the convolution kernel is 5*6, and the step size is 1*2, and the output vector x3 is obtained. k [1, 32, 249, 64].

[0156] Calculate the eigenvector x4 k : Pair with f4 k The dimension [249,256] is expanded to [1, 1, 249,256], and a 2D convolution operation is performed. The output channel is 32, the convolution kernel is 5*6, and the step size is 1*4, and the output vector x4 is obtained. k [1, 32, 249, 64].

[0157] After being processed by the encoder, the characteristic vector xi of each subband k The dimensions are all [1, 32, 249, 64]. In order to facilitate the fusion of features obtained with the full-band encoder, the feature dimensions of each sub-band encoder are set to be consistent with those of the full-band encoder.

[0158] In the disclosed embodiment, a full-band-subband encoder is used to encode a complex spectrum. The full-band encoder can use a 3-layer CNN to extract features from the complex spectrum. CNN can effectively extract features from audio signals, but considering that deep CNN and a large number of channels cannot be used in lightweight models, the features extracted become a bottleneck for network performance. In the disclosed embodiment, a sub-band encoder is used to assist the full-band encoder in feature extraction. In addition, considering that low-frequency signals occupy more sub-bands than high-frequency signals, the human ear is more sensitive to low-frequency signals. Therefore, if Fig.13b As shown, the subbands can be divided into N groups, and the subbands in the same group have the same number of frequency bands. The subband signals are encoded using N 1-layer Conv encoders. Accordingly, the full-band encoder can be used to extract the global feature U at the frame level. ful , and the subband encoder can be used to extract the local features U at the subband level sub .

[0159] Optionally, before inputting the frequency domain features into the full-band-subband encoder, feature processing can also be performed on the frequency domain features of the target audio to obtain its amplitude features; then, one encoder in the full-band-subband encoder can input the frequency domain features, and the other encoder can input the amplitude features, so that the two encoders process different features, enriching the features learned by the network and improving network performance.

[0160] Among them, the feature X can be expressed as X(k)=a_k+b_k*j, which contains the signal amplitude M(k)=(a_k^2+b_k^2)^0.5 and phase P information a_k=M(k)*cos(P), b_k=M(k)*sin(P), k indicates the frequency point. a_k indicates the real part of the corresponding frequency point, and b_k*j indicates the imaginary part of the corresponding frequency point.

[0161] Optionally, taking the absolute value of feature X can obtain the amplitude feature of the signal, and the amplitude feature can be described as M=abs(X).

[0162] Optionally, when the amplitude feature is used as the input of the sub-band encoder, the amplitude feature is first divided to obtain at least two groups of sub-frequency domain features, and then the corresponding sub-band encoder is used to perform sub-band encoding for each group of sub-frequency domain features.

[0163] In a possible embodiment, if Fig.11 and 12 As shown, in step S002, frequency domain coding features are obtained based on the full-band coding features and the sub-band coding features, including steps B1 and B2:

[0164] Step B1: Fusing the full-band coding feature and the sub-band coding feature to obtain a fused coding feature.

[0165] Optionally, the fusion method may include two methods:

[0166] First, if Fig. 9 As shown, the sub-band coding features output by each sub-band encoder can be fused first to obtain fused sub-band coding features (the sub-band coding features belong to a whole at this time), and then the fused sub-band coding features can be fused with the full-band coding features.

[0167] Second, when multiple sub-band coding features are obtained, all sub-band coding features are directly fused with the full-band coding features.

[0168] The fusion method can be weighted or fused through a network, such as feature fusion through a full connection layer (FC). The fused features (fused coding features) include the audio features extracted by the full-band encoder and the audio features extracted by the sub-band encoder.

[0169] Optionally, in step B1, the full-band coding feature and the sub-band coding feature are fused to obtain a fused coding feature, including steps B11 and B12:

[0170] Step B11: Perform feature processing on all-band coding features to obtain global coding features.

[0171] Among them, Fig.13b As shown, considering that the embodiment of the present disclosure adopts a lightweight model, in order to enable the network to learn more comprehensive global features, after obtaining the full-band coding features output by the full-band encoder, the full-band coding features can be feature processed (such as 1x1 convolution) to obtain global coding features.

[0172] Step B12: Connect the global coding feature and the sub-band coding feature, and fuse the connected features to obtain a fused coding feature.

[0173] Among them, Fig.13b As shown, the global coding feature is first connected with at least two groups of sub-band coding features through a concatenation layer, and then the connected features are processed through a bottleneck layer to obtain a fused coding feature.

[0174] In the present disclosure, Fig.13b As shown in Figure 1, the feature fusion module is used to merge global features (full-band encoding features) and local features (sub-band encoding features). Since the full-band encoder uses fewer layers, resulting in a smaller receptive field and difficulty in obtaining global information of the frequency band, a 1x1 convolution can be used to process the features of the full-band encoder along the frequency dimension. Then, the processed full-band encoding features and sub-band encoding features are connected in series and fused through the bottleneck layer. Finally, the fused feature U∈R L×M×P Sent to the feature processing module (such as DPE module).

[0175] Step B2: performing dimension reduction on the fused coding features to obtain frequency domain coding features.

[0176] In order to reduce the complexity of calculation, the fused features can be reduced in dimension through a convolution layer (Conv). For example, the input feature vector is [1, 32, 249, 64], and the new vector [1, 16, 249, 64] is obtained through the convolution operation.

[0177] Optionally, when the fused coding features do not need to be reduced in dimension, the fused coding features can be directly regarded as frequency domain coding features.

[0178] Correspondingly, in step S104, full-band decoding and sub-band decoding are performed on the target audio features to obtain full-band decoding features and sub-band decoding features, including: dimensionality upgrading is performed on the target audio features, and full-band decoding and sub-band decoding are performed based on the target audio features after dimensionality upgrading to obtain full-band decoding features and sub-band decoding features.

[0179] Among them, the dimension of the target audio feature after dimensionality upgrade is consistent with the dimension of the fused coding feature.

[0180] Alternatively, if Fig.11 and Fig.12 As shown in the figure, the feature dimension can be restored to the same dimension as the encoder through the convolution layer, that is, the dimension can be increased. An example of the operation of increasing the dimension through a convolution layer is as follows: the input feature vector [1, 16, 249, 64] is converted into a new vector [1, 32, 249, 64] through the convolution operation.

[0181] In a possible embodiment, if Figure 7 , 10 As shown in FIG. 13 , the full-band-sub-band decoder includes a full-band decoder and at least two sub-band decoders connected in parallel. The number of sub-band decoders is the same as the number of sub-band encoders. Optionally, the decoders are connected to their respective corresponding encoders through a skip connection. Through the corresponding full-band encoder and full-band decoder, the network can learn more global information. Through the corresponding sub-band encoder and sub-band decoder, the network will focus on learning more local or mid- and low-frequency information.

[0182] Optionally, in step S104, full-band decoding and sub-band decoding are respectively performed on the target audio feature to obtain full-band decoding features and sub-band decoding features, including steps C1 to C3:

[0183] Step C1: split the target audio feature to obtain a full-band audio feature and at least two groups of sub-band audio features.

[0184] Among them, Fig.11 and Fig.12 As shown, after obtaining the target audio features, before entering the decoding stage, the target audio features are first split to obtain full-band audio features and sub-band audio features.

[0185] Among them, Fig.13a As shown, adapted to the connection relationship between the sub-band encoders and the sub-band decoders (the encoders and decoders included are connected one-to-one), at least two groups of sub-band audio features can be obtained based on the number of sub-band coding features.

[0186] Optionally, the target audio features may be split into full-band audio features and sub-band audio features through a fully connected layer.

[0187] Step C2: Perform full-band decoding based on the full-band encoding features and the full-band audio features to obtain full-band decoding features.

[0188] Alternatively, if Fig.13a As shown in the figure, the input of the full-band decoder is the output of the feature processing module and the output of the full-band encoder of the corresponding layer. The two outputs are concatenated and can be used as the input of this module. The number of convolutional layers arranged by the full-band decoder is the same as that of the full-band encoder.

[0189] Among them, adapted to the jump connection structure of each level in the full-band decoder and each level in the full-band encoder, the embodiment of the present disclosure can perform full-band decoding based on the full-band coding features and full-band audio features output by each level of the full-band encoder through the full-band decoder to obtain the full-band mask feature M ful ; Then, based on the full-band mask feature and the full-band encoding feature U ful Full-band decoding features are obtained, or full-band decoding features are obtained based on full-band mask features and frequency domain features.

[0190] The embodiment of the present disclosure is provided with a feature division module after the DPE module, such as Fig.13b As shown in the figure, it is used to restore the features output by DPE into global features (full-band audio features) and local features (sub-band audio features). Then, the features processed by the DPE module are decoded by the full-band-sub-band decoder. The full-band decoder is connected to the full-band encoder in a jump connection manner, and a three-layer deconvolution method (the number of convolution layers used by the full-band decoder and the full-band encoder is the same) is used to obtain the complexity spectrum mask (full-band mask feature, M ful ). To reduce complexity, 1x1 convolution can be used before each deconvolution layer to reduce the number of feature channels from 2 to 1.

[0191] Optionally, the full-band enhanced complex spectrum (full-band decoding feature) Z ful ∈R L×K×H byU ful and the full-band mask feature M ful The elements between are multiplied to obtain the following formula (2):

[0192] Z ful =U ful ·M ful ...Formula (2)

[0194] Optionally, the full-band decoding feature Z ful (k) is composed of the frequency domain feature X(k) of the target audio and the full-band mask feature M ful (k) is obtained by multiplying the elements between them, as shown in the following formula (3):

[0195] Z ful (k) = X(k)·M ful (k) .....Formula (3)

[0197] Step C3: performing sub-band decoding based on the sub-band coding feature and the sub-band audio feature to obtain sub-band decoding features.

[0198] Optionally, the sub-band decoder input is the output of the feature processing module and the output of the corresponding sub-band encoder, and the two outputs are concatenated and used as the input of this module. The number of sub-band decoders is the same as the number of sub-band encoders.

[0199] Among them, Fig.13b As shown, it is adapted to the one-to-one connection structure between each parallel-connected sub-band decoder and each parallel-connected sub-band encoder. In the embodiment of the present disclosure, each sub-band decoder can perform sub-band decoding based on the corresponding sub-band coding feature and sub-band audio feature to obtain a sub-band mask feature; then, based on the sub-band mask feature M sub and subband coding feature U sub The subband decoding feature is obtained, or the subband decoding feature is obtained based on the subband mask feature and the frequency domain feature.

[0200] Among them, through the processing of the subband decoder, the subband mask feature M can be obtained sub The structure of the subband decoder is similar to that of the full-band decoder, but the subband decoder uses a deconvolution layer. Subband enhanced complex spectrum (subband decoding feature) Z sub ∈R L×K×H byU sub and M sub The elements between are multiplied to obtain the following formula (4):

[0201] Z sub =U sub ·M sub ...Formula (4)

[0203] Among them, through the processing of the subband decoder, the subband mask feature M can be obtained sub (k). Subband decoding feature Z sub (k) can be obtained by the frequency domain features X(k) and M of the target audio. sub (k) is obtained by multiplying the elements between them, as shown in the following formula (5):

[0204] Z sub (k) = X(k)·M sub (k) ......Formula (5)

[0206] Optionally, since the subband mask features output by at least two subband decoders and the subband coding features output by at least two subband encoders are included, when the features output by the subband decoders are processed to obtain the subband decoding features, one of the following operations may be performed:

[0207] Operation 1: First, concatenate the mask features of each sub-band to obtain a total sub-band mask feature, and then multiply the total sub-band mask feature with the frequency domain feature of the target audio to obtain the total sub-band decoding feature; or, synchronously concatenate the coding features of each sub-band to obtain a total sub-band coding feature, and then multiply the total sub-band mask feature and the vertical sub-band coding feature to obtain the total sub-band decoding feature.

[0208] Operation 2: For each sub-band mask feature, multiply it with the corresponding sub-band coding feature to obtain the corresponding sub-band decoding feature; or multiply it with the corresponding sub-frequency domain feature to obtain the corresponding sub-band decoding feature; then, concatenate each group of sub-band decoding features to obtain the total sub-band decoding feature.

[0209] In a feasible embodiment, obtaining the processed target audio based on the full-band decoding feature and the sub-band decoding feature in step S105 includes steps D1 and D2:

[0210] Step D1: Obtain frequency domain decoding features based on the full-band decoding features and the sub-band decoding features.

[0211] Alternatively, if Fig.13a As shown, the full-band decoding feature Z ful and subband decoding feature Z sub Perform weighted sum processing to obtain frequency domain decoding features.

[0212] Optionally, at least two groups of sub-band decoding features Z may be included sub , the sub-band decoding features of each group can be concatenated first, and the total sub-band decoding features can be added to the full-band decoding features to obtain the frequency domain decoding features.

[0213] Optionally, the outputs of the two decoders are fused, and the fusion method can be weighted or through a fully connected network. To reduce computational complexity, this embodiment can use a weighted method to fuse, and the weights of the two decoders can be set to 0.5 and 0.5 (only as an example, and the embodiment of the present disclosure is not limited to this).

[0214] Step D2: performing time-frequency transformation on the frequency domain decoding features to obtain processed audio information.

[0215] Alternatively, if Fig.13a As shown, the fused features are subjected to ISTFT (inverse Fourier transform) to obtain the processed target audio with time domain features.

[0216] like Fig.13bAs shown in Figure 2, after full-band decoding and sub-band decoding, the results of the two decoders (full-band decoding features and sub-band decoding features) are added together to obtain the final frequency domain signal Z. Finally, the final output waveform y∈R can be obtained through iSTFT. 1×T , as shown in the following formula (6):

[0217] y=iSTFT(Z ful +Z sub )...Formula (6)

[0219] Optionally, if both the full-band encoder and the sub-band encoder encode based on frequency domain features, the corresponding decoding features can be expressed as Z_x=X*mask1 and Z_x=X*mask2; accordingly, formula (6) corresponds to y=iSTFT(X*mask1+X*mask2).

[0220] Optionally, if one of the full-band encoders or sub-band encoders encodes based on the amplitude feature M, the corresponding decoded feature can be expressed as: Z_m = M*mask1, in which case the amplitude information is m_1; when the other encoder encodes based on the frequency domain feature X, the corresponding decoded feature can be expressed as: Z_x = X*mask2, in which case the amplitude information is m_2 and the phase information is phase_2. Accordingly, m_1, m_2 and phase_2 are fused (e.g., the fused amplitude is (m_1+m_2) / 2 and the phase is phase_2) to obtain the fused feature Z. Finally, the final time domain signal is obtained by calculating y = iSTFT(Z) corresponding to formula (6).

[0221] Combine the following Fig.14a and Fig.14b The effects that can be achieved by the embodiments of the present disclosure are described.

[0222] in, Fig.14a It can be a signal diagram of an input target audio. Fig.14b The following is a schematic diagram of a target audio signal output after being processed by the method provided in an embodiment of the present disclosure. Comparing the two schematic diagrams, it can be seen that a lot of noise is reduced in the processed target audio, and the part outside the noise is effectively retained.

[0223] The method provided by the embodiment of the present disclosure has strong robustness in processing non-stationary noise and audio signals with low signal-to-noise ratio, and can be applied to tasks such as speech enhancement, speech noise reduction, speech separation, and speech extraction. Exemplarily, when applied to speech noise reduction tasks, it can be applied to the scenario of voice communication through headphones, and the voice signal of voice communication can be denoised in real time to improve the performance of speech noise reduction and the quality of voice calls. Optionally, the embodiment of the present disclosure can also be applied to video communication scenarios to process audio signals in video communication in real time to improve the quality of video communication.

[0224] Optionally, adapted to instant call scenarios, the target audio in the embodiments of the present disclosure may be input audio data. Accordingly, the method provided in the embodiments of the present disclosure further includes: based on the target audio features, performing enhancement processing on the audio data in the instant call, and outputting the enhanced audio data.

[0225] For example, when objects A and B in a scene wear headphones to talk on the phone, the headphones can execute the method provided by the embodiment of the present disclosure, and the audio data (target audio) input into the headphones of object A is the voice data output by object B, and the audio data (target audio) input into the headphones of object B is the voice data output by object A; wherein, when the headphones of object A execute the method provided by the embodiment of the present disclosure to obtain the target audio features, the audio data in the instant call is enhanced and processed, and the enhanced audio data (processed target audio) is output, which corresponds to the voice data output from the headphones of object A; when the headphones of object B execute the method provided by the embodiment of the present disclosure to obtain the target audio features, the audio data in the instant call is enhanced and processed, and the enhanced audio data (processed target audio) is output, which corresponds to the voice data output from the headphones of object B.

[0226] An embodiment of the present disclosure also provides an electronic device, which includes a processor and, optionally, may also include a transceiver and / or a memory coupled to the processor, and the processor is configured to execute the steps of the method provided in any optional embodiment of the present disclosure.

[0227] Fig.15 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present disclosure is shown in FIG. Fig.15 As shown, Fig.15The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present disclosure.

[0228] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0229] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0230] The memory 4003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0231] The memory 4003 is used to store the computer program for executing the embodiment of the present disclosure, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0232] An embodiment of the present disclosure provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0233] The embodiments of the present disclosure also provide a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiments when executed by a processor.

[0234] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than that shown or described in the text.

[0235] It should be understood that, although the flowchart of the embodiment of the present disclosure indicates each operation step by arrows, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present disclosure, the implementation steps in each flowchart can be executed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times. In scenarios with different execution times, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present disclosure does not limit this.

[0236] The above text and drawings are provided only as examples to help readers understand the present disclosure. They are not intended and should not be interpreted as limiting the scope of the present disclosure in any way. Although certain embodiments and examples have been provided, based on the contents disclosed herein, it is obvious to those skilled in the art that the embodiments and examples shown can be changed without departing from the scope of the present disclosure, and other similar implementation means based on the technical ideas of the present disclosure are adopted, which also fall within the protection scope of the embodiments of the present disclosure.

Claims

1. A method performed by an electronic device, characterized in that: include: Dividing the first audio feature of the target audio to obtain at least two groups of sub-audio features; For each group of sub-audio features, perform inter-block and / or inter-frame feature processing through a corresponding neural network to obtain a second audio feature corresponding to the group of sub-audio features; A target audio feature is obtained based on the second audio feature.

2. The method according to claim 1, characterized in that: Before the step of dividing the first audio feature of the target audio, the method further includes: Performing full-band coding and sub-band coding on the frequency domain features of the target audio to obtain full-band coding features and sub-band coding features; Based on the full-band coding feature and the sub-band coding feature, a frequency domain coding feature is obtained; Obtaining the first audio feature based on the frequency domain coding feature; The method further comprises: Performing full-band decoding and sub-band decoding on the target audio features respectively to obtain full-band decoding features and sub-band decoding features; A processed target audio is obtained based on the full-band decoding features and the sub-band decoding features.

3. The method according to claim 2, characterized in that The full-band coding and sub-band coding are respectively performed on the frequency domain features of the target audio to obtain full-band coding features and sub-band coding features, including: Performing feature extraction on the acquired target audio to obtain frequency domain features of the target audio; Performing full-band coding on the frequency domain features to obtain full-band coding features; The frequency domain features are divided to obtain at least two groups of sub-frequency domain features, and sub-band coding is performed on each group of sub-frequency domain features to obtain corresponding sub-band coding features.

4. The method according to claim 3, characterized in that The performing full-band encoding on the frequency domain feature to obtain the full-band encoding feature includes: Performing feature processing on the frequency domain features to obtain amplitude features; Performing full-band coding on the amplitude feature to obtain a full-band coding feature; Or, the frequency domain features are divided to obtain at least two groups of sub-frequency domain features, including: Performing feature processing on the frequency domain features to obtain amplitude features; The amplitude feature is divided to obtain at least two groups of sub-frequency domain features.

5. The method according to claim 2, characterized in that: The performing full-band decoding and sub-band decoding on the target audio feature to obtain full-band decoding features and sub-band decoding features includes: Splitting the target audio feature to obtain a full-band audio feature and at least two groups of sub-band audio features; Perform full-band decoding based on the full-band coding feature and the full-band audio feature to obtain a full-band decoding feature; Sub-band decoding is performed based on the sub-band coding feature and the sub-band audio feature to obtain sub-band decoding features.

6. The method according to claim 5, characterized in that The performing full-band decoding based on the full-band coding feature and the full-band audio feature to obtain the full-band decoding feature includes: By using a full-band decoder, full-band decoding is performed based on the full-band coding features output by each level of the full-band encoder and the full-band audio features to obtain a full-band mask feature; A full-band decoding feature is obtained based on the full-band mask feature and the full-band encoding feature, or a full-band decoding feature is obtained based on the full-band mask feature and the frequency domain feature.

7. The method according to claim 5, characterized in that The performing sub-band decoding based on the sub-band coding feature and the sub-band audio feature to obtain the sub-band decoding feature includes: By each sub-band decoder, sub-band decoding is performed based on the corresponding sub-band coding feature and the sub-band audio feature to obtain a sub-band mask feature; A subband decoding feature is obtained based on the subband mask feature and the subband coding feature, or a subband decoding feature is obtained based on the subband mask feature and the frequency domain feature.

8. The method according to claim 2, characterized in that: The step of obtaining the processed target audio based on the full-band decoding feature and the sub-band decoding feature includes: Obtaining frequency domain decoding features based on the full-band decoding features and the sub-band decoding features; A time-frequency transform is performed on the frequency domain decoding features to obtain processed audio information.

9. The method according to claim 2, characterized in that: The obtaining of frequency domain coding features based on the full-band coding features and the sub-band coding features includes: The full-band coding feature and the sub-band coding feature are fused to obtain a fused coding feature; Performing dimensionality reduction on the fused coding features to obtain frequency domain coding features; The performing full-band decoding and sub-band decoding on the target audio feature to obtain full-band decoding features and sub-band decoding features includes: The target audio features are dimensionally upgraded, and full-band decoding and sub-band decoding are performed based on the dimensionally upgraded target audio features to obtain full-band decoding features and sub-band decoding features.

10. The method according to claim 9, characterized in that The fusing the full-band coding feature and the sub-band coding feature to obtain a fused coding feature includes: Perform feature processing on all-band coding features to obtain global coding features; The global coding feature and the sub-band coding feature are connected, and the connected features are fused to obtain a fused coding feature.

11. The method according to claim 1, characterized in that The first audio feature includes feature information obtained by performing intra-block and / or intra-frame feature processing on the audio feature of the target audio; The method further comprises: iteratively performing a preset number of intra-block and / or intra-frame, inter-block and / or inter-frame feature processing operations, and using the target audio feature obtained in the last iteration as the target audio feature finally outputted; The audio features of the target audio in the current iteration are the target audio features obtained in the previous iteration.

12. The method according to any one of claims 1 to 11, characterized in that The target audio includes audio data input in the instant call; The method further comprises: Based on the target audio features, the audio data in the instant call is enhanced and the enhanced audio data is output.

13. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images