Sound source identification method and device and storage medium
Through the codec of multi-layer loop feature learning, time-domain and frequency-domain features are extracted, the problem of real-time recognition and separation of audio sources in the prior art is solved, real-time processing of complex acoustic signals is realized, and low-latency and efficient processing is met. Requirements have improved terminal functions and user experience.
Patent Information
- Application Number
- CN202310870313.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-14
- Publication Date
- 2025-01-14
AI Technical Summary
The existing technology is difficult to realize real-time identification and separation of sound sources in complex acoustic signals. Traditional Unet networks cannot process streaming audio signals in real time, and the non-causal modeling of time and frequency domain dimensions by the encoding network cannot meet low latency and high efficiency. To handle the requirements.
Using a codec with multi-layer loop feature learning, the real-time processing of streaming audio signals is realized by extracting time domain features and combining frequency domain features, establishing a time domain causal model and non-causal feature learning of frequency domain dimensions to achieve fine-grainedness Feature learning.
Real-time identification and separation of complex acoustic signals is realized, model performance is improved, low-latency and efficient processing needs such as mobile phones and smart homes, and terminal functions and user experience are enhanced.
Smart Images
Figure CN119314508A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the fields of signal science and artificial intelligence, and in particular to a sound source recognition method, device and storage medium. Background Art
[0002] Our daily lives are full of various complex sound signals. Although the human ear can easily distinguish complex sound signals, it is very difficult for our smart devices to distinguish the various sound sources in complex acoustic signals. In recent years, with the development of intelligent signal processing technology, sound source recognition and separation technology has attracted more and more attention. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the present disclosure provides a sound source recognition method, device and storage medium.
[0004] According to a first aspect of an embodiment of the present disclosure, a method for identifying a sound source is provided, the method comprising:
[0005] In response to obtaining an audio signal to be subjected to sound source identification, spectral feature data of the audio signal is determined; a network model for sound source identification is called, the network model comprising an encoding network, a feature transition network and a decoding network; the spectral feature data is input into the encoding network of the network model, and the encoding network extracts the time domain features of the audio signal based on the spectral feature data, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal; the sound source embedding layer features are input into the feature transition network of the network model, and the feature transition network performs splicing of the multiple sound source embedding layer features and then inputs them into the decoding network, and the decoding network performs feature restoration on the sound source embedding layer features to obtain the sound source signal.
[0006] In one embodiment, the coding network includes M network modules, each of which includes a recurrent neural network and at least one convolutional neural network, the value of M is a positive integer greater than or equal to 1, and the number of convolutional neural networks included in different network modules is the same or different; the coding network extracts the time domain features of the audio signal based on the spectral feature data, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, to obtain the sound source embedding layer features of the audio signal, including: extracting the time domain features based on the recurrent neural network in the Lth network module, and extracting the frequency domain features based on at least one convolutional neural network in the Lth network module; repeating the above process until the Mth completes the time domain feature extraction and the convolutional neural network completes the frequency domain feature extraction, to obtain the sound source embedding layer features of the audio signal; wherein the value of L is a positive integer between 1 and M, and the time domain feature dimensions and frequency domain feature dimensions extracted by the 1 to M network modules decrease in sequence.
[0007] In one embodiment, the recurrent neural network includes a first fully connected layer, a recurrent neural network layer and a second fully connected layer; the extraction of time domain features based on the recurrent neural network in the Lth network module includes: performing dimensionality reduction processing on the spectral feature data based on the first fully connected layer of the recurrent neural network in the Lth network module, and obtaining the reduced dimensionality time domain features after passing through the recurrent neural network layer; inputting the reduced dimensionality time domain features into the second fully connected layer, and restoring the reduced dimensionality time domain features by the second fully connected layer to obtain the time domain features of the spectral feature data.
[0008] In one embodiment, the convolutional neural network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer; the first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer, and the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected via a residual connection; frequency domain features are extracted based on at least one convolutional neural network in the Lth network module, including: based on the spectrum feature data processed by the recurrent neural network in the Lth network module, sliding the spectrum feature data through the convolution kernel of the first two-dimensional convolutional layer with a specific step size The first two-dimensional convolution layer is used to perform a convolution operation on the spectral feature data to obtain a first convolution result; the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the second two-dimensional convolution layer slides on the spectral feature data with a specific step size, and the spectral feature data is convolved to obtain a second convolution result; the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectral feature data with a specific step size, and the spectral feature data is convolved to obtain the spectral feature data after the frequency domain feature is extracted; the first two-dimensional convolution layer and the third two-dimensional convolution layer are connected by residuals to ensure the effective transmission of the spectral feature data. In one embodiment, the time domain dimension of the convolution kernel during the convolution operation in the convolutional neural network is 1.
[0009] According to a second aspect of an embodiment of the present disclosure, a sound source recognition device is provided, which comprises: an extraction device, in response to obtaining an audio signal to be subjected to sound source recognition, determining spectral feature data of the audio signal; an identification device, calling a network model for sound source recognition, the network model comprising a coding network, a feature transition network and a decoding network; inputting the spectral feature data into the coding network of the network model, the coding network extracting the time domain features of the audio signal based on the spectral feature data, and extracting the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, to obtain a sound source embedding layer feature of the audio signal; inputting the sound source embedding layer feature into the feature transition network of the network model, the feature transition network performing splicing of the multiple sound source embedding layer features and then inputting the features into the decoding network, the decoding network performing feature restoration on the sound source embedding layer features, to obtain a sound source signal.
[0010] In one embodiment, the coding network includes M network modules, each of which includes a recurrent neural network and at least one convolutional neural network, the value of M is a positive integer greater than or equal to 1, and the number of convolutional neural networks included in different network modules is the same or different; the sound source identification device adopts the following method: the coding network extracts the time domain features of the audio signal based on the spectrum feature data, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal: extracting the time domain features based on the recurrent neural network in the Lth network module, and extracting the frequency domain features based on at least one convolutional neural network in the Lth network module; repeating the above process until the Mth completes the time domain feature extraction and the convolutional neural network completes the frequency domain feature extraction, so as to obtain the sound source embedding layer features of the audio signal; wherein the value of L is a positive integer between 1 and M, and the time domain feature dimensions and frequency domain feature dimensions extracted by the 1 to M network modules are reduced in sequence.
[0011] In one embodiment, the recurrent neural network includes a first fully connected layer, a recurrent neural network layer and a second fully connected layer; the recognition device extracts time domain features based on the recurrent neural network in the Lth network module in the following manner: the spectral feature data is subjected to dimension reduction processing based on the first fully connected layer of the recurrent neural network in the Lth network module, and the reduced dimension time domain features are obtained after passing through the recurrent neural network layer; the reduced dimension time domain features are input into the second fully connected layer, and the reduced dimension time domain features are restored by the second fully connected layer to obtain the time domain features of the spectral feature data.
[0012] In one embodiment, the convolutional neural network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer; the first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, and the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer; the recognition device extracts frequency domain features based on at least one convolutional neural network in the Lth network module in the following manner: based on the spectral feature data processed by the recurrent neural network in the Lth network module, the convolution kernel of the first two-dimensional convolutional layer slides on the spectral feature data with a specific step size, and the spectral features are extracted. The first convolution operation is performed on the data to obtain a first convolution result; the first convolution result is input into a second two-dimensional convolution layer, and the convolution kernel of the second two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain a second convolution result; the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain spectrum feature data after frequency domain feature extraction; the first two-dimensional convolution layer and the third two-dimensional convolution layer are connected through a residual connection to ensure the effective transmission of the spectrum feature data.
[0013] In one embodiment, the time domain dimension of the convolution kernel during the convolution operation in the convolutional neural network is 1.
[0014] According to a third aspect of an embodiment of the present disclosure, there is provided a sound source recognition device, comprising:
[0015] processor;
[0016] a memory for storing processor-executable instructions;
[0017] The processor is configured to: execute the sound source recognition method described in the first aspect or any one of the implementations of the first aspect.
[0018] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal can perform the method described in the first aspect or any one of the implementations of the first aspect.
[0019] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: based on the acquired audio signal to be identified, the spectral feature data of the audio signal is determined, and a network model for sound source identification is called to implement streaming processing of the spectral feature data, thereby realizing real-time processing of the audio signal, achieving the effect of real-time audio recognition, enhancing the functionality of the terminal, and enhancing the user experience.
[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0022] Figure 1 The figure is a flow chart showing an application scenario according to an exemplary embodiment.
[0023] Figure 2 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.
[0024] Figure 3 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.
[0025] Figure 4 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.
[0026] Figure 5 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.
[0027] Figure 6 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.
[0028] Figure 7 It is a block diagram of a network model according to an exemplary embodiment.
[0029] Figure 8 The present invention is a block diagram of a sound source recognition device according to an exemplary embodiment.
[0030] Fig. 9 The present invention is a block diagram of a device for sound source recognition according to an exemplary embodiment. DETAILED DESCRIPTION
[0031] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.
[0032] In the accompanying drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limitations on the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.
[0033] The sound source recognition method provided by the embodiment of the present disclosure can be applied to application scenarios where multiple sound sources in a mixed sound source signal need to be recognized. By designing a sound source recognition and separation algorithm that can be processed in real time, single-frame streaming reasoning can be achieved; by building a learning network with multiple layers of cyclic features, the learning of effective features of multiple sound sources is improved, and the overall performance of the model is improved; the embodiment of the present disclosure can achieve low-latency processing on mobile phones and smart home products.
[0034] Figure 1 FIG. 2 shows a schematic diagram of a multi-source scene in an exemplary embodiment of the present disclosure. Figure 1 As shown, there are multiple sound sources including sound source 1, sound source 2, ..., sound source n.
[0035] In the related technology, sound source recognition can be applied to a variety of scenarios. For example, in the figure, a mixed audio signal is captured by a terminal microphone, and the mixed audio signal contains complex sound source data, such as: people talking, music, noise generated by vehicles, etc. By performing sound source processing operations on the mixed audio signal, one or more groups of single sound source data are obtained, that is, the target sound source described in the figure. The sound source recognition method provided by the embodiment of the present disclosure can be applied to a variety of scenarios such as sound source positioning, extracting recordings of speakers in conference application scenarios, identifying and classifying specific sound sources, and separating musical instruments, and can be expanded in technical fields such as speech separation, speaker recognition, and sound source recognition.
[0036] In the related art, there are many methods that can be used to identify specific sound sources. These include non-negative matrix decomposition, sound source separation methods based on sparse matrices, independent component analysis, and sound source separation methods based on neural networks. Among them, the sound source separation methods based on neural networks commonly include Unet models. However, there are the following technical problems in the related art: the encoding network constructed by the encoder and decoder of the sound source separation method based on the traditional Unet network is a non-causal model. Although it has good algorithm performance, it cannot perform real-time processing on streaming audio signals, and thus cannot meet actual product requirements; the encoding network of the sound source separation method based on the traditional Unet network performs non-causal modeling of the time domain dimension and the frequency domain dimension with the same weight, and cannot perform fine-grained feature learning for specific dimensions alone; the codec structure constructed based on the traditional Unet network sound source separation method is complicated, and most of them are composed of multi-scale convolutional neural networks (CNN) stacked, which cannot meet the low-latency and high-efficiency processing availability requirements of mobile phones, smart home products, and wearable products.
[0037] In view of this, an embodiment of the present disclosure provides a sound source recognition method. In the sound source recognition method provided by the present disclosure, a codec based on multi-layer cyclic feature learning extracts time domain features and then identifies streaming audio signal features, thereby extracting effective streaming feature information of the target sound source.
[0038] The sound source recognition method involved in the embodiments of the present disclosure can be understood as a sound source recognition service that receives audio signals in real time, processes the audio signals and outputs them in real time. For example, sound source positioning, based on the clean sound source information, the location of the sound source can be estimated, such as the direction angle and relative distance of the target sound source. In the conference scene, after being processed by this technical solution, the clean voice signal of the conference speaker can be obtained, and the speaker recording, long-distance signal transmission, audio translation, etc. can be performed. Specific sound source recognition and classification, in a multi-target sound source scenario, separation and recognition can be performed based on the estimated sound source signal, and the sound perception scene information in the current environment can be output to assist the perception function of mobile phones and smart homes. Instrument separation, in a multi-target sound source scenario, separation operations can be performed on music, such as separation of vocals and accompaniment, separation of multiple instrument tracks, etc., which can be applied to mobile phone karaoke scenes and audio editing scenes.
[0039] In one implementation, the stream processing reasoning method is implemented by establishing a time domain causal model. Causal features are learned in the time domain dimension while non-causal features are learned in the frequency domain dimension, and the two features are layered and weighted to achieve fine-grained feature learning for specific dimensions, thereby compensating for the performance loss of the causal model and improving the performance of the model as a whole.
[0040] Figure 2 is a flow chart of a sound source recognition method according to an exemplary embodiment. Figure 2 As shown, the sound source identification method includes the following steps:
[0041] In step S21, in response to obtaining an audio signal to be subjected to sound source identification, frequency spectrum feature data of the audio signal is determined.
[0042] In the embodiments of the present disclosure, the audio signal to be identified as a sound source generally refers to a multi-source mixed audio signal. The multi-source mixed audio signal contains multiple sound sources, each of which has its own unique and distinguishing sound source characteristics. Of course, if the mixed audio signal includes a single sound source, the solution provided in the embodiments of the present disclosure can also be applied to finally identify the single sound source.
[0043] In the disclosed embodiment, the terminal detects the triggering of the sound source recognition instruction, starts to receive the audio signal, and converts the received audio signal into spectral feature data. For example, the terminal can capture the audio signal through a microphone (Mics), and extract the spectral feature data of the audio signal through a short-time Fourier transform (STFT) of the captured audio signal. The spectral feature data can be a spectrogram or a spectrogram.
[0044] In step S22, a network model for sound source recognition is called, where the network model includes an encoding network, a feature transition network, and a decoding network.
[0045] In the disclosed embodiment, the coding network mainly includes one or more network modules, and the spectrum feature data is processed by a recurrent neural network and at least one convolutional neural network in the network module. The convolutional neural networks included in different network modules can be the same or different. Through this structure, real-time processing of spectrum feature data is achieved.
[0046] In step S23, the spectral feature data is input into the encoding network of the network model, and the encoding network extracts the time domain features of the spectral feature data based on the spectral feature data, and obtains the sound source embedding layer features of the audio signal based on the time domain features of the spectral feature data obtained after extracting the time domain features.
[0047] In step S24, the sound source embedding layer features are input into the feature transition network of the network model, and the feature transition network performs multi-sound source embedding layer features concatenation and inputs into the decoding network, and the decoding network performs feature restoration on the sound source embedding layer features to obtain the sound source signal. It should be understood that the encoding network and the decoding network are symmetrical structures, and the convolutional neural network in the encoding network is jump-connected with the symmetrical deconvolutional neural network in the decoding network.
[0048] In the disclosed embodiment, the feature transition network is composed of two layers of recurrent neural networks connected by residual connections, and the number of hidden layer nodes is 128, so that the features output by the encoding network are effectively transferred to the decoding network.
[0049] In the disclosed embodiment, the decoding network has a similar structure to the encoding network and uses the same parameters for restoration. The convolutional neural network is the same as the convolutional neural network in the encoding network and uses an upsampling operation for feature restoration.
[0050] In the disclosed embodiment, based on the real-time recognition of the frequency spectrum feature data, the recognized audio features are output.
[0051] The following embodiment of the present disclosure describes the coding network of the network model, assuming that M is the number of network modules, and L is the network module currently processing spectrum feature data, where L is a positive integer from 1 to M, where M is the number of network modules.
[0052] Figure 3 is a flow chart of a sound source recognition method according to an exemplary embodiment. Figure 3 As shown, the sound source identification method includes the following steps:
[0053] In step S31, time domain features are extracted based on the recurrent neural network in the Lth network module, and frequency domain features are extracted based on at least one convolutional neural network in the Lth network module. In the disclosed embodiment, the time domain features and frequency domain features in the audio features are extracted by the recurrent neural network and the convolutional neural network in the network module, thereby achieving one-time processing of the audio features.
[0054] In step S32, it is determined whether the time domain feature extraction and the frequency domain feature extraction by the convolutional neural network are completed for the Mth time domain feature extraction and the frequency domain feature extraction by the convolutional neural network are completed. If the time domain feature extraction and the frequency domain feature extraction by the convolutional neural network are not completed for the Mth time domain feature extraction and the convolutional neural network are not completed, the above process is repeated.
[0055] In step S33, if the Mth time domain feature extraction is completed and the convolutional neural network completes the frequency domain feature extraction, the sound source embedding layer features of the audio signal are obtained.
[0056] In the disclosed embodiment, the time domain features of the audio signal are extracted based on the spectrum feature data through the coding network, and the frequency domain features of the audio signal are extracted based on the audio time domain data obtained after the time domain features are extracted, so as to obtain the sound source embedding layer features of the audio signal. The audio features are processed by multiple network modules to achieve the dimension reduction and feature extraction of the spectrum feature data.
[0057] In the disclosed embodiment, the recurrent neural network in the network module is composed of a first fully connected layer, a recurrent neural network layer and a second fully connected layer.
[0058] In the disclosed embodiment, the extraction of time domain features and frequency domain features of spectral feature data is achieved through recurrent neural networks and convolutional neural networks in one or more network modules.
[0059] The following is an explanation of the recurrent neural network in the network module in the embodiment of the present disclosure.
[0060] Figure 4 is a flow chart of a sound source recognition method according to an exemplary embodiment. Figure 4 As shown, the sound source identification method includes the following steps:
[0061] In step S41, the spectral feature data is subjected to dimension reduction processing based on the first fully connected layer of the recurrent neural network in the Lth network module, and the reduced-dimensional time domain features are obtained after passing through the recurrent neural network layer.
[0062] In the disclosed embodiment, the spectral feature data is subjected to dimension reduction processing through the first fully connected layer, so that the processed spectral feature data can enter the recurrent neural network; and the time domain feature is extracted through the recurrent neural network layer. For example, the dimension of the original spectral feature data is 4, and the original spectral feature data is compressed through the first fully connected layer, and the dimension of the compressed spectral feature data is 3, which can then be sent to the recurrent neural network layer for time domain feature extraction.
[0063] In step S42, the reduced-dimensional time domain features are input into the second fully connected layer, and the reduced-dimensional time domain features are restored by the second fully connected layer to obtain the time domain features of the spectral feature data.
[0064] In the disclosed embodiment, the reduced-dimensional time domain features are restored through the second fully connected layer to restore the time domain features of the spectrum feature data to the original dimension. For example, for 3-dimensional spectrum data with an original dimension of 4, after dimensionality reduction compression and time domain feature extraction through the first fully connected layer and the recurrent neural network layer, the second fully connected layer can restore it to 4-dimensional spectrum data to ensure effective data transmission.
[0065] In the disclosed embodiment, the time domain features are processed by the recurrent neural network in the sound source recognition network to achieve modeling and learning of the time domain dimension of the sound source features. It should be understood that for the number of hidden layer nodes of the recurrent neural network, in order to ensure that the needs of terminal use are met, the number of hidden layer nodes should not be too large. For example, the number of hidden layer nodes can be set to 128, 64, 64, and 128 respectively, and the number of layers can be set to 2 layers. Combined with the codec, the construction method of the recurrent neural network (Long Short-Term Memory, LSTM) is simplified, the reverse dimension mapping of the recurrent neural network output is simplified, the number of parameters is reduced, and the reasoning efficiency is improved; a single-frame real-time reasoning method is established, with low latency and efficient reasoning. In this way, the model structure is simplified to meet the availability requirements of low latency and efficient processing.
[0066] In the disclosed embodiment, the convolutional neural network in the network module includes multiple two-dimensional convolutional layers (ConvBlock). For example, it includes 3 two-dimensional convolutional layers. The three two-dimensional convolutional layers are respectively referred to as the first two-dimensional convolutional layer, the second two-dimensional convolutional layer, and the third two-dimensional convolutional layer. The first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer, and the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected by residuals. For example, for each Conv Block, it is composed of three groups of two-dimensional convolutions, the first layer and the third layer have the same number of channels (Channel), and the second layer has 1 / 2 of the number of channels of the previous convolutional layer.
[0067] Figure 5 is a flow chart of a sound source recognition according to an exemplary embodiment. Figure 5 As shown, the sound source identification method includes the following steps:
[0068] In step S51, based on the spectral feature data processed by the recurrent neural network in the Lth network module, the convolution kernel of the first two-dimensional convolutional layer slides on the spectral feature data with a specific step size, and a convolution operation is performed on the spectral feature data to obtain a first convolution result.
[0069] In the disclosed embodiment, a convolution operation is performed on the spectral data features by sliding the convolution kernel on the spectral feature data, thereby achieving a feature extraction of the frequency domain features.
[0070] In step S52, the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the second two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain a second convolution result.
[0071] In the disclosed embodiment, a convolution operation on the spectral data features is implemented by sliding the convolution kernel on the spectral feature data. The number of channels of the second two-dimensional convolution layer is half of the number of channels of the first two-dimensional convolution layer, thereby implementing a feature extraction and compression of the frequency domain features.
[0072] In step S53, the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain the spectrum feature data after frequency domain feature extraction.
[0073] In the disclosed embodiment, a convolution operation on the spectral data features is implemented by sliding the convolution kernel on the spectral feature data. The number of channels of the third two-dimensional convolution layer is the same as that of the first two-dimensional convolution layer, and a residual connection is used to ensure the effective transmission of the compressed feature data to avoid problems such as gradient disappearance and gradient explosion.
[0074] In the disclosed embodiment, if real-time processing of audio data is to be achieved, streaming processing of frequency domain features of spectral data features needs to be achieved.
[0075] Figure 6 is a flow chart of a sound source recognition method according to an exemplary embodiment. Figure 6 As shown, the sound source identification method includes the following steps:
[0076] In step S61, based on the spectral feature data processed by the recurrent neural network in the Lth network module, a convolution operation is performed on the spectral feature data by sliding a convolution kernel with a time domain dimension of 1 in the first two-dimensional convolution layer on the spectral feature data with a specific step size to obtain a first convolution result.
[0077] In the disclosed embodiment, by setting the time domain dimension of the convolution kernel in the first two-dimensional convolution layer to 1, the streaming processing of the spectrum feature data by the first two-dimensional convolution layer is achieved.
[0078] In step S62, the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel with a time domain dimension of 1 in the second two-dimensional convolution layer slides on the spectrum feature data with a specific step size to perform a convolution operation on the spectrum feature data to obtain a second convolution result.
[0079] In the disclosed embodiment, by setting the time domain dimension of the convolution kernel in the second two-dimensional convolution layer to 1, the streaming processing of the spectrum feature data by the second two-dimensional convolution layer is achieved.
[0080] In step S63, the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel with a time domain dimension of 1 in the third two-dimensional convolution layer slides on the spectrum feature data with a specific step size to perform a convolution operation on the spectrum feature data to obtain the spectrum feature data after frequency domain feature extraction.
[0081] In the disclosed embodiment, by setting the time domain dimension of the convolution kernel in the third two-dimensional convolution layer to 1, the streaming processing of the spectrum feature data by the third two-dimensional convolution layer is achieved.
[0082] In the disclosed embodiment, after the network module extracts the time domain features and frequency domain features of the spectrum feature data, it is necessary to downsample the spectrum feature data.
[0083] Figure 7 It is a schematic diagram of a network model for training or identifying a sound source identification method according to an exemplary embodiment.
[0084] For the convenience of description in the following embodiments of the present disclosure, the encoding network may be referred to as an encoder, the decoding network may be referred to as a decoder, and the feature transition network may be referred to as an intermediate transition network.
[0085] The basic architecture of the network model of this exemplary embodiment is as follows: Figure 7 As shown in the figure, it is mainly a model structure based on the Unet network. The complete model structure includes an encoder, a decoder, an intermediate transition network, and a sound source recovery network. Figure 7 , left) to extract features, that is, to calculate the short-time Fourier transform (STFT) and obtain the frequency domain spectral features. The spectral features are sent to the constructed Unet network ( Figure 7 , middle) to obtain the embedding layer features of different sound sources. The embedding layer features are passed through the sound source recovery network ( Figure 7 , and then the final output of the model is obtained. The audio signals of each sound source are restored by short-time inverse Fourier transform.
[0086] In this exemplary embodiment, during the model training process, the data is preprocessed. For example, all data (44.1kHz) are resampled to a sampling rate of 16kHz, and 3 seconds of audio are randomly extracted from each 10-second audio in each training iteration, and the short-time Fourier transform (STFT) is calculated using a window length of 20 milliseconds and 3 milliseconds and a 50% overlap with the Hamming window to obtain the frequency domain features of the speech signal, and the real and imaginary parts of the frequency domain are spliced. In order to speed up the model training speed, the generated features are saved and used as features of the input model. It should be understood that the preprocessing operation may be different from the above operation, which is only used for illustration.
[0087] In this exemplary embodiment, the separation deep neural network includes a convolutional neural network (CNN), a recurrent neural network (LSTM), and other common node calculation functions, including Relu activation function, batch normalization (Batch Normalization, BN), etc. The overall structure of the model is a Unet structure, that is, the network includes an encoding network (Encoder) and a decoding network (Decoder) and an intermediate feature transition network (Mid). For the Encoder part, it mainly includes 4 groups of convolutional neural network (CNN) frequency domain feature extraction modules (ConvBlock1-4) and 3 groups of LSTM time domain dimension feature extraction modules (LSTM Block1-3). For each Conv Block, it is composed of three groups of two-dimensional convolutions, the first layer and the third layer are composed of the same number of channels (Channel), and the second layer is composed of 1 / 2 of the number of channels of the previous convolution layer. In addition, the first and third layers are added by residual connections to ensure the effective transmission of features. In addition, after each Conv Block, the convolution features are reduced in dimension by using average pooling and down sampling operations, which is beneficial for high-dimensional feature extraction and reduces the amount of computation in the subsequent network. For each LSTMBlock, it consists of a fully connected layer (FC1), two LSTM layers, and a fully connected layer (FC2). FC1 is used to compress the 4-dimensional data of the convolution features so that they can be fed into the LSTM, and FC2 is used to restore the 3-dimensional features output by the LSTM to 4-dimensional features and feed them into the subsequent network. For the parameters of the CNN Block part, the convolution kernel is (1, 3), and the number of channels is: 32, 64, 64, 128 respectively; for the parameters of the LSTM Block part, the number of hidden layer nodes is: 128, 64, 64, 128 respectively, and the number of layers is 2. For the Mid part, it consists of 2 layers of LSTM with 128 hidden layer nodes, and the features are added and fed into the Decoder through residual connections. For the Decoder part, its structure is similar to that of the Encoder, and the same parameters are used for feature restoration. The DeConv Block part is the same as the Conv Block of the Encoder part, and the Up sample operation is used to gradually restore the features. After the Decoder, a layer of two-dimensional convolution is used to obtain the final multi-source features, where the number of separation sources is the same as the channel dimension of the convolution layer.
[0088] In the disclosed embodiment, the training settings of the model can be set according to the needs. For example, the batch size of the model training is set to 16, and the Adam optimizer is applied. The learning rate is set to 0.001. The model training loss function is the mean square error loss function (MSE). In order to improve the effectiveness of the training, the learning rate is multiplied by 0.9 after every 15,000 steps. In the end, the complete training process contains about 300,000 steps.
[0089] In the disclosed embodiment, the inference test of the model should be different from the training test, and the inference test should be aimed at meeting the application requirements of the streaming test. For example, to meet the application requirements of real-time streaming processing, the test duration can be 20ms, and the overlap duration can be 10ms.
[0090] In the disclosed embodiment, the network model parameter volume is 1.53M, and the Flops is 0.02G, which meets the application requirements of the mobile phone. Based on the self-built test set data, the network model can improve the SDR by 0.6-1.0 (dB), the SDR of the simple separation of human voice is improved by 0.4-0.8 (dB), and the SDR of the simple separation of music is improved by 1.4-1.7 (dB). The algorithm designed by the network model can be applied to mobile phones, tablets and other platforms, and the delay is less than about 5ms.
[0091] Based on the same concept, the embodiment of the present disclosure also provides a sound source identification device.
[0092] It is understandable that the sound source recognition device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.
[0093] Figure 8 The block diagram of a sound source recognition device according to an exemplary embodiment is shown in FIG. 1 . The device 100 includes an extraction unit 101 and a recognition unit 102 .
[0094] The extraction unit 101 is configured to determine frequency spectrum feature data of the audio signal in response to obtaining the audio signal to be subjected to sound source recognition.
[0095] The recognition unit 102 is used to call a network model for sound source recognition, wherein the network model includes a coding network, a feature transition network, and a decoding network.
[0096] In one embodiment, the extraction unit 101 and the recognition unit 102 use the following method to call the network model for sound source recognition, including: inputting the spectral feature data into the encoding network of the network model, and the encoding network extracting the time domain features of the audio signal based on the spectral feature data, and extracting the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, to obtain the sound source embedding layer features of the audio signal; inputting the sound source embedding layer features into the feature transition network of the network model, and the feature transition network performs splicing of multiple sound source embedding layer features and then inputs them into the decoding network, and the decoding network performs feature restoration on the sound source embedding layer features to obtain the sound source signal.
[0097] In one embodiment, the encoding network in the recognition unit 102 includes: M network modules, each network module includes a recurrent neural network and at least one convolutional neural network, the value of M is a positive integer greater than or equal to 1, and the number of convolutional neural networks included in different network modules is the same or different;
[0098] In one embodiment, the recognition unit 102 extracts the time domain features of the audio signal based on the spectral feature data by the encoding network, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal: extracting the time domain features based on the recurrent neural network in the Lth network module, and extracting the frequency domain features based on at least one convolutional neural network in the Lth network module; repeating the above process until the Mth completes the time domain feature extraction and the convolutional neural network completes the frequency domain feature extraction, so as to obtain the sound source embedding layer features of the audio signal; wherein the value of L is a positive integer between 1 and M, and the time domain feature dimensions and frequency domain feature dimensions extracted by the 1 to M network modules decrease successively.
[0099] In one embodiment, the recurrent neural network includes a first fully connected layer, a recurrent neural network layer and a second fully connected layer; the recognition unit extracts time domain features based on the recurrent neural network in the Lth network module in the following manner: the spectral feature data is subjected to dimension reduction processing based on the first fully connected layer of the recurrent neural network in the Lth network module, and the reduced dimension time domain features are obtained after passing through the recurrent neural network layer; the reduced dimension time domain features are input into the second fully connected layer, and the reduced dimension time domain features are restored by the second fully connected layer to obtain the time domain features of the spectral feature data.
[0100] In one embodiment, the convolutional neural network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer, the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected by a residual connection; the first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer, and the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected by a residual connection; the recognition unit 102 extracts frequency domain features based on at least one convolutional neural network in the Lth network module in the following manner: based on the frequency domain features processed by the recurrent neural network in the Lth network module, The spectral feature data is subjected to convolution operation by sliding the convolution kernel of the first two-dimensional convolution layer on the spectral feature data with a specific step size, and a first convolution result is obtained; the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the second two-dimensional convolution layer slides on the spectral feature data with a specific step size, and a second convolution result is obtained; the first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectral feature data with a specific step size, and a convolution operation is performed on the spectral feature data to obtain the spectral feature data after frequency domain feature extraction.
[0101] In one embodiment, the time domain dimension of the convolution kernel during the convolution operation in the convolutional neural network is 1.
[0102] Fig. 9 2 is a block diagram of a device 200 for sound source recognition according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0103] Reference Fig. 9 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .
[0104] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.
[0105] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0106] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.
[0107] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0108] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.
[0109] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0110] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
[0111] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0112] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.
[0113] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0114] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.
[0115] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.
[0116] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.
[0117] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.
[0118] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.
[0119] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.
[0120] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.
Claims
1. A sound source recognition method, characterized in that: include: In response to acquiring an audio signal to be subjected to sound source identification, determining frequency spectrum feature data of the audio signal; Calling a network model for sound source recognition, wherein the network model includes an encoding network, a feature transition network, and a decoding network; The frequency spectrum feature data is input into the encoding network of the network model, and the encoding network extracts the time domain features of the audio signal based on the frequency spectrum feature data, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal; The sound source embedding layer features are input into the feature transition network of the network model, the feature transition network performs concatenation of the sound source embedding layer features and then input into the decoding network, the decoding network performs feature restoration on the sound source embedding layer features to obtain the sound source signal.
2. The sound source recognition method according to claim 1, characterized in that: The encoding network includes M network modules, each of which includes a recurrent neural network and at least one convolutional neural network, the value of M is a positive integer greater than or equal to 1, and the number of convolutional neural networks included in different network modules is the same or different; The step of extracting the time domain features of the audio signal based on the frequency spectrum feature data by the encoding network, and extracting the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, to obtain the sound source embedding layer features of the audio signal, includes: Extracting time domain features based on a recurrent neural network in the Lth network module, and extracting frequency domain features based on at least one convolutional neural network in the Lth network module; Repeat the above process until the Mth time domain feature extraction is completed and the convolutional neural network completes the frequency domain feature extraction, thereby obtaining the sound source embedding layer features of the audio signal; Among them, the value of L is a positive integer between 1 and M, and the time domain feature dimensions and frequency domain feature dimensions extracted by the 1 to M network modules decrease in sequence.
3. The method according to claim 2, characterized in that The recurrent neural network includes a first fully connected layer, a recurrent neural network layer and a second fully connected layer; The extracting of time domain features based on the recurrent neural network in the Lth network module includes: Based on the first fully connected layer of the recurrent neural network in the Lth network module, the spectral feature data is subjected to dimension reduction processing, and the dimension-reduced time domain features are obtained after passing through the recurrent neural network layer; The reduced-dimensional time domain features are input into a second fully connected layer, and the reduced-dimensional time domain features are restored by the second fully connected layer to obtain the time domain features of the spectral feature data.
4. The method according to claim 2, characterized in that: The convolutional neural network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer; The first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer, and the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected via a residual connection; Extracting frequency domain features based on at least one convolutional neural network in the Lth network module includes: Based on the frequency spectrum feature data processed by the recurrent neural network in the L-th network module, a convolution operation is performed on the frequency spectrum feature data by sliding the convolution kernel of the first two-dimensional convolution layer on the frequency spectrum feature data with a specific step size to obtain a first convolution result; Inputting the first convolution result into a second two-dimensional convolution layer, sliding the convolution kernel of the second two-dimensional convolution layer on the spectrum feature data with a specific step length, performing a convolution operation on the spectrum feature data, and obtaining a second convolution result; The first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain the spectrum feature data after frequency domain feature extraction.
5. The method according to claim 4, characterized in that The time domain dimension of the convolution kernel when performing a convolution operation in the convolutional neural network is 1.
6. A sound source recognition device, characterized in that: The device comprises: An extraction unit, configured to determine frequency spectrum feature data of the audio signal in response to acquiring the audio signal to be subjected to sound source identification; An identification unit, used to call a network model for sound source identification, wherein the network model includes an encoding network, a feature transition network, and a decoding network; The frequency spectrum feature data is input into the encoding network of the network model, and the encoding network extracts the time domain features of the audio signal based on the frequency spectrum feature data, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal; The sound source embedding layer features are input into the feature transition network of the network model, the feature transition network performs concatenation of the sound source embedding layer features and then input into the decoding network, the decoding network performs feature restoration on the sound source embedding layer features to obtain the sound source signal.
7. The sound source recognition device according to claim 6, characterized in that: The encoding network includes M network modules, each of which includes a recurrent neural network and at least one convolutional neural network, the value of M is a positive integer greater than or equal to 1, and the number of convolutional neural networks included in different network modules is the same or different; The recognition unit extracts the time domain features of the audio signal based on the frequency spectrum feature data by the encoding network, and extracts the frequency domain features of the audio signal based on the audio time domain data obtained after extracting the time domain features, so as to obtain the sound source embedding layer features of the audio signal in the following manner: Extracting time domain features based on a recurrent neural network in the Lth network module, and extracting frequency domain features based on at least one convolutional neural network in the Lth network module; Repeat the above process until the Mth time domain feature extraction is completed and the convolutional neural network completes the frequency domain feature extraction, thereby obtaining the sound source embedding layer features of the audio signal; Among them, the value of L is a positive integer between 1 and M, and the time domain feature dimensions and frequency domain feature dimensions extracted by the 1 to M network modules decrease in sequence.
8. The device according to claim 7, characterized in that The recurrent neural network includes a first fully connected layer, a recurrent neural network layer and a second fully connected layer; The recognition device extracts time domain features based on the recurrent neural network in the Lth network module in the following manner: Based on the first fully connected layer of the recurrent neural network in the Lth network module, the spectral feature data is subjected to dimension reduction processing, and the dimension-reduced time domain features are obtained after passing through the recurrent neural network layer; The reduced-dimensional time domain features are input into a second fully connected layer, and the reduced-dimensional time domain features are restored by the second fully connected layer to obtain the time domain features of the spectral feature data.
9. The device according to claim 7, characterized in that The convolutional neural network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer and a third two-dimensional convolutional layer; The first two-dimensional convolutional layer and the third two-dimensional convolutional layer have the same number of channels, the number of channels of the second two-dimensional convolutional layer is half of the number of channels of the first two-dimensional convolutional layer, and the first two-dimensional convolutional layer and the third two-dimensional convolutional layer are connected via a residual connection; The recognition device extracts frequency domain features based on at least one convolutional neural network in the Lth network module in the following manner: Based on the frequency spectrum feature data processed by the recurrent neural network in the L-th network module, a convolution operation is performed on the frequency spectrum feature data by sliding the convolution kernel of the first two-dimensional convolution layer on the frequency spectrum feature data with a specific step size to obtain a first convolution result; Inputting the first convolution result into a second two-dimensional convolution layer, sliding the convolution kernel of the second two-dimensional convolution layer on the spectrum feature data with a specific step length, performing a convolution operation on the spectrum feature data, and obtaining a second convolution result; The first convolution result is input into the second two-dimensional convolution layer, and the convolution kernel of the third two-dimensional convolution layer slides on the spectrum feature data with a specific step size, and a convolution operation is performed on the spectrum feature data to obtain the spectrum feature data after frequency domain feature extraction.
10. The device according to claim 9, characterized in that The time domain dimension of the convolution kernel when performing a convolution operation in the convolutional neural network is 1.
11. A sound source recognition device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 5.
12. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 5.