Sound source identification method and device and storage medium

Abstract

The invention relates to a sound source identification method and device and a storage medium. The sound source identification method comprises the following steps: extracting mixed sound source characteristics of an audio signal to be subjected to sound source identification; sound source recognition is executed based on the mixed sound source features in sequence according to the following mode until the nth sound source is recognized, and n is the number of sound sources included in the mixed sound source features; obtaining the ith sound source based on the mixed sound source feature recognition, wherein the value of i is a positive integer from 1 to n-1; and removing the feature of the ith sound source from the mixed sound source features, and identifying to obtain an (i + 1) th sound source based on the mixed sound source features and the feature of removing the feature of the ith sound source. According to the invention, multiple sound sources of the mixed sound source signal can be identified.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio processing, and in particular to a sound source recognition method, device and storage medium. Background Art

[0002] With the development of sound source separation technology, sound source recognition technology that can identify audio signals of specific sound sources from mixed audio signals has been applied in many fields, such as scene (environment) perception, intelligent conference assistance, and instrument separation.

[0003] In the related art, there are many methods to identify specific sound sources. For example, non-negative matrix decomposition can be used to decompose the spectrum into dictionaries and activations, and the separated sources can be obtained by multiplying the activations with different dictionaries. The independent component analysis method can be used to distinguish between different sources based on the differences in the principal components. It is also possible to establish a frequency domain recognition network or a time domain recognition network based on a neural network. General networks are divided into masking methods and mapping methods. Both have their own advantages, but also disadvantages. Generally, the modeling method is selected according to the application scenario. However, in scenarios where multiple sound sources need to be identified in mixed audio signals, the extraction effects of the above methods are poor. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a sound source recognition method, device and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, a sound source identification method is provided, the sound source identification method comprising: extracting mixed sound source features of an audio signal to be subjected to sound source identification. Performing sound source identification based on the mixed sound source features in sequence in the following manner until the nth sound source is identified, where n is the number of sound sources included in the mixed sound source features. Identifying the i-th sound source based on the mixed sound source features, where the value of i is a positive integer from 1 to n-1. Removing the features of the i-th sound source from the mixed sound source features, and identifying the i+1-th sound source based on the mixed sound source features and the features after removing the i-th sound source features.

[0006] In one embodiment, the performing of sound source recognition based on the mixed sound source feature includes: performing sound source recognition on the mixed sound source feature based on the BGRU network in the sound source recognition network within the mth cycle to obtain a first recognition result, inputting the first recognition result into the sound source recognition subnetwork in the sound source recognition network to obtain a first recognition result of the i-th sound source, wherein the duration corresponding to the first recognition result is the duration having a first degree of overlap with the m-th cycle from the start time of the m-th cycle, and m is a positive integer greater than or equal to 1. Starting from the end time of the duration corresponding to the first recognition result, performing sound source recognition on the mixed sound source feature based on the BGRU network within the m+1th cycle to obtain a second recognition result, inputting the second recognition result into the sound source recognition subnetwork to obtain a second recognition result of the i-th sound source, wherein the duration of the second recognition result is the duration having the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle. Repeating the above process until the recognition of the mixed sound source feature is completed.

[0007] In one embodiment, the extracting of mixed sound source features of an audio signal to be subjected to sound source recognition comprises: extracting mixed sound source features of an audio signal to be subjected to sound source recognition based on a feature extraction network in a sound source recognition network. The feature extraction network comprises a plurality of subnetworks and a batch normalization network, and each of the plurality of subnetworks comprises a two-dimensional convolutional network and an average pooling layer. Adjacent convolutional layers in the two-dimensional convolutional network have receptive fields of different sizes. The two-dimensional convolutional network comprises a first portion of adjacent convolutional layers and a second portion of adjacent convolutional layers. The first portion of adjacent convolutional layers is before the second portion of adjacent convolutional layers, and the corresponding receptive field of the first portion of adjacent convolutional layers is larger than that of the second portion of adjacent convolutional layers.

[0008] In one embodiment, before performing sound source recognition based on the mixed sound source features, the method further includes: performing linear dimensional transformation on the mixed sound source features extracted based on the feature extraction network, and performing sound source recognition as the mixed sound source features for sound source recognition.

[0009] In one embodiment, the sound source recognition network is trained in the following manner: a mixed sound source training data set is input into a feature extraction network included in the sound source recognition network, and a mixed sound source feature of the mixed sound source training data set is obtained through the feature extraction network. The mixed sound source features are respectively input into a plurality of bidirectional gated recurrent unit BGRU networks of different scales and a plurality of sound source recognition subnetworks included in the sound source recognition network to obtain N sound source recognition results. The number of the plurality of BGRU networks of different scales is the same as the number of preset sound sources. Each of the plurality of sound source recognition subnetworks is used to identify a sound source, wherein the input of the n+1th sound source recognition subnetwork for sound source recognition is the mixed sound source feature output by the BGRU network, and the mixed sound source feature corresponding to the sound source identified by the first n sound source recognition subnetworks removed from the mixed sound source feature. Based on the loss of the N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

[0010] According to a second aspect of the embodiments of the present disclosure, a sound source identification device is provided, the sound source identification device comprising: an extraction unit, for extracting mixed sound source features of an audio signal to be subjected to sound source identification. An identification unit, for performing sound source identification based on the mixed sound source features in sequence in the following manner until the nth sound source is identified, where n is the number of sound sources included in the mixed sound source features. The i-th sound source is identified based on the mixed sound source features, where the value of i is a positive integer from 1 to n-1. The features of the i-th sound source are removed from the mixed sound source features, and the i+1-th sound source is identified based on the mixed sound source features and the features after removing the i-th sound source features.

[0011] In one embodiment, the recognition unit performs sound source recognition based on the mixed sound source feature in the following manner: within the mth period, the mixed sound source feature is recognized based on the BGRU network in the sound source recognition network to obtain a first recognition result, the first recognition result is input into the sound source recognition sub-network in the sound source recognition network to obtain a first recognition result of the i-th sound source, the duration corresponding to the first recognition result is the duration with the first degree of overlap with the m-th period from the start time of the m-th period, and m is a positive integer greater than or equal to 1. From the end time of the duration corresponding to the first recognition result, within the m+1th period, the mixed sound source feature is recognized based on the BGRU network to obtain a second recognition result, the second recognition result is input into the sound source recognition sub-network to obtain a second recognition result of the i-th sound source, the duration of the second recognition result is the duration with the first degree of overlap with the m+1th period from the start time of the m+1th period. Repeat the above process until the recognition of the mixed sound source feature is completed.

[0012] In one embodiment, the extraction unit extracts mixed sound source features of the audio signal to be subjected to sound source recognition in the following manner: based on a feature extraction network in a sound source recognition network, the mixed sound source features of the audio signal to be subjected to sound source recognition are extracted. The feature extraction network includes multiple subnetworks and a batch normalization network, and each of the multiple subnetworks includes a two-dimensional convolutional network and an average pooling layer. Adjacent convolutional layers in the two-dimensional convolutional network have receptive fields of different sizes. The two-dimensional convolutional network includes a first portion of adjacent convolutional layers and a second portion of adjacent convolutional layers. The first portion of adjacent convolutional layers is before the second portion of adjacent convolutional layers, and the corresponding receptive field of the first portion of adjacent convolutional layers is larger than that of the second portion of adjacent convolutional layers.

[0013] In one embodiment, the extraction unit is further used to perform sound source recognition by performing linear dimensional transformation on the mixed sound source features extracted based on the feature extraction network and using the mixed sound source features as the mixed sound source features for sound source recognition.

[0014] In one embodiment, the sound source recognition device further includes: a training unit, which is used to train the sound source separation network in the following manner: input the mixed sound source training data set into the feature extraction network included in the sound source recognition network, and obtain the mixed sound source features of the mixed sound source training data set through the feature extraction network. The mixed sound source features are respectively input into multiple bidirectional gated recurrent unit BGRU networks of different scales and multiple sound source recognition subnetworks included in the sound source recognition network to obtain N sound source recognition results. The number of the multiple BGRU networks of different scales is the same as the number of preset sound sources. Each of the multiple sound source recognition subnetworks is used to identify a sound source, wherein the input of the n+1th sound source recognition subnetwork when performing sound source recognition is the mixed sound source features output by the BGRU network, and the mixed sound source features corresponding to the sound sources recognized by the first n sound source recognition subnetworks are removed from the mixed sound source features. Based on the loss of the N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

[0015] According to a third aspect of the embodiments of the present disclosure, there is provided a sound source recognition device, comprising: a processor, and a memory for storing instructions executable by the processor, wherein the processor is configured to: execute the method in the first aspect or any one of the implementations of the first aspect.

[0016] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the method in the first aspect or any one of the implementations of the first aspect.

[0017] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: sound source recognition is performed based on the mixed sound source features. Each time a sound source is recognized, its features are removed from the mixed sound source features, and then the next sound source is recognized based on the remaining mixed sound source features and the features of the removed sound source. Through continuous iteration, the features of each individual sound source in the mixed sound source can be gradually separated, and each sound source is recognized, thereby realizing multi-sound source recognition of the mixed sound source signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] Figure 1 It is a schematic diagram of a sound source recognition scheme.

[0020] Figure 2 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.

[0021] Figure 3 It is a schematic diagram of a sound source recognition method according to an exemplary embodiment.

[0022] Figure 4 The figure is a flow chart of a sound source recognition method according to an exemplary embodiment.

[0023] Figure 5 It is a schematic diagram showing a feature extraction method according to an exemplary embodiment.

[0024] Figure 6 The present invention is a flowchart showing a method of performing linear dimensional transformation on mixed sound source features according to an exemplary embodiment.

[0025] Figure 7 The figure is a schematic diagram showing a process of performing sound source recognition by using a trained sound source recognition method according to an exemplary embodiment.

[0026] Figure 8 The figure is a flowchart of training a sound source recognition network according to an exemplary embodiment.

[0027] Fig. 9 The present invention is a block diagram of a sound source recognition device according to an exemplary embodiment.

[0028] Fig.10 The present invention is a block diagram of a sound source recognition device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0030] In the accompanying drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limitations on the present disclosure. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present disclosure. The embodiments of the present disclosure are described in detail below in conjunction with the accompanying drawings.

[0031] The sound source recognition method provided by the embodiments of the present disclosure can be applied to application scenarios where multiple sound sources in a mixed sound source signal need to be recognized. For example: based on the recognized sound source information, the current environment is analyzed or the voice information of multiple speakers and the sound source information contained in the environment are distinguished, and the target sound source is extracted and retained. In the multi-target sound source scenario, separation operations can be performed on music, such as separation of human voice and accompaniment, separation of tracks of multiple instruments, etc.

[0032] In the related art, there are many methods that can be used to identify a specific sound source. These include non-negative matrix decomposition, independent component analysis, and frequency domain recognition and time domain recognition based on neural networks. However, these methods can only achieve recognition of a single sound source. When faced with audio signals mixed with multiple sound sources, the solutions used in the related art are as follows: Figure 1 shown.

[0033] Figure 1 A sound source identification scheme is shown. Figure 1 In the sound source recognition scheme shown, it is necessary to perform a sound source separation operation on the mixed sound source to obtain the separated single sound source, and then perform sound source recognition on each single sound source. This sound source recognition scheme that first separates the sound source and then recognizes the separated single sound sources one by one has the following problems. First: As the number of sound source recognitions increases, the network size will gradually expand, resulting in an increase in computing power and device power consumption. Second: Due to the high complexity of the network, real-time streaming processing cannot be achieved, and the delay is large. Third: This method requires the establishment of a network based on a data set in a special environment, so its adaptability is poor.

[0034] In view of this, an embodiment of the present disclosure provides a sound source identification method. In the identification method provided by the present disclosure, an iterative process is used: in each round, a new sound source is identified and its features are removed from the mixed sound source features, and then based on the remaining mixed sound source features and the features of the removed sound source, the next sound source is identified. This process will continue until all sound sources are identified. In each round, since a sound source has been identified, the features of the sound source can be removed from the mixed sound source features, thereby highlighting the features of other sound sources. Because each sound source has unique features in the mixed sound source, after removing the features of one sound source, the features of the remaining sound sources will be more clearly presented in the remaining mixed sound source features. Therefore, through continuous iteration, the features of each individual sound source in the mixed sound source can be gradually separated, and each sound source can be identified.

[0035] Figure 2 is a flow chart of a method for identifying a sound source according to an exemplary embodiment. Figure 2 As shown, the steps include the following steps S11 to S14.

[0036] In step S11, the mixed sound source features of the audio signal to be subjected to sound source identification are extracted.

[0037] Among them, the audio signal to be identified as a sound source usually refers to a mixed audio signal of multiple sound sources. In some scenarios, there are a large number of sound sources that emit sound at the same time. These sounds will overlap with each other, making it difficult for the sound emitted by each sound source to be clearly received. In this case, the sound signal received by the sound receiving device is a mixed audio signal of multiple sound sources.

[0038] The mixed audio signal contains multiple sound sources, each of which has its own unique and distinguishing sound source features. In the mixed audio signal, although the audio signals emitted by each sound source are superimposed on each other and difficult to be clearly received, the sound source features of each sound source are completely preserved when combined to form the mixed sound source features. Therefore, the mixed sound source features of the audio signal to be identified contain the sound source features of all the sound sources in the audio signal to be identified.

[0039] In step S12, the i-th sound source is obtained based on the mixed sound source feature recognition, where the value of i is a positive integer from 1 to n-1.

[0040] The audio features of the known sound source are used to identify a specific sound source in the mixed sound source. By comparing the similarity between the mixed sound source features and the known sound source features, the specific sound source can be accurately identified. The audio features of the known sound source can be determined during the training process of the sound source recognition network.

[0041] In step S13, the features of the i-th sound source are removed from the mixed sound source features, and the (i+1)-th sound source is identified based on the mixed sound source features and the features after the i-th sound source features are removed.

[0042] Among them, the mixed sound source feature refers to the mixed sound source feature including all sound source features, and the feature removing the i-th sound source feature refers to the remaining mixed sound source feature after removing all sound source features from the 1st sound source to the i-th sound source from the mixed sound source feature including all sound source features.

[0043] In step S14, sound source recognition is performed based on the mixed sound source feature in sequence according to the above method until the nth sound source is recognized, where n is the number of sound sources included in the mixed sound source feature.

[0044] For example, the mixed sound source contains a total of n sound sources, where n is a positive integer greater than or equal to 2. After the first sound source is identified based on the mixed sound source feature, the first sound source is removed from the mixed sound source feature, and the second sound source is identified based on the complete mixed sound source feature and the mixed sound source feature with the feature of the first sound source removed. After the second sound source is identified, the features of the first and second sound sources are removed from the mixed sound source feature, and the third sound source is identified based on the complete mixed sound source feature and the mixed sound source feature with the feature of the first and second sound sources removed. This process is repeated until the nth sound source is identified.

[0045] By performing feature extraction and sound source identification on the mixed audio signal, multiple sound sources of the mixed audio signal can be identified, and the audio signals of each sound source in the mixed audio signal can be obtained.

[0046] The sound source identification method provided by the embodiment of the present disclosure uses an iterative process: in each round, a new sound source is identified and its features are removed from the mixed sound source features, and then based on the remaining mixed sound source features and the features of the removed sound source, the next sound source is continuously identified until all sound sources are identified, thereby realizing multi-sound source identification of mixed audio. At the same time, the sound source separation method provided by the embodiment of the present disclosure adopts a lightweight network structure and an optimized model design, which can meet the needs of sound source separation on a mobile terminal.

[0047] Figure 3 It is a schematic diagram of a sound source recognition method according to an exemplary embodiment.

[0048] See Figure 3As shown, in one scenario, the received signal is a mixed audio signal reverberated from audio signals emitted by multiple sound sources, wherein sound sources 1 to n are different sound sources with distinguishability, and the total number of sound sources is n. When sound sources 1 to n emit sounds at the same time, the audio signals emitted by these sound sources will reverberate, making it difficult for the sounds emitted by each sound source to be clearly received. Therefore, in order to obtain the audio signals emitted by sound sources 1 to n in the mixed audio signal, it is necessary to identify sound sources 1 to n in the mixed audio signal separately. Since the audio signals of sound sources 1 to n in the mixed audio signal affect each other, the single sound source recognition method in the related art cannot obtain accurate recognition results. In view of this, the present disclosure provides a sound source recognition method.

[0049] The sound source identification method provided by the present disclosure identifies the mixed sound source features of a mixed audio signal. When multiple sound sources emit sounds at the same time, these sounds will overlap with each other, making it difficult for the sound emitted by each sound source to be clearly received. However, the sound source features of each sound source are fully preserved when forming the mixed sound source features. Therefore, by identifying the mixed sound source features of a mixed audio signal, each sound source contained in the mixed audio signal can be identified.

[0050] The sound source identification method provided by the present disclosure adopts an iterative method to perform multi-sound source identification: in each round, a new sound source is first identified, and then the characteristics of the sound source are removed from the mixed sound source characteristics. Next, based on the remaining mixed sound source characteristics and the characteristics of the removed sound source, the next sound source is identified until all the sound sources are finally identified.

[0051] The present disclosure embodiment hereinafter describes the sound source recognition method involved in the above embodiment.

[0052] Figure 4 is a flow chart of a method for identifying a sound source according to an exemplary embodiment. Figure 4 As shown, the following steps are included from S21 to S2n-i+5.

[0053] In step S21, in the mth cycle, sound source recognition is performed on the mixed sound source features based on a bidirectional gated recurrent unit (BGRU) network in the sound source recognition network to obtain a first recognition result.

[0054] The BGRU network is a bidirectional recurrent neural network that can utilize input information in both the forward and backward directions. At each time step, the outputs of the forward and backward BGRU networks are connected to form a new feature representation that contains all the information before and after the current time step. The BGRU network has the following characteristics: In one cycle, each BGRU network receives two partial feature representations. From the start of this cycle, during the duration that has the first degree of overlap with this cycle, the BGRU network receives features that are a combination of the features output backward from the previous cycle and the features output forward from this cycle. During the remaining duration of this cycle, the BGRU network receives features that are a combination of the features output backward from this cycle and the features output forward from the next cycle.

[0055] For the duration that has the first degree of overlap with the mth cycle from the start time of the mth cycle, the BGRU network will receive a feature representation composed of the features output backward from the m-1th cycle and the features output forward from the mth cycle. Since the m-1th cycle has completed the recognition of the sound source features of the m-1th cycle, the BGRU network can use the recognition results of the m-1th cycle to eliminate the interference of the features output backward from the m-1th cycle. This allows the BGRU network to perform sound source recognition based on the features output forward from this cycle in the first half of the mth cycle, so the recognition result of the duration that has the first degree of overlap with the mth cycle from the start time of the mth cycle is credible. On the contrary, in the remaining duration of the mth cycle, the BGRU network will receive a feature representation composed of the features output backward from the mth cycle and the features output forward from the m+1th cycle. Since the BGRU network cannot receive the recognition result of the m+1th cycle in the mth cycle, the interference of the features output forward from the m+1th cycle cannot be eliminated, so the recognition result of the remaining duration of the mth cycle is unreliable. Therefore, the first recognition result is a recognition result of a time length having a first degree of overlap with the mth period from the start time of the mth period.

[0056] In step S22, the first recognition result is input into the sound source recognition subnetwork in the sound source recognition network to obtain the first recognition result of the i-th sound source, and the time length corresponding to the first recognition result is the time length having a first degree of overlap with the m-th period from the start time of the m-th period, where m is a positive integer greater than or equal to 1.

[0057] For the i-th sound source, its first recognition result corresponds to the recognition result of the time length having a first degree of overlap with the m-th period from the start time of the m-th period.

[0058] In step S23, starting from the end time of the duration corresponding to the first recognition result, sound source recognition is performed on the mixed sound source features based on the BGRU network within the (m+1)th cycle to obtain a second recognition result.

[0059] As in the case of the mth cycle, only the duration of the m+1th cycle that has the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle is credible. Therefore, the second recognition result is the recognition result of the duration of the m+1th cycle that has the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle.

[0060] In order to obtain reliable recognition results, the BGRU network uses an overlapping processing strategy. The starting time of the m+1th cycle is located at the moment when the duration of the first overlap with the mth cycle from the starting time of the mth cycle ends. Therefore, in the mth cycle and the m+1th cycle, the first recognition result and the second recognition result have the following relationship in timing: the duration of the first recognition result is from the starting time of the mth cycle to the end of the mth cycle and the first overlap. The duration of the second recognition result is from the starting time of the m+1th cycle to the end of the m+1th cycle and the first overlap, and the end time of the second recognition result is exactly the end time of the mth cycle. Therefore, by performing timing modeling on the first recognition result and the second recognition result, the correct recognition result for the entire mth cycle can be obtained.

[0061] In step S24, the second recognition result is input into the sound source recognition subnetwork to obtain the second recognition result of the ith sound source. The duration of the second recognition result is the duration from the start time of the m+1th cycle to the first overlap with the m+1th cycle.

[0062] In step S25 to step S2n-i+5, the above process is repeated until the identification of the mixed sound source features is completed.

[0063] In the disclosed embodiment, the sound source features of the mixed audio signal are modeled in time series and multiple sound sources are identified based on the BGRU network. Among them, since the BGRU network is a bidirectional gated recurrent unit, a bidirectional feature input will be introduced. In order to ensure the accuracy of the recognition result, an overlapping processing strategy can be used. That is, a certain overlapping area is maintained between adjacent cycles, so that the audio data of adjacent cycles are partially overlapped to obtain a more accurate recognition result. For a continuous mixed audio signal, the separation results of each cycle need to be spliced ​​to obtain an overall separation result. This can be achieved by processing the overlapping area.

[0064] The above process is repeated to continuously perform sound source recognition based on the first recognition result of the current cycle and the second recognition result of the next cycle. The recognition result of each cycle is used as the input of the next cycle to gradually improve the accuracy of separation.

[0065] Here, m is a positive integer greater than or equal to 1.

[0066] Through this process, the BGRU network realizes real-time recognition of sound sources in different frequency bands in mixed audio. Each cycle uses the previous recognition results and the current sub-band frequency domain features for recognition, and continuously iterates to improve the reliability of recognition. In this way, accurate real-time recognition of multiple sound sources in mixed audio can be achieved.

[0067] Further, based on the above embodiments, when the method in the embodiments of the present disclosure is used to perform sound source recognition, it is necessary to extract the mixed sound source features of the audio signal to be used for sound source recognition. In view of this, the embodiments of the present disclosure provide a mixed sound source feature extraction method.

[0068] Figure 5 is a schematic diagram of a feature extraction method according to an exemplary embodiment. Figure 5 As shown, the process includes the following steps S31 to S32.

[0069] In step S31, the audio signal to be subjected to sound source recognition is input into the feature extraction network in the sound source recognition network.

[0070] In step S32, based on the feature extraction network in the sound source recognition network, the mixed sound source features of the audio signal to be subjected to sound source recognition are extracted.

[0071] In the disclosed embodiment, the feature extraction network is mainly composed of a batch normalization layer (Batch Normalization) and multiple sub-networks.

[0072] The batch normalization layer normalizes the input of each layer to make the distribution of input data more stable, reduce the risk of gradient vanishing or exploding in the network, and help improve the training speed and effect of the model.

[0073] The subnetwork is mainly composed of a two-dimensional convolutional network (Conv2D) and an average pooling layer (Average Pooling). Among them, the two-dimensional convolutional network is mainly used to extract features from the input audio signal. Through the convolution operation, some features in the input audio signal can be identified. In order to obtain features of different resolutions, the two-dimensional convolutional layer adopts a layered convolution method. In order to increase the amount of data for the two-dimensional convolutional network to perceive the characteristics of the sound source and avoid excessive calculation, adjacent convolutional layers use receptive fields of different sizes, and relatively different receptive fields are used in different layers. In this way, features of different resolutions can be obtained while controlling the amount of calculation within an acceptable range.

[0074] The introduction of the average pooling layer can improve the translation invariance of the output features, reduce the computational complexity of the network, and prevent overfitting. The sub-network structure in the disclosed embodiment enables the self-network to better extract the mixed sound source features of the audio signal to be identified and reduce the computational complexity of the network.

[0075] In the sound source identification method provided by the embodiment of the present disclosure, the sound source features of the mixed audio signal are obtained through a feature extraction network. In a mixed audio signal, when each sound source emits sound at the same time, these sounds will overlap with each other, making it difficult to clearly receive the sound emitted by each sound source. However, the sound source features of each sound source are fully preserved when combined to form a mixed sound source feature. Compared with the mixed audio signal itself, the information contained in the sound source features of the mixed audio signal is richer and more complete. Extracting the sound source features of the mixed audio signal is conducive to improving the accuracy of sound source identification.

[0076] Further, based on the above embodiment, when the method in the embodiment of the present disclosure is used for sound source recognition, the mixed sound source features of the sound source recognition audio signal extracted by the feature extraction network need to be input into the BGRU network. However, after the initial processing of the feature extraction network, the key feature information of the mixed sound source features of the sound source recognition audio signal is not obvious enough and is difficult to be directly used for calculation and processing. In view of this, the embodiment of the present disclosure provides a method for extracting key feature information from the sound source recognition audio signal.

[0077] Figure 6 FIG. 1 is a flow chart showing a method for performing linear dimension transformation on mixed sound source features according to an exemplary embodiment. Figure 6 As shown, the steps include the following steps S41 to S42.

[0078] In step S41, the mixed sound source features of the sound source recognition audio signal after passing through the second batch of standardized networks are input into the linear layer to obtain the mixed sound source features of the sound source recognition audio signal after linearization by the linear layer.

[0079] Among them, the linear layer is the most basic layer type in a deep neural network. It can multiply the input tensor with the weight matrix and add the bias vector to obtain an output tensor. The main function of the linear layer is to extract key feature information from the input information.

[0080] In step S42, the mixed sound source features of the sound source recognition audio signal linearized by the linear layer are input into the sound source recognition network.

[0081] By using the method for linear dimensional transformation of mixed sound source features provided by the embodiment of the present disclosure, the mixed sound source features of the sound source recognition audio signal after preliminary processing by the feature extraction network are linearly transformed, thereby helping to improve the accuracy of sound source recognition.

[0082] The feature extraction method and the method for linear dimensional transformation of mixed sound source features provided in the embodiments of the present disclosure can establish distinctive sound source features, improve the recognition ability of the network, and thus achieve effective recognition of a larger number of sound sources.

[0083] Figure 7 is a flow chart of a method for training a sound source recognition network according to an exemplary embodiment. Figure 7 As shown, the steps include the following steps S51 to S52.

[0084] In step S51, the mixed sound source training data set is input into a feature extraction network included in the sound source recognition network, and the mixed sound source features of the mixed sound source training data set are obtained through the feature extraction network.

[0085] The mixed audio training data set is formed by randomly mixing multiple pure sources of different durations. The more pure sources there are in the mixed audio training data set, the better the training effect.

[0086] In step S52, the mixed sound source features are respectively input into a plurality of bidirectional gated recurrent unit BGRU networks of different scales and a plurality of sound source recognition sub-networks included in the sound source recognition network to obtain N sound source recognition results.

[0087] Among them, the number of multiple BGRU networks of different scales is the same as the number of preset sound sources. During the sound source recognition process, the mixed audio signal input to the sound source recognition network will be processed into an audio signal segment of a fixed scale. For some special sound sources, after the mixed audio signal is processed into an audio signal segment of a shorter or longer duration, the use of a fixed-scale BGRU network for recognition may cause the recognized audio signal to be distorted. Therefore, using multiple BGRU networks of different scales to train the sound source recognition network can enable the model to better recognize audio signals of various lengths, and effectively avoid the distortion problem caused by changes in the duration of the audio signal. Each of the multiple sound source recognition subnetworks is used to identify a sound source, wherein the input of the n+1th sound source recognition subnetwork for sound source recognition is the mixed sound source feature output by the BGRU network, and the mixed sound source feature corresponding to the sound source recognized by the first n sound source recognition subnetworks removed from the mixed sound source feature.

[0088] In step S53, based on the losses of the N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

[0089] Among them, some training parameters can be preset according to the usage scenario of the sound source recognition network. For example, if the sound source recognition network is used on the terminal, the training data can be set with reference to the following data: a sampling rate of 16kHz, a training audio duration of 3 seconds, a 20-ms window length and a 50% overlap with the Hamming window are used to calculate the Fourier transform (FFT), and the amplitude spectrum and phase spectrum of the frequency domain features are concatenated as the input features of the model.

[0090] Among them, some training parameters need to be set according to the training results of the sound source separation network, such as setting the batch size to 64 and applying the Adam optimizer. The learning rate is set to 0.001.

[0091] The model training loss function is the MAE loss function. MAE is a common regression model loss function used to evaluate the difference between the predicted value and the true value. It calculates the average of the absolute difference between the predicted value and the true value, that is: Among them, n is the number of samples in the data set, y_i is the true value, and is the corresponding predicted value. The advantage of MAE is that it has a better tolerance for outliers, because it only considers the distance between the predicted value and the true value, and does not consider the square of this distance. In addition, MAE gives the same penalty to samples with large or small error values, so it can avoid being dominated by outliers.

[0092] In the disclosed embodiment, the audio recognition network is trained by mixing a plurality of known pure sound sources of different durations into a mixed audio signal. Based on the absolute error loss function, the model training constraints are performed to obtain the sound source recognition network. The more pure sound sources there are in the mixed audio signal for training, the more accurate the recognition result of the sound source recognition network obtained by training.

[0093] Figure 8 The figure is a schematic diagram showing a process of performing sound source recognition by using a trained sound source recognition method according to an exemplary embodiment.

[0094] The basic network architecture of this exemplary embodiment is as follows: Figure 8 As shown, it can be divided into three parts: feature extraction, time series modeling, and sound source identification.

[0095] In the disclosed embodiment, the feature extraction network completes the feature extraction part. The feature extraction network extracts audio features by using a two-dimensional convolutional network, batch normalization and average pooling layers, and a parameterized rectified linear unit activation function (pRelu).

[0096] In order to obtain features of different resolutions, the two-dimensional convolution layer adopts a layered convolution method. In order to increase the amount of data that the two-dimensional convolution network uses to perceive the characteristics of the sound source and avoid excessive calculation, adjacent convolution layers use receptive fields of different sizes, and relatively different receptive fields are used in different layers. In this way, features of different resolutions can be obtained while controlling the amount of calculation within an acceptable range. For example: the receptive field sizes of the first two layers of convolution can be (2, 5) and (2, 3) respectively, and all subsequent convolutions can use receptive fields of (3, 3) and (1, 1).

[0097] In the disclosed embodiment, the BGRU network completes the time series modeling and sound source recognition part. According to the preset number of sound sources, the sound source features extracted by the feature extraction network are sent to the BGRU network of different scales. This part uses the BGRU network to perform time series modeling on different sound source branches, which is conducive to identifying mixed audio signals of different durations.

[0098] After the time series modeling is completed, the i-th sound source contained in the mixed audio signal is identified in the BGRU network. For the i-th sound source, the final feature representation of the current sound source is obtained through feature integration and activation function processing.

[0099] In the disclosed embodiment, the sound source features of the mixed audio signal are modeled in time series and multiple sound sources are identified based on the BGRU network. Among them, since the BGRU network is a bidirectional gated recurrent unit, a bidirectional feature input will be introduced. In order to ensure the accuracy of the recognition result, an overlapping processing strategy can be used. That is, a certain overlapping area is maintained between adjacent cycles, so that the audio data of adjacent cycles are partially overlapped to obtain a more accurate recognition result. For a continuous mixed audio signal, the separation results of each cycle need to be spliced ​​to obtain an overall separation result. This can be achieved by processing the overlapping area. For example: the recognition duration can be set to 100ms, and the overlapping time length (Overlap) can be set to 50ms, that is, the network recognizes once every 100ms, and gives the recognition result within the current 50ms to meet the application requirements of real-time streaming reasoning.

[0100] The above process is repeated to continuously perform sound source recognition based on the first recognition result of the current cycle and the second recognition result of the next cycle. The recognition result of each cycle is used as the input of the next cycle to gradually improve the accuracy of separation.

[0101] Here, m is a positive integer greater than or equal to 1.

[0102] Through this process, the BGRU network realizes real-time recognition of sound sources in different frequency bands in mixed audio. Each cycle uses the previous recognition results and the current sub-band frequency domain features for recognition, and continuously iterates to improve the credibility of recognition. In this way, accurate real-time recognition of sound sources in mixed audio can be achieved.

[0103] For subsequent sound sources, in addition to their own features, they are also concatenated (Concat) as reference features after the masking process (M) of the previous sound source, and finally the audio features of the current sound source are obtained. This approach can eliminate the interference of known sound sources, increase the recognition accuracy of subsequent sound sources, and also reduce the recognition burden of subsequent sound sources.

[0104] In the recognition method provided by the present disclosure, an iterative process is used: in each round, a new sound source is identified and its features are removed from the mixed sound source features, and then the next sound source is identified based on the remaining mixed sound source features and the features of the removed sound source. This process will continue until all sound sources are identified. In each round, since a sound source has been identified, the features of the sound source can be removed from the mixed sound source features, thereby highlighting the features of other sound sources. Because each sound source has unique features in the mixed sound source, after removing the features of one sound source, the features of the remaining sound sources will be more clearly presented in the remaining mixed sound source features. Therefore, through continuous iteration, the features of each individual sound source in the mixed sound source can be gradually separated, and each sound source can be identified.

[0105] After obtaining the final feature representations of different sound sources, the corresponding audio is obtained through inverse Fourier transform.

[0106] In order to meet the application requirements of the terminal, the sound source recognition method in this exemplary embodiment can estimate the model size by referring to the following data: for example, the model parameter quantity is 138k, and the number of floating point operations per second (Flops) is 328M.

[0107] The sound source recognition method in this exemplary embodiment can reduce the interference between various sound sources, improve the recognition accuracy, and increase the number of sound source recognition, thereby realizing the recognition of multiple sound sources of mixed audio signals. The algorithm is applicable to terminals such as mobile phones, tablets, and smart mobile home platforms, with a delay of less than 8ms.

[0108] The sound source recognition method provided by the embodiments of the present disclosure can be applied to application scenarios that require identification of multiple sound sources in mixed sound source signals. For example: The main application scenarios are: Scene (environment) perception: Based on the identified sound source information, the current environment (quiet, noisy, bedroom, living room, subway station) can be analyzed to assist other home products or smart wearable devices in converting to the corresponding working state; Intelligent conference assistance: Distinguish the voice information of multiple speakers and the sound source information contained in the environment, and extract, retain, and post-process (noise reduction, translation, remote transmission, etc.) the target sound source. Instrument separation: In a multi-target sound source scenario, separation operations can be performed on music, such as separation of vocals and accompaniment, separation of multiple instrument tracks, etc., which can be applied to mobile phone karaoke scenes and audio editing scenes, etc.

[0109] Based on the same concept, the embodiment of the present disclosure also provides a sound source identification device.

[0110] It is understandable that the sound source recognition device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0111] Fig. 9 is a block diagram of a sound source recognition device according to an exemplary embodiment. Fig. 9 , the device includes an extraction unit 101 and a recognition unit 102.

[0112] The extraction unit 101 is used to extract mixed sound source features of the audio signal to be subjected to sound source recognition.

[0113] The identification unit 102 is used to perform sound source identification based on the mixed sound source features.

[0114] In one embodiment, the extraction unit 101 and the recognition unit 102 perform sound source recognition based on the mixed sound source features in sequence in the following manner until the nth sound source is recognized, where n is the number of sound sources included in the mixed sound source features, including: recognizing the i-th sound source based on the mixed sound source features, where i is a positive integer from 1 to n-1; removing the features of the i-th sound source from the mixed sound source features, and recognizing the i+1-th sound source based on the mixed sound source features and the features after removing the i-th sound source features.

[0115] In one embodiment, the extraction unit 101 extracts mixed sound source features of an audio signal to be subjected to sound source recognition in the following manner, including: extracting mixed sound source features of an audio signal to be subjected to sound source recognition based on a feature extraction network in a sound source recognition network; the feature extraction network includes multiple subnetworks and a batch normalization network, and each of the multiple subnetworks includes a two-dimensional convolutional network and an average pooling layer; adjacent convolutional layers in the two-dimensional convolutional network have receptive fields of different sizes; the two-dimensional convolutional network includes a first portion of adjacent convolutional layers and a second portion of adjacent convolutional layers; the first portion of adjacent convolutional layers is before the second portion of adjacent convolutional layers, and the corresponding receptive field of the first portion of adjacent convolutional layers is larger than that of the second portion of adjacent convolutional layers.

[0116] In one embodiment, the extraction unit 101 and the recognition unit 102 perform sound source recognition using the mixed sound source features extracted based on the feature extraction network as the mixed sound source features for sound source recognition in the following manner, including: performing linear dimensional transformation on the mixed sound source features extracted based on the feature extraction network and using them as the mixed sound source features for sound source recognition.

[0117] In one embodiment, the recognition unit 102 performs sound source recognition based on the mixed sound source features in the following manner, including: performing sound source recognition on the mixed sound source features based on the BGRU network in the sound source recognition network within the mth period to obtain a first recognition result, inputting the first recognition result into the sound source recognition sub-network in the sound source recognition network to obtain a first recognition result of the i-th sound source, wherein the duration corresponding to the first recognition result is the duration having a first degree of overlap with the m-th period from the start time of the m-th period, where m is a positive integer greater than or equal to 1; performing sound source recognition on the mixed sound source features based on the BGRU network within the m+1th period from the end time of the duration corresponding to the first recognition result to obtain a second recognition result, inputting the second recognition result into the sound source recognition sub-network to obtain a second recognition result of the i-th sound source, wherein the duration of the second recognition result is the duration having a first degree of overlap with the m+1th period from the start time of the m+1th period;

[0118] In one embodiment, the training unit 103 is trained to obtain a sound source separation network in the following manner: a mixed sound source training data set is input into a feature extraction network included in a sound source recognition network, and a mixed sound source feature of the mixed sound source training data set is obtained through the feature extraction network; the mixed sound source features are respectively input into a plurality of bidirectional gated recurrent unit BGRU networks of different scales and a plurality of sound source recognition subnetworks included in the sound source recognition network to obtain N sound source recognition results; wherein the number of the plurality of BGRU networks of different scales is the same as the number of preset sound sources. Each of the plurality of sound source recognition subnetworks is used to identify a sound source, wherein the input of the n+1th sound source recognition subnetwork for sound source recognition is the mixed sound source feature output by the BGRU network, and the mixed sound source feature corresponding to the sound source recognized by the first n sound source recognition subnetworks removed from the mixed sound source feature;

[0119] Based on the losses of N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

[0120] Fig.10 2 is a block diagram of a device 200 for sound source recognition according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0121] Reference Fig.10 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0122] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0123] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0124] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0125] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0126] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0127] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0128] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0129] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0130] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0131] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0132] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0133] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar thereto. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0134] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0135] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0136] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0137] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0138] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A sound source recognition method, characterized in that: include: Extracting mixed sound source features of the audio signal to be identified; Performing sound source recognition based on the mixed sound source feature in sequence in the following manner until the nth sound source is recognized, where n is the number of sound sources included in the mixed sound source feature; Obtaining an i-th sound source based on the mixed sound source feature recognition, where the value of i is a positive integer from 1 to n-1; The feature of the i-th sound source is removed from the mixed sound source feature, and the (i+1)-th sound source is identified based on the mixed sound source feature and the feature after removing the i-th sound source feature.

2. The method according to claim 1, characterized in that The performing of sound source identification based on the mixed sound source feature comprises: In the mth cycle, a sound source is identified on the mixed sound source features based on the bidirectional gated recurrent unit (BGRU) network in the sound source identification network to obtain a first identification result, and the first identification result is input into the sound source identification subnetwork in the sound source identification network to obtain a first identification result of the i-th sound source, wherein the duration corresponding to the first identification result is the duration having a first degree of overlap with the mth cycle from the start time of the mth cycle, and m is a positive integer greater than or equal to 1; Starting from the end time of the duration corresponding to the first recognition result, performing sound source recognition on the mixed sound source features based on the BGRU network in the m+1th cycle to obtain a second recognition result, inputting the second recognition result into the sound source recognition subnetwork to obtain a second recognition result of the i-th sound source, wherein the duration of the second recognition result is the duration having the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle; The above process is repeated until the mixed sound source features are identified.

3. The method according to claim 1, characterized in that The step of extracting mixed sound source features of an audio signal to be subjected to sound source recognition comprises: Based on the feature extraction network in the sound source recognition network, extract the mixed sound source features of the audio signal to be subjected to sound source recognition; the feature extraction network includes a plurality of sub-networks and a batch normalization network, and each of the plurality of sub-networks includes a two-dimensional convolutional network and an average pooling layer; Adjacent convolutional layers in the two-dimensional convolutional network have receptive fields of different sizes; The two-dimensional convolutional network includes a first part of adjacent convolutional layers and a second part of adjacent convolutional layers; The first portion of adjacent convolutional layers is before the second portion of adjacent convolutional layers, and a corresponding receptive field of the first portion of adjacent convolutional layers is larger than that of the second portion of adjacent convolutional layers.

4. The method according to claim 3, characterized in that Before performing sound source recognition based on the mixed sound source feature, the method further includes: After the mixed sound source features extracted based on the feature extraction network are linearly transformed, the mixed sound source features are used as the mixed sound source features for sound source recognition to perform sound source recognition.

5. The method according to claim 2 or 3, characterized in that: The sound source recognition network is trained in the following way: Inputting the mixed sound source training data set into the feature extraction network included in the sound source recognition network, and obtaining the mixed sound source features of the mixed sound source training data set through the feature extraction network; Inputting the mixed sound source features into a plurality of bidirectional gated recurrent unit BGRU networks of different scales and a plurality of sound source recognition sub-networks included in the sound source recognition network, respectively, to obtain N sound source recognition results; The number of the multiple BGRU networks of different scales is the same as the number of preset sound sources; Each of the multiple sound source identification subnetworks is used to identify a sound source, wherein the input of the n+1th sound source identification subnetwork for sound source identification is the mixed sound source feature received by the BGRU network, and the mixed sound source feature corresponding to the sound source identified by the first n sound source identification subnetworks is removed from the mixed sound source feature; Based on the losses of the N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

6. A sound source recognition device, characterized in that: include: An extraction unit, used to extract mixed sound source features of an audio signal to be subjected to sound source recognition; an identification unit, configured to perform sound source identification based on the mixed sound source feature in sequence in the following manner until an nth sound source is identified, where n is the number of sound sources included in the mixed sound source feature; Based on the mixed sound source feature, an i-th sound source is identified, where the value of i is a positive integer from 1 to n-1; the feature of the i-th sound source is removed from the mixed sound source feature, and based on the mixed sound source feature and the feature after removing the i-th sound source feature, an i+1-th sound source is identified.

7. The sound source recognition device according to claim 6, characterized in that: The recognition unit performs sound source recognition based on the mixed sound source features in the following manner: In the mth cycle, a sound source is identified on the mixed sound source features based on the bidirectional gated recurrent unit (BGRU) network in the sound source identification network to obtain a first identification result, and the first identification result is input into the sound source identification subnetwork in the sound source identification network to obtain a first identification result of the i-th sound source, wherein the duration corresponding to the first identification result is the duration having a first degree of overlap with the mth cycle from the start time of the mth cycle, and m is a positive integer greater than or equal to 1; Starting from the end time of the duration corresponding to the first recognition result, performing sound source recognition on the mixed sound source features based on the BGRU network within the m+1th cycle to obtain a second recognition result, inputting the second recognition result into the sound source recognition subnetwork to obtain a second recognition result of the i-th sound source, wherein the duration of the second recognition result is the duration having the first degree of overlap with the m+1th cycle from the start time of the m+1th cycle; The above process is repeated until the mixed sound source features are identified.

8. The sound source recognition device according to claim 6, characterized in that: The extraction unit extracts the mixed sound source features of the audio signal to be identified by the following method: Based on the feature extraction network in the sound source recognition network, extract the mixed sound source features of the audio signal to be used for sound source recognition; The feature extraction network includes a plurality of sub-networks and a batch normalization network, and each of the plurality of sub-networks includes a two-dimensional convolutional network and an average pooling layer; Adjacent convolutional layers in the two-dimensional convolutional network have receptive fields of different sizes; The two-dimensional convolutional network includes a first part of adjacent convolutional layers and a second part of adjacent convolutional layers; The first portion of adjacent convolutional layers is before the second portion of adjacent convolutional layers, and a corresponding receptive field of the first portion of adjacent convolutional layers is larger than that of the second portion of adjacent convolutional layers.

9. The sound source recognition device according to claim 8, characterized in that: The identification unit is also used for: Before performing sound source recognition based on the mixed sound source features, the mixed sound source features extracted based on the feature extraction network are linearly transformed and then used as the mixed sound source features for performing sound source recognition to perform sound source recognition.

10. The sound source recognition device according to claim 7 or 8, characterized in that: The sound source recognition device further includes: a training unit, which is used to train the sound source separation network in the following manner: Inputting the mixed sound source training data set into the feature extraction network included in the sound source recognition network, and obtaining the mixed sound source features of the mixed sound source training data set through the feature extraction network; Inputting the mixed sound source features into a plurality of bidirectional gated recurrent unit BGRU networks of different scales and a plurality of sound source recognition sub-networks included in the sound source recognition network, respectively, to obtain N sound source recognition results; The number of the multiple BGRU networks of different scales is the same as the number of preset sound sources; Each of the multiple sound source identification subnetworks is used to identify a sound source, wherein the input of the n+1th sound source identification subnetwork for sound source identification is the mixed sound source feature received by the BGRU network, and the mixed sound source feature corresponding to the sound source identified by the first n sound source identification subnetworks is removed from the mixed sound source feature; Based on the losses of the N sound source recognition results, the sound source recognition network is trained to obtain a sound source recognition network.

11. A sound source recognition device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 5.

12. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 5.

Images