Method for training audio processing model, audio processing method and related device

Abstract

The application provides a training method of an audio processing model, an audio processing method and related equipment. The training method of the audio processing model comprises inputting audio sample data into a to-be-trained audio processing model for iterative training, the to-be-trained audio processing model comprising an audio processing network and a speaker recognition network; determining a loss value of the Nth training by using a loss function, N being a positive integer; adjusting parameters of the to-be-trained audio processing model based on the loss value; and in the case that the loss value meets a loss convergence condition, determining the audio processing network in the to-be-trained audio processing model of the Nth training as an audio processing model. The application solves the problem of low accuracy of speaker separation.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a training method for an audio processing model, an audio processing method, and related equipment. Background Technology

[0002] With the continuous development of technology, speaker separation technology has been applied in scenarios such as conferences, voice calls, and broadcast audio. Speaker separation technology, also known as speaker segmentation, speaker role separation, speaker logging, and speaker archiving, is mainly used to mark the speaking time of each speaker in an audio segment.

[0003] Existing technologies for speaker separation typically extract speaker vectors from each audio segment, then cluster the extracted speaker vectors in an unsupervised manner, and obtain the speaking time segment of each speaker through continuous iterative operations.

[0004] Because existing technologies lack explicit reference feature vectors when performing speaker separation, the accuracy of extracting speaker vectors from audio segments is low, which in turn leads to low accuracy in speaker separation. Summary of the Invention

[0005] This invention provides a training method for an audio processing model, an audio processing method, and related equipment to solve the problem of low accuracy in speaker separation.

[0006] In a first aspect, embodiments of the present invention provide a method for training an audio processing model, comprising:

[0007] Audio sample data is input into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network.

[0008] The loss value for the Nth training iteration is determined using the loss function, where N is a positive integer.

[0009] The parameters of the audio processing model to be trained are adjusted based on the loss value;

[0010] If the loss value satisfies the loss convergence condition, the audio processing network in the audio processing model to be trained in the Nth training iteration is determined as the audio processing model.

[0011] The audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a sequence of statistical feature vectors representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector. The audio frame sequence includes multiple audio frames. The speaker recognition network is used to perform classification processing on the sequence of statistical feature vectors obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame. The loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and a pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and a pre-labeled second category.

[0012] Optionally, the audio processing network includes:

[0013] The feature extraction layer is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0014] The feature statistics layer is used to perform statistical processing on at least two consecutive first feature vectors in the first sequence to obtain the statistical feature vector sequence, which includes multiple statistical feature vectors.

[0015] An identity vector extraction layer is used to extract vectors from the statistical feature vector sequence to obtain a second sequence, the second sequence including multiple second feature vectors, and each second feature vector corresponds to an audio frame;

[0016] A classification processing layer is used to classify the second sequence to obtain a first category corresponding to each audio frame.

[0017] Optionally, the feature statistics layer includes:

[0018] An audio segmentation sub-layer is used to combine the first feature vectors in the first sequence to obtain feature groups, wherein each feature group includes at least two consecutive first feature vectors.

[0019] A pooling operation sublayer is used to perform a pooling operation on the first feature vector within each feature group to obtain a statistical feature vector for each feature group, wherein the statistical feature vector includes a higher-order statistic.

[0020] Secondly, embodiments of the present invention provide an audio processing method, including:

[0021] The audio frame sequence corresponding to the audio data to be processed is input into the audio processing model to obtain the first category corresponding to each audio frame. The audio frame sequence includes multiple audio frames.

[0022] The audio processing model is trained using the training method for the audio processing model as described in the first aspect.

[0023] Optionally, the step of inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame includes:

[0024] The audio frame sequence corresponding to the audio data to be processed is input into the feature extraction layer for feature extraction to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0025] The first sequence is input into the feature statistics layer, and multiple statistical processing is performed on at least two consecutive first feature vectors to obtain a statistical feature vector sequence, wherein the statistical feature vector sequence includes multiple statistical feature vectors.

[0026] The statistical feature vector sequence is input into the identity vector extraction layer for vector extraction to obtain a second sequence. The second sequence includes multiple second feature vectors, and each second feature vector corresponds to an audio frame.

[0027] The second sequence is input into a classification network layer for classification processing to obtain the first category corresponding to each audio frame.

[0028] Thirdly, embodiments of the present invention also provide a training apparatus for an audio processing model, comprising:

[0029] The training module is used to input audio sample data into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network.

[0030] The first determining module is used to determine the loss value for the Nth training iteration using the loss function, where N is a positive integer;

[0031] The adjustment module is used to adjust the parameters of the audio processing model to be trained based on the loss value;

[0032] The second determining module is used to determine the audio processing network in the audio processing model to be trained in the Nth training as the audio processing model when the loss value satisfies the loss convergence condition.

[0033] The audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a sequence of statistical feature vectors representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector. The audio frame sequence includes multiple audio frames. The speaker recognition network is used to perform classification processing on the sequence of statistical feature vectors obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame. The loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and a pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and a pre-labeled second category.

[0034] Optionally, the audio processing network includes:

[0035] The feature extraction layer is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0036] The feature statistics layer is used to perform statistical processing on at least two consecutive first feature vectors in the first sequence to obtain the statistical feature vector sequence, which includes multiple statistical feature vectors.

[0037] An identity vector extraction layer is used to extract vectors from the statistical feature vector sequence to obtain a second sequence, the second sequence including multiple second feature vectors, and each second feature vector corresponds to an audio frame;

[0038] A classification processing layer is used to classify the second sequence to obtain a first category corresponding to each audio frame.

[0039] Fourthly, embodiments of the present invention also provide an audio processing apparatus, comprising:

[0040] The input module is used to input the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame, wherein the audio frame sequence includes multiple audio frames.

[0041] The audio processing model is trained using the training method for the audio processing model as described in the first aspect.

[0042] Fifthly, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a program stored in the memory and executable on the processor. When executed by the processor, the program implements the steps in the training method of the audio processing model as described in the first aspect, or implements the steps in the audio processing method as described in the second aspect.

[0043] In a sixth aspect, embodiments of the present invention also provide a readable storage medium storing a program that, when executed by a processor, implements the steps in the training method of the audio processing model as described in the first aspect, or implements the steps in the audio processing method as described in the second aspect.

[0044] In this embodiment of the invention, the audio processing model to be trained includes an audio processing network and a speaker recognition network. The audio processing network outputs a first category corresponding to each audio frame, and the speaker recognition network obtains a second category corresponding to each statistical feature vector. During training, the speaker recognition network fully utilizes pre-obtained speaker audio feature information, improving the accuracy of the output results. Because the speaker recognition network improves the accuracy of the statistical feature vector and the second feature vector, the feature vector used by the audio processing model for classification is more consistent with the audio data, thereby improving the output accuracy of the audio processing network, i.e., improving the output accuracy of the audio processing model. Simultaneously, the audio processing model can perform end-to-end speaker separation. Since the output of the second feature vector is jointly influenced by the audio processing network and the speaker recognition network, speaker audio information is fully utilized during speaker separation, improving information utilization. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart illustrating the training method of the audio processing model provided in an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the structure of the audio processing model to be trained provided in an embodiment of the present invention;

[0048] Figure 3 This is a flowchart illustrating the audio processing method provided in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the structure of the training device for the audio processing model provided in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the structure of the audio processing device provided in an embodiment of the present invention;

[0051] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship also changes accordingly.

[0054] like Figures 1-2 As shown, this embodiment of the invention provides a training method for an audio processing model, including:

[0055] Step 101: Input the audio sample data into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network.

[0056] It should be understood that, in this embodiment, the number of audio sample data is multiple. Inputting the audio sample data into the audio processing model to be trained for iterative training can be understood as using multiple sets of the audio sample data to perform multiple iterative training iterations on the audio processing model.

[0057] In this embodiment of the invention, the audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a statistical feature vector sequence representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector, wherein the audio frame sequence includes multiple audio frames; the speaker recognition network is used to perform classification processing on the statistical feature vector sequence obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame; the loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and the pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and the pre-labeled second category.

[0058] It should be understood that, in some embodiments, the first category may also be referred to as a relative category, which can be understood as the speaking time of speaker one relative to speaker two. In the first category, it is not necessary to determine the specific speaker. In some embodiments, the second category may also be referred to as an absolute category, which can determine the specific speaker in the sample set corresponding to each time period.

[0059] It should be understood that the audio processing network obtains the statistical feature vector based on the audio frame sequence corresponding to the audio sample data. The statistical feature vector is input into the speaker recognition network, which can output a second category corresponding to each statistical feature vector, and simultaneously output a second feature vector corresponding to each audio frame.

[0060] It should be understood that before iteratively training the audio processing model to be trained, the sample set typically includes audio recordings of multiple known speakers. Each speaker also corresponds to a second category. In specific implementations, the audio sample data used for iterative training of the audio processing model to be trained is typically audio recordings of dialogues between multiple speakers. The audio processing network outputting the first category corresponding to each audio frame can be understood as obtaining the speaking time segment of each speaker from the audio sample data. The speaker recognition network obtaining the second category corresponding to each statistical feature vector can be understood as the speaker recognition network classifying the statistical feature vectors to determine the second category corresponding to the speaker in each time segment of the audio.

[0061] Step 102: Determine the loss value for the Nth training iteration using the loss function, where N is a positive integer.

[0062] During iterative training, the loss function is used to calculate the output of each training iteration to determine the loss value for that iteration.

[0063] It should be understood that the loss function includes a first loss function and a second loss function. The first loss function is used to calculate the first loss value for each output result of the audio processing network to determine the first loss value for that training iteration; the second loss function is used to calculate the second loss value for each output result of the speaker recognition network to determine the second loss value for that training iteration. The specific type of the loss function is not limited here. For example, in some embodiments, the first loss function can be a permutation-invariant training (PIT) loss function. In some embodiments, the second loss function can be an AAM-Softmax loss function formula.

[0064] It should be understood that the loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and the pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and the pre-labeled second category. The loss value can be understood as including the first loss value calculated by the first loss function and the second loss value calculated by the second loss function.

[0065] Step 103: Adjust the parameters of the audio processing model to be trained based on the loss value.

[0066] It should be understood that adjusting the parameters of the audio processing model to be trained based on the loss value means simultaneously adjusting the parameters of the audio processing model to be trained using both the first loss value and the second loss value, in order to train the audio processing model. In some embodiments, adjusting the parameters of the audio processing model to be trained based on the loss value can also be understood as adjusting the weights of the audio processing model to be trained based on the loss value.

[0067] Step 104: If the loss value satisfies the loss convergence condition, the audio processing network in the audio processing model to be trained in the Nth training iteration is determined as the audio processing model.

[0068] It should be understood that the loss value satisfying the loss convergence condition can be interpreted as the first loss value and / or the second loss value satisfying the convergence condition. If the loss value satisfies the loss convergence condition, the training of the audio processing model to be trained can be considered complete.

[0069] It should be understood that the method for determining the completion of iterative training is not limited in the specific implementation. For example, in some embodiments, if the loss value is greater than a threshold, the audio processing network in the audio processing model to be trained in the Nth training iteration is determined as the audio processing model. In other embodiments, if the number of iterative training iterations reaches a preset value, the audio processing network in the audio processing model to be trained in the Nth training iteration is determined as the audio processing model.

[0070] To facilitate understanding, the training process of the audio processing model will be illustrated below with a specific example. Before iteratively training the audio processing model to be trained, audio from four speakers is first recorded. By processing the audio from the four speakers, feature vectors corresponding to each speaker are obtained. The training process of the audio processing model will be illustrated below with an audio sample dataset. For ease of description, the second category of the four speakers is denoted as A, B, C, and D, respectively. The audio sample dataset is input into the audio processing model to be trained for iterative training. Assuming that the audio sample dataset is the audio of a dialogue between A and C, the first category can be divided into four subcategories: the first subcategory is when only speaker one is speaking; the second subcategory is when only speaker two is speaking; the third subcategory is when both speaker one and speaker two are speaking; and the fourth subcategory is when neither speaker one nor speaker two is speaking. In this case, one of speaker one and speaker two is A, and the other is C.

[0071] The audio processing network can output a first category corresponding to each audio frame. By comparing the first category output by the audio processing network for each audio frame with a pre-known first category corresponding to each audio frame, the difference between the two is obtained. The first loss value is then determined using the first loss function, and the parameters of the audio processing model to be trained are adjusted based on the first loss value. By training the audio processing model to be trained, the accuracy of the first category output by the audio processing network can be improved.

[0072] The speaker recognition network can output a second category corresponding to each statistical feature vector. Each statistical feature vector may correspond to multiple categories, namely one or more of A, B, C, and D. By comparing the second category corresponding to each statistical feature vector output by the speaker recognition network with the pre-known second category corresponding to each statistical feature vector, the difference between the two is obtained. The second loss value is then determined using the second loss function, and the parameters of the audio processing model to be trained are adjusted using the second loss value. By training the audio processing model to be trained, the accuracy of the second category output by the speaker recognition network can be improved.

[0073] Therefore, the audio processing network can perform speaker separation processing on audio, marking the speaking time of each speaker in an audio segment. Since the audio processing network does not need to identify a specific second category, users do not need to register in advance. That is, when processing audio, the speakers in the audio do not need to record their audio information in advance, which improves the convenience of use and the scope of application.

[0074] In some embodiments, the audio processing model trained for the Nth time can be determined as the audio processing model if the loss value satisfies the loss convergence condition. In this case, the audio processing model can output either a first category corresponding to each audio frame or a second category corresponding to each audio frame or each statistical feature vector. Although the audio processing method provided in this embodiment can output both the speaking time segment of each speaker in the audio and the speaker identity of each time segment, when using the audio processing model to process the audio, the user needs to be a speaker from the sample set used during training. It should be understood that, in specific implementations, the audio processing network and the speaker recognition network are separable networks.

[0075] In this embodiment of the invention, the audio processing model to be trained includes an audio processing network and a speaker recognition network. The audio processing network outputs a first category corresponding to each audio frame, and the speaker recognition network obtains a second category corresponding to each statistical feature vector. During training, the speaker recognition network fully utilizes pre-obtained speaker audio feature information, improving the accuracy of the output results. Because the speaker recognition network improves the accuracy of the statistical feature vector and the second feature vector, the feature vector used by the audio processing model for classification is more consistent with the audio data, thereby improving the output accuracy of the audio processing network, i.e., improving the output accuracy of the audio processing model. Simultaneously, the audio processing model can perform end-to-end speaker separation. Since the output of the second feature vector is jointly influenced by the audio processing network and the speaker recognition network, speaker audio information is fully utilized during speaker separation, improving information utilization.

[0076] like Figure 2 As shown, optionally, the audio processing network includes:

[0077] The feature extraction layer is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0078] It should be understood that the audio frame sequence can be understood as a set of multiple audio frames arranged in order. For example, each audio frame of the audio sample data can be extracted and arranged in the time sequence corresponding to the audio frames to obtain the audio frame sequence.

[0079] It should be understood that the specific operations for obtaining the audio frame sequence corresponding to the audio sample data are not limited here. For example, in some embodiments, the input speech signal is considered to be a short-time stationary signal within 10ms-30ms. Therefore, in this embodiment, the audio sample data can be divided into frames using a frame length of 25ms and a frame shift of 10ms to obtain the audio frame sequence.

[0080] It should be understood that the first sequence includes a plurality of first feature vectors, each of which corresponds to an audio frame, and the first feature vector can be used to represent the corresponding audio frame.

[0081] It should be understood that the specific operations for feature extraction from the audio frame sequence to obtain the first sequence are not limited here. For example, in some embodiments, firstly, a short-time Fourier transform is performed on the audio frame sequence to obtain multiple frequency feature points. Optionally, in some embodiments, the short-time Fourier transform can be a 512-point short-time Fourier transform, in which case the number of frequency feature points obtained is 257. Secondly, the amplitudes of the multiple frequency feature points are obtained and a logarithmic operation is performed to obtain multiple corresponding first spectrograms, each of which represents a corresponding audio frame. Then, multidimensional features are extracted from the first spectrograms to obtain multiple corresponding second spectrograms. It should be understood that the multidimensional features can be Mel-Frequency Analysis (Fbank) multidimensional features. It should be understood that the dimension of the extracted features is not limited here; for example, the features can be 64-dimensional features, 128-dimensional features, or 32-dimensional features. Finally, the second spectrograms are input into a neural network to obtain the first sequence. It should be understood that the specific structure of the neural network is not limited here. For example, the neural network can be a convolutional neural network (CNN), a deep neural network (DNN), a time-delay neural network (TDNN), or a long short-term memory artificial neural network (LSTM).

[0082] The feature statistics layer is used to perform statistical processing on at least two consecutive first feature vectors in the first sequence to obtain the statistical feature vector sequence, which includes multiple statistical feature vectors.

[0083] It should be understood that the first feature vector typically includes not only the feature information of its corresponding audio frame, but also the feature information of other audio frames related to its corresponding audio frame. For example, when the neural network is a CNN, the receptive field of each layer of the CNN is the context information of the current layer, and the total receptive field is all acquired context information. It should be understood that the context information refers to other audio frames that are consecutive to the audio frame corresponding to the current layer.

[0084] In some embodiments, the first sequence may also be processed by an attention module to obtain a related first sequence of the same shape, wherein each related first feature vector in the related first sequence contains correlation information between each audio frame and other audio frames.

[0085] When the audio frame is short, the first feature vector is insufficient to represent the speaker's features. To obtain more feature information, the feature statistics layer can be used to statistically process at least two consecutive first feature vectors in the first sequence to obtain the statistical feature vector sequence. The statistical feature vector not only contains the characteristic information of the first feature vector, but also includes statistical quantities of multiple first feature vector information.

[0086] It should be understood that the statistical feature vector sequence includes multiple statistical feature vectors. Therefore, the feature statistics layer needs to perform statistical processing on at least two different consecutive first feature vectors.

[0087] It should be understood that the specific operations for statistical processing of at least two consecutive first feature vectors in the first sequence are not limited here. For example, in one embodiment, the statistical processing of at least two consecutive first feature vectors in the first sequence includes: performing a sliding statistical pooling operation using a window with a window length of 150 frames and a window shift of 1 frame. Specifically, the first feature vectors corresponding to the audio frames included in the window are pooled to obtain the corresponding statistical feature vectors. Then, the window is moved, with each move step being 1 frame. After the move, the first feature vectors corresponding to the audio frames included in the current window are pooled to obtain the corresponding statistical feature vectors. It should be understood that the value of the window length and the value of the window shift are not limited here. When using a 10ms frame shift to perform frame segmentation on the audio sample data, a window with a window length of 150 frames and a window shift of 1 frame can be used for sliding statistical pooling. In this case, each pooling operation is performed on 150 frames, that is, the audio frames within a 1.5-second time period are pooled.

[0088] An identity vector extraction layer is used to extract vectors from the statistical feature vector sequence to obtain a second sequence. The second sequence includes multiple second feature vectors, and each second feature vector corresponds to an audio frame.

[0089] It should be understood that the specific method for vector extraction from the statistical feature vector sequence is not limited here. For example, in one embodiment, vector extraction from the statistical feature vector sequence can be understood as mapping the statistical feature vector sequence to the second sequence.

[0090] It should be understood that the statistical feature vectors output by the feature statistics layer are input into the speaker recognition network. The speaker recognition network outputs a speaker representation corresponding to each statistical feature vector, and also outputs the second sequence via the identity vector extraction layer. Therefore, during the training of the speaker recognition network, data also passes through the identity vector extraction layer. In some embodiments, the identity vector extraction layer can be considered to be shared by the audio processing network and the speaker recognition network.

[0091] A classification processing layer is used to classify the second sequence to obtain a first category corresponding to each audio frame.

[0092] It should be understood that the method for classifying the second sequence is not limited here. Typically, the audio processing model is a neural network model, which can use the classification function of the neural network to classify the second sequence, thereby obtaining the category corresponding to each second feature vector, that is, the first category corresponding to the audio frame corresponding to each second feature vector.

[0093] It should be understood that the speaker recognition network includes a second classification processing layer, which is used to classify the statistical feature vector sequence to obtain a second category corresponding to each audio frame.

[0094] In practice, the speaking time period of each person can be obtained by recording consecutive audio frames of the same category as a time period.

[0095] In this embodiment of the invention, the audio processing network includes a feature extraction layer, a feature statistics layer, an identity vector extraction layer, and a classification processing layer. The feature statistics layer improves the completeness of the acquired feature information and reduces the dimensionality of the first feature vector, thus increasing the speed of data computation. Simultaneously, the identity vector extraction layer maps the statistical feature vector to a second feature vector. Compared to the first feature vector, the second feature vector corresponds more closely to the corresponding audio frame, resulting in a more accurate classification result.

[0096] It should be noted that the various optional implementation methods described in the embodiments of this application can be combined with each other or implemented individually, and the embodiments of this application do not limit this.

[0097] Optionally, the feature statistics layer includes:

[0098] An audio segmentation sublayer is used to combine the first feature vectors in the first sequence to obtain feature groups, wherein each feature group includes at least two consecutive first feature vectors.

[0099] It should be understood that consecutive first feature vectors can be interpreted as the audio frames represented by the first feature vectors being consecutive in time. There are multiple feature groups, and the number of first feature vectors within each feature group may be different. In embodiments of the present invention, two different feature groups may include the same first feature vector.

[0100] A pooling operation sublayer is used to perform a pooling operation on the first feature vector within each feature group to obtain a statistical feature vector for each feature group, wherein the statistical feature vector includes a higher-order statistic.

[0101] It should be understood that pooling the first feature vector within each feature group can be understood as performing statistical calculations on the first feature vector within each feature group, by calculating first-order, second-order, and higher-order statistics. For example, in one embodiment, pooling the first feature vector within each feature group yields the maximum value and / or average value of multiple first feature vectors. In another embodiment, pooling the first feature vector within each feature group yields the standard deviation and / or variance of multiple first feature vectors.

[0102] In this embodiment, the feature statistics layer includes the audio segmentation sublayer and the pooling operation sublayer. Through the audio segmentation sublayer and the pooling operation layer, statistical operations can be performed on at least two consecutive first feature vectors. Since the audio time corresponding to the feature group is relatively long and contains more information, the accuracy of the output speaker identifier is improved. Simultaneously, the system parameters can be adjusted using the second loss value, making the statistical feature vector more consistent with the corresponding audio frame, thereby improving the accuracy of the audio processing model's processing results.

[0103] like Figure 3 As shown, this embodiment of the invention also provides an audio processing method, including:

[0104] Step 301: Input the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame. The audio frame sequence includes multiple audio frames.

[0105] The audio processing model is trained using the audio processing model training method described above.

[0106] Optionally, the step of inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame includes:

[0107] The audio frame sequence corresponding to the audio data to be processed is input into the feature extraction layer for feature extraction to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0108] The first sequence is input into the feature statistics layer, and multiple statistical processing is performed on at least two consecutive first feature vectors to obtain a statistical feature vector sequence, wherein the statistical feature vector sequence includes multiple statistical feature vectors.

[0109] The statistical feature vector sequence is input into the identity vector extraction layer for vector extraction to obtain a second sequence. The second sequence includes multiple second feature vectors, and each second feature vector corresponds to an audio frame.

[0110] The second sequence is input into a classification network layer for classification processing to obtain the first category corresponding to each audio frame.

[0111] It should be understood that during the training process of the audio processing model to be trained, the audio processing model is defined as the audio processing model. Therefore, in this embodiment, inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing model can be understood as inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing network.

[0112] It should be understood that inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame can be understood as follows: inputting the audio frame sequence into a feature extraction layer for feature extraction to obtain the first sequence; inputting the obtained first sequence into a feature statistics layer for feature extraction to obtain the statistical feature vector sequence; inputting the obtained statistical feature vector sequence into an identity vector extraction layer for vector extraction to obtain the second sequence; and inputting the second sequence into a classification network layer for classification processing to obtain the first category corresponding to each audio frame.

[0113] It should be understood that, as described above, the audio processing network includes a feature extraction layer, a feature statistics layer, an identity vector extraction layer, and a classification processing layer. Therefore, the specific implementation method for audio processing using the audio processing model can refer to the foregoing content and will not be repeated here.

[0114] See Figure 4 , Figure 4 This is a structural diagram of the training device 400 for the audio processing model provided in this application embodiment, as shown below. Figure 4 As shown, this embodiment of the invention also provides a training device 400 for an audio processing model, comprising:

[0115] Training module 401 is used to input audio sample data into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network.

[0116] The first determining module 402 is used to determine the loss value of the Nth training iteration using the loss function, where N is a positive integer;

[0117] Adjustment module 403 is used to adjust the parameters of the audio processing model to be trained based on the loss value;

[0118] The second determining module 404 is used to determine the audio processing network in the audio processing model to be trained in the Nth training as the audio processing model when the loss value satisfies the loss convergence condition.

[0119] The audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a sequence of statistical feature vectors representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector. The audio frame sequence includes multiple audio frames. The speaker recognition network is used to perform classification processing on the sequence of statistical feature vectors obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame. The loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and a pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and a pre-labeled second category.

[0120] Optionally, the audio processing network includes:

[0121] The feature extraction layer is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame.

[0122] The feature statistics layer is used to perform statistical processing on at least two consecutive first feature vectors in the first sequence to obtain the statistical feature vector sequence, which includes multiple statistical feature vectors.

[0123] An identity vector extraction layer is used to extract vectors from the statistical feature vector sequence to obtain a second sequence, the second sequence including multiple second feature vectors, and each second feature vector corresponds to an audio frame;

[0124] A classification processing layer is used to classify the second sequence to obtain a first category corresponding to each audio frame.

[0125] The training device 400 for the audio processing model provided in this application embodiment can achieve Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0126] See Figure 5 , Figure 5This is a structural diagram of the audio processing device 500 provided in the embodiments of this application, as shown below. Figure 5 As shown, this embodiment of the invention also provides an audio processing device 500, comprising:

[0127] Input module 501 is used to input the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame, wherein the audio frame sequence includes multiple audio frames;

[0128] The audio processing model 500 is trained using the above-described audio processing model training method.

[0129] The audio processing device 500 provided in this application embodiment can achieve... Figure 3 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0130] Figure 6 A schematic diagram of the hardware structure of an electronic device for implementing the various embodiments of this application.

[0131] The electronic device 600 includes, but is not limited to, components such as: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and a power supply 611. Those skilled in the art will understand that... Figure 6 The electronic device structures shown are not intended to limit the electronic device. An electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements. In the embodiments of this application, the electronic device includes, but is not limited to, mobile phones, tablets, laptops, PDAs, in-vehicle terminals, wearable devices, and pedometers.

[0132] The processor 610 is used to perform the following operations:

[0133] Audio sample data is input into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network.

[0134] The loss value for the Nth training iteration is determined using the loss function, where N is a positive integer.

[0135] The parameters of the audio processing model to be trained are adjusted based on the loss value;

[0136] If the loss value satisfies the loss convergence condition, the audio processing network in the audio processing model to be trained in the Nth training iteration is determined as the audio processing model.

[0137] The audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a sequence of statistical feature vectors representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector. The audio frame sequence includes multiple audio frames. The speaker recognition network is used to perform classification processing on the sequence of statistical feature vectors obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame. The loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and a pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and a pre-labeled second category.

[0138] Alternatively, processor 610 is used to perform the following operations:

[0139] The audio frame sequence corresponding to the audio data to be processed is input into the audio processing model to obtain the first category corresponding to each audio frame. The audio frame sequence includes multiple audio frames.

[0140] The audio processing model is trained using the aforementioned audio processing model training method.

[0141] It should be understood that, in this embodiment, the radio frequency unit 601 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink data from the base station and processes it with the processor 610; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 601 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 601 can also communicate with networks and other devices via a wireless communication system.

[0142] The electronic device provides users with wireless broadband internet access through the network module 602, such as helping users send and receive emails, browse web pages, and access streaming media.

[0143] The audio output unit 603 can convert audio data received by the radio frequency unit 601 or the network module 602 or stored in the memory 609 into audio signals and output them as sound. Furthermore, the audio output unit 603 can also provide audio output related to specific functions performed by the electronic device 600 (e.g., call signal reception sound, message reception sound, etc.). The audio output unit 603 includes a speaker, a buzzer, and a receiver, etc.

[0144] Input unit 604 is used to receive audio or video signals. Input unit 604 may include a graphics processing unit (GPU) 6041 and a microphone 6042. The GPU 6041 processes audio data from still images or videos acquired by an audio capture device (such as a camera) in video capture mode or audio capture mode. The processed audio frames can be displayed on display unit 606. The audio frames processed by GPU 6041 can be stored in memory 609 (or other storage media) or transmitted via radio frequency unit 601 or network module 602. Microphone 6042 can receive sound and process such sound into audio data. The processed audio data can be converted into a format that can be transmitted to a mobile communication base station via radio frequency unit 601 in telephone call mode.

[0145] The electronic device 600 also includes at least one sensor 605, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 6061 according to the ambient light level, and the proximity sensor can turn off the display panel 6061 and / or backlight when the electronic device 600 is moved to the ear. As a type of motion sensor, an accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used to identify the posture of the electronic device (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. The sensor 605 may also include a fingerprint sensor, pressure sensor, iris sensor, molecular sensor, gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc., which will not be described in detail here.

[0146] The display unit 606 is used to display information input by the user or information provided to the user. The display unit 606 may include a display panel 6061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0147] User input unit 607 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of electronic devices. Specifically, user input unit 607 includes a touch panel 6071 and other input devices 6072. Touch panel 6071, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 6071). Touch panel 6071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 610, which receives and executes commands from the processor 610. In addition, touch panel 6071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. Besides touch panel 6071, user input unit 607 may also include other input devices 6072. Specifically, other input devices 6072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here.

[0148] Furthermore, the touch panel 6071 can cover the display panel 6061. When the touch panel 6071 detects a touch operation on or near it, it transmits the information to the processor 610 to determine the type of touch event. Subsequently, the processor 610 provides corresponding visual output on the display panel 6061 based on the type of touch event. Although in Figure 6 In this embodiment, the touch panel 6071 and the display panel 6061 are two independent components to realize the input and output functions of the electronic device. However, in some embodiments, the touch panel 6071 and the display panel 6061 can be integrated to realize the input and output functions of the electronic device. The specific implementation is not limited here.

[0149] Interface unit 608 serves as an interface for connecting external devices to electronic device 600. For example, external devices may include a wired or wireless headphone port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 608 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within electronic device 600, or it can be used to transmit data between electronic device 600 and external devices.

[0150] The memory 609 can be used to store software programs and various data. The memory 609 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 609 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0151] The processor 610 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 609, and by calling data stored in the memory 609, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. The processor 610 may include one or more processing units; preferably, the processor 610 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 610.

[0152] The electronic device 600 may also include a power supply 611 (such as a battery) for supplying power to various components. Preferably, the power supply 611 is logically connected to the processor 610 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system.

[0153] In addition, the electronic device 600 includes some functional modules not shown, which will not be described in detail here.

[0154] Preferably, this application embodiment also provides an electronic device, including a processor 610, a memory 609, and a computer program stored in the memory 609 and executable on the processor 610. When the computer program is executed by the processor 610, it implements the training method of the above-described audio processing model and the various processes of the above-described audio processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0155] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the training method of the aforementioned audio processing model and the various processes of the aforementioned audio processing method embodiments, achieving the same technical effects. To avoid repetition, further details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0158] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A training method for an audio processing model, characterized in that, Comprise: input audio sample data to the to-be-trained audio processing model for iterative training, the to-be-trained audio processing model comprises an audio processing network and a speaker recognition network; Determine the loss value of the Nth training by using the loss function, N is a positive integer; Adjust the parameters of the to-be-trained audio processing model based on the loss value; In the case where the loss value meets the loss convergence condition, the audio processing network in the to-be-trained audio processing model of the Nth training is determined as an audio processing model; Wherein, the audio processing network is used for extracting features from the audio frame sequence corresponding to the audio sample data to obtain a statistical feature vector sequence for representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputting a first category corresponding to each audio frame according to the second feature vector, the audio frame sequence comprises a plurality of audio frames;The speaker recognition network is used for classifying the statistical feature vector sequence obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and the second feature vector corresponding to each audio frame;The loss value comprises a first loss value determined based on the difference between the first category output by the audio processing network and the pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and the pre-labeled second category.

2. The method of claim 1, wherein, The audio processing network comprises: Feature extraction layer, the feature extraction layer is used for extracting features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence, the first sequence comprises a plurality of first feature vectors, and each first feature vector corresponds to an audio frame; Feature statistics layer, the feature statistics layer is used for statistical processing of at least two first feature vectors in the first sequence to obtain a statistical feature vector sequence, the statistical feature vector sequence comprises a plurality of statistical feature vectors; Identity vector extraction layer, the identity vector extraction layer is used for vector extraction of the statistical feature vector sequence to obtain a second sequence, the second sequence comprises a plurality of second feature vectors, and each second feature vector corresponds to an audio frame; Classification processing layer, the classification processing layer is used for classifying the second sequence to obtain a first category corresponding to each audio frame.

3. The method of claim 2, wherein, The feature statistics layer comprises: Audio segmentation sublayer, the audio segmentation sublayer is used for combining the first feature vectors in the first sequence to obtain a feature group, wherein each feature group comprises at least two continuous first feature vectors; Pooling operation sublayer, the pooling operation sublayer is used for pooling operation of the first feature vectors in each feature group to obtain a statistical feature vector of each feature group, wherein the statistical feature vector comprises a high-order statistical quantity.

4. An audio processing method, characterized by, Comprise: Input the audio frame sequence corresponding to the to-be-processed audio data to the audio processing model to obtain a first category corresponding to each audio frame, the audio frame sequence comprises a plurality of audio frames; The audio processing model is obtained by training using the training method for the audio processing model as described in any one of claims 1 to 3.

5. The method of claim 4, wherein, The step of inputting the audio frame sequence corresponding to the audio data to be processed into the audio processing model to obtain the first category corresponding to each audio frame includes: The audio frame sequence corresponding to the audio data to be processed is input into the feature extraction layer for feature extraction to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame. The first sequence is input into the feature statistics layer, and multiple statistical processing is performed on at least two consecutive first feature vectors to obtain a statistical feature vector sequence, wherein the statistical feature vector sequence includes multiple statistical feature vectors. The statistical feature vector sequence is input into the identity vector extraction layer for vector extraction to obtain a second sequence. The second sequence includes multiple second feature vectors, and each second feature vector corresponds to an audio frame. The second sequence is input into a classification network layer for classification processing to obtain the first category corresponding to each audio frame.

6. An apparatus for training an audio processing model, comprising: include: The training module is used to input audio sample data into the audio processing model to be trained for iterative training. The audio processing model to be trained includes an audio processing network and a speaker recognition network. The first determining module is used to determine the loss value for the Nth training iteration using the loss function, where N is a positive integer; The adjustment module is used to adjust the parameters of the audio processing model to be trained based on the loss value; The second determining module is used to determine the audio processing network in the audio processing model to be trained in the Nth training as the audio processing model when the loss value satisfies the loss convergence condition. The audio processing network is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a sequence of statistical feature vectors representing the audio frame sequence and a second feature vector corresponding to each audio frame, and outputs a first category corresponding to each audio frame based on the second feature vector. The audio frame sequence includes multiple audio frames. The speaker recognition network is used to perform classification processing on the sequence of statistical feature vectors obtained by the audio processing network to obtain a second category corresponding to each statistical feature vector and a second feature vector corresponding to each audio frame. The loss value includes a first loss value determined based on the difference between the first category output by the audio processing network and a pre-labeled first category, and a second loss value determined based on the difference between the second category output by the speaker recognition network and a pre-labeled second category.

7. The apparatus for training an audio processing model according to claim 6, wherein, The audio processing network includes: The feature extraction layer is used to extract features from the audio frame sequence corresponding to the audio sample data to obtain a first sequence. The first sequence includes multiple first feature vectors, and each first feature vector corresponds to one audio frame. a feature statistics layer configured to perform statistical processing on at least two continuous first feature vectors in the first sequence to obtain a sequence of statistical feature vectors, the sequence of statistical feature vectors comprising a plurality of statistical feature vectors; an identity vector extraction layer configured to perform vector extraction on the sequence of statistical feature vectors to obtain a second sequence, the second sequence comprising a plurality of second feature vectors, and each second feature vector corresponding to an audio frame; a classification processing layer configured to perform classification processing on the second sequence to obtain a first category corresponding to each audio frame.

8. An audio processing apparatus, characterized by comprising: comprising: an input module configured to input an audio frame sequence corresponding to audio data to be processed into an audio processing model to obtain a first category corresponding to each audio frame, the audio frame sequence comprising a plurality of audio frames; wherein the audio processing model is trained by using the training method of the audio processing model according to any one of claims 1 to 3.

9. An electronic device, comprising: comprising a processor, a memory, and a program stored in the memory and executable on the processor, and when the program is executed by the processor, the steps in the training method of the audio processing model according to any one of claims 1 to 3 are implemented, or the steps in the audio processing method according to claim 4 or 5 are implemented.

10. A readable storage medium, characterized by, the readable storage medium stores a program, and when the program is executed by a processor, the steps in the training method of the audio processing model according to any one of claims 1 to 3 are implemented, or the steps in the audio processing method according to claim 4 or 5 are implemented.

Images