Audio separation method, system and related device

Abstract

The invention discloses an audio separation method, an audio separation system and a related device. The method comprises the following steps: acquiring an audio to be separated; inputting the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio; wherein the target separation model is obtained by training a plurality of target training samples, the target training samples are determined based on a plurality of initial training audios and candidate sound parts respectively matched with a plurality of audio track categories, and the initial training audios comprise reference sound parts respectively matched with the plurality of audio track categories; the candidate sound part is used for replacing at least one reference sound part in the initial training audio. In this way, the accuracy of audio separation can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to an audio separation method, system and related devices. Background Art

[0002] Audio separation technology aims to accurately extract the target sound signal from a complex multi-source environment. It is widely used in speech recognition in noisy environments, background noise removal in music production, speaker separation in conference records, and sound event detection in security monitoring. With the rapid development of artificial intelligence and machine learning technologies, modern audio separation technology continues to break through the limitations of traditional methods, achieving more accurate and real-time sound separation effects, greatly improving the naturalness and efficiency of human-computer interaction. Existing audio separation methods mainly rely on neural network models, and the training effect of the neural network model will directly affect the accuracy of audio separation. In order to improve the model training effect, it is mainly trained by constructing a large number of training samples, which requires a lot of cost.

[0003] In view of this, how to improve the efficiency of audio separation and reduce the cost of audio separation has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide an audio separation method, system and related devices, which can improve the accuracy of audio separation.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an audio separation method, comprising: obtaining audio to be separated; inputting the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio; wherein the target separation model is trained using multiple target training samples, and the target training samples are determined based on multiple initial training audios and candidate parts that match multiple audio track categories, respectively, and the initial training audio includes reference parts that match multiple audio track categories, respectively, and the candidate parts are used to replace at least one of the reference parts in the initial training audio.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide an audio separation system, comprising: an acquisition module, used to acquire the audio to be separated; a separation module, used to input the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio; wherein the target separation model is trained using multiple target training samples, and the target training samples are determined based on multiple initial training audios and candidate parts that match multiple audio track categories, respectively, and the initial training audio includes reference parts that match multiple audio track categories, respectively, and the candidate parts are used to replace at least one of the reference parts in the initial training audio.

[0007] To solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, comprising: a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio separation method mentioned in the above technical solution.

[0008] In order to solve the above technical problems, another technical solution adopted in the present application is: providing a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the audio separation method mentioned in the above technical solution is implemented.

[0009] The beneficial effect of the present application is that, different from the prior art, the audio separation method proposed in the present application increases the number of target training samples used for model training by using reference parts and initial training samples that match multiple track categories to expand samples, thereby improving the subsequent training effect of the target separation model and the stability of the trained target separation model. Therefore, after obtaining the audio to be separated, the trained target separation model has high stability and accuracy when separating the audio to be separated. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0011] Figure 1 It is a flowchart of an implementation method of the audio separation method of the present application;

[0012] Figure 2 is a flowchart of an implementation method corresponding to constructing a target training sample;

[0013] Figure 3 yes Figure 2Step S204 corresponds to a flow chart of another implementation method;

[0014] Figure 4 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method;

[0015] Figure 5 yes Figure 4 Step S402 corresponds to a flow chart of another implementation method;

[0016] Figure 6 is a schematic structural diagram of a first processing network corresponding to an implementation mode;

[0017] Figure 7 yes Figure 4 Step S404 corresponds to a flow chart of another implementation method;

[0018] Figure 8 It is a flowchart of an implementation method of a training method of a target separation model of the present application;

[0019] Fig. 9 It is a structural schematic diagram of an implementation method of the audio separation system of the present application;

[0020] Fig.10 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0021] Fig.11 It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of an audio separation method of the present application, the method comprising:

[0024] S101: Acquire the audio to be separated.

[0025] In one implementation, audio to be separated that needs to be separated is obtained.

[0026] In one implementation scenario, the audio to be separated is a musical work including music accompaniment and vocals, and the separation task is to separate the music accompaniment part and the vocal part in the audio to be separated.

[0027] In another embodiment, after the audio to be separated is obtained, noise reduction processing is performed on the audio to be separated to help improve recognition accuracy subsequently.

[0028] Of course, in other embodiments, the audio to be separated may also include other types of sounds, and the above separation task may also be determined according to the types of different sounds in the audio to be separated; for example, when the audio to be separated includes expressive speech and ambient noise, the separation task is to separate the expressive speech and ambient noise.

[0029] S102: Input the audio to be separated into the trained target separation model to obtain the first sub-audio and the second sub-audio; wherein the target separation model is trained using multiple target training samples, and the target training samples are determined based on multiple initial training audios and candidate parts that match multiple audio track categories, respectively. The initial training audio includes reference parts that match multiple audio track categories, and the candidate parts are used to replace at least one reference part in the initial training audio.

[0030] In one implementation, a trained target separation model is obtained, and the obtained audio to be separated is input into the target separation model to obtain a first sub-audio and a second sub-audio corresponding to the audio to be separated.

[0031] In one implementation scenario, the audio to be separated is a music song, the first sub-audio obtained after separation is the accompaniment part, and the second sub-audio is the vocal part other than the accompaniment part. The target separation model is trained using multiple target training samples, and the target training samples are obtained by sample expansion using multiple initial training audios and candidate voice parts that match multiple track categories.

[0032] Specifically, all initial training audios are used as target training samples. Also, since each initial training audio includes reference parts that match multiple track categories, the corresponding reference parts in the initial training audio are replaced with candidate parts to generate new target training samples. The track categories are determined based on vocal categories and instrument categories. For example, the track categories include vocals, drums, bass, zither, and flute.

[0033] The audio separation method proposed in this application uses reference parts and initial training samples that match multiple audio track categories for sample expansion, thereby increasing the number of target training samples used for model training, improving the subsequent training effect of the target separation model and the stability of the trained target separation model. Therefore, after obtaining the audio to be separated, the trained target separation model has high stability and accuracy when separating the audio to be separated.

[0034] See also Figure 2 , Figure 2 : is a flowchart of an implementation method for constructing a target training sample. Specifically, the process of constructing the target training sample includes:

[0035] S201: Construct a candidate database; wherein the candidate database includes a plurality of candidate voice parts that are matched with different audio track categories.

[0036] In one embodiment, a plurality of audio track categories are predetermined. For each audio track category, a plurality of corresponding candidate voice parts are obtained. A candidate database is constructed based on the candidate voice parts matching each audio track category.

[0037] In one implementation scenario, when the audio track category corresponds to an instrument, the step of obtaining the corresponding candidate voice part includes: obtaining the corresponding performance audio from an open source database for the name of the instrument, and using the performance audio as the candidate voice part. Alternatively, the performance audio of the corresponding instrument is collected on-site, and the collected performance audio is used as the corresponding candidate voice part.

[0038] And, when the audio track category is human voice, the step of obtaining the corresponding candidate voice part includes: obtaining multiple target texts, using relevant personnel to read the target texts, and using the collected reading audio as the corresponding candidate voice part. Alternatively, obtaining multiple target texts, converting the target texts into audio through a speech conversion model, and using the audio as the corresponding candidate voice part.

[0039] S202: Acquire multiple initial training audios, and use at least one candidate voice part in the initial training audios as a target voice part.

[0040] In one embodiment, a plurality of initial training audios are obtained, each of which includes reference parts respectively matched with a plurality of audio track categories.

[0041] Furthermore, for each initial training audio, at least one is selected from all corresponding reference parts as the target part.

[0042] In a specific application scenario, the initial training audio is obtained through an open source database.

[0043] In another embodiment, a pre-set quantity threshold is obtained. For each initial training audio, the number of target parts selected from all corresponding reference parts is less than or equal to the quantity threshold. This method helps to avoid too many reference parts in the initial training audio being replaced, resulting in poor correlation between different reference parts. For example, in response to each initial training audio including four different track categories, the above quantity threshold is set to 3.

[0044] S203: Based on the first track category matched by the target part, obtain a candidate part matching the first track category from a candidate database.

[0045] In one embodiment, the track category that matches the target voice part is used as the first track category, and any reference voice part that matches the first track category is selected from the candidate database.

[0046] In a specific application scenario, if the first audio track category that matches the selected target part is "drums", a candidate part that matches "drums" is selected from the candidate database. Alternatively, if the first audio track category that matches the selected target part includes "drums" and "human voice", a candidate part that matches "drums" and a candidate part that matches "human voice" are selected from the candidate database.

[0047] S204: Replace the target part with the candidate part matched with the first audio track category to obtain a target training sample.

[0048] In one embodiment, the candidate voice part selected from the candidate database and matching the first audio track category is used to replace the corresponding target voice part to obtain a target training sample.

[0049] In a specific application scenario, when the first audio track category is "drum", the target part matching the first audio track category is replaced with the candidate part matching "drum", and the replaced initial training audio is used as the target training audio.

[0050] The above scheme constructs a candidate database in advance so that after obtaining multiple initial training audios, at least one reference part in the initial training audio is replaced with a candidate part in the candidate database to expand the number of training samples.

[0051] See also Figure 3 , Figure 3 yes Figure 2 Step S204 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S204 includes:

[0052] S301: Adjusting parameter information corresponding to the candidate part matching the first audio track category to obtain an adjusted part; wherein the parameter information includes at least one of a playback speed, an audio amplitude, an audio sampling rate, and an audio format of the candidate part.

[0053] In one embodiment, in response to determining the candidate part that matches the first track category, parameter information corresponding to the candidate part that matches the first track category is obtained, and the parameter information is adjusted to obtain an adjusted part.

[0054] In one implementation scenario, the reference information includes at least one of a playback speed, an audio amplitude, an audio sampling rate, and an audio format of the corresponding candidate voice part.

[0055] In a specific application scenario, when the reference information includes the playback speed, audio amplitude, audio sampling rate and audio format of the candidate part, the adjustment of the reference information includes: adjusting the playback speed of the candidate part; adjusting the audio amplitude of the candidate part to adjust the playback loudness of the candidate part; adjusting the audio sampling rate corresponding to the candidate part; and converting the audio format corresponding to the candidate part, for example, from flac format to mp3 format, etc.

[0056] Optionally, in other implementations, only part of the data in the reference information may be adjusted. For example, when the reference information includes the playback speed, audio amplitude, audio sampling rate, and audio format of the candidate voice part, only the audio amplitude is adjusted.

[0057] S302: Replace the corresponding target part with the adjusted part to obtain a target training sample.

[0058] In one implementation, the corresponding target voice part is replaced with the adjusted voice part to obtain the target training sample.

[0059] The above scheme helps to improve the flexibility of obtaining target training samples by adjusting the parameter information corresponding to the corresponding candidate voice parts.

[0060] In another embodiment, after the candidate database is constructed, the step of constructing the target training sample may further include: determining multiple second track categories, selecting candidate parts matching each second track category from the candidate database, and obtaining the target training sample based on the candidate parts matching the second track category.

[0061] Specifically, a plurality of audio track categories corresponding to the candidate database are randomly selected as the second audio track categories. For each second audio track category, any matching candidate voice part is selected. All the selected candidate voice parts matching the second audio track category are fused to obtain the target training sample.

[0062] It should be noted that, in order to achieve a higher degree of integration between different candidate parts, before integrating all candidate parts matching the second track category, the playback speed of each candidate part matching the second track category is adjusted so that the time required to play different candidate parts is the same. Alternatively, after selecting any candidate part matching each second track category, the length of at least part of the candidate parts is trimmed so that the lengths of the candidate parts matching each second track category are the same.

[0063] See also Figure 4 , Figure 4 yes Figure 1 Step S102 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S102 includes:

[0064] S401: Input the audio to be separated into the feature extraction network in the target separation model to obtain the initial frequency domain features corresponding to the audio to be separated.

[0065] In one embodiment, the audio to be separated is input into a feature extraction network in a target separation model, so that the feature extraction network is used to extract features of the audio to be separated, and extracted features corresponding to the initial frequency domain dimension are obtained.

[0066] Furthermore, in order to save computational effort in subsequent processing, the extracted features are compressed in the frequency domain dimension to obtain initial frequency domain features corresponding to the first reference dimension.

[0067] In a specific application scenario, the sampling rate of the audio to be separated is 48K, and the Hanning window with a frame length of 2048 and a frame shift of 1024 is used to extract features of the audio to be separated to obtain extracted features. In addition, the extracted features are dimensionally compressed to obtain initial frequency domain features. The formula for calculating the initial frequency domain features is as follows:

[0068] Y(T,F)=FFT(x(t)) 2048 ,F∈(0,1024)

[0069] Y'(T,F)=log(FtoBark(Y(T,F)) 2 )

[0070] Among them, Y(T,F) represents the extracted features, x(t) represents the audio to be separated, T represents the time frame index, F represents the frequency domain point index, and Y′(T,F) represents the initial frequency domain features, and its corresponding first reference dimension is 128.

[0071] S402: Input the initial frequency domain features into the first processing network in the target separation model to obtain a real number mask matching the initial frequency domain features.

[0072] In one embodiment, the initial frequency domain features are input into the first processing network in the target separation model, so as to use the first processing network to extract time domain and frequency domain related features from the initial frequency domain features, and obtain a real number mask for audio separation after decoding.

[0073] S403: Based on the real number mask and the initial frequency domain feature, obtain an enhanced feature corresponding to the initial frequency domain feature.

[0074] In one implementation, the obtained real number mask is dimensionally enhanced, and enhanced features are obtained based on the real number mask after dimension enhancement and the extracted features obtained in step S401.

[0075] Specifically, the real number mask after dimension enhancement is multiplied by the above-mentioned extracted features to obtain enhanced features.

[0076] S404: Input the enhanced features into the second processing network in the target separation model to obtain a complex mask.

[0077] In one embodiment, the obtained enhanced features are input into a second processing network in the target separation model, so that the second processing network further extracts features from the enhanced features to obtain a complex mask.

[0078] S405: Based on the complex mask and the enhanced feature, obtain a first sub-audio and a second sub-audio corresponding to the audio to be separated.

[0079] In one implementation, the complex mask and the enhanced feature are multiplied to obtain a target feature, and the target feature is subjected to an inverse Fourier transform to obtain a first sub-audio.

[0080] Furthermore, the second sub-audio is obtained by using the audio to be separated and the first sub-audio, that is, the part of the audio to be separated except the first sub-audio is used as the second sub-audio.

[0081] See also Figure 5 , Figure 5 yes Figure 4 Step S402 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S402 includes:

[0082] S501: Split the initial frequency domain feature into multiple frequency domain sub-features.

[0083] In one embodiment, the initial frequency domain feature is divided according to the first reference dimension corresponding to the frequency domain point index in the initial frequency domain feature, and the dimension of the feature obtained after the division is increased to the second reference dimension to obtain the corresponding multiple frequency domain sub-features. The feature obtained after the division is increased in dimension to improve the expressive power of the feature in the subsequent process.

[0084] Specifically, see Figure 6 , Figure 6 The initial frequency domain features are input to the frequency band separation layer in the first processing network, so as to divide and enhance the dimension of the initial frequency domain features by using the frequency band separation layer to obtain corresponding multiple frequency domain sub-features.

[0085] In one implementation scenario, in response to the first reference dimension of the frequency domain point index of the initial frequency domain feature being 128, the initial frequency domain feature is divided into 32 sub-band features according to each adjacent four frequency domain points as a sub-band, and the dimension corresponding to each sub-band feature is 4. The feature dimension of each sub-band feature is raised to the second reference dimension to obtain the frequency domain sub-feature corresponding to each sub-band feature. The second reference dimension is 64.

[0086] S502: Input the plurality of frequency domain sub-features into a first coding layer in a first processing network in sequence to obtain a first coding feature corresponding to the frequency domain sub-feature.

[0087] In one embodiment, a plurality of frequency domain sub-features are sequentially input into a first coding layer in a first processing network in a time sequence, so that the frequency domain sub-features are encoded by the first coding layer to obtain a first coded feature.

[0088] Specifically, Figure 6 As shown, the first coding layer is composed of a normalization layer, a bidirectional GRU (Gate Recurrent Unit) network and a full connection. By sequentially inputting multiple frequency domain sub-features into the first coding layer, the first coding layer further extracts the input frequency domain sub-features, and enhances the features of the frequency domain sub-features related to the time domain and the frequency domain, respectively, to obtain a first coding feature with stronger expression ability, so as to help improve the effect of subsequent audio separation.

[0089] S503: Input the first encoding features into the first decoding layer in the first processing network to obtain a first decoding feature corresponding to each first encoding feature.

[0090] In one embodiment, the first encoding feature is input into a first decoding layer in a first processing network, so as to perform dimensionality reduction processing on the first encoding feature using the first decoding layer to obtain a corresponding first decoding feature.

[0091] Specifically, the first decoding layer is composed of a normalization layer and a fully connected layer, and the first decoding layer performs decoding and dimensionality reduction processing on the input first coding feature to obtain a first decoding feature, wherein the dimension of the first decoding feature is consistent with the dimension corresponding to each sub-band feature.

[0092] S504: Obtain a real number mask based on all first decoding features.

[0093] In one implementation, the first coding features obtained in step S503 are concatenated in sequence according to the order in the frequency domain to obtain a real number mask of the first reference dimension.

[0094] In another embodiment, in order to save computing resources, the above step S501 may not perform dimension enhancement on the features obtained after the division during execution.

[0095] The above scheme uses the first processing network to obtain a real number mask to help the subsequent extraction of enhanced features that match the audio to be separated, thereby improving the accuracy of audio separation.

[0096] In another embodiment, in response to obtaining the real number mask through the above-mentioned corresponding embodiment, the specific implementation process of step S403 includes: in order to make the dimension of the real number mask consistent with the dimension of the extracted features obtained in step S401, the real number mask is subjected to dimensionality upgrading processing, that is, the dimension of the real number mask after dimensionality upgrading processing is consistent with the initial frequency domain dimension of the extracted features.

[0097] Furthermore, the real number mask after dimension enhancement is multiplied by the above-mentioned extracted features to obtain enhanced features. The specific calculation formula of the enhanced features is as follows:

[0098] Y(T, F) = Y real (T, F)+j·Y imag (T, F)

[0099] Y1(T, F) = [Y real (T, F)+j·Y imag (T, F)]·E(T, F)

[0100] Among them, Y(T, F) represents the extracted features, E(T, F) represents the real mask, and Y1(T, F) represents the enhanced features.

[0101] See also Figure 7 , Figure 7 yes Figure 4 Step S404 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S404 includes:

[0102] S601: Obtain the real feature and the imaginary feature corresponding to the enhanced feature, and obtain the fusion feature based on the real feature and the imaginary feature.

[0103] In one implementation, the enhanced features obtained in step S403 are converted into a complex form, that is, including corresponding real and imaginary features.

[0104] Furthermore, the real part feature and the imaginary part feature are concatenated to obtain a fusion feature. The specific calculation formula of the fusion feature is as follows:

[0105] Y1(T, F) = Y 1-real (T, F)+j·Y 1-imag (T, F)

[0106] Y1′(T, F)=cat[Y 1-real (T, F), Y 1-imag (T, F)]

[0107] Among them, Y1(T, F) represents the enhanced feature, Y 1-real (T, F) represents the real part of the enhanced feature, Y 1-imag (T, F) represents the imaginary feature of the enhanced feature; Y1′(T, F) represents the fused feature, and cat represents the concatenation operation.

[0108] S602: Obtain a real number mask based on the fused features.

[0109] In one embodiment, the fused features are extracted using the second processing network, and the fused features after feature extraction are subjected to dimensionality reduction processing to obtain a target real number mask. The target real number mask is converted into a complex number form to obtain a complex number mask.

[0110] In one implementation scenario, the second processing network also includes a GRU network and a fully connected layer. In the process of forward processing, the fusion features are extracted using the above-mentioned GRU network and the fully connected layer so that the fusion features combine the information in the time dimension. Further, in the process of reverse processing, the extracted features are subjected to dimensionality reduction processing to obtain a target real number mask. The target real number mask is converted into a complex form to obtain a complex number mask. Among them, the specific calculation formula of the complex number mask is as follows:

[0111] E′(T, F)=cat[E real (T, F), E imag (T, F)]

[0112] E1(T, F) = e real (T, F)+j·E imag (T, F)

[0113] Among them, E′(T, F) represents the target real number mask, and E1(T, F) represents the complex number mask.

[0114] In another embodiment, the structure of the second processing network may also refer to other neural network structures capable of feature extraction.

[0115] In the above scheme, the complex mask matching the enhanced feature format is obtained by using the second processing network, so that the separated first sub-audio and second sub-audio are obtained based on multiplying the enhanced feature with the complex mask.

[0116] See also Figure 8 , Figure 8 : is a flowchart of an implementation method of the target separation model training method of the present application. The specific implementation process of the training method includes:

[0117] S701: Acquire multiple first target training samples, input the first target training samples into an initial separation model to obtain a first separation result, and adjust the parameters of a first processing network in the initial separation model based on the first separation result to obtain a reference separation model.

[0118] In one embodiment, a plurality of first target training samples are obtained, and the first target training samples are input into a constructed initial separation model, so that the initial separation model performs audio separation on the first target training samples to obtain a first separation result.

[0119] Specifically, the process of obtaining the first target training sample can refer to the process of obtaining the target training sample in the above-mentioned corresponding implementation mode. According to the track category corresponding to each voice part in the first target training sample, the training label corresponding to the first target training sample is determined. For example, when the audio separation task is to separate the accompaniment and the vocals, according to the voice parts in the first target training sample, the first reference sub-audio and the second reference sub-audio are determined, the first reference sub-audio is the accompaniment part corresponding to the first target training sample, and the second reference sub-audio is the vocal part corresponding to the first target training sample.

[0120] In addition, the specific structure of the initial separation model can refer to the structure of the target separation model mentioned in the above corresponding implementation manner, and the specific process of audio separation by the initial separation model can refer to the audio separation method mentioned in any of the above implementation manners.

[0121] Further, in response to the first separation result output by the initial separation model, the first separation result includes an enhanced feature corresponding to the first target training sample. According to the first separation result and the corresponding training label, a first training loss is calculated, and the parameters of the first processing network in the initial separation model are adjusted according to the first training loss to obtain a reference separation model after preliminary training.

[0122] In one implementation scenario, the first training loss is calculated by using the MSE loss function, and the specific calculation formula is as follows:

[0123] Loss1=MSE(|Y a (T, F)|,|Z a (T, F)|)

[0124] Among them, Loss1 represents the first training loss, Y a (T, F) represents the enhanced features corresponding to the first target training sample, Z a (T, F) is obtained by Fourier transforming the first reference sub-audio. In addition, the initial separation model is preliminarily trained, and the model learning rate is set to 1e -3 .

[0125] S702: Obtain multiple second target training samples, input the second target training samples into the reference training model to obtain a second separation result, and based on the second separation result, adjust at least the parameters in the first processing network and the second processing network in the reference separation model to obtain a trained target separation model.

[0126] In one embodiment, a plurality of second target training samples are obtained. The process of obtaining the second target training samples can refer to the process of obtaining the first target training samples in the above corresponding embodiment. According to the track categories corresponding to each voice part in the second target training samples, the training labels corresponding to the second target training samples are determined. Among them, the training labels corresponding to the second target training samples include a third reference sub-audio and a fourth reference sub-audio, the third reference sub-audio is the accompaniment part corresponding to the second target training sample, and the fourth reference sub-audio is the vocal part corresponding to the second target training sample.

[0127] Furthermore, the second target training sample is input into the reference training model after preliminary training to obtain a second separation result, wherein the second separation result includes the enhanced features corresponding to the second target training sample generated during the separation process, and the third predicted sub-audio finally output by the reference training model corresponding to the second target training sample, the third predicted sub-audio being the accompaniment part in the second target training sample predicted by the reference separation model.

[0128] Further, a second training loss is calculated based on the enhanced features corresponding to the second target training sample, the third reference sub-audio, and the third predicted sub-audio. The second training loss is used to adjust at least the parameters of the first processing network and the second processing network in the reference separation model until the model convergence condition is met, thereby obtaining a trained target separation model. The specific calculation formula of the second training loss is as follows:

[0129] l1=MSE(|Y b (T, F)|,|Z b (T, F)|)

[0130] l2=SISDR(y(t),z(t))

[0131] Loss2=l1+α·l2

[0132] Among them, l1 represents the first sub-loss, Y b (T, F) represents the enhanced features corresponding to the second target training sample, Z b (T, F) is obtained by Fourier transforming the third reference sub-audio; l2 represents the second sub-loss, y(t) represents the third predicted sub-audio, and z(t) represents the third reference sub-audio; Loss2 represents the second training loss, SISDR represents the loss function, and α represents the weight value corresponding to the second sub-loss. The specific value of the weight value can be obtained by estimation by relevant technical personnel, or can also be obtained by reverse deduction after multiple experiments. In addition, during the training of the reference separation model, the model learning rate is set to 5e -4 .

[0133] The above scheme, after constructing the initial training model, performs tests at different stages until the target separation model is obtained, which optimizes the training effect and helps to improve the effect of the target separation model in audio separation.

[0134] Of course, in other embodiments, in order to improve the efficiency of training, the model training process may also include: obtaining a target training sample, inputting the target training sample into a reference training model, obtaining a separation result, and calculating a training loss based on the separation result. The training loss is used to adjust at least the parameters in the first processing network and the second processing network in the initial separation model to obtain a trained target separation model. The specific calculation process of the above training loss can refer to the calculation process of the second training loss in step S702.

[0135] See also Fig. 9 , Fig. 9 1 is a schematic diagram of the structure of an embodiment of the audio separation system of the present application. Specifically, the audio separation system includes an acquisition module 10 and a separation module 20 coupled to each other.

[0136] Specifically, the acquisition module 10 is used to acquire the audio to be separated.

[0137] The separation module 20 is used to input the audio to be separated into the trained target separation model to obtain the first sub-audio and the second sub-audio; wherein the target separation model is obtained by training using multiple target training samples, and the target training samples are determined based on multiple initial training audios and candidate parts that match multiple audio track categories respectively. The initial training audio includes reference parts that match multiple audio track categories respectively, and the candidate parts are used to replace at least one reference part in the initial training audio.

[0138] In one embodiment, please continue to refer to Fig. 9The audio separation system proposed in the present application also includes a sample construction module 30 coupled to the separation module 20. The sample construction module 30 constructs the target training sample in the following steps: constructing a candidate database; wherein the candidate database includes a plurality of candidate parts respectively matched with different audio track categories; obtaining a plurality of initial training audios, and taking at least one candidate part in the initial training audio as the target part; based on the first audio track category matched with the target part, obtaining the candidate part matching the first audio track category from the candidate database; replacing the target part with the candidate part matching the first audio track category to obtain the target training sample.

[0139] In one embodiment, the sample construction module 30 replaces the target part with the candidate part matching the first audio track category to obtain the target training sample, including: adjusting the parameter information corresponding to the candidate part matching the first audio track category to obtain the adjusted adjusted part; wherein the parameter information includes at least one of the playback rate, audio amplitude, audio sampling rate and audio format of the candidate part; and replacing the corresponding target part with the adjusted part to obtain the target training sample.

[0140] In one embodiment, the step of constructing the target training sample by the sample construction module 30 further includes: determining multiple second track categories, selecting candidate parts matching each second track category from the candidate database; and obtaining the target training sample based on the candidate parts matching the second track categories.

[0141] In one embodiment, the separation module 20 inputs the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio, including: inputting the audio to be separated into a feature extraction network in the target separation model to obtain initial frequency domain features corresponding to the audio to be separated; inputting the initial frequency domain features into a first processing network in the target separation model to obtain a real number mask matching the initial frequency domain features; based on the real number mask and the initial frequency domain features, obtaining enhanced features corresponding to the initial frequency domain features; inputting the enhanced features into a second processing network in the target separation model to obtain a complex number mask; based on the complex number mask and the enhanced features, obtaining the first sub-audio and the second sub-audio corresponding to the audio to be separated.

[0142] In one embodiment, the separation module 20 inputs the initial frequency domain features into the first processing network in the target separation model to obtain a real number mask that matches the initial frequency domain features, including: dividing the initial frequency domain features into multiple frequency domain sub-features; inputting the multiple frequency domain sub-features into the first coding layer in the first processing network in sequence to obtain the first coding features corresponding to the frequency domain sub-features; inputting the first coding features into the first decoding layer in the first processing network to obtain the first decoding features corresponding to each first coding feature; and obtaining a real number mask based on all first decoding features.

[0143] In one embodiment, the separation module 20 inputs the enhanced feature into the second processing network in the target separation model to obtain a complex mask, including: obtaining the real feature and the imaginary feature corresponding to the enhanced feature, and obtaining a fused feature based on the real feature and the imaginary feature; and obtaining a complex mask based on the fused feature.

[0144] See also Fig.10 , Fig.10 : It is a structural diagram of an embodiment of an electronic device of the present application. The electronic device includes: a memory 40 and a processor 50 coupled to each other. The memory 40 stores program instructions, and the processor 50 is used to execute the program instructions to implement the method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 50 can also be called a CPU (Center Processing Unit). The processor 50 may be an integrated circuit chip with signal processing capabilities. The processor 50 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 50 can be implemented by an integrated circuit chip.

[0145] See also Fig.11 , Fig.11 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. The computer-readable storage medium 60 stores program instructions 70 that can be executed by a processor. When the program instructions 70 are executed by the processor, the method mentioned in any of the above embodiments is implemented.

[0146] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0148] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0150] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An audio separation method, characterized in that: include: Get the audio to be separated; The audio to be separated is input into a trained target separation model to obtain a first sub-audio and a second sub-audio; wherein the target separation model is trained using a plurality of target training samples, and the target training samples are determined based on a plurality of initial training audios and candidate parts that match a plurality of audio track categories, respectively, the initial training audio includes reference parts that match a plurality of audio track categories, and the candidate parts are used to replace at least one of the reference parts in the initial training audio.

2. The method according to claim 1, characterized in that The step of constructing the target training sample includes: Constructing a candidate database; wherein the candidate database includes a plurality of candidate voice parts respectively matched with different audio track categories; Acquire a plurality of the initial training audios, and use at least one of the candidate voice parts in the initial training audios as a target voice part; Based on the first track category matched by the target part, acquiring the candidate parts matching the first track category from the candidate database; The target part is replaced by the candidate part matched with the first audio track category to obtain the target training sample.

3. The method according to claim 2, characterized in that The step of replacing the target part with the candidate part matched with the first audio track category to obtain the target training sample includes: Adjusting parameter information corresponding to the candidate part matched with the first audio track category to obtain an adjusted part; wherein the parameter information includes at least one of a playback rate, an audio amplitude, an audio sampling rate, and an audio format of the candidate part; The target part is replaced with the adjusted part to obtain the target training sample.

4. The method according to claim 2, characterized in that: The step of constructing the target training sample further includes: Determine a plurality of second audio track categories, and select the candidate voice part matching each of the second audio track categories from the candidate database; The target training sample is obtained based on the candidate voice part matched by the second audio track category.

5. The method according to claim 1, characterized in that The step of inputting the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio comprises: Inputting the audio to be separated into a feature extraction network in the target separation model to obtain initial frequency domain features corresponding to the audio to be separated; Inputting the initial frequency domain features into a first processing network in the target separation model to obtain a real number mask matching the initial frequency domain features; Based on the real number mask and the initial frequency domain feature, obtaining an enhanced feature corresponding to the initial frequency domain feature; Inputting the enhanced features into a second processing network in the target separation model to obtain a complex mask; Based on the complex mask and the enhanced feature, the first sub-audio and the second sub-audio corresponding to the audio to be separated are obtained.

6. The method according to claim 5, characterized in that The step of inputting the initial frequency domain features into a first processing network in the target separation model to obtain a real number mask matching the initial frequency domain features comprises: Splitting the initial frequency domain feature into a plurality of frequency domain sub-features; Inputting the plurality of frequency domain sub-features into a first coding layer in the first processing network in sequence to obtain a first coding feature corresponding to the frequency domain sub-feature; Inputting the first encoding features into a first decoding layer in the first processing network to obtain a first decoding feature corresponding to each of the first encoding features; Based on all the first decoded features, the real number mask is obtained.

7. The method according to claim 5, characterized in that The step of inputting the enhanced features into a second processing network in the target separation model to obtain a complex mask comprises: Acquire a real feature and an imaginary feature corresponding to the enhanced feature, and obtain a fused feature based on the real feature and the imaginary feature; Based on the fused features, the complex mask is obtained.

8. An audio separation system, characterized in that: include: An acquisition module, used to acquire the audio to be separated; A separation module is used to input the audio to be separated into a trained target separation model to obtain a first sub-audio and a second sub-audio; wherein the target separation model is trained using multiple target training samples, and the target training samples are determined based on multiple initial training audios and candidate parts that match multiple track categories respectively, the initial training audio includes reference parts that match multiple track categories respectively, and the candidate parts are used to replace at least one of the reference parts in the initial training audio.

9. An electronic device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the audio separation method according to any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the audio separation method according to any one of claims 1 to 7 is implemented.

Images