Sound source separation method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a sound source separation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed audio and prompt information of a target sound source, and enabling the prompt information of the target sound source to be obtained after the initial prompt information of the target sound source is updated; the initial prompt information is obtained based on the audio of the target sound source, and the updating is carried out based on the initial prompt information to obtain a separation result of the target sound source from the first sample audio; and based on the to-be-processed audio and the prompt information of the target sound source, determining the audio of the target sound source in the to-be-processed audio.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a sound source separation method, device, electronic device and storage medium. Background Art

[0002] Sound source separation refers to separating one or more target sound sources from an audio stream that is a mixture of multiple sounds. It can be applied to audio extraction, speech enhancement, and so on. Summary of the invention

[0003] In view of this, the present disclosure provides a sound source separation method, device, electronic device and storage medium to solve the problem of sound source separation.

[0004] In a first aspect, the present disclosure provides a method for separating sound sources, the method comprising:

[0005] Acquire prompt information of the audio to be processed and the target sound source, wherein the prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source, the initial prompt information is obtained based on the target sound source, and the updating is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information;

[0006] Based on the prompt information of the target sound source, the target sound source is separated from the audio to be processed.

[0007] In a second aspect, the present disclosure provides a sound source separation device, the device comprising:

[0008] An acquisition module, used to acquire prompt information of the audio to be processed and a target sound source, wherein the prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source, the initial prompt information is obtained based on the target sound source, and the update is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information;

[0009] The sound source separation module is used to separate the target sound source from the audio to be processed based on the prompt information of the target sound source.

[0010] In a third aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, computer instructions are stored in the memory, and the processor executes the sound source separation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0011] In a fourth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the sound source separation method of the first aspect or any corresponding embodiment thereof.

[0012] The sound source separation method provided by the present disclosure can enhance the prompt information of the target sound source by updating the initial prompt information, and improves the separation performance of the target sound source by using the prompt information updated from the initial prompt information of the target sound source. Compared with directly using the initial prompt information obtained from the audio of the target sound source to identify the target sound source, this method enhances the prompt information of the target sound source, and therefore can improve the separation effect of the target sound source. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 is a flow chart of a sound source separation method according to an embodiment of the present disclosure;

[0015] Figure 2 is a flow chart of a sound source separation method according to an embodiment of the present disclosure;

[0016] Figure 3 is a structural block diagram of sound source separation according to an embodiment of the present disclosure;

[0017] Figure 4 is a flow chart of a method for determining prompt information of a target sound source according to an embodiment of the present disclosure;

[0018] Figure 5 is a structural block diagram of determining initial prompt information of a sound source according to an embodiment of the present disclosure;

[0019] Figure 6 is a structural block diagram of prompt information for determining a sound source according to an embodiment of the present disclosure;

[0020] Figure 7 is a structural block diagram of a sound source separation device according to an embodiment of the present disclosure;

[0021] Figure 8 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0023] In the related art, the solution adopted for sound source separation is: for each type of sound source, a sound source separation model is obtained, so as to extract the corresponding type of sound source from the mixed audio. For example, for human voice, a sound source separation model that can separate human voice is trained; for music, a sound source separation model that can separate music is trained, and so on. However, in this solution, a sound source separation model needs to be obtained for each type. If a sound source separation model needs to separate multiple sound sources, it is necessary to first train multiple sound source separation models, and then use each sound source separation model to perform sound source separation, which results in low efficiency of sound source separation.

[0024] In another related technology, the scheme adopted for sound source separation is: obtaining prompt information (prompt) of each sound source, using the prompt information to train the sound source separation model, and updating the parameters of the sound source separation model while keeping the prompt information unchanged, so that the trained sound source separation model is a universal sound source separation model. In this scheme, although the trained sound source separation model is universal, the data set used to train the sound source separation model is limited in size and the data distribution is uneven, resulting in the general sound source separation model having poor performance in separating less common sound sources in specific downstream tasks.

[0025] Based on this, the sound source separation method provided by the embodiment of the present disclosure separates the target sound source of the audio to be processed based on the prompt information of the target sound source, wherein the prompt information of the target sound source is updated on the basis of the initial prompt information of the target sound source, and the initial prompt information is obtained based on the target sound source. Since the prompt information of the target sound source is updated on the basis of the initial prompt information, the obtained prompt information can have more detailed features of the target audio, thereby improving the separation performance of the target sound source.

[0026] According to an embodiment of the present disclosure, an embodiment of a sound source separation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a sound source separation method is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 1 is a flow chart of a sound source separation method according to an embodiment of the present disclosure, such as Figure 1 As shown, the process includes the following steps:

[0028] Step S101, obtaining prompt information of the audio to be processed and the target audio source.

[0029] Among them, the prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source, the initial prompt information is obtained based on the audio of the target sound source, and the update is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information.

[0030] The audio to be processed is the audio collected in the actual scene, which may include audio from multiple sound sources. For example, in a public scene, it may include vehicle sounds, human voices, and wind sounds, etc.; in an entertainment scene, it may include human voices and music sounds, etc. Alternatively, the audio to be processed may also be stored offline audio, which may also include audio from multiple sound sources.

[0031] The target sound source is the sound source to be separated, which can be one or more. For example, in a public scene, the sound of a vehicle needs to be separated, and the sound of the vehicle is the target sound source. For another example, in an entertainment scene, the sound of music needs to be separated, and the sound of music is the target sound source.

[0032] The prompt information of the target sound source is used to characterize the relevant information of the target sound source, such as the sound source characteristics and the like. Specifically, audio detection is performed on the audio of the target sound source to obtain initial prompt information; and the initial prompt information is iteratively updated to obtain the prompt information of the target sound source. The update is based on the difference between the predicted result of the sound source separation model for the target sound source and the actual result, that is, the first sample audio and the initial prompt information are input into the sound source separation model to obtain the predicted result of the target audio, and then the initial prompt information is iteratively updated using the difference between the predicted result and the actual result. The stopping condition for the iterative update may be that the current number of iterations reaches the maximum number of iterations, or that the difference between the predicted result and the actual result is less than a preset value, and the like.

[0033] Step S102: separating the target sound source from the audio to be processed based on the prompt information of the target sound source.

[0034] When separating the target sound source of the audio to be processed, the prompt information of the target sound source is used as a reference to separate the target sound source from the audio to be processed including multiple sound sources. For example, sound source separation is achieved based on a sound source separation model, the prompt information of the audio to be processed and the target sound source is the input of the sound source separation model, and the target sound source is the output of the sound source separation model. The sound source separation model is a general sound source separation model, and the separation of multiple sound sources can be achieved using this model. For example, the sound source separation model can separate vehicle sounds and music sounds.

[0035] The sound source separation model can be implemented based on a multimodal generative adversarial network, or by introducing a self-attention mechanism into a convolutional neural network, etc. Of course, the results of the sound source separation model are not limited to those described above, and other structures can also be used. The specific structure is set according to actual needs, and no limitation is made here.

[0036] The sound source separation method provided in this embodiment improves the separation performance of the target sound source by using the updated prompt information of the target sound source to separate the target sound source, because the update of the initial prompt information can strengthen the prompt information of the target sound source. Compared with directly using the initial prompt information obtained from the audio of the target sound source to identify the target sound source, this method can improve the separation effect of the target sound source because it strengthens the prompt information of the target sound source.

[0037] In this embodiment, a sound source separation method is provided, which can be used in electronic devices such as computers, mobile terminals, etc. Figure 2 is a flow chart of a sound source separation method according to an embodiment of the present disclosure, such as Figure 2 As shown, the process includes the following steps:

[0038] Step S201, obtaining prompt information of the audio to be processed and the target audio source.

[0039] The target sound source prompt information is obtained by updating the target sound source's initial prompt information, which is obtained based on the target sound source's audio. Figure 1 The detailed description of step S101 in the illustrated embodiment is not repeated here.

[0040] Step S202: Separate the target sound source from the audio to be processed based on the prompt information of the target sound source.

[0041] Specifically, the above step S202 includes:

[0042] Step S2021, input the audio to be processed and the prompt information of the target sound source into the target sound source separation model, and the target sound source separation model outputs the target sound source.

[0043] Among them, the target sound source separation model is used to separate multiple sound sources.

[0044] The target sound source separation model is a universal sound source separation model, and the separation of multiple sound sources can be achieved by using this model. Among them, the separation of multiple sound sources can be to separate multiple sound sources at the same time, or to separate only one at a time, but multiple sound sources can be separated in sequence. For example, the audio to be processed includes human voices, animal sounds, vehicle sounds, and music sounds, and the target sound sources include human voices and animal sounds. The target sound source separation model can be used to separate human voices and animal sounds at the same time, or to separate human voices and animal sounds separately. Of course, for the separation of human voices, human voices with different timbres can be further separated, for example, speaker 1, speaker 2, and so on.

[0045] The input of the target sound source separation model is the audio to be processed and the prompt information of the target sound source, and the output is the target sound source. If the target sound source is one, the input includes the prompt information of one target sound source; if the target sound source is multiple, the input includes the prompt information of the corresponding multiple target sound sources.

[0046] For example, Figure 3 FIG. 1 shows a schematic diagram of the processing structure of the sound source separation. Figure 3 In the target sound source separation model, the input includes the audio to be processed and the prompt information of the target sound source, and the output is the target sound source. If the target sound source is one, the output is the audio of one sound source; if the target sound source is multiple, the output is the audio corresponding to each target sound source.

[0047] In some optional implementations, the target sound source separation model is a pre-trained sound source separation model. For example, by collecting a large amount of sample data corresponding to various sound sources, a sound source separation model is pre-trained based on the large amount of sample data, i.e., the target sound source separation model. Since the target sound source separation model is trained based on sample data of multiple sound sources, it can be applied to the separation of multiple sound sources.

[0048] Since the pre-trained sound source separation model is obtained through large-scale data training, it has better separation performance, thereby improving the separation effect of the target audio.

[0049] The sound source separation method provided in this embodiment is a target sound source separation model for separating multiple sound sources, which can also be called a universal sound source separation model. The universal sound source separation model is used in the separation of the target sound source. On the one hand, the generalization ability of the universal sound source separation model can be utilized. On the other hand, due to the use of prompt information of the target sound source, the separation performance of the target sound source can also be improved.

[0050] The above-mentioned sound source separation method is obtained based on the prompt information of the target sound source, wherein: Figure 4 A flow chart of a method for obtaining prompt information of a target sound source is shown. Figure 4 The method for determining the prompt information of the target sound source includes:

[0051] Step S401, obtaining initial prompt information and a first sample audio of a target sound source.

[0052] The first sample audio includes a target sound source.

[0053] The initial prompt information of the target sound source is obtained based on the target sound source, for example, by audio encoding the target sound source. Since the initial prompt information needs to be updated in the subsequent process, the initial prompt information can also be called trainable prompt information.

[0054] The first sample audio includes the target sound source, and of course, may also include audio from other sound sources. The first sample audio may be obtained by mixing other background audio based on the target sound source, or may be obtained by collecting audio in an actual scene. That is, the first sample audio may include fused audio, or may include audio in a real scene, and so on.

[0055] In some optional implementations, the step S401 of acquiring the initial prompt information of the target sound source includes:

[0056] Step a1, obtaining at least one original audio containing a target sound source.

[0057] Step a2: performing audio event detection of a target sound source on at least one original audio to obtain an audio code of the original audio.

[0058] Step a3, based on the audio encoding of the original audio, obtain the initial prompt information of the target sound source.

[0059] For the same target sound source, at least one original audio is obtained. Alternatively, the same target sound source may also include audio of multiple subcategories. For example, for the same animal A, different species of animal A make different sounds. Therefore, the sound source of animal sound may include audio of multiple species of animal A as the original audio of animal A.

[0060] Perform audio event detection of the target sound source on at least one acquired original audio to obtain the audio code of the original audio. Audio event detection can be detecting the audio of the active event from the original audio, while removing the static background sound, etc. There is no limitation on the audio event detection method, which is specifically set according to actual needs. The audio code obtained by the audio event detection is used as the initial prompt information of the target sound source.

[0061] In some optional implementations, if the number of original audios is greater than 1, the above step a3 includes: performing mean processing on the audio encodings of all original audios to obtain initial prompt information of the target audio source.

[0062] Specifically, if the number of original audios of the target sound source is N, and N is greater than 1, then a corresponding audio code is obtained for each original audio, that is, N audio codes are obtained for the target sound source. Based on this, it is necessary to fuse the N audio codes to obtain the initial prompt information of the target sound source. The way to fuse the N audio codes includes but is not limited to the mean processing described above, and can also be other processing methods to obtain 1 audio code, which is called the initial prompt information of the target sound source.

[0063] Figure 5 The processing process of multiple original audios is shown. Audio event detection is performed on multiple original audios respectively to obtain audio codes corresponding to the original audios one by one, and the multiple audio codes are averaged to obtain initial prompt information.

[0064] For example, we obtain the original videos of K sound sources, each of which includes N original videos. We perform audio event detection on the original videos of each sound source and obtain N audio codes corresponding to each sound source, which are represented as [e 11 , e 12 ,…,e 1N ],[e 21 , e 22 ,…,e 2N ],…,[e k1 , e k2 ,…,e kN ], where e ij represents the audio code of the jth original audio in sound source i, i = 1, 2, ..., k, i = 1, 2, ..., N. The mean value of the N audio codes of each sound source is calculated to obtain the initial prompt information corresponding to each sound source, i.e., p1, p2, ..., p k , p i It is the initial prompt information of sound source i.

[0065] By performing audio event detection of the target sound source from at least one original audio, an audio code is obtained, and on this basis, initial prompt information of the target sound source is obtained; that is, the initial prompt information is obtained by audio event detection, and since the audio event can guarantee the characteristics of the target audio, the reliability of the initial prompt information can be guaranteed. In the case of multiple original audios, since each original audio can obtain a corresponding audio code, by performing mean processing on multiple audio codes, the data volume of the initial prompt information of the target sound source is reduced, and the robustness of subsequent sound source separation is improved.

[0066] Step S402: input the first sample audio and the initial prompt information of the target sound source into a sound source separation model, and the sound source separation model outputs the predicted audio of the target sound source.

[0067] The sound source separation model can be obtained by pre-training or by other means, etc. The first sample audio and the initial prompt information of the target sound source are input into the sound source separation model, and the sound source separation model outputs the predicted audio of the target sound source. Since the input of the sound source separation model includes the initial prompt information of the target sound source, corresponding guidance is given to the processing of the sound source separation, and therefore, the accuracy of the predicted audio obtained is relatively high.

[0068] Step S403: determining a prediction loss based on the predicted audio of the target sound source and the target audio of the target sound source in the first sample audio.

[0069] When the first sample audio is obtained, the target audio of the target sound source in the first sample audio is also included. The predicted audio output by the sound source separation model and the target audio are subjected to loss calculation to obtain the predicted loss. The specific type of the loss function used in the loss calculation is set according to actual needs and is not limited here.

[0070] Step S404: fix the parameters of the sound source separation model, and update the initial prompt information of the target sound source based on the predicted loss to obtain the prompt information of the target sound source.

[0071] The purpose of the above calculation to obtain the prediction loss is to train the initial prompt information, while the parameters of the sound source separation model remain unchanged. That is, the parameters of the sound source separation model need to be fixed, and the initial prompt information of the target sound source is updated based on the determined prediction loss until the iteration cutoff condition is met. The iteration cutoff condition includes but is not limited to the current number of iterations reaching the maximum number of iterations, or the current prediction loss is less than the preset value.

[0072] Based on this, the process of determining the prompt information of the target sound source includes two parts: initializing the prompt information and fine-tuning the initial prompt information.

[0073] For example, Figure 6 A block diagram for updating the prompt information of the sound source is shown, in which the first sample audio and the prompt information of the audio are input into the sound source separation model to obtain the corresponding predicted audio; the loss of the predicted audio and the target audio is then calculated to obtain the predicted loss, and on this basis the prompt information of the sound source is updated to determine the prompt information of the sound source.

[0074] The method for determining the prompt information of the target sound source provided in this embodiment is based on the initial sound source prompt information and the sound source separation model when determining the prompt information of the target sound source. When the initial sound source prompt information is updated, the parameters of the sound source separation model are fixed, which can not only reduce the amount of data processing, but also retain the performance of the sound source separation model, thereby improving the efficiency of determining the prompt information of the target sound source.

[0075] In some optional implementations, the above-mentioned sound source separation method further includes:

[0076] Step b1, obtaining a sound source separation task, the sound source separation task includes separation of multiple sound sources to be processed.

[0077] Step b2, querying in the sound source prompt information library to obtain prompt information of multiple sound sources to be processed, the sound source prompt information library includes prompt information of multiple sound sources, and the prompt information of multiple sound sources is obtained based on initial prompt information of multiple sound sources.

[0078] Step b3, based on the audio to be processed of the sound source separation task and the prompt information of multiple sound sources to be processed, the audio to be processed is separated into audio sources to obtain audio of multiple sound sources.

[0079] For the sound source separation task, the sound source to be separated is not limited to one, but may be multiple. That is, the sound source separation task includes the separation of multiple sound sources to be processed, which are respectively called sound source to be processed 1 to sound source to be processed k.

[0080] The sound source prompt information library includes prompt information of M sound sources, where M is greater than k. The sound source prompt information of each sound source is obtained based on the initial prompt information of each sound source. The prompt information of each sound source is stored in the sound source prompt information library using a corresponding identifier, such as the sound source name, or the unique identifier of the sound source. Therefore, after determining multiple sound sources to be separated from the sound source separation task, the prompt information of the sound source 1 to the sound source k to be processed can be obtained by querying the sound source prompt information library.

[0081] The audio to be processed and the prompt information of the sound source 1 to be processed to the prompt information of the sound source k to be processed are input into the sound source separation model to obtain the audio of the sound source 1 to be processed to the audio of the sound source k to be processed.

[0082] Of course, if only one sound source needs to be separated, such as the sound source i to be processed, the sound source prompt information library is queried using the sound source i to be processed to obtain the prompt information of the sound source i to be processed. Then the prompt information of the sound source i to be processed and the audio to be processed are input into the sound source separation model to separate the audio of the sound source i to be processed from the audio to be processed.

[0083] For a sound source separation task involving separation of multiple sound sources, the prompt information of the corresponding multiple sound sources can be queried and obtained by implementing an established sound source prompt information database, thereby improving the efficiency of obtaining the prompt information of the sound sources.

[0084] In some optional implementations, the method for determining the sound source prompt information library in the above step b2 includes:

[0085] Step b21, obtaining a second sample audio and initial prompt information of multiple audio sources, wherein the second sample audio includes audio of multiple audio sources.

[0086] Step b22: input the second sample audio and the initial prompt information of multiple sound sources into the sound source separation model, and the sound source separation model outputs the predicted audio of the multiple sound sources.

[0087] Step b23, determining the audio prediction loss based on the predicted audios of the multiple audio sources and the target audios of the multiple audio sources in the second sample audio.

[0088] Step b23, fixing the parameters of the sound source separation model, and updating the initial prompt information of the target sound source based on the audio prediction loss, so as to obtain the prompt information of each sound source to form a sound source prompt information library.

[0089] Since the sound source prompt information library is used to include prompt information of M sound sources, the determination method is to obtain prompt information of M sound sources and then form the sound source prompt information library. The determination method of the prompt information of M sound sources is similar to the determination method of the prompt information of a single sound source, and both are obtained by updating the initial prompt information.

[0090] Specifically, the second sample audio and initial prompt information of multiple sound sources are obtained, and the second sample audio includes audio of multiple sound sources. For example, it is necessary to obtain prompt information of sound sources 1 to sound source M, and the second sample audio includes audio of one or more sound sources from sound source 1 to sound source M. It should be noted that the second sample audio is a general term for multiple audios, and all second sample audios need to include audio of M sound sources from sound source 1 to sound source M.

[0091] The second sample audio and the initial prompt information of the M sound sources are input into the sound source separation model to obtain the predicted audio of the M sound sources. The loss of the predicted audio of the M sound sources and the corresponding target audio is calculated respectively to obtain M loss values, and the M loss values ​​are merged to obtain the audio prediction loss. The parameters of the sound source separation model are fixed, and the initial prompt information of the M sound sources is updated based on the sound source prediction loss to obtain the prompt information of the M sound sources.

[0092] The sound source separation model can simultaneously obtain prompt information of multiple sound sources, thereby improving the efficiency of obtaining prompt information of the sound sources.

[0093] As a specific application embodiment of the embodiment of the present disclosure, if it is necessary to separate the vehicle sound in a public scene, the vehicle sound is used as the target sound source. The target sound source is determined in an interactive manner in the electronic device, and accordingly, the prompt information of the vehicle sound is obtained by using the target sound source. The audio collected in the public scene and the prompt information of the vehicle sound are input into the target sound source separation model to obtain the audio of the vehicle sound. Among them, the prompt information of the vehicle sound is obtained based on the initial prompt information of the vehicle sound. Specifically, the sample audio containing the vehicle sound and the target audio of the vehicle sound in the sample audio are obtained, and the sample audio and the initial prompt information of the vehicle sound are input into the audio separation model to obtain the predicted audio of the vehicle sound. The loss between the predicted audio and the target audio is calculated, and the parameters of the audio separation model are fixed and the initial prompt information is updated based on the calculated loss. After multiple rounds of iterative processing, the prompt information of the vehicle sound is obtained.

[0094] In this embodiment, a sound source separation device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0095] This embodiment provides a sound source separation device, such as Figure 7 As shown, including:

[0096] The acquisition module 701 is used to obtain the prompt information of the audio to be processed and the target sound source. The prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source. The initial prompt information is obtained based on the audio of the target sound source, and the update is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information.

[0097] The sound source separation module 702 is used to separate the target sound source from the audio to be processed based on the audio to be processed and the prompt information of the target sound source.

[0098] In some optional implementations, the sound source separation module 702 includes:

[0099] The sound source separation unit is used to input the prompt information of the audio to be processed and the target sound source into the target sound source separation model. The target sound source separation model outputs the determined target sound source. The target sound source separation model is used to separate multiple sound sources.

[0100] In some optional implementations, the target sound source separation model is a pre-trained sound source separation model.

[0101] In some optional implementations, the acquisition module 701 includes:

[0102] The first prompt information acquisition unit is used to acquire initial prompt information of a target sound source and a first sample audio, where the first sample audio includes the target sound source.

[0103] The first prediction unit is used to input the first sample audio and the initial prompt information of the target sound source into the sound source separation model, and the sound source separation model outputs the predicted audio of the target sound source.

[0104] The first loss determination unit is used to determine the prediction loss based on the predicted audio of the target sound source and the target audio of the target sound source in the first sample audio.

[0105] The first prompt information updating unit is used to fix the parameters of the sound source separation model and update the initial prompt information of the target sound source based on the prediction loss to obtain the prompt information of the target sound source.

[0106] In some optional implementations, the prompt information obtaining unit includes:

[0107] The audio acquisition subunit is used to acquire at least one original audio containing a target sound source.

[0108] The audio event detection subunit is used to perform audio event detection of a target sound source on at least one original audio to obtain an audio code of the original audio.

[0109] The audio encoding subunit is used to obtain initial prompt information of the target sound source based on the audio encoding of the original audio.

[0110] In some optional implementations, if the number of original audios is greater than 1, the audio encoding subunit includes:

[0111] The mean processing subunit is used to perform mean processing on the audio encoding of all original audios to obtain initial prompt information of the target audio source.

[0112] In some optional embodiments, the device further comprises:

[0113] The task acquisition module is used to acquire a sound source separation task, where the sound source separation task includes the separation of multiple sound sources to be processed.

[0114] The query module is used to query in the sound source prompt information library to obtain prompt information of multiple sound sources to be processed. The sound source prompt information library includes prompt information of multiple sound sources, and the prompt information of multiple sound sources is obtained based on initial prompt information of multiple sound sources.

[0115] The separation module is used to separate the audio to be processed based on the audio to be processed of the sound source separation task and the prompt information of multiple sound sources to obtain audio of multiple sound sources.

[0116] In some optional implementations, the determination module of the sound source prompt information library includes:

[0117] The second prompt information acquisition unit is used to acquire the second sample audio and initial prompt information of multiple sound sources, and the second sample audio includes audio of multiple sound sources.

[0118] The second prediction unit is used to input the second sample audio and the initial prompt information of multiple sound sources into the sound source separation model, and the sound source separation model outputs the predicted audio of the multiple sound sources.

[0119] The second loss determination unit is used to determine the audio prediction loss based on the predicted audios of the multiple audio sources and the target audios of the multiple audio sources in the second sample audio.

[0120] The second prompt information updating unit is used to fix the parameters of the sound source separation model and update the initial prompt information of the target sound source based on the audio prediction loss to obtain the prompt information of each sound source to form a sound source prompt information library.

[0121] The sound source separation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0122] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0123] The present disclosure also provides an electronic device having the above Figure 7 The sound source separation device shown.

[0124] See also Figure 8 , Figure 8 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 8As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0125] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0126] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0127] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0129] The electronic device further comprises a communication interface 30 for the electronic device to communicate with other devices or a communication network.

[0130] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0131] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0132] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0133] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0134] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0135] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A sound source separation method, characterized in that: The method comprises: Acquire prompt information of the audio to be processed and the target sound source, wherein the prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source, the initial prompt information is obtained based on the audio of the target sound source, and the update is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information; Based on the prompt information of the target sound source, the target sound source is separated from the audio to be processed.

2. The method according to claim 1, characterized in that The step of separating the target sound source from the audio to be processed based on the prompt information of the target sound source includes: The audio to be processed and the prompt information of the target sound source are input into a target sound source separation model, and the target sound source separation model outputs the target sound source. The target sound source separation model is used to separate multiple sound sources.

3. The method according to claim 1, characterized in that: The prompt information for obtaining the target sound source includes: Acquire initial prompt information of the target sound source and a first sample audio, where the first sample audio includes the target sound source; Inputting the first sample audio and the initial prompt information of the target sound source into a sound source separation model, and the sound source separation model outputs the predicted audio of the target sound source; Determining a prediction loss based on the predicted audio of the target sound source and the target audio of the target sound source in the first sample audio; The parameters of the sound source separation model are fixed, and the initial prompt information of the target sound source is updated based on the predicted loss to obtain the prompt information of the target sound source.

4. The method according to claim 3, characterized in that The obtaining of initial prompt information of the target sound source includes: Acquire at least one original audio containing the target sound source; Performing audio event detection of the target sound source on the at least one original audio to obtain audio coding of the original audio; Based on the audio encoding of the original audio, initial prompt information of the target sound source is obtained.

5. The method according to claim 4, characterized in that If the number of the original audio is greater than 1, the audio encoding based on the original audio to obtain the initial prompt information of the target audio source includes: The audio codes of all the original audios are averaged to obtain initial prompt information of the target sound source.

6. The method according to any one of claims 1 to 5, characterized in that The method further comprises: Acquire a sound source separation task, wherein the sound source separation task includes separation of multiple sound sources to be processed; Searching in a sound source prompt information library to obtain prompt information of the multiple sound sources to be processed, wherein the sound source prompt information library includes prompt information of multiple sound sources, and the prompt information of the multiple sound sources is obtained based on initial prompt information of the multiple sound sources; Based on the audio to be processed of the sound source separation task and the prompt information of the multiple sound sources to be processed, the audio to be processed is separated into sound sources to obtain the audio of the multiple sound sources to be processed.

7. The method according to claim 6, characterized in that The method for determining the sound source prompt information library includes: Acquire a second sample audio and initial prompt information of the multiple sound sources, wherein the second sample audio includes the audio of the multiple sound sources; Inputting the second sample audio and the initial prompt information of the multiple sound sources into a sound source separation model, and the sound source separation model outputs predicted audio of the multiple sound sources; Determining audio prediction loss based on the predicted audios of the multiple audio sources and the target audios of the multiple audio sources in the second sample audio; The parameters of the sound source separation model are fixed, and the initial prompt information of the target sound source is updated based on the audio prediction loss to obtain the prompt information of each sound source to form the sound source prompt information library.

8. A sound source separation device, characterized in that: The device comprises: An acquisition module, used to acquire prompt information of the audio to be processed and a target sound source, wherein the prompt information of the target sound source is obtained by updating the initial prompt information of the target sound source, the initial prompt information is obtained based on the target sound source, and the update is performed based on the separation result of the target sound source obtained from the first sample audio based on the initial prompt information; The sound source separation module is used to separate the target sound source from the audio to be processed based on the prompt information of the target sound source.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the sound source separation method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the sound source separation method according to any one of claims 1 to 7.

Images