Audio separation method, electronic device and computer-readable storage medium

Abstract

The embodiments of the present application disclose an audio separation method, electronic device, and computer-readable storage medium, wherein the method includes: obtaining multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1; performing multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, the first separation result including N single-track audio data; performing single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, the second separation result including N single-track audio data; and fusing the same type of single-track audio data in the first separation result and the second separation result based on a fusion model to obtain a target separation result. By fusing the processing results of the multi-track audio separation model and the single-track audio extraction model, the present application can effectively improve the overall performance of audio separation.

Description

Technical Field

[0001] The present invention relates to the technical field of audio processing, and in particular to an audio separation method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Typically, an audio file is a mix of data corresponding to multiple different tracks. For example, music can be created by mixing data corresponding to a vocal track and data corresponding to an accompaniment track. The accompaniment track data may also include data corresponding to different instrument tracks such as drums, bass, guitar, and piano. When performing secondary creation on an audio file obtained by mixing different tracks, the audio file needs to be separated to obtain the data corresponding to each track.

[0003] Currently, audio files can be separated by using model processing. For example, extraction models or separation models. Among them, the extraction model can usually only independently obtain the data corresponding to a certain track among multiple tracks, and cannot separate multiple tracks at the same time. The separation model can obtain the data corresponding to each track among multiple tracks at the same time, but because the separation model needs to consider the correlation between each track, when performing audio separation through the separation model, it is difficult to achieve the best separation effect for each track at the same time. Therefore, if the separation effect and correlation of each track are considered at the same time when measuring the overall performance of audio separation, the overall audio separation performance of the above method is poor. Summary of the Invention

[0004] The embodiments of the present application provide an audio separation method, an electronic device, and a computer-readable storage medium, which can effectively improve the overall performance of audio separation.

[0005] In a first aspect, an embodiment of the present application provides an audio separation method, the method comprising:

[0006] Acquire multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1;

[0007] Performing multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, where the first separation result includes N single-track audio data;

[0008] Performing single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data;

[0009] Based on the fusion model, single-track audio data of the same type in the first separation result and the second separation result are fused to obtain a target separation result.

[0010] When implementing the method described in the first aspect, a fusion model can be used to fuse the first separation result of the multi-track audio separation model with the second separation results of the N single-track audio extraction models to obtain a target separation result. Since the separation effect of each single-track audio data in the first separation result is average but the correlation is good, and the separation effect of each single-track audio data in the second separation result is good but the correlation is poor, the separation effect and correlation of each single-track audio data in the fused target separation result can both achieve a good effect, thereby improving the overall performance of audio separation.

[0011] In one possible implementation, the fusing of the same type of single-track audio data in the first separation result and the second separation result based on the fusion model to obtain a target separation result includes:

[0012] Inputting the single-track audio data of the same type in the first separation result and the second separation result into the fusion model;

[0013] Based on the first weight and the second weight in the fusion model, weight operations and addition operations are performed on the single-track audio data of the same type in the first separation result and the second separation result to obtain the target separation result; the first weight is the fusion weight corresponding to the first separation result, and the second weight is the fusion weight corresponding to the second separation result.

[0014] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated;

[0015] The method further comprises:

[0016] Determining target requirements for audio separation of the multi-track audio data;

[0017] If the target requirement is to request that the correlation between the separated single-track audio data is stronger, then the fusion model is obtained, and the first weight of the fusion model is greater than the second weight.

[0018] Based on this approach, when the first separation result and the second separation result are fused, a higher weight can be assigned to the first separation result according to target requirements, thereby making the correlation between each single-track audio data in the target separation result stronger.

[0019] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated;

[0020] The method further comprises:

[0021] Determining target requirements for audio separation of the multi-track audio data;

[0022] If the target requirement is used to request that the correlation between the separated single-track audio data is weak, then the fusion model is obtained, and the first weight of the fusion model is less than the second weight.

[0023] Based on this approach, when the first separation result and the second separation result are fused, a higher weight can be assigned to the second separation result according to target requirements, thereby making the separation effect of each single-track audio data in the target separation result stronger.

[0024] In one possible implementation, the method further includes:

[0025] Read the preset demand information;

[0026] Determining a target requirement for performing audio separation on the multi-track audio data includes:

[0027] The target requirement for performing audio separation on the multi-track audio data is determined according to the preset requirement information read.

[0028] Based on this approach, target requirements can be determined based on preset demand information, and target separation results that meet the preset demand information can be obtained. For example, when music is released online, target requirements can be determined based on the various mixed sound effect types preset by the platform, and target separation results that match each mixed sound effect type can be obtained in advance.

[0029] In one possible implementation, the method further includes:

[0030] Receive user input requirement information sent by the client;

[0031] Determining a target requirement for performing audio separation on the multi-track audio data includes:

[0032] The target requirement for performing audio separation on the multi-track audio data is determined according to the requirement information.

[0033] Based on this approach, a target requirement can be determined based on the user's input requirement information, thereby obtaining a target separation result that satisfies the user's requirement information. For example, the type of mixed sound effects set by the user when listening to music can be obtained, and the target requirement can be determined based on the set mixed sound effects type, thereby obtaining a target separation result that matches the set mixed sound effects type in advance.

[0034] In one possible implementation, the method further includes:

[0035] Acquire multi-track audio training data, multiple real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to each real single-track audio data;

[0036] Inputting the multi-track audio training data into an initial multi-track audio separation model to obtain a plurality of predicted single-track audio data and frequency domain data corresponding to each predicted single-track audio data;

[0037] Determining a first time-domain loss value based on the plurality of real single-track audio data and the plurality of predicted single-track audio data;

[0038] Determining a first frequency domain loss value based on the frequency domain data corresponding to the multiple real single-track audio data and the frequency domain data corresponding to the multiple predicted single-track audio data;

[0039] Model parameters in the initial multi-track audio separation model are determined based on the first time domain loss value and the first frequency domain loss value to obtain the multi-track audio separation model.

[0040] Based on this method, by considering the loss values of the initial multi-track audio separation model in the time domain and frequency domain, the model training effect can be effectively improved, thereby improving the separation effect of the multi-track audio separation model on multi-track audio data.

[0041] In one possible implementation, the method further includes:

[0042] Acquire multi-track audio training data, a real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to the real single-track audio data;

[0043] Inputting the multi-track audio training data into an initial single-track audio extraction model to obtain a predicted single-track audio data and frequency domain data corresponding to the predicted single-track audio data;

[0044] Determine a second time domain loss value based on the one real single-track audio data and the one predicted single-track audio data;

[0045] Determining a second frequency domain loss value based on the frequency domain data corresponding to the one real single-track audio data and the frequency domain data corresponding to the one predicted single-track audio data;

[0046] Model parameters in the initial single-track audio extraction model are determined based on the second time-domain loss value and the second frequency-domain loss value to obtain the single-track audio extraction model.

[0047] Based on this method, by considering the loss values of the initial single-track audio extraction model in the time domain and frequency domain, the effect of model training can be effectively improved, thereby improving the extraction effect of the single-track audio extraction model on single-track audio data in multi-track audio data.

[0048] In a second aspect, an embodiment of the present application provides an audio separation device, the device comprising:

[0049] An acquisition module is used to acquire multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1;

[0050] a processing module, configured to perform multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, wherein the first separation result includes N single-track audio data;

[0051] The processing module is further configured to perform single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data;

[0052] The processing module is further configured to perform fusion processing on the same type of single-track audio data in the first separation result and the second separation result based on a fusion model to obtain a target separation result.

[0053] In one possible implementation, when the processing module performs fusion processing on the single-track audio data of the same type in the first separation result and the second separation result based on the fusion model to obtain the target separation result, it is specifically configured to:

[0054] Inputting the single-track audio data of the same type in the first separation result and the second separation result into the fusion model;

[0055] Based on the first weight and the second weight in the fusion model, weight operations and addition operations are performed on the single-track audio data of the same type in the first separation result and the second separation result to obtain the target separation result; the first weight is the fusion weight corresponding to the first separation result, and the second weight is the fusion weight corresponding to the second separation result.

[0056] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated;

[0057] The processing module is further configured to determine a target requirement for audio separation of the multi-track audio data;

[0058] If the target requirement is to request that the correlation between the separated single-track audio data is stronger, the acquisition module is further used to obtain the fusion model, and the first weight of the fusion model is greater than the second weight.

[0059] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated;

[0060] The processing module is further configured to determine a target requirement for audio separation of the multi-track audio data;

[0061] If the target requirement is used to request that the correlation between the separated single-track audio data is weak, the acquisition module is further used to obtain the fusion model, and the first weight of the fusion model is less than the second weight.

[0062] In one possible implementation, the processing module is further configured to read preset demand information;

[0063] When determining the target requirement for audio separation for the multi-track audio data, the processing module is specifically configured to: determine the target requirement for audio separation for the multi-track audio data according to the read preset requirement information.

[0064] In a possible implementation manner, the acquisition module is further configured to receive user input requirement information sent by the client;

[0065] When determining the target requirement for audio separation for the multi-track audio data, the processing module is specifically configured to: determine the target requirement for audio separation for the multi-track audio data according to the requirement information.

[0066] In one possible implementation, the acquisition module is further configured to acquire multi-track audio training data, a plurality of real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to each real single-track audio data;

[0067] The processing module is also used to: input the multi-track audio training data into the initial multi-track audio separation model to obtain multiple predicted single-track audio data, and frequency domain data corresponding to each predicted single-track audio data; determine a first time domain loss value based on the multiple real single-track audio data and the multiple predicted single-track audio data; determine a first frequency domain loss value based on the frequency domain data corresponding to the multiple real single-track audio data and the frequency domain data corresponding to the multiple predicted single-track audio data; determine the model parameters in the initial multi-track audio separation model based on the first time domain loss value and the first frequency domain loss value to obtain the multi-track audio separation model.

[0068] In a possible implementation, the acquisition module is further configured to acquire multi-track audio training data, a real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to the real single-track audio data;

[0069] The processing module is also used to input the multi-track audio training data into the initial single-track audio extraction model to obtain a predicted single-track audio data and frequency domain data corresponding to the predicted single-track audio data; determine a second time domain loss value based on the real single-track audio data and the predicted single-track audio data; determine a second frequency domain loss value based on the frequency domain data corresponding to the real single-track audio data and the frequency domain data corresponding to the predicted single-track audio data; determine the model parameters in the initial single-track audio extraction model based on the second time domain loss value and the second frequency domain loss value to obtain the single-track audio extraction model.

[0070] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor; the memory is used to store a computer program, and the computer program comprises program instructions; the processor is used to call the program instructions from the memory, so that the electronic device executes any one of the methods described in the first aspect above.

[0071] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes any one of the methods in the first aspect above.

[0072] The beneficial effects of each possible implementation method in the second to fourth aspects can be found in the corresponding description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments.

[0074] Figure 1 is a schematic diagram of an audio separation system provided in an embodiment of the present application;

[0075] Figure 2 1 is a schematic diagram of a processing flow of audio separation provided in an embodiment of the present application;

[0076] Figure 3 This is a flowchart of an audio separation method provided in an embodiment of the present application;

[0077] Figure 4This is a schematic diagram of a process for obtaining single-track audio data based on a multi-track audio separation model provided by an embodiment of the present application;

[0078] Figure 5 This is a schematic diagram of the training process of a multi-track audio separation model provided in an embodiment of the present application;

[0079] Figure 6 This is a structural diagram of an audio separation device provided in an embodiment of the present application;

[0080] Figure 7 This is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0082] In the specification, claims, and drawings of this application, the terms "first," "second," and the like are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements, but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.

[0083] The present invention provides an audio separation method that can be applied to Figure 1 The audio separation system shown in FIG. Figure 1 As shown, the audio separation system includes at least one client, such as client 10, and at least one server, such as server 11. The client 10 and the server 11 can establish a communication connection through a network, which can be a wired network or a wireless network.

[0084] For the server 11 in the audio separation system, the server 11 can store a multi-track audio separation model, a fusion model, and multiple single-track audio extraction models. After the server 11 obtains the multi-track audio data, the server 11 can perform multi-track separation on the multi-track audio data according to the multi-track audio separation model to obtain a first separation result including multiple single-track audio data; at the same time, the server 11 can perform single-track extraction on the multi-track audio data according to the multiple single-track audio extraction models to obtain a second separation result including multiple single-track audio data; after the server 11 obtains the first separation result and the second separation result, the server 11 can use the fusion model to fuse the first separation result and the second separation result to obtain the target separation result.

[0085] For example, Figure 2 This is a schematic diagram of a processing flow of audio separation provided by an embodiment of the present application. Figure 2 As shown, the server 11 can first input the multi-track audio data into the single-track audio extraction model 1 to the single-track audio extraction model N and the multi-track audio separation model for processing. The multi-track audio separation model can output a first separation result after processing, and the N single-track audio extraction models can output a second separation result after processing. Next, the server 11 inputs the first and second separation results into the fusion model for processing. The fusion model outputs N single-track audio data, which are the target separation results.

[0086] Furthermore, the server 11 can send the target separation result to the client 10, and the client 10 can perform secondary creation such as mixing and tuning according to the obtained target separation result and play it. Alternatively, the server 11 first performs secondary creation such as mixing and tuning according to the obtained target separation result, and stores the audio processing result of the secondary creation in the server 11; when the client 10 needs to request the audio processing result of the secondary creation, the server 11 can send the audio processing result of the secondary creation to the client 10, so that the client 10 can play the processed audio based on the audio processing result of the secondary creation.

[0087] Optionally, when the server 11 processes the first separation result and the second separation result using the fusion model, the server 11 may use the fusion weight to multiply the first separation result and the second separation result and then add them together.

[0088] Optionally, for the first separation result and the second separation result, the server 11 may adopt different fusion weights according to different target requirements to achieve different audio separation effects.

[0089] Optionally, for the client 10 in the audio separation system, a multi-track audio separation model, a fusion model, and multiple single-track audio extraction models may also be stored in the client 10. After the client 10 obtains the multi-track audio data, it can process the multi-track audio data according to the stored multi-track audio separation model, fusion model, and multiple single-track audio extraction models to obtain the target separation result, and perform secondary creation such as mixing and audio correction based on the target separation result. Alternatively, the client 10 can obtain the multi-track audio separation model, fusion model, and multiple single-track audio extraction models, etc. from the server 11 in real time.

[0090] It should be noted that the client 10 may be a terminal device, which may be a smart phone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device, smart home appliance, vehicle-mounted terminal, etc., but is not limited thereto. The server 11 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0091] The above is a brief introduction to the audio separation system provided in the embodiment of the present application. Figures 3 to 7 The audio separation method, audio separation device, electronic device, and computer-readable storage medium provided in the embodiments of the present application are described in detail.

[0092] See Figure 3 , Figure 3 This is a flow chart of an audio separation method provided by an embodiment of the present application. The method includes steps S301 to S304, and its execution subject can be a server or a chip in the server. The following is an example of the execution subject of the method, which can be the above-mentioned server. Figure 1 Server 11 described in . Among them:

[0093] S301. The server obtains multi-track audio data, where the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1.

[0094] In an embodiment of the present application, the multi-track audio data is data obtained by mixing N single-track audio data. The multi-track audio data can be data in a song file, a recording file, a radio audio file, etc.

[0095] Single-track audio data can be broadly categorized into vocals and accompaniment. For vocals, when multi-track audio data includes audio data for multiple different vocals, each vocal data item can be considered a single-track audio data item. For accompaniment, when multi-track audio data includes audio data for multiple instrumental accompaniments, each instrumental accompaniment data item can be considered a single-track audio data item. For example, if multi-track audio data includes singer A's vocals, piano, and drums, the multi-track audio data can be considered a mixture of three single-track audio data items.

[0096] Optionally, the server can obtain multi-track audio data from other devices or local devices. For example, the server can obtain multi-track audio data from the client, and the multi-track audio data can be pre-stored in the client or collected in real time by the client's audio collection device; in another example, the server can obtain multi-track audio data from other servers, and the other servers can be servers dedicated to storing multi-track audio data, such as the background servers of music software; in another example, the server locally stores multi-track audio data, and the server can obtain multi-track audio data from the local storage, etc. The embodiments of the present application do not limit the source of the multi-track audio data.

[0097] S302: The server performs multi-track separation on the multi-track audio data based on the multi-track audio separation model to obtain a first separation result, where the first separation result includes N single-track audio data.

[0098] The multi-track audio separation model is a model that can simultaneously output multiple single-track audio data, and the multi-track audio separation model can be a model based on time domain and frequency domain processing. Specifically, when the server performs multi-track separation on multi-track audio data based on the multi-track audio separation model, the server can use the multi-track audio separation model to extract the time domain features and frequency domain features of the multi-track audio data, and output multiple single-track audio data based on the determined time domain features and frequency domain features.

[0099] In an embodiment of the present application, the multi-track audio data is time domain data. In order for the multi-track audio separation model to extract the time domain features and frequency domain features of the multi-track audio data, the server can first perform time-frequency conversion on the multi-track audio data to obtain the frequency domain data corresponding to the multi-track audio data; then, the server inputs the multi-track audio data and the frequency domain data corresponding to the multi-track audio data into the multi-track audio separation model for processing. Moreover, when the multi-track audio separation model processes according to the determined time domain features and frequency domain features, it can refer to the relationship between different single-track audio data in the multi-track audio data, so that each single-track audio data output by the multi-track audio separation model has a certain correlation. This correlation can make each single-track audio data have a better mixing effect in the mixing processing of secondary creation. This model processing method that integrates time domain features and frequency domain features can improve the separation effect between the output multiple single-track audio data while ensuring that the multiple single-track audio data output by the multi-track audio separation model have correlation.

[0100] Optionally, the multi-track audio separation model can be a neural network model based on a deep learning algorithm, for example, the multi-track audio separation model can be a hybrid multiplexer hybrid-demucs. This application does not limit the specific structure of the multi-track audio separation model.

[0101] For example, Figure 4 This is a flow chart of obtaining single-track audio data based on a multi-track audio separation model provided by an embodiment of the present application. Figure 4As shown, a short-time Fourier transform (STFT) is first performed on the multi-track audio data to obtain the frequency domain data corresponding to the multi-track audio data. Then, the multi-track audio data and the frequency domain data corresponding to the multi-track audio data are input into the multi-track audio separation model for processing. The multi-track audio data is sequentially processed through time domain encoder 1, time domain encoder 2, ..., and time domain encoder 5 for feature extraction to obtain the time domain features of the multi-track audio data, with the dimension of the time domain features extracted by each time domain encoder gradually increasing. Simultaneously, the frequency domain data corresponding to the multi-track audio data is sequentially processed through frequency domain encoder 1, frequency domain encoder 2, ..., and frequency domain encoder 5 for feature extraction to obtain the frequency domain features of the multi-track audio data, with the dimension of the frequency domain features extracted by each frequency domain encoder gradually increasing. Furthermore, the time domain features output by the time domain encoder 5 and the frequency domain features output by the frequency domain encoder 5 are fused via a time-frequency domain encoder to obtain time-frequency domain features, which are then feature mapped via a time-frequency domain decoder to obtain mapped features. The mapping features and frequency domain features then undergo feature mapping in sequence through frequency domain decoder 1, frequency domain decoder 2, ..., and frequency domain decoder 5, yielding frequency domain data corresponding to the three single-track audio data. Since the frequency domain data corresponding to the three single-track audio data is obtained through feature mapping based on the mapping features and frequency domain features, and since the mapping features are derived from the time-frequency domain features and contain partial information about the time domain features, the resulting frequency domain data corresponding to the three single-track audio data simultaneously incorporates both the frequency domain information and partial time domain information of the multi-track audio data. Next, the frequency domain data corresponding to the three single-track audio data undergo an inverse short-time Fourier transform (ISTFT) and are then superimposed with the mapping features and time domain features output by time domain decoder 1, time domain decoder 2, ..., and time domain decoder 5, yielding the single-track audio data. Since the data output by time domain decoder 5 is obtained through feature mapping based on the mapping features and frequency domain features and since the mapping features contain partial information about the frequency domain features, the data output by time domain decoder 5 simultaneously incorporates both the time domain information and partial frequency domain information of the multi-track audio data. For example, the single-track audio data may be Figure 4 Single-track audio data 1, single-track audio data 2, and single-track audio data 3.

[0102] S303: The server performs single-track extraction on the multi-track audio data based on the N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data.

[0103] In the embodiments of the present application, a single-track audio extraction model is a model used to extract a single track of audio data for a particular audio track. For example, a single-track audio extraction model used to extract piano sounds can extract the corresponding single-track audio data from multi-track audio data. Similarly, a single-track audio extraction model used to extract vocals can extract the corresponding single-track audio data from multi-track audio data.

[0104] In the present application, N is an integer greater than 1, and N single-track audio extraction models can perform single-track extraction on multi-track audio data respectively, and output a total of N single-track audio data. These N single-track audio data are also the second separation results. Among them, one single-track audio data among the N single-track audio data is the output of one single-track audio extraction model among the N single-track audio extraction models. Due to the characteristics that a single-track extraction model is only used to extract single-track audio data of one audio track, a single-track extraction model will not refer to the relationship between different single-track audio data in the multi-track audio data during extraction, so that the extracted single-track audio data is relatively accurate and the separation effect is good. Furthermore, the separation effect of the N single-track audio data output by the N single-track audio extraction models is good and has no correlation.

[0105] Optionally, the single-track audio extraction model can be a model based on time domain and frequency domain processing. When the server performs single-track extraction on multi-track audio data based on a single-track audio extraction model, the server can use the single-track audio extraction model to extract the time domain features and frequency domain features of the multi-track audio data respectively, and output a single-track audio data based on the determined time domain features and frequency domain features. This model processing method that integrates time domain features and frequency domain features can improve the separation effect of the single-track audio data output by the single-track audio extraction model.

[0106] Optionally, the single-track audio extraction model can be a neural network model based on a deep learning algorithm. For example, the single-track audio extraction model can use the same model structure as the multi-track audio separation model, or a different model structure. Figure 4 The model structure of the multi-track audio separation model has the same functions as the time domain encoder, frequency domain encoder, time domain decoder, and frequency domain decoder. The difference is that the decoder maps the features into a single track of audio data instead of mapping them into multiple single tracks of audio data during feature mapping. This application does not limit the specific structure of the single-track audio extraction model.

[0107] Optionally, N single-track audio extraction models can be determined based on the first separation result and matched with the type of single-track audio data contained in the first separation result. For example, five single-track audio extraction models are stored in the server, which are used to extract audio data of human voice, piano sound, drum sound, guitar sound, and electronic keyboard sound, respectively. When the number of single-track audio data included in the first separation result is three, and the categories to which the three single-track audio data belong are human voice, piano sound, and drum sound, the server can determine the three single-track audio extraction models corresponding to human voice, piano sound, and drum sound from the five stored single-track audio extraction models, and obtain the second separation result based on these three single-track audio extraction models.

[0108] Optionally, the N single-track audio extraction models may also be determined based on association information of the multi-track audio data, where the association information is used to indicate the type of audio track corresponding to the multi-track audio data. For example, when the server obtains the multi-track audio data, it also obtains the association information of the multi-track audio data. The server may then determine, based on the association information of the multi-track audio data, a multi-track audio separation model and N single-track audio separation models that match the type of audio track corresponding to the multi-track audio data.

[0109] It should be noted that the N single-track audio extraction models can also be determined according to other methods, or all single-track audio extraction models stored in the server, etc. This application does not limit this.

[0110] S304: The server performs fusion processing on the single-track audio data of the same type in the first separation result and the second separation result based on the fusion model to obtain a target separation result.

[0111] In the embodiment of the present application, the fusion model is used to fuse single-track audio data of the same type. The same type refers to the single-track audio data belonging to the same track type. For example, if the first separation result includes single-track audio data of human voice, and the second separation result also includes single-track audio data of human voice, then these two single-track audio data of human voice are single-track audio data of the same type.

[0112] In one possible implementation, the server may input single-track audio data of the same type in the first separation result and the second separation result into a fusion model; then the server performs weight operations and addition operations on the single-track audio data of the same type in the first separation result and the second separation result based on the first weight and the second weight in the fusion model to obtain the target separation result, where the first weight is the fusion weight corresponding to the first separation result, and the second weight is the fusion weight corresponding to the second separation result.

[0113] For example, the first separation result is {x1,x2,...,x N}, the second separation result is {y1,y2,...,y N}, the target separation result is {z1,z2,...,z N}, x N 、y N 、z N The Nth single-track audio data in the first separation result, the second separation result, and the target separation result, respectively, x N with y N With z N The same type as z N It can be obtained by the following formula 1:

[0114] z N =W1*x N +W2*y N (Formula 1)

[0115] Where the first weight is W1 and the second weight is W2. This formula means that the Nth single-track audio data in the target separation result is obtained by multiplying the weights of the Nth single-track audio data in the first and second separation results and adding them together. Based on this formula, the good correlation of the first separation result and the good separation effect of the second separation result can be combined to achieve a good separation effect and correlation of the single-track audio data in the target separation result, thereby improving the overall performance of audio separation.

[0116] In one possible implementation, the server may also first determine the target requirement for audio separation of multi-track audio data; if the target requirement is used to request that the correlation between the single-track audio data obtained by separation is stronger, then a fusion model is obtained, and the first weight of the fusion model is greater than the second weight.

[0117] In this possible implementation, the target requirement is to request a stronger correlation between the individual single-track audio data obtained through separation. Therefore, when the fusion model fuses the first separation result with the second separation result, it can assign a higher weight to the first separation result. For example, when using the single-track audio data corresponding to each audio track to mix audio data for panoramic sound effects, it is necessary for the single-track audio data corresponding to each audio track to have a strong correlation. In this case, a fusion model with a first weight greater than the second weight can be used to obtain the target separation result.

[0118] In another possible implementation, the server may also determine the target requirement for audio separation of multi-track audio data; if the target requirement is used to request weak correlation between the individual single-track audio data obtained by separation, a fusion model is obtained, and the first weight of the fusion model is less than the second weight.

[0119] In this possible implementation, the target demand is used to request that the correlation between the individual single-track audio data obtained by separation is weak, so when the fusion model fuses the first separation result and the second separation result, it can give a higher weight to the second separation result. Of course, this method can also be applied to situations where the target demand is used to request that the separation effect of the individual single-track audio data obtained by separation is strong. For example, if the audio data corresponding to the vocal track in a song needs to be re-created to satisfy a better listening experience, such as amplifying the vocals in the song, then the audio data corresponding to the vocal track needs to have a stronger separation effect or a weaker correlation with the remaining single-track audio data. In this case, a fusion model in which the second weight is greater than the first weight can be used.

[0120] Based on the above method, the server can perform audio separation on multi-track audio data according to the target requirements, thereby obtaining the target separation result. The correlation and separation effect of the target separation result obtained by this method can match the target requirements, thereby improving the overall performance of audio separation.

[0121] With respect to the above possible implementation methods, the target requirements can be determined in a variety of ways. The following describes the methods for determining the target requirements.

[0122] Method 1: The server may read preset demand information; and then the server determines a target demand for audio separation for multi-track audio data based on the read preset demand information.

[0123] The preset demand information may be demand information input by a developer into a server and stored by the server. When the server obtains the multi-track audio data, the multi-track audio data may be processed according to the preset demand information.

[0124] For example, the preset demand information can be the type of mixed sound effects supported by the music platform, which includes but is not limited to panoramic sound effects, 3D vocals, wide-angle surround, etc., and for different types of mixed sound effects, the requirements for the relevance or separation effect of each single-track audio data in the target separation result are different. The server can trigger the execution of the audio separation method proposed in this application when the music is online, that is, for each online music, a matching fusion model is determined according to the target demand corresponding to the preset demand information, and the audio of the online music is separated according to each model to obtain the target separation result corresponding to the online music. For example, when song A is online, the target demand is determined according to the stored "panoramic sound effect", and the target separation result corresponding to the "panoramic sound effect" is obtained according to the target demand. The target separation result determined by this process can be stored by the server. Furthermore, the server can obtain mixed audio data based on the stored target separation result and the corresponding mixing method, and send the mixed audio data to the client for playback. Alternatively, the server sends the target separation result to the client, and the client performs the mixing operation to obtain the mixed audio data and plays it. This application does not limit the further application of the target separation results, and the above content is only an example.

[0125] Method 2: The server may receive user input requirement information sent by the client; and then the server may determine a target requirement for audio separation for multi-track audio data based on the requirement information.

[0126] Among them, the demand information input by the client can be information input by the user in the client for multi-track audio data, which can be setting information for the mixing sound effect type of multi-track audio data, or setting information for the audio repair type of multi-track audio data, etc.

[0127] For example, a user can set the type of mixed sound effects for a song when listening to it, and the server can obtain the type of mixed sound effects set by the user by detecting the user's setting operation. Furthermore, the server can determine the target demand based on the type of mixed sound effects set by the user, determine the fusion model based on the target demand, and perform audio separation on the multi-track audio data according to each model to obtain the target separation result. For another example, a user can use the client's recording device to sing karaoke and obtain multi-track audio data, and then the user can perform audio adjustment operations on the multi-track audio data; when the server detects the user's audio adjustment operation, it can determine the target demand based on the audio adjustment operation, determine the fusion model based on the target demand, and perform audio separation on the multi-track audio data according to each model to obtain the target separation result.

[0128] based on Figure 3In the embodiment described, the server can use a fusion model to fuse the first separation result of the multi-track audio separation model and the second separation results of N single-track audio extraction models to obtain a target separation result. Since the separation effect of each single-track audio data in the first separation result is average but the correlation is good, and the separation effect of each single-track audio data in the second separation result is good but the correlation is poor, the separation effect and correlation of each single-track audio data in the target separation result obtained by fusion can both achieve a good effect, thereby improving the overall performance of audio separation.

[0129] against Figure 3 In the embodiment shown, in a specific implementation, the client can also store a multi-track audio separation model, N single-track audio extraction models, and a fusion model, or the client can obtain these models from the server upon detecting input demand information. The client can then process the multi-track audio data according to these models to obtain the target separation result.

[0130] The above introduces the audio separation method in this application. The following is a supplementary introduction to the training methods of the multi-track audio separation model, single-track audio extraction model, and fusion model in the audio separation method.

[0131] See Figure 5 , Figure 5 This is a schematic diagram of the training process of a multi-track audio separation model provided by an embodiment of the present application. The method includes steps S501 to S505, and its execution subject can be a server or a chip in the server. The following is an example of the execution subject of the method, which can be the above-mentioned server. Figure 1 Server 11 described in . Among them:

[0132] S501: The server obtains multi-track audio training data, multiple real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to each real single-track audio data.

[0133] Among them, the multi-track audio training data is obtained by mixing multiple real single-track audio data, and the multi-track audio training data and multiple real single-track audio data are both time domain data.

[0134] S502: The server inputs the multi-track audio training data into the initial multi-track audio separation model to obtain a plurality of predicted single-track audio data and frequency domain data corresponding to each predicted single-track audio data.

[0135] In an embodiment of the present application, the initial multi-track audio separation model can perform audio separation on the multi-track audio training data to obtain multiple predicted single-track audio data and frequency domain data corresponding to each predicted single-track audio data. The multiple predicted single-track audio data can be the data output by the initial multi-track audio separation model, and the frequency domain data corresponding to each predicted single-track audio data can be the intermediate processing data of the initial multi-track audio separation model, for example, Figure 4 The data before ISTFT was performed.

[0136] S503: The server determines a first time-domain loss value based on the multiple real single-track audio data and the multiple predicted single-track audio data.

[0137] For example, if the real single-track audio data and the predicted single-track audio data are J, J is an integer greater than 1, and the J real single-track audio data are {m1(t), m2(t), ..., m J (t)}, the J predicted single-track audio data are {m′1(),m′2(t),...,m′ J (t)}, the first time domain loss value can be obtained by the following formula 2:

[0138]

[0139] in, is the first time domain loss value, ||m j (t)-m′ j (t)||1 represents m j (t)-m′ j The first-order norm of (t), ∑ t ||m j (t)-m′ j (t)||1 is used to represent m′ J (t) and m J The time domain error of (t), the meaning expressed by Formula 2 is: the first time domain loss value is equal to the sum of the errors between J real single-track audio data and the corresponding predicted single-track audio data.

[0140] S504: The server determines a first frequency domain loss value based on the frequency domain data corresponding to the multiple real single-track audio data and the frequency domain data corresponding to the multiple predicted single-track audio data.

[0141] For example, the first frequency domain loss value can be obtained by the following formula 3:

[0142]

[0143] in, is the first frequency domain loss value, and S is the number of STFTs with different resolutions. A resolution is a set of window lengths and frame shifts. For example, when STFT is framed, the frame shift can be 1024, 512, or 256, and when windowing, the window length can be 4096, 2048, or 1024. A frame shift of 1024 and a tail window length of 4096 can be a set of resolutions. Indicates: First, m′ is calculated according to different groups of resolutions J (t) and m J (t) First perform inverse short-time Fourier transform to obtain m′ J (t) and m J (t) The corresponding frequency domain data, and then calculate m' J (t) and m J (t) The error of the corresponding frequency domain data. Therefore, Formula 3 means that the first frequency domain loss value is equal to the sum of the errors between the frequency domain data of the J real single-track audio data and the frequency domain data of the corresponding predicted single-track audio data.

[0144] Specifically, It can be obtained according to the following formulas 4 to 6:

[0145]

[0146]

[0147]

[0148] Among them, |||STFTm j (t)-|STFTm′ j (t)||| F |STFTm j (t)|-|STFTm′ j The second-order norm of (t), |||STFTm j (t)||| F |STFTm j The second-order norm of (t)|, Represents m′ J (t) and m J (t) The relative error of the corresponding frequency domain data, Represent m′ respectively J (t) and m J (t) The log error of the corresponding frequency domain data, is the sum of these two errors.

[0149] S505: The server determines model parameters in the initial multi-track audio separation model based on the first time domain loss value and the first frequency domain loss value to obtain the multi-track audio separation model.

[0150] In an embodiment of the present application, the server may first process the first time domain loss value and the first frequency domain loss value to obtain a training loss value, and then determine the model parameters in the initial multi-track audio separation model based on the training loss value to obtain the multi-track audio separation model. For example, the sum of the first time domain loss value and the first frequency domain loss value may be used as the training loss value, or the weighted sum of the first time domain loss value and the first frequency domain loss value may be used as the training loss value, etc.

[0151] For example, if α is a weight value, the server can obtain the training loss value according to the following formula 7:

[0152]

[0153] Among them, L loss is a training loss value, which may be a weighted average loss value of the first time-domain loss value and the first frequency-domain loss value.

[0154] Compared with the method of training the model only for time domain loss or frequency domain loss, this time-frequency domain mixing method can enable the trained multi-track audio separation model to fully utilize the time domain and frequency domain processing methods, and improve the separation effect between multiple single-track audio data output by the multi-track audio separation model.

[0155] Similarly, the single-track audio extraction model in this application can also be trained according to the above principles. The method of the single-track audio extraction model is summarized below. The training can be performed by a server or other device. Specifically, the server obtains multi-track audio training data, a real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to the real single-track audio data; then, the server inputs the multi-track audio training data into the initial single-track audio extraction model to obtain a predicted single-track audio data, and frequency domain data corresponding to the predicted single-track audio data; the server determines the second time domain loss value based on the real single-track audio data and the predicted single-track audio data, and determines the second frequency domain loss value based on the frequency domain data corresponding to the real single-track audio data and the frequency domain data corresponding to the predicted single-track audio data; finally, the server determines the model parameters in the initial single-track audio extraction model based on the second time domain loss value and the second frequency domain loss value to obtain the single-track audio extraction model.

[0156] Among them, the multi-track audio training data is obtained by mixing multiple real single-track audio data, but because the single-track audio extraction model is used to extract one of the multiple real single-track audio data, this application needs to obtain a real single-track audio data corresponding to the single-track audio extraction model and the frequency domain data corresponding to the real single-track audio data. The multi-track audio training data and this real single-track audio data are both time domain data. At the same time, the server can refer to the above formulas 2 to 7 when determining the second time domain loss value, the second frequency domain loss value, and the training loss value of the single-track audio extraction model, and the value of J in formula 2 and formula 7 is 1.

[0157] It should be noted that this application does not elaborate on the specific training process of the single-track audio extraction model, and the specific training process of the multi-track audio separation model mentioned above can be referred to. By training the initial single-track audio extraction model in this time-frequency domain mixing method, the trained single-track audio extraction model can give full play to the processing methods of the time domain and frequency domain, and improve the extraction effect of the single-track audio extraction model on the single-track audio data in the multi-track audio data. In other words, the single-track audio data output by the single-track audio extraction model can be made more accurate and more similar to the real single-track audio data.

[0158] Furthermore, the server can also train the fusion model using the multi-track audio separation model and the single-track audio extraction model obtained by the above training. For example, the input of the initial fusion model is the output of the above multi-track audio separation model and the output of the above multiple single-track audio extraction models, and the output of the initial fusion model is multiple predicted single-track audio data. Since the fusion model performs weighted addition processing on the output of the multi-track audio separation model and the output of the multiple single-track audio extraction models, this addition is only for time domain data. Therefore, multiple real single-track audio data and multiple predicted single-track audio data output by the initial fusion model can be brought into the above formula 2 to determine the training loss value of the initial fusion model, and the parameters in the initial fusion model can be determined according to the training loss value of the initial fusion model to obtain the fusion model.

[0159] It should be noted that when training the fusion model, the parameters in the fusion model can also be adjusted according to different preset requirements. The preset requirement can be the above-mentioned target requirement, and the parameter can be the fusion weight, so that the trained fusion model can be applicable to the different determined target requirements.

[0160] See Figure 6 , Figure 6 6 is a schematic diagram of the structure of an audio separation device provided in an embodiment of the present application. The audio separation device 600 includes an acquisition module 601 and a processing module 602.

[0161] The acquisition module 601 is used to acquire multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1;

[0162] A processing module 602 is configured to perform multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, where the first separation result includes N single-track audio data;

[0163] The processing module 602 is further configured to perform single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data;

[0164] The processing module 602 is further configured to perform fusion processing on the same type of single-track audio data in the first separation result and the second separation result based on a fusion model to obtain a target separation result.

[0165] In one possible implementation, when the processing module 602 performs fusion processing on the single-track audio data of the same type in the first separation result and the second separation result based on the fusion model to obtain the target separation result, it is specifically used to: input the single-track audio data of the same type in the first separation result and the second separation result into the fusion model; based on the first weight and the second weight in the fusion model, perform weight operation and addition operation on the single-track audio data of the same type in the first separation result and the second separation result to obtain the target separation result; the first weight is the fusion weight corresponding to the first separation result, and the second weight is the fusion weight corresponding to the second separation result.

[0166] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated; the processing module 602 is further used to determine the target requirement for audio separation of the multi-track audio data; if the target requirement is used to request that the correlation between the single-track audio data obtained by separation is stronger, the acquisition module 601 is further used to obtain the fusion model, and the first weight of the fusion model is greater than the second weight.

[0167] In one possible implementation, the single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated; the processing module 602 is further used to determine the target requirement for audio separation of the multi-track audio data; if the target requirement is used to request that the correlation between the single-track audio data obtained by separation is weak, the acquisition module 601 is further used to obtain the fusion model, and the first weight of the fusion model is less than the second weight.

[0168] In one possible implementation, the processing module 602 is also used to read preset demand information; when determining the target requirement for audio separation for the multi-track audio data, the processing module 602 is specifically used to: determine the target requirement for audio separation for the multi-track audio data based on the read preset demand information.

[0169] In one possible implementation, the acquisition module 601 is also used to receive user input demand information sent by the client; when determining the target demand for audio separation for the multi-track audio data, the processing module 602 is specifically used to: determine the target demand for audio separation for the multi-track audio data based on the demand information.

[0170] In one possible implementation, the acquisition module 601 is also used to acquire multi-track audio training data, multiple real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to each real single-track audio data; the processing module 602 is also used to: input the multi-track audio training data into the initial multi-track audio separation model to obtain multiple predicted single-track audio data, and frequency domain data corresponding to each predicted single-track audio data; determine a first time domain loss value based on the multiple real single-track audio data and the multiple predicted single-track audio data; determine a first frequency domain loss value based on the frequency domain data corresponding to the multiple real single-track audio data and the frequency domain data corresponding to the multiple predicted single-track audio data; determine the model parameters in the initial multi-track audio separation model based on the first time domain loss value and the first frequency domain loss value to obtain the multi-track audio separation model.

[0171] In one possible implementation, the acquisition module 601 is also used to acquire multi-track audio training data, a real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to the real single-track audio data; the processing module 602 is also used to input the multi-track audio training data into the initial single-track audio extraction model to obtain a predicted single-track audio data, and frequency domain data corresponding to the predicted single-track audio data; based on the real single-track audio data and the predicted single-track audio data, a second time domain loss value is determined; based on the frequency domain data corresponding to the real single-track audio data and the frequency domain data corresponding to the predicted single-track audio data, a second frequency domain loss value is determined; based on the second time domain loss value and the second frequency domain loss value, the model parameters in the initial single-track audio extraction model are determined to obtain the single-track audio extraction model.

[0172] It should be noted that the functions of each module of the audio separation device in the embodiment of the present application can be specifically implemented according to the method in the above method embodiment. Its specific implementation process and beneficial effects can refer to the relevant description of the above method embodiment, which will not be repeated here.

[0173] See Figure 7 , Figure 7 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 700 may include: one or more processors 701 and a memory 702. Optionally, the electronic device 700 may further include a transceiver 703. The processor 701, memory 702, and transceiver 703 may be connected via a bus 704. The memory 702 is used to store a computer program, which includes program instructions. The processor 701 and transceiver 703 are used to execute the program instructions stored in the memory 502 to perform the following operations:

[0174] Acquire multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1;

[0175] Performing multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, where the first separation result includes N single-track audio data;

[0176] Performing single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data;

[0177] Based on the fusion model, single-track audio data of the same type in the first separation result and the second separation result are fused to obtain a target separation result.

[0178] It should be understood that in some feasible implementations, the processor 701 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The memory 702 may include a read-only memory and a random access memory, and provide instructions and data to the processor 701. A portion of the memory 702 may also include a non-volatile random access memory. For example, the memory 702 may also store device type information.

[0179] In a specific implementation, the electronic device can execute the above-mentioned functions through its built-in functional modules. Figure 3-Figure 5 The provided implementation method, specific implementation process and beneficial effects can be found in the specific content of the above method embodiment, which will not be repeated here.

[0180] The embodiment of the present application further provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the aforementioned audio separation device, and the computer program includes program instructions. When the processor executes the above program instructions, it can execute the above Figure 3-Figure 5 Therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, the program instructions can be deployed on an electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple locations and interconnected by a communication network. Multiple electronic devices distributed in multiple locations and interconnected by a communication network can constitute a blockchain system.

[0181] According to one aspect of the present application, a computer program product is also provided, the computer program product including a computer program, the computer program stored in a computer-readable storage medium, including program instructions. A processor of an electronic device reads the program instructions from the computer-readable storage medium, and the processor executes the program instructions, so that the electronic device can perform the above-mentioned Figure 3-Figure 5 Therefore, I will not elaborate on the content here.

[0182] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0183] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. An audio separation method, characterized in that: The method comprises: Acquire multi-track audio data; the multi-track audio data is obtained by mixing N single-track audio data, where N is an integer greater than 1; Performing multi-track separation on the multi-track audio data based on a multi-track audio separation model to obtain a first separation result, where the first separation result includes N single-track audio data; Performing single-track extraction on the multi-track audio data based on N single-track audio extraction models to obtain a second separation result, where the second separation result includes N single-track audio data; Based on the fusion model, single-track audio data of the same type in the first separation result and the second separation result are fused to obtain a target separation result.

2. The method according to claim 1, characterized in that The fusing the single-track audio data of the same type in the first separation result and the second separation result based on the fusion model to obtain a target separation result includes: Inputting the single-track audio data of the same type in the first separation result and the second separation result into the fusion model; Based on the first weight and the second weight in the fusion model, weight operations and addition operations are performed on the single-track audio data of the same type in the first separation result and the second separation result to obtain the target separation result; the first weight is the fusion weight corresponding to the first separation result, and the second weight is the fusion weight corresponding to the second separation result.

3. The method according to claim 2, characterized in that The single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated; The method further comprises: Determining target requirements for audio separation of the multi-track audio data; If the target requirement is to request that the correlation between the separated single-track audio data is stronger, then the fusion model is obtained, and the first weight of the fusion model is greater than the second weight.

4. The method according to claim 2, characterized in that The single-track audio data included in the first separation result are associated, and the single-track audio data included in the second separation result are not associated; The method further comprises: Determining target requirements for audio separation of the multi-track audio data; If the target requirement is to request that the separation of the individual single-track audio data is stronger or less correlated, then the fusion model is obtained, and the first weight of the fusion model is less than the second weight.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Read the preset demand information; Determining a target requirement for performing audio separation on the multi-track audio data includes: The target requirement for performing audio separation on the multi-track audio data is determined according to the preset requirement information read.

6. The method according to any one of claims 3 or 4, characterized in that The method further comprises: Receive user input requirement information sent by the client; Determining a target requirement for performing audio separation on the multi-track audio data includes: The target requirement for performing audio separation on the multi-track audio data is determined according to the requirement information.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Acquire multi-track audio training data, multiple real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to each real single-track audio data; Inputting the multi-track audio training data into an initial multi-track audio separation model to obtain a plurality of predicted single-track audio data and frequency domain data corresponding to each predicted single-track audio data; Determining a first time-domain loss value based on the plurality of real single-track audio data and the plurality of predicted single-track audio data; Determining a first frequency domain loss value based on the frequency domain data corresponding to the multiple real single-track audio data and the frequency domain data corresponding to the multiple predicted single-track audio data; Model parameters in the initial multi-track audio separation model are determined based on the first time domain loss value and the first frequency domain loss value to obtain the multi-track audio separation model.

8. The method according to any one of claims 1 to 4, characterized in that The method further comprises: Acquire multi-track audio training data, a real single-track audio data corresponding to the multi-track audio training data, and frequency domain data corresponding to the real single-track audio data; Inputting the multi-track audio training data into an initial single-track audio extraction model to obtain a predicted single-track audio data and frequency domain data corresponding to the predicted single-track audio data; Determine a second time domain loss value based on the one real single-track audio data and the one predicted single-track audio data; Determining a second frequency domain loss value based on the frequency domain data corresponding to the one real single-track audio data and the frequency domain data corresponding to the one predicted single-track audio data; Model parameters in the initial single-track audio extraction model are determined based on the second time-domain loss value and the second frequency-domain loss value to obtain the single-track audio extraction model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor; The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is configured to call the program instructions from the memory, so that the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 8.

Images