Music separation method, music separation device, electronic equipment and storage medium

Abstract

The present application discloses a music separation method, a music separation device, an electronic device and a storage medium. The method includes obtaining a first audio feature of the current audio; processing the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio, wherein the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio, wherein the multiple second audio features at least include the first audio feature. The method realizes separation of the current audio from the current mixed audio based on the first audio based on the target diffusion model. Since the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, the target diffusion model can perform music separation on different mixed audios according to the audio features, thereby simplifying the operation process of music separation.

Description

Technical Field

[0001] The present application relates to the technical field of audio processing, and in particular to a music separation method, a music separation device, an electronic device and a storage medium. Background Art

[0002] Music is a complex audio signal composed of the sound of musical instruments and singing voices, which includes the accompaniment of various instruments and the singing voices of different people. With the continuous development of computer signal processing technology and Internet technology, the separation of human voice and music and the separation of various instruments have gradually received more and more attention. It can be widely used in music mixing, music information retrieval, music education and other fields.

[0003] At present, in the process of music separation, a neural network (NN) is trained in advance according to the historical mixed audio to obtain a trained NN, and the trained NN is used to perform music separation on the current mixed audio. Among them, the audio source type of the current mixed audio is the same as the audio source type of the historical mixed audio.

[0004] However, during the music separation process, if the audio source type of the current mixed audio is different from the audio source type of the historical mixed audio, when using the trained NN to perform music separation on the current mixed audio, it is necessary to use the NN to pre-train mixed audio samples with the same audio source type as the current mixed audio, and the operation process is relatively cumbersome. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a music separation method, a music separation device, an electronic device and a storage medium to overcome the above problems of the prior art.

[0006] In a first aspect, an embodiment of the present application provides a music separation method, the music separation method comprising:

[0007] Get the first audio feature of the current audio;

[0008] The first audio feature and the current mixed audio are processed by a target diffusion model to obtain the current audio, the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features at least include the first audio feature.

[0009] The solution provided by the present application realizes that in the process of music separation, the current audio is separated from the current mixed audio according to the first audio based on the target diffusion model. Since the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audios with the same audio source type as each mixed audio, which simplifies the operation process of music separation. In addition, the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, which ensures that the music separation process has a high separation quality.

[0010] Wherein, in some optional embodiments, obtaining the first audio feature of the current audio includes:

[0011] Get the current audio source type of the current audio;

[0012] According to the current audio source type, the first audio feature is acquired from an audio feature library.

[0013] The solution provided in this embodiment can ensure that the audio source required by the user can be subsequently separated from the current mixed audio according to the first audio feature, since the current audio source type is the audio source from which the user needs to separate the music, thereby improving the user's music separation experience.

[0014] In some optional embodiments, before acquiring the first audio feature from an audio feature library according to the current audio source type, the music separation method further includes:

[0015] Get multiple audio sources, each audio source corresponds to an audio source type;

[0016] Extract audio features from multiple audio sources using an audio feature extraction model to obtain multiple third audio features, each third audio feature corresponds to an audio source, and the multiple third audio features include the first audio feature;

[0017] An audio feature library is generated according to the plurality of third audio features.

[0018] The solution provided in this embodiment realizes the construction of an audio high-dimensional feature library based on multiple third audio features extracted by the audio feature extraction model, simplifies the extraction process of multiple third audio features, and thus simplifies the construction process of the audio feature library.

[0019] In some optional embodiments, the audio feature extraction model is any one of a convolutional neural network, a deep neural network or a long short-term memory network.

[0020] In some optional embodiments, the first audio feature, the second audio feature and the third audio feature are all high-dimensional audio features.

[0021] In some optional embodiments, the current audio includes a plurality of sub-audios, the first audio feature includes a plurality of sub-audio features, and the processing of the first audio feature and the current mixed audio by the target diffusion model to obtain the current audio includes:

[0022] The multiple sub-audio features and the current mixed audio are processed through a target diffusion model to obtain multiple sub-audios.

[0023] The solution provided in this embodiment realizes separation of multiple sub-audios from the current mixed audio based on the target diffusion model according to multiple sub-audio features, thereby improving the user's music separation experience.

[0024] In some optional embodiments, before the first audio feature and the current mixed audio are processed by the target diffusion model to obtain the current audio, the music separation method further includes:

[0025] Acquire a historical mixed audio and a plurality of second audio features of the historical mixed audio;

[0026] The initial diffusion model is trained according to the historical mixed audio and the plurality of second audio features to obtain a target diffusion model.

[0027] The solution provided in this embodiment realizes the training of the initial diffusion model based on the historical mixed audio and multiple second audio features to obtain the target diffusion model. The diffusion model itself has the ability to adapt to the generation of different mixed audios, ensuring that the target diffusion model has a strong generalized music separation ability.

[0028] In some optional embodiments, before the initial diffusion model is trained according to the historical mixed audio and the plurality of second audio features to obtain the target diffusion model, the music separation method further includes:

[0029] Get the noise audio;

[0030] The step of training the initial diffusion model according to the historical mixed audio and the plurality of second audio features to obtain a target diffusion model includes:

[0031] An initial diffusion model is trained according to the noise audio, the historical mixed audio, and a plurality of second audio features to obtain a target diffusion model.

[0032] The solution provided in this embodiment realizes the training of the initial diffusion model based on the noisy audio, the historical mixed audio and the multiple second audio features to obtain the target diffusion model, so that the target diffusion model further learns the ability to generate noise-free audio from the noisy audio, and further improves the generalized music separation ability of the target diffusion model.

[0033] In some optional embodiments, the target diffusion model is obtained based on any one of a Unet network, a noise conditional scoring network or a noise conditional scoring network++.

[0034] In some optional embodiments, the target diffusion model is formed based on any one of a denoised diffusion probability model algorithm, a denoised diffusion implicit model algorithm, a stochastic differential equation algorithm or a score-based generative model algorithm.

[0035] In a second aspect, an embodiment of the present application provides a music separation device, the music separation device comprising:

[0036] A feature acquisition module, used to acquire a first audio feature of the current audio;

[0037] A processing module is used to process the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio, wherein the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features at least include the first audio feature.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device executes the music separation method provided in the first aspect above.

[0039] In a fourth aspect, an embodiment of the present application provides a chip for use in an electronic device, the chip including a processor, the processor being used to read and execute a computer program stored in a memory, and when the computer program is executed by the processor, the electronic device can execute the music separation method provided in the first aspect above.

[0040] In some optional embodiments, the chip also includes a memory, and the memory is connected to the processor via a circuit or wire.

[0041] In some optional embodiments, the chip also includes a communication interface.

[0042] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored. The program code can be called by an electronic device to execute the music separation method provided in the first aspect above.

[0043] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the music separation method provided in the first aspect above.

[0044] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0046] Figure 1 A schematic diagram of the structure of a software system of an electronic device provided in an embodiment of the present application is shown.

[0047] Figure 2 A flow chart of a music separation method provided in an embodiment of the present application is shown.

[0048] Figure 3 A schematic diagram of an application scenario of the diffusion model in the music separation method provided in an embodiment of the present application is shown.

[0049] Figure 4 A schematic diagram of an application scenario in which the diffusion model in the music separation method provided in an embodiment of the present application processes audio samples is shown.

[0050] Figure 5 A schematic diagram of an application scenario of the current mixed audio in the music separation method provided in an embodiment of the present application is shown.

[0051] Figure 6 A schematic diagram of an application scenario of multiple current audios obtained by music separation of current mixed audio in the music separation method provided in an embodiment of the present application is shown.

[0052] Figure 7 A schematic diagram of an application scenario of forward diffusion processing and reverse sampling processing of a target diffusion model in the music separation method provided in an embodiment of the present application is shown.

[0053] Figure 8Another flow chart of the music separation method provided in an embodiment of the present application is shown.

[0054] Fig. 9 Another flow chart of the music separation method provided in the embodiment of the present application is shown.

[0055] Fig.10 A structural block diagram of a music separation device provided in an embodiment of the present application is shown.

[0056] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.

[0057] Fig.12 A functional block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0059] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0060] Music is a complex audio signal composed of the sound of musical instruments and singing voices, which includes the accompaniment of various instruments and the singing voices of different people. With the continuous development of computer signal processing technology and Internet technology, the separation of human voice and music and the separation of various instruments have gradually received more and more attention. It can be widely used in music mixing, music information retrieval, music education and other fields.

[0061] At present, in the process of music separation, a neural network (NN) is trained in advance according to the historical mixed audio to obtain a trained NN, and the trained NN is used to perform music separation on the current mixed audio. Among them, the audio source type of the current mixed audio is the same as the audio source type of the historical mixed audio.

[0062] However, in the process of music separation, if the audio source type of the current mixed audio is different from the audio source type of the historical mixed audio, when using the trained NN to perform music separation on the current mixed audio, it is necessary to use the NN to pre-train the mixed audio samples with the same audio source type as the current mixed audio, and the operation process is relatively cumbersome. In addition, the separated audio obtained by separating the historical mixed audio with the trained NN still contains other mixed audio, or / and, the separated audio has audio data missing, which reduces the separation quality of music separation.

[0063] In view of the above problems, in the music separation method, music separation device, electronic device and storage medium provided by the embodiment of the present application, the music separation method obtains the first audio feature of the current audio, and processes the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio, the audio source type of the current mixed audio includes the audio source type of the current audio, the target diffusion model is obtained by training the initial diffusion model based on the historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features include at least the first audio feature, so that in the music separation process, the current audio is separated from the current mixed audio according to the first audio based on the target diffusion model, because the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, so that the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audio with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation. And the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, which ensures that the music separation process has a high separation quality.

[0064] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0065] The music separation method provided in the embodiment of the present application can be applied to electronic devices. The electronic devices can include various terminal devices, which can also be called terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc.

[0066] The terminal device may be a mobile phone, a sweeping robot, a drone, a smart TV, a wearable device, a PDA, a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The type of terminal device is not limited here, and it can be set according to actual needs.

[0067] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.

[0068] See also Figure 1 , which shows a schematic diagram of the structure of the software system of an electronic device provided by an embodiment of the present application. The software system includes several layers, each layer has a clear role and division of labor, and the layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, namely, the application layer, the application framework layer, the system library, and the kernel layer from top to bottom.

[0069] The application layer may include a series of applications. For example, the application layer may include a camera application, a gallery application, a call application, a Wireless Local Area Networks (WLAN) application, a music separation application, a video application, a Media Provider application, a FUSE (Filesystem in Userspace) file system application, etc.

[0070] The media provider is used to create multimedia files in the FUSE file system, or access multimedia files in the FUSE file system. Each application in the application layer can create multimedia files in the FUSE file system through the media provider MediaProvider, or access multimedia files in the FUSE file system through the media provider MediaProvider.

[0071] The FUSE file system is used to store the multimedia files that the media provider creates. Of course, in other embodiments, the FUSE file system may also be used to store other data.

[0072] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a window manager, a content provider, a resource manager, a view system, a package manager service (PMS), and an activity manager service (AMS).

[0073] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0074] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0075] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0076] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0077] As a package manager service, the package management service is mainly responsible for installing, managing, and uninstalling applications on Android devices. The package management service scans the specified directory in the system, finds the files ending with APK, and parses these files to obtain all the information of the application and store it in packages.xml.

[0078] When a new application is installed, the package management service will identify all components of the application (such as Activity, Service, Broadcast Receiver, etc.) and assign corresponding permissions to these components. At the same time, the package management service also monitors the status of installed applications to ensure the integrity and security of the applications.

[0079] The package management service is also used to manage the DE (Device Encrypted) data and CE (Credential Encrypted) data of the application. Among them, the key of the DE data can only be obtained after the system performs a verifiable startup. The CE directory uses a key associated with the user authentication (such as a pattern, password, etc.) to encrypt the data. This key can only be obtained after the user performs authentication first.

[0080] The CE directory of the application may include the original uid of the application. The package management service is used to execute the DE data and CE data of the music separation application in this embodiment.

[0081] As an activity manager service, the activity management service is mainly responsible for managing and tracking the activity tasks and life cycles of all applications. When an application is opened, the activity management service starts the process of the application and allocates processor resources and memory to the application. When the application is no longer in the foreground or background, or when the system memory is insufficient, the activity management service terminates or kills the process of the application.

[0082] For example, the activity management service can be responsible for managing and tracking the activity tasks and lifecycle of a music separation application. When the music separation application is opened, the activity management service starts the music separation application process and allocates processor resources and memory to the music separation application. When the music separation application is no longer in the foreground or background, or when the system is low on memory, the activity management service terminates or kills the music separation application process.

[0083] System libraries may include Surface Manager, Media Libraries, Android Rruntime, etc.

[0084] The Android runtime consists of a core library and a virtual machine. The Android runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one is the function that the Java language needs to call, and the other is the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0085] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0086] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0087] The kernel layer may include modules such as audio driver, display driver, Wi-Fi driver, Bluetooth driver, and sensor driver.

[0088] Understandably, Figure 1 The layers in the software structure shown and the components contained in each layer do not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which is not limited in the present application.

[0089] It should be noted that although the embodiment of the present application is described using the Android system as an example, its basic principle is also applicable to ( ), ( ), ( ), ( ), and electronic devices with operating systems such as Hongmeng (Harmony).

[0090] See also Figure 2 , which shows a flowchart of a music separation method provided by an embodiment of the present application. In a specific embodiment, the music separation method can be applied to electronic devices. Figure 2 The process shown is described in detail, and the music separation method may include the following steps S110 to S120.

[0091] Step S110: Acquire the first audio feature of the current audio.

[0092] In the embodiment of the present application, when the user needs to perform music separation, a separation instruction can be sent to the electronic device, and the electronic device receives and responds to the separation instruction to obtain the first audio feature of the current audio. The first audio feature can be used to characterize the target music source from which the user needs to separate the music, and the first audio feature can be a high-dimensional audio feature or a low-dimensional audio feature, etc., which is not limited here.

[0093] In some embodiments, when a user needs to perform music separation, a separation instruction can be sent to an electronic device, which receives and responds to the separation instruction, obtains the current audio source type of the current audio, and matches the current audio source type with multiple audio features pre-stored in an audio feature library to obtain a matching degree, and determines an audio feature whose matching degree is greater than or equal to a matching degree threshold as a first audio feature. Since the current audio source type is the audio source from which the user needs to separate music, it can be ensured that the audio source required by the user can be subsequently separated from the current mixed audio according to the first audio feature, thereby improving the user's music separation experience.

[0094] Among them, the current audio source type is the audio source from which the user needs to separate the music. The current audio source type may include bass music source, piano music source, guitar music source, singing music source, and violin music source, etc., which are not limited here.

[0095] The matching degree threshold can be used to characterize the minimum matching degree when the current audio source type matches the audio features in the audio feature library. The matching degree threshold can be a matching degree pre-set by the user, or a matching degree automatically generated by the electronic device based on a process of separating music multiple times. The setting method of the matching degree threshold is not limited here, and can be set according to actual needs.

[0096] As an implementation mode, the electronic device pre-stores the current audio source type of the current audio. When the user needs to separate music, a separation instruction can be sent to the electronic device. The electronic device receives and responds to the separation instruction and reads the pre-stored current audio source type.

[0097] As an implementation method, when a user needs to separate music, a separation instruction can be sent to an electronic device. The electronic device receives and responds to the separation instruction, sends a first acquisition instruction to a server via a network, the server receives and responds to the first acquisition instruction, reads the current audio source type of the current audio pre-stored on the server, and sends the current audio source type to the electronic device via the network. The electronic device receives the current audio source type returned by the server.

[0098] The server is connected to the electronic device through the network and exchanges data with the electronic device through the network. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), big data or artificial intelligence platforms, etc. The type of server is not limited here and can be set according to actual needs.

[0099] The network can be any one of a ZigBee network, a Bluetooth (BT) network, a Wireless Fidelity (Wi-Fi) network, a Thread network, a Long Range Radio (LoRa) network, a Low-Power Wide-Area Network (LPWAN), an infrared network, a Narrow Band Internet of Things (NB-IoT), a Controller Area Network (CAN), a Digital Living Network Alliance (DLNA) network, a Wide Area Network (WAN), a Local Area Network (LAN), a Metropolitan Area Network (MAN) or a Wireless Personal Area Network (WPAN), etc. The type of network is not limited here and can be set according to actual needs.

[0100] As an implementation method, when a user needs to separate music, a separation instruction can be sent to an electronic device. The electronic device receives and responds to the separation instruction, generates a first upload prompt message, and receives the current audio source type of the current audio uploaded by the user according to the first upload prompt message.

[0101] The first upload prompt information can be used to prompt the user to upload the current audio source type of the current audio to the electronic device according to the first upload prompt information. The first upload prompt information can be at least one of sound prompt information, text prompt information, or light prompt information, etc. The type of the first upload prompt information is not limited here and can be set according to actual needs.

[0102] In some embodiments, when a user needs to separate music, a separation instruction carrying the first audio feature of the current audio can be sent to an electronic device. The electronic device receives and responds to the separation instruction and obtains the first audio feature according to the separation instruction.

[0103] In some embodiments, the electronic device may be provided with an input panel. When the user needs to separate music, the user may input a separation instruction on the input panel of the electronic device, for example, by handwriting the separation instruction on the input panel of the electronic device, or by pressing a button on the input panel of the electronic device. The electronic device receives the separation instruction through the input panel.

[0104] In some embodiments, the electronic device may be provided with a voice recognition module. When a user needs to perform music separation, the user may send voice information within the voice collection range of the voice recognition module. The voice recognition module collects the voice information sent by the user, and performs voice recognition on the collected voice information to obtain a voice recognition result. When it is determined that the voice recognition result contains keywords for instructing the electronic device to perform music separation, for example, the keyword is "music separation", or for another example, the keywords are "music" and "separation", etc., it is determined that the separation instruction has been received.

[0105] As an example, the voice message sent by the user is: to separate the music, and the voice recognition result of the voice recognition contains the keywords "music" and "separation", then it is determined that the separation instruction is received.

[0106] In some embodiments, when a user needs to separate music, a separation instruction can be sent to the client. The client receives and responds to the separation instruction, forwards the separation instruction to the electronic device through the network, and the electronic device receives the separation instruction forwarded by the client.

[0107] The client is connected to the electronic device through a network and exchanges data with the electronic device through the network. The client can be a mobile client (for example, a mobile client, a PDA client, a Tablet PC client, a laptop client, a smart watch client, a smart bracelet client, or a wearable client) or a fixed client (for example, a desktop computer client, a smart panel client, etc.), etc. The type of the client is not limited here and can be set according to actual needs.

[0108] Step S120: Processing the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio.

[0109] In an embodiment of the present application, after acquiring the first audio feature of the current audio, the electronic device can process the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio, the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training the initial diffusion model based on the historical mixed audio and multiple second audio features of the historical mixed audio, the multiple second audio features include at least the first audio feature, so that in the music separation process, the current audio is separated from the current mixed audio according to the first audio based on the target diffusion model. Since the initial diffusion model learns the audio characteristics in the mixed audio according to the audio characteristics during the training process, and the diffusion model has the characteristic of parameter sharing, the target diffusion model can perform music separation on different mixed audios according to the audio characteristics, and there is no need to use the initial diffusion model to train the training sample audio with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation.

[0110] And the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, ensuring that the music separation process has a high separation quality. For example, if the current audio source type is a piano music source, then the current mixed audio includes the audio of the piano music source, the current audio is a piano music source, and the audio data distribution of the piano music source in the mixed audio is the same as the audio data distribution of the piano music source in the current audio.

[0111] Among them, the second audio feature can be a high-dimensional audio feature or a low-dimensional audio feature, etc., which is not limited here.

[0112] The diffusion model is a generative model that can learn to approximate unknown data distribution given independent and identically distributed sample data from the position data distribution. The diffusion model mainly includes the forward diffusion process and the reverse sampling (inference) process.

[0113] The forward diffusion process adds Gaussian noise to the samples slowly and orderly, and learns to recover from the noisy samples to clean samples in the reverse sampling process. Therefore, the diffusion model learns the reverse sampling process by learning the forward diffusion process. As a high-quality generation model, the diffusion model generates high-quality and pleasant-sounding speech signals.

[0114] As an example, Figure 3 As shown, sample x0 is a noise-free sample, x t-1 is the noise sample obtained after the sample x0 is processed by the diffusion model t times of forward diffusion, x t is the noise sample obtained after the sample x0 is processed by the diffusion model t+1 times of forward diffusion, x TIt is the noise sample obtained after the sample x0 is processed by the diffusion model T+1 times of forward diffusion.

[0115] Noise sample x T After T+1 times of reverse sampling, we can get the noise-free sample x0; the noise sample x T After T-t+1 times of reverse sampling, the noise sample x can be obtained. t ; Noise sample x T After Tt times of reverse sampling, the noise sample x can be obtained. t-1 .

[0116] q data (x0) is the noise distribution in sample x0, q(x t |x t-1 ) is the noise distribution added in the t+1th forward diffusion process, q latent (x T )=N(0,1) represents the noise sample x T The noise distribution in q latent (x T ) satisfies the normal distribution N(0,1). θ (x t-1 |x t ) is the noise distribution added in the Ttth reverse sampling process.

[0117] In one application scenario, such as Figure 4 As shown, the audio sample X0 is a noise-free audio sample. After the forward diffusion processing of the diffusion model, the audio sample X0 is gradually added with Gaussian noise, and the noisy audio sample X0 can be obtained. T Noise Audio Sample X T After the reverse sampling process of the diffusion model, a noise-free audio sample X0 can be obtained.

[0118] In some embodiments, the electronic device may pre-store a target diffusion model. After acquiring the first audio feature of the current audio, the electronic device may input the current mixed audio and the first audio feature into the target diffusion model. The target diffusion model receives and responds to the current mixed audio and the first audio feature, generates the current audio, and outputs the current audio to the electronic device. The electronic device receives the current audio output by the target diffusion model.

[0119] In some embodiments, the target diffusion model is pre-stored in a server. After the electronic device obtains the first audio feature of the current audio, it can send the current mixed audio and the first audio feature to the server via the network. The server receives and responds to the current mixed audio and the first audio feature, inputs the current mixed audio and the first audio feature to the target diffusion model, the target diffusion model receives and responds to the current mixed audio and the first audio feature, generates the current audio, and outputs the current audio to the server. The server receives the current audio output by the target diffusion model and sends the current audio to the electronic device via the network. The electronic device receives the current audio returned by the server.

[0120] In some implementations, the first audio feature may include multiple sub-audio features, indicating that the user needs to separate multiple target music sources. The current audio may include multiple sub-audios, indicating that multiple target music sources separated based on the target diffusion model are present.

[0121] After obtaining multiple high-dimensional features of the current audio, the electronic device can process the multiple sub-audio features and the current mixed audio through the target diffusion model to obtain multiple sub-audios, thereby separating the multiple sub-audios from the current mixed audio according to the multiple sub-audio features based on the target diffusion model, thereby improving the user's music separation experience.

[0122] As an implementation method, the electronic device may pre-store a target diffusion model. After acquiring multiple sub-audio features of multiple sub-audios, the electronic device may input the current mixed audio and the multiple sub-audio features into the target diffusion model. The target diffusion model receives and responds to the current mixed audio and the multiple sub-audio features, generates a sub-audio according to each sub-audio feature and the current mixed audio, and outputs the multiple sub-audios to the electronic device. The electronic device receives the multiple sub-audios output by the target diffusion model.

[0123] As an implementation method, the target diffusion model is pre-stored in a server. After the electronic device obtains multiple sub-audio features of multiple sub-audios, it can send the current mixed audio and the multiple sub-audio features to the server through the network. The server receives and responds to the current mixed audio and the multiple sub-audio features, inputs the current mixed audio and the multiple sub-audio features to the target diffusion model, the target diffusion model receives and responds to the current mixed audio and the multiple sub-audio features, generates a sub-audio according to each sub-audio feature and the current mixed audio, and outputs multiple sub-audios to the server. The server receives the multiple sub-audios output by the target diffusion model and sends the multiple sub-audios to the electronic device through the network. The electronic device receives the multiple sub-audios returned by the server.

[0124] In one application scenario, such as Figure 5As shown, the current mixed audio may include vocal audio, bass audio, drum audio, guitar audio, etc. Figure 5 The frequency distribution of singing audio, bass audio, drum audio, and guitar audio corresponding to different sampling durations are shown.

[0125] The multiple sub-audio features may include singing audio features, bass audio features, drum audio features, and guitar audio features, etc. The multiple sub-audios may include current singing audio, current bass audio, current drum audio, and current guitar audio, etc.

[0126] The target diffusion model can generate the current singing audio according to the singing audio features and the current mixed audio, generate the current bass audio according to the bass audio features and the current mixed audio, generate the current drum audio according to the drum audio features and the current mixed audio, and generate the current guitar audio according to the guitar audio features and the current mixed audio, such as Figure 6 shown.

[0127] The frequency distribution in the current singing audio is the same as the frequency distribution of the singing audio in the current mixed audio, the frequency distribution in the current bass audio is the same as the frequency distribution of the bass audio in the current mixed audio, the frequency distribution in the current drum audio is the same as the frequency distribution of the drum audio in the current mixed audio, and the frequency distribution in the current guitar audio is the same as the frequency distribution of the guitar audio in the current mixed audio.

[0128] In some embodiments, the target diffusion model can be obtained based on any one of the Unet network, the Noise Condition Score Network (NCSN) or the Noise Condition Score Network++ (NCSN++), that is, the target diffusion model can select any one of the Unet network, NCSN or NCSN++ as the model architecture, which is not limited here.

[0129] In some embodiments, the target diffusion model can be formed based on any one of a denoising diffusion probability model (Denoising Diffusion Probabilistic Model, DDPM) algorithm, a denoising diffusion implicit model (Denoising Diffusion Implicit Models, DDIM) algorithm, a stochastic differential equation (Stochastic Differential Equation, SDE) algorithm, or a score-based generative model (Score-based Generative Model, SGM) algorithm, etc., that is, the target diffusion model can select any one of the DDPM algorithm, DDIM algorithm, SDE algorithm, or SGM algorithm, etc. as a model framework, which is not limited here.

[0130] In one application scenario, such as Figure 7 As shown, the current audio may include the current piano audio, the current guitar audio, and the current drum audio, and the current mixed audio may include the singing audio, the bass audio, the drum audio, and the guitar audio.

[0131] The current piano audio, the current guitar audio and the current drum audio are forward diffused by the target diffusion model to obtain the current mixed audio; the current mixed audio is reverse sampled by the target diffusion model to obtain the current piano audio, the current guitar audio and the current drum audio.

[0132] This embodiment provides a solution, by obtaining the first audio feature of the current audio, and processing the first audio feature and the current mixed audio through the target diffusion model, the current audio is obtained, the audio source type of the current mixed audio includes the audio source type of the current audio, the target diffusion model is obtained by training the initial diffusion model based on the historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features at least include the first audio feature, so that in the process of music separation, the current audio is separated from the current mixed audio according to the first audio based on the target diffusion model, because the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, so that the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audio with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation. Moreover, the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, which ensures that the music separation process has a high separation quality.

[0133] See also Figure 8, which shows a flowchart of a music separation method provided by another embodiment of the present application. In a specific embodiment, the music separation method can be applied to electronic devices. Figure 8 The process shown is described in detail, and the music separation method may include the following steps S210 to S260.

[0134] Step S210: Acquire multiple audio sources.

[0135] In this embodiment, when the user needs to separate music, a separation instruction can be sent to the electronic device, and the electronic device receives and responds to the separation instruction to obtain multiple audio sources.

[0136] Each audio source may correspond to an audio source type, and the audio source may include bass, piano, guitar, singer, violin, etc. The type of the audio source is not limited here and may be set according to actual needs.

[0137] In some embodiments, the electronic device pre-stores multiple audio sources. When the user needs to separate music, a separation instruction can be sent to the electronic device. The electronic device receives and responds to the separation instruction and reads the pre-stored multiple audio sources.

[0138] In some embodiments, when a user needs to separate music, a separation instruction can be sent to an electronic device. The electronic device receives and responds to the separation instruction, sends a second acquisition instruction to a server via a network, the server receives and responds to the second acquisition instruction, reads multiple audio sources pre-stored on the server, and sends the multiple audio sources to the electronic device via the network. The electronic device receives the multiple audio sources returned by the server.

[0139] In some embodiments, when the user needs to separate music, a separation instruction can be sent to the electronic device, and the electronic device receives and responds to the separation instruction, generates a second upload prompt message, and receives multiple audio sources uploaded by the user according to the second upload prompt message.

[0140] The second upload prompt information can be used to prompt the user to upload multiple audio sources to the electronic device according to the second upload prompt information. The second upload prompt information can be at least one of sound prompt information, text prompt information, or light prompt information, etc. The type of the second upload prompt information is not limited here and can be set according to actual needs.

[0141] Step S220: extracting audio features from multiple audio sources using an audio feature extraction model to obtain multiple third audio features.

[0142] In this embodiment, after acquiring multiple audio sources, the electronic device can extract audio features from the multiple audio sources through an audio feature extraction model to obtain multiple third audio features, thereby realizing construction of an audio high-dimensional feature library based on the multiple third audio features extracted by the audio feature extraction model. There is no need for the user to manually extract audio features from the audio sources, and the process of manual extraction of third audio features by the user is relatively complicated, thereby simplifying the extraction process of audio features.

[0143] Among them, the audio feature extraction model can be any one of Convolutional Neural Networks (CNN), Deep Neural Networks (DNN) or Long Short-Term Memory (LSTM) network, etc. The type of audio feature extraction model is not limited here and can be set according to actual needs.

[0144] Each third audio feature corresponds to an audio source. The third audio feature may include the first audio feature. The third audio feature may be a high-dimensional audio feature or a low-dimensional audio feature, etc., which is not limited here.

[0145] Specifically, after acquiring multiple audio sources, the electronic device can input the multiple audio sources into a pre-trained audio feature extraction model. The audio feature extraction model receives and responds to the multiple audio sources, extracts audio features for each audio source, obtains a third audio feature, and outputs each third audio feature to the electronic device. The electronic device receives multiple third audio features output by the audio feature extraction model.

[0146] In some embodiments, the audio feature extraction model may be a CNN. After acquiring multiple audio sources, the electronic device may input the multiple audio sources into a pre-trained CNN. The CNN receives and responds to the multiple audio sources, extracts audio features from each audio source, obtains a third audio feature, and outputs each third audio feature to the electronic device. The electronic device receives the multiple third audio features output by the CNN.

[0147] In some embodiments, the audio feature extraction model may be a DNN. After acquiring multiple audio sources, the electronic device may input the multiple audio sources into a pre-trained DNN. The DNN receives and responds to the multiple audio sources, extracts audio features from each audio source, obtains a third audio feature, and outputs each third audio feature to the electronic device. The electronic device receives multiple third audio features output by the DNN.

[0148] In some embodiments, the audio feature extraction model can be an LSTM network. After obtaining multiple audio sources, the electronic device can input the multiple audio sources into a pre-trained LSTM network. The LSTM network receives and responds to the multiple audio sources, extracts audio features from each audio source, obtains a third audio feature, and outputs each third audio feature to the electronic device. The electronic device receives the multiple third audio features output by the LSTM network.

[0149] Step S230: Generate an audio feature library according to the plurality of third audio features.

[0150] In this embodiment, after the electronic device extracts audio features from each audio source through an audio feature extraction model to obtain multiple third audio features, it can generate an audio feature library based on the multiple third audio features, thereby realizing the construction of the audio feature library based on the multiple third audio features extracted by the audio feature extraction model, and simplifying the construction process of the audio feature library.

[0151] Step S240: Obtain the current audio source type of the current audio.

[0152] Step S250: Acquire a first audio feature from an audio feature library according to the current audio source type.

[0153] Step S260: Process the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio.

[0154] In this embodiment, step S240, step S250 and step S260 may refer to the contents of the corresponding steps in the aforementioned embodiments, which will not be described in detail here.

[0155] This embodiment provides a solution, by obtaining multiple audio sources, and extracting audio features from the multiple audio sources through an audio feature extraction model to obtain multiple third audio features, and generating an audio feature library based on the multiple third audio features, and obtaining the current audio source type of the current audio, and according to the current audio source type, obtaining the first audio feature from the audio feature library, and processing the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio, so as to achieve separation of the current audio from the current mixed audio based on the first audio in the music separation process based on the target diffusion model, because the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, so that the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audios with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation. Moreover, the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, thereby ensuring that the music separation process has a high separation quality.

[0156] Furthermore, an audio feature library is constructed based on multiple third audio features extracted by the audio feature extraction model, thereby simplifying the construction process of the audio feature library.

[0157] See also Fig. 9 , which shows a flowchart of a music separation method provided by another embodiment of the present application. In a specific embodiment, the music separation method can be applied to electronic devices. Fig. 9 The process shown is described in detail, and the music separation method may include the following steps S310 to S340.

[0158] Step S310: Acquire historical mixed audio and multiple second audio features of the historical mixed audio.

[0159] In this embodiment, when the user needs to separate music, a separation instruction can be sent to the electronic device, and the electronic device receives and responds to the separation instruction to obtain the historical mixed audio and multiple second audio features of the historical mixed audio.

[0160] In some embodiments, the electronic device pre-stores historical mixed audio and multiple second audio features of the historical mixed audio. When the user needs to separate music, a separation instruction can be sent to the electronic device. The electronic device receives and responds to the separation instruction and reads the pre-stored historical mixed audio and multiple second audio features of the historical mixed audio.

[0161] In some embodiments, when a user needs to separate music, a separation instruction can be sent to an electronic device, the electronic device receives and responds to the separation instruction, sends a third acquisition to a server via a network, the server receives and responds to the third acquisition instruction, reads the historical mixed audio and multiple second audio features of the historical mixed audio pre-stored by the server, and sends the historical mixed audio and multiple second audio features of the historical mixed audio to the electronic device via the network, and the electronic device receives the historical mixed audio and multiple second audio features of the historical mixed audio returned by the server.

[0162] In some embodiments, when a user needs to separate music, a separation instruction can be sent to an electronic device, which receives and responds to the separation instruction, generates a third upload prompt message, and receives the historical mixed audio uploaded by the user according to the third upload prompt message and multiple second audio features of the historical mixed audio.

[0163] The third upload prompt information can be used to prompt the user to upload the historical mixed audio and multiple second audio features of the historical mixed audio to the electronic device according to the third upload prompt information. The third upload prompt information can be at least any one of sound prompt information, text prompt information, or light prompt information, etc. The type of the third upload prompt information is not limited here, and can be set according to actual needs.

[0164] Step S320: training the initial diffusion model according to the historical mixed audio and the plurality of second audio features to obtain a target diffusion model.

[0165] In this embodiment, after acquiring the historical mixed audio and multiple second audio features of the historical mixed audio, the electronic device can input the historical mixed audio and the multiple second audio features into the initial diffusion model for training to obtain a target diffusion model, thereby realizing the training of the initial diffusion model based on the historical mixed audio and the multiple second audio features to obtain the target diffusion model. The diffusion model itself has the ability to adapt to the generation of different mixed audios, thereby ensuring that the target diffusion model has a strong generalized music separation capability.

[0166] In some embodiments, after obtaining historical mixed audio and multiple second audio features of the historical mixed audio, the electronic device can obtain noise audio, and input the noise audio, the historical mixed audio, and the multiple second audio features into the initial diffusion model for training to obtain a target diffusion model, thereby achieving the target diffusion model obtained by training the initial diffusion model based on the noise audio, the historical mixed audio, and the multiple second audio features, so that the target diffusion model further learns the ability to generate noise-free audio from noisy audio, further improving the generalized music separation ability of the target diffusion model.

[0167] Step S330: Acquire the first audio feature of the current audio.

[0168] Step S340: Process the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio.

[0169] In this embodiment, step S330 and step S340 may refer to the contents of the corresponding steps in the aforementioned embodiments, which will not be described in detail here.

[0170] The solution provided in this embodiment is to obtain the historical mixed audio and multiple second audio features of the historical mixed audio, and train the initial diffusion model according to the historical mixed audio and multiple second audio features to obtain the target diffusion model, and obtain the first audio feature of the current audio, and process the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio, so as to achieve separation of the current audio from the current mixed audio according to the first audio based on the target diffusion model during the music separation process. Since the initial diffusion model learns the audio features in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audios with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation. Moreover, the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, thereby ensuring that the music separation process has a high separation quality.

[0171] Furthermore, the initial diffusion model is trained based on the historical mixed audio and multiple second audio features of the historical mixed audio to obtain the target diffusion model. The diffusion model itself has the ability to adapt to the generation of different mixed audios, ensuring that the target diffusion model has a strong generalized music separation ability.

[0172] See also Fig.10 , which shows a music separation device 300 provided by an embodiment of the present application. The music separation device 300 can be applied to electronic devices. The following takes electronic devices as an example. Fig.10 The music separation device 300 shown is described in detail. The music separation device 300 may include a feature acquisition module 310 and a processing module 320.

[0173] The feature acquisition module 310 can be used to acquire the first audio feature of the current audio; the processing module 320 can be used to process the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio. The audio source type of the current mixed audio can include the audio source type of the current audio. The target diffusion model can be obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio. The multiple second audio features can include at least the first audio feature.

[0174] In some implementations, the feature acquisition module 310 may include a first acquisition unit and a second acquisition unit.

[0175] The first acquisition unit may be used to acquire the current audio source type of the current audio; the second acquisition unit may be used to acquire the first audio feature from the audio feature library according to the current audio source type.

[0176] In some implementations, the music separation device 300 may further include an audio source acquisition module, an extraction module, and a generation module.

[0177] The audio source acquisition module can be used for the second acquisition unit to acquire multiple audio sources before acquiring the first audio feature from the audio feature library according to the current audio source type, and each audio source can correspond to an audio source type; the extraction module can be used to extract audio features from the multiple audio sources through the audio feature extraction model to obtain multiple third audio features, each third audio feature can correspond to an audio source, and the multiple third audio features can include the first audio feature; the generation module can be used to generate an audio feature library based on the multiple third audio features.

[0178] In some embodiments, the audio feature extraction model may be any one of a convolutional neural network, a deep neural network, or a long short-term memory network.

[0179] In some implementations, the first audio feature, the second audio feature, and the third audio feature may all be high-dimensional audio features.

[0180] In some implementations, the current audio may include multiple sub-audios, the first audio feature may include multiple sub-audio features, and the processing module 320 may include a processing unit.

[0181] The processing unit can be used to process multiple sub-audio features and the current mixed audio through a target diffusion model to obtain multiple sub-audios.

[0182] In some implementations, the music separation device 300 may further include a historical mixed audio acquisition module and a training module.

[0183] The historical mixed audio acquisition module can be used for the processing module 320 to process the first audio feature and the current mixed audio through the target diffusion model, and obtain the historical mixed audio and multiple second audio features of the historical mixed audio before obtaining the current audio; the training module can be used to train the initial diffusion model according to the historical mixed audio and multiple second audio features to obtain the target diffusion model.

[0184] In some implementations, the music separation device 300 may further include a noise audio acquisition module.

[0185] The noise audio acquisition module can be used for the training module to train the initial diffusion model according to the historical mixed audio and multiple second audio features, and obtain the noise audio before obtaining the target diffusion model.

[0186] In some embodiments, a training module may enroll in a training unit.

[0187] The training unit can be used to train the initial diffusion model according to the noise audio, the historical mixed audio and the plurality of second audio features to obtain the target diffusion model.

[0188] In some implementations, the target diffusion model may be obtained based on any one of a Unet network, a noise-conditioned scoring network, or a noise-conditioned scoring network++.

[0189] In some embodiments, the target diffusion model may be formed based on any one of a denoised diffusion probability model algorithm, a denoised diffusion implicit model algorithm, a stochastic differential equation algorithm, or a score-based generative model algorithm.

[0190] The solution provided in this embodiment obtains the first audio feature of the current audio, and processes the first audio feature and the current mixed audio through the target diffusion model to obtain the current audio, the audio source type of the current mixed audio includes the audio source type of the current audio, the target diffusion model is obtained by training the initial diffusion model based on the historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features include at least the first audio feature, so that in the process of music separation, the current audio is separated from the current mixed audio according to the first audio based on the target diffusion model, because the initial diffusion model learns the audio characteristics in the mixed audio according to the audio features during the training process, and the diffusion model has the characteristic of parameter sharing, so that the target diffusion model can perform music separation on different mixed audios according to the audio features, and there is no need to use the initial diffusion model to train the training sample audio with the same audio source type as each mixed audio, thereby simplifying the operation process of music separation. Moreover, the audio source type of the current mixed audio includes the audio source type of the current audio, and the audio data distribution of the current audio is the same as the audio data distribution corresponding to the current audio source type in the current mixed audio, which ensures that the music separation process has a high separation quality.

[0191] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Any processing method described in the method embodiment can be implemented by the corresponding processing module in the device embodiment, and will not be repeated one by one in the device embodiment.

[0192] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0193] See also Fig.11 , which shows a hardware structure diagram of an electronic device 600 provided by an embodiment of the present application. Fig.11As shown, the electronic device 600 may include a processor 610, an external memory interface 620, an internal memory 621, a Universal Serial Bus (USB) interface 630, a charging management module 640, a power management module 641, a battery 642, an antenna 1, an antenna 2, a mobile communication module 650, a wireless communication module 660, an audio module 670, a speaker 670A, a receiver 670B, a microphone 670C, an earphone interface 670D, a sensor module 680, a button 690, a motor 6191, an indicator 692, a camera 693, a display screen 694, and a Subscriber Identification Module (SIM) card interface 695, etc. Among them, the sensor module 680 may include a pressure sensor 680A, a gyroscope sensor 680B, an air pressure sensor 680C, a magnetic sensor 680D, an acceleration sensor 680E, a distance sensor 680F, a proximity light sensor 680G, a fingerprint sensor 680H, a temperature sensor 680J, a touch sensor 680K, an ambient light sensor 680L, a bone conduction sensor 680M, etc.

[0194] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 600. In other embodiments of the present application, the electronic device 600 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0195] For example, Fig.11 The processor 610 shown may include one or more processing units, for example, the processor 610 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors.

[0196] The AP may be used to control and manage the music separation application. For example, the AP may control the music separation application to separate the music.

[0197] The controller may be the nerve center and command center of the electronic device 600. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0198] The processor 610 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 610 is a cache memory. The memory may store instructions or data that the processor 610 has just used or cyclically used. If the processor 610 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 610, and thus improves the efficiency of the system.

[0199] In some embodiments, the processor 610 may include one or more interfaces. The interface may include an Inter-Integrated Circuit (I2C) interface, an Inter-Integrated Circuit Sound (I2S) interface, a Pulse Code Modulation (PCM) interface, a Universal Asynchronous Receiver / Transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a General Purpose Input / Output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and / or a Universal Serial Bbus (USB) interface, etc.

[0200] In some embodiments, the I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). The processor 610 may include multiple groups of I2C buses. The processor 610 may be coupled to the touch sensor 680K, the charger, the flash, the camera 693, etc. through different I2C bus interfaces. For example, the processor 610 may be coupled to the touch sensor 680K through the I2C interface, so that the processor 610 communicates with the touch sensor 680K through the I2C bus interface to realize the touch function of the electronic device 600.

[0201] In some embodiments, the I2S interface can be used for audio communication. The processor 610 can include multiple I2S buses. The processor 610 can be coupled to the audio module 670 via the I2S bus to achieve communication between the processor 610 and the audio module 670.

[0202] In some embodiments, the audio module 670 can transmit audio signals to the wireless communication module 660 through the I2S interface to implement the function of answering calls through a Bluetooth headset.

[0203] In some embodiments, the PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. The audio module 670 and the wireless communication module 660 can be coupled via a PCM bus interface.

[0204] In some embodiments, the audio module 670 can also transmit audio signals to the wireless communication module 660 through the PCM interface to implement the function of answering calls through a Bluetooth headset. It should be understood that both the I2S interface and the PCM interface can be used for audio communication.

[0205] The electronic device 600 can implement audio functions such as music playing and recording through the audio module 670, the speaker 670A, the receiver 670B, the microphone 670C, the headphone jack 670D, and the application processor.

[0206] The audio module 670 can be used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signal. The audio module 670 can also be used to encode and decode audio signals. In some embodiments, the audio module 670 can be arranged in the processor 610, or some functional modules of the audio module 670 can be arranged in the processor 610.

[0207] The speaker 670A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 600 can listen to music or listen to a hands-free call through the speaker 670A.

[0208] The receiver 670B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 600 receives a call or voice message, the voice can be received by placing the receiver 670B close to the human ear.

[0209] Microphone 670C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 670C to input the sound signal into the microphone 670C. The electronic device 600 can be provided with at least one microphone 670C. In other embodiments, the electronic device 600 can be provided with two microphones 670C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 600 can also be provided with three, four or more microphones 670C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0210] The touch sensor 680K is also called a "touch panel". The touch sensor 680K can be set on the display screen 694, and the touch sensor 680K and the display screen 694 form a touch screen, also called a "touch screen". The touch sensor 680K is used to detect touch operations acting on or near it. The touch sensor 680K can pass the detected touch operation to the application processor 610 to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 694. In other embodiments, the touch sensor 680K can also be set on the surface of the electronic device 600, which is different from the position of the display screen 694.

[0211] The key 690 includes a power key, a volume key, etc. The key 690 may be a mechanical key or a touch key. The electronic device 600 may receive key input and generate key signal input related to user settings and function control of the electronic device 600.

[0212] See also Fig.12 , which shows a functional block diagram of an electronic device 600 according to an embodiment of the present application. Fig.12 As shown, the electronic device 600 includes: at least one processor 610 ( Fig.12 Only one processor is shown in the figure), a memory 696, and a computer program 697 stored in the memory 696 and executable on at least one processor 610. When the processor 610 executes the computer program 697, the electronic device 600 implements the steps of any one of the above methods.

[0213] Those skilled in the art will understand that Fig.12 This is merely an example of the electronic device 600 and does not constitute a limitation on the electronic device 600. In practice, the electronic device 600 may include more or fewer components than those shown in the figure, or a combination of certain components, or different components, for example, it may also include input and output devices, network access devices, etc.

[0214] The processor 610 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0215] In some embodiments, the memory 696 may be an internal storage unit of the electronic device 600, such as a hard disk or memory of the electronic device 600. In other embodiments, the memory 696 may also be an external storage device of the electronic device 600, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 600. Optionally, the memory 696 may also include both an internal storage unit of the electronic device 600 and an external storage device. The memory 696 is used to store an operating system, an application program, a boot loader, data, and other programs, such as program codes of a computer program. The memory 696 may also be used to temporarily store data that has been output or is to be output.

[0216] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0217] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0218] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is called by an electronic device, it can implement the methods described in the above-mentioned various method embodiments.

[0219] An embodiment of the present application also provides a computer program product, wherein the computer-readable storage medium stores program code. When the computer program product is run on an electronic device, the electronic device executes the above-mentioned related steps to implement the methods described in the above-mentioned various method embodiments.

[0220] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0221] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0222] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0223] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0224] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0225] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0226] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0227] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0228] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some implementations", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A music separation method, characterized in that: The music separation method comprises: Get the current audio source type of the current audio; Get multiple audio sources, each audio source corresponds to an audio source type; Extracting audio features from multiple audio sources using an audio feature extraction model to obtain multiple third audio features, each third audio feature corresponding to an audio source, and the multiple third audio features include the first audio feature; generating an audio feature library according to a plurality of third audio features; Acquire the first audio feature from the audio feature library according to the current audio source type; The first audio feature and the current mixed audio are processed by a target diffusion model to obtain the current audio, the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features at least include the first audio feature.

2. The music separation method according to claim 1, characterized in that: The audio feature extraction model is any one of a convolutional neural network, a deep neural network or a long short-term memory network.

3. The music separation method according to claim 1, characterized in that: The first audio feature, the second audio feature, and the third audio feature are all high-dimensional audio features.

4. The music separation method according to claim 1, characterized in that: The current audio includes a plurality of sub-audios, the first audio feature includes a plurality of sub-audio features, and the processing of the first audio feature and the current mixed audio by the target diffusion model to obtain the current audio includes: The multiple sub-audio features and the current mixed audio are processed through a target diffusion model to obtain multiple sub-audios.

5. The music separation method according to claim 1, characterized in that: Before the first audio feature and the current mixed audio are processed by the target diffusion model to obtain the current audio, the music separation method further includes: Acquire a historical mixed audio and a plurality of second audio features of the historical mixed audio; The initial diffusion model is trained according to the historical mixed audio and the plurality of second audio features to obtain a target diffusion model.

6. The music separation method according to claim 5, characterized in that: Before the initial diffusion model is trained according to the historical mixed audio and the plurality of second audio features to obtain the target diffusion model, the music separation method further comprises: Get the noise audio; Training an initial diffusion model according to the historical mixed audio and the plurality of second audio features to obtain a target diffusion model includes: An initial diffusion model is trained according to the noise audio, the historical mixed audio, and a plurality of second audio features to obtain a target diffusion model.

7. The music separation method according to claim 1, characterized in that: The target diffusion model is obtained based on any one of a Unet network, a noise conditional scoring network or a noise conditional scoring network++.

8. The music separation method according to any one of claims 1 to 7, characterized in that: The target diffusion model is formed based on any one of a denoised diffusion probability model algorithm, a denoised diffusion implicit model algorithm, a stochastic differential equation algorithm, or a score-based generative model algorithm.

9. A music separation device, characterized in that: The music separation device comprises: A first acquisition unit, used to acquire a current audio source type of a current audio; An audio source acquisition module is used to acquire multiple audio sources, each audio source corresponds to an audio source type; An extraction module, configured to extract audio features from multiple audio sources using an audio feature extraction model to obtain multiple third audio features, each third audio feature corresponding to one audio source, and the multiple third audio features including the first audio feature; A generating module, used for generating an audio feature library according to a plurality of third audio features; A second acquisition unit, configured to acquire the first audio feature from the audio feature library according to the current audio source type; A processing module is used to process the first audio feature and the current mixed audio through a target diffusion model to obtain the current audio, wherein the audio source type of the current mixed audio includes the audio source type of the current audio, and the target diffusion model is obtained by training an initial diffusion model based on historical mixed audio and multiple second audio features of the historical mixed audio, and the multiple second audio features at least include the first audio feature.

10. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device executes the music separation method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by an electronic device to execute the music separation method as described in any one of claims 1 to 8.

Images