Blind source separation method, model training method, and electronic device

Abstract

Embodiments of the present application provide a blind source separation method, a model training method and an electronic device. The blind source separation method comprises: obtaining a first audio signal; inputting the first audio signal into a first model; the first model is trained based on an initial diffusion model; the first model performs a reverse diffusion process, separates the first audio signal, and outputs a second audio signal; and a single speech signal in the first audio signal is determined according to the second audio signal. The method can improve the blind source separation effect.

Description

Technical Field

[0001] This application relates to the field of audio technology, specifically to a blind source separation method, a model training method, and an electronic device. Background Technology

[0002] Blind source separation (BSS) is a signal processing technique that separates the source signals from the mixed audio output signal when neither the source signals nor the mixing method are known. Blind source separation has important applications in speech recognition, music analysis, and security monitoring.

[0003] Blind source separation methods in related technologies suffer from poor separation performance. Summary of the Invention

[0004] This application provides a blind source separation method, a model training method, and an electronic device that can improve the blind source separation effect.

[0005] In a first aspect, this application provides a blind source separation method, which is executed by an electronic device. The method includes: acquiring a first audio signal; inputting the first audio signal into a first model; the first model being trained based on an initial diffusion model; the first model performing a back diffusion process to separate the first audio signal and output a second audio signal; and determining a single speech signal in the first audio signal based on the second audio signal.

[0006] The first aspect of the blind source separation method involves performing an anti-diffusion process using a first model (also known as a single-speech separation model) to separate speech signals. This single-speech separation model is trained based on a diffusion model. Firstly, this method eliminates the need for phase and spectral estimation, resulting in more accurate separation. Secondly, it does not rely on the independence assumption; therefore, it can effectively extract source signals regardless of whether the source signals satisfy the independence assumption, resulting in good signal separation. Thirdly, this method does not require calculations based on multi-microphone arrays, thus eliminating reliance on multi-microphone pickup; audio from a single microphone can also be separated, making it highly applicable. Fourthly, this method uses a diffusion model to separate source signals, fully utilizing the separation capability of the diffusion model without damaging the signal, improving separation quality, robustness, and fidelity.

[0007] In one possible implementation, determining a single speech signal in the first audio signal based on the second audio signal includes: splitting the second audio signal into a third audio signal and a fourth audio signal; the third audio signal and the fourth audio signal have equal lengths, the third audio signal is used to store data of the single speech signal separated from the first audio signal, and the fourth audio signal is used to store data of the remaining audio signal after the third audio signal is separated from the first audio signal; if the data of the third audio signal is not empty, then the third audio signal is determined to be a single speech signal in the first audio signal.

[0008] In this implementation, the second audio signal is split to determine individual speech signals. The specific position of the individual speech signal within the second audio signal depends on the position of the target single sample signal during model training.

[0009] In one possible implementation, the method further includes: if the data of the third audio signal is not empty, then using a target adapter to repair the fourth audio signal to obtain a fifth audio signal; copying the fifth audio signal to obtain another fifth audio signal; splicing the two fifth audio signals to obtain a sixth audio signal; using the sixth audio signal as the first audio signal, and returning to the execution step: inputting the first audio signal into the first model.

[0010] In this implementation, the fifth audio signal is copied, spliced, and then re-input into the first model to achieve repeated separation of the first audio signal, thereby separating each individual speech signal in the first audio signal and achieving more thorough blind source separation.

[0011] In one possible implementation, the target adapter is obtained by training the initial adapter based on the first model.

[0012] In this implementation, the target adapter is trained based on the first model, which improves the coupling between the trained target adapter and the first model, thereby improving the quality of blind source separation.

[0013] In one possible implementation, the third audio signal is the first half of the second audio signal and the fourth audio signal is the second half of the second audio signal; or, the third audio signal is the second half of the second audio signal and the fourth audio signal is the first half of the second audio signal.

[0014] The positions of the third and fourth audio signals within the second audio signal are consistent with the positions of the target single sample signal and the reverberant residual sample signal within the positive input signal during model training.

[0015] In one possible implementation, acquiring the first audio signal includes: receiving a first instruction input by a user through a first application, the first instruction being used to instruct recording; in response to the first instruction, recording a seventh audio signal; and, when a first function of the first application is enabled, using the seventh audio signal as the first audio signal; the first function being a function implemented based on a single voice signal.

[0016] The first instruction can be implemented, for example, by clicking on the specific implementation method. Figure 1 The recording control 103 shown is used for input.

[0017] In one possible implementation, the first application is a note-taking application, and the first function includes the ability to display text in the speech according to the speaker.

[0018] Based on the function of displaying the text in the speaker's voice, that is, the speech-to-text function and the speaker display function in the note-taking application.

[0019] In this implementation, the blind source separation method is applied to the speaker display function in the note-taking application to separate the voices of different speakers in the audio, making it easier for users to record content, facilitating user use, and improving user experience.

[0020] Secondly, this application provides a model training method, which is executed by an electronic device. The method includes: acquiring multiple sets of speech sample groups, wherein a first speech sample group includes a first mixed sample of equal length, a first target single sample, and a first mixed residual sample; the first speech sample group is any one of the multiple speech sample groups; the first mixed sample is a mixed signal of k segments of single speech signals, where k is an integer greater than or equal to 2; the first target single sample is one of the k segments of single speech signals; and the first mixed residual sample is the speech signal remaining after removing the first target single sample from the first mixed sample; and executing a first training process; the first training process... The process includes: concatenating a first target single sample and a first mixed sample to obtain a positive input signal; copying the first mixed sample to obtain another first mixed sample; concatenating the two first mixed samples to obtain a first concatenated sample; inputting the positive input signal into an initial diffusion model, using the first concatenated sample as the target label, and performing a positive diffusion process through the initial diffusion model to train the initial diffusion model; and repeatedly performing the first training process by taking other speech sample groups from multiple speech sample groups as first speech sample groups until the initial diffusion model converges or the number of training iterations reaches a first preset threshold to obtain a first model.

[0021] The second aspect provides a model training method where the target single sample and the mixed residue are concatenated to form the target domain signal of the diffusion model, or target distribution. The diffusion model then performs a forward diffusion process, outputting a mixed prediction signal, called the mixed distribution. In other words, the initial diffusion model performs a forward diffusion process to transform the target distribution into a mixed distribution, and the separated signals into a mixed signal. This process utilizes the target domain modeling capability of the diffusion model, enabling the model to learn the ability to transform the mixed signal into a separated signal, that is, to learn the ability to separate a single speech signal from the mixed distribution, thus obtaining a first model with single speech separation capability. The first model trained by this method can achieve blind source separation, and the model has high robustness, producing high fidelity and quality of separated speech.

[0022] In one possible implementation, the first target single sample is located in the first half of the positive input signal, and the first mixed sample is located in the second half; or, the first mixed sample is located in the first half of the positive input signal, and the first target single sample is located in the second half.

[0023] In one possible implementation, after obtaining the first model, the method further includes: performing a second training process; the second training process includes: copying the second mixed sample to obtain another second mixed sample; the second mixed sample is any one of the mixed samples in multiple groups of speech samples; splicing the two second mixed samples to obtain a second spliced ​​sample; inputting the second spliced ​​sample into the first model; the first model performs a back-diffusion process to separate the second spliced ​​sample and output a separation prediction signal; training the initial adapter based on the separation prediction signal; using other mixed samples in the multiple groups of speech samples as second mixed samples, repeating the second training process until the initial adapter converges, or the number of training iterations reaches a second preset threshold, to obtain the target adapter.

[0024] In this implementation, a target adapter with high robustness and accurate output results is trained based on the first model. The combined use of the first model and the target adapter can improve the separation effect of blind source separation, reduce the deviation between the mixed residual signal and the signal under ideal conditions, reduce the accumulation of separation deviation in iterative cycles, and improve the accuracy of signal separation.

[0025] In one possible implementation, in the forward input signal, the first target single sample is located in the first half, and the first mixed sample is located in the second half. Based on the separation prediction signal, the initial adapter is trained, including: splitting the separation prediction signal into a single prediction signal and a mixed residual prediction signal; the single prediction signal and the mixed residual prediction signal have equal lengths, the single prediction signal is the first half of the separation prediction signal, and the mixed residual prediction signal is the second half of the separation prediction signal; the single prediction signal is used to store data of the single speech signal separated from the separation prediction signal, and the mixed residual prediction signal is used to store data of the audio signal remaining after separating the single prediction signal from the separation prediction signal; determining the target single sample most similar to the single prediction signal from multiple groups of speech samples; removing the most similar target single sample from the second mixed sample to obtain the mixed residual target signal; inputting the mixed residual prediction signal into the initial adapter, using the mixed residual target signal as the target label, and training the initial adapter.

[0026] In one possible implementation, determining the target single sample most similar to the single predicted signal from multiple sets of speech samples includes: if the data of the single predicted signal is not empty, then determining the target single sample most similar to the single predicted signal from multiple sets of speech samples; the second training process further includes: if the data of the single predicted signal is not empty, then using the signal output by the initial adapter as the second mixed sample, and returning to the execution step: copying the second mixed sample to obtain another second mixed sample.

[0027] In this implementation, the signal output from the initial adapter is input back into the first model for separation. During the process of the first model repeatedly separating single speech signals, the initial adapter is repeatedly trained. This accelerates the adapter's convergence speed and improves the coupling between the final target adapter and the first model.

[0028] In one possible implementation, determining the target single sample most similar to a single predicted signal from multiple groups of speech samples includes: calculating the similarity between the single predicted signal and the target single sample in each group of speech samples; and determining the most similar target single sample based on the similarity.

[0029] In this implementation, by calculating similarity, actual sample signals that match a single predicted signal are found from the speech sample library. Based on this, the initial adapter is trained, resulting in a more accurate target adapter.

[0030] Thirdly, this application provides an apparatus included in an electronic device, which has the function of implementing the behaviors of the electronic device in the first aspect and its possible implementations, as well as the second aspect and its possible implementations. The function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0031] Fourthly, this application provides an electronic device, which includes a processor, a memory, and an interface; the processor, memory, and interface cooperate with each other to enable the electronic device to execute any one of the technical solutions of the first and second aspects.

[0032] Fifthly, this application provides a chip system including a processor. The processor is configured to read and execute a computer program stored in a memory to perform the methods of the first aspect and any possible implementation thereof, and the second aspect and any possible implementation thereof.

[0033] Optionally, the chip system may also include memory, which is connected to the processor via circuitry or wires.

[0034] Alternatively, the chip system may also include a communication interface.

[0035] Sixthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform any one of the methods in the first and second aspects of the technical solutions.

[0036] In a seventh aspect, this application provides a computer program product, which includes computer program code that, when executed on an electronic device, causes the electronic device to perform any one of the technical solutions of the first and second aspects. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating an application scenario of a blind source separation method provided in an embodiment of this application;

[0038] Figure 2 This is a schematic diagram illustrating the working principle of blind source separation methods based on statistical signals in related technologies;

[0039] Figure 3 This is a schematic diagram illustrating the working principle of blind source separation methods based on discriminative neural networks in related technologies;

[0040] Figure 4 This is a schematic diagram of the structure of an example electronic device 100 provided in an embodiment of this application;

[0041] Figure 5 This is a software structure block diagram of an example electronic device 100 provided in an embodiment of this application;

[0042] Figure 6 This is a schematic diagram illustrating the working principle of an example diffusion model provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram illustrating the principle of training a single-speech separation model provided in an embodiment of this application;

[0044] Figure 8 This is a flowchart illustrating an example model training method provided in an embodiment of this application;

[0045] Figure 9 This is a schematic diagram of an example voice sample group provided in an embodiment of this application;

[0046] Figure 10 This is a schematic diagram of the principle of a training target adapter provided in an embodiment of this application;

[0047] Figure 11 This is a flowchart illustrating another model training method provided in an embodiment of this application;

[0048] Figure 12 This is a schematic diagram illustrating the back diffusion principle of an example single-speech separation model provided in this application embodiment;

[0049] Figure 13 This is a flowchart illustrating an example of a blind source separation method provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0051] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0052] References to "one embodiment" or "some embodiments" as described in this application specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0053] To better understand the embodiments of this application, the terms or concepts that may be involved in the embodiments are explained below.

[0054] 1. Cocktail party effect and cocktail party problems

[0055] The cocktail party effect refers to a person's ability to selectively focus their attention on a particular person's conversation while ignoring other background noises, even in a noisy environment like a cocktail party.

[0056] Based on this, the scenario where the desired source signal needs to be separated from complex audio is called the cocktail party problem.

[0057] 2. Source signal and mixed signal

[0058] In the field of audio, a source signal refers to the original, unprocessed audio signal. A source signal can be understood as the raw recording of sound. Source signals can originate from various audio sources, such as musical signals from musical instruments or human speech.

[0059] In the field of audio, a mixed signal is a signal formed by combining two or more source signals in some way. This combination can be intentional, such as the mixing of signals during music production, or unintentional, such as the superposition of sounds in a noisy environment.

[0060] This application mainly separates the speech source signal from the mixed signal. The speech source signal, also known as the single speech signal, refers to a speech signal that comes from a single sound source and is not mixed with other speech signals.

[0061] 3. Independent Vector Analysis (IVA)

[0062] Independent vector analysis (IVA) is a blind source separation method. The principle of IVA is based on the independence assumption to estimate the source signals. Specifically, it assumes there are n source signals s(t) = [s1(t), s2(t), ... sn(t)]. n (t)] T T denotes transpose. These source signals are mixed through a mixing matrix A to obtain m observed signals x(t) = [x1(t), x2(t), ... x... n (t)] T The relationship between them can be represented by a linear mixture model x(t) = A s (t) is shown below. The purpose of independent vector analysis is to find a separation matrix W such that the separated signal y(t) = W x (t) is as close as possible to the source signal s(t).

[0063] Before introducing the methods provided in the embodiments of this application, the application scenarios and technical problems faced by this application will be explained first.

[0064] Blind source separation refers to separating individual source signals from a mixed signal. It enables machines to possess auditory capabilities similar to humans, making it a crucial tool for solving cocktail party problems. Consequently, blind source separation has found wide application in practical products, such as speech recognition, music analysis, and security monitoring.

[0065] For example, blind source separation can be applied to recording applications (APPs), note-taking apps, and meeting apps on electronic devices to process recordings and separate individual speech signals from each speaker, thereby displaying or recording the content according to the speaker. Here, a speaker, also called a voicer, refers to the person who makes the sound. The following explanation is illustrated with accompanying diagrams.

[0066] For example, Figure 1 This is a schematic diagram illustrating an application scenario of a blind source separation method provided in an embodiment of this application. Taking the application of blind source separation in a note-taking app as an example, as... Figure 1 As shown in Figure (a), the desktop of the electronic device includes an icon 101 for a note-taking app. In response to the user clicking the note-taking app icon 101, the electronic device displays the main interface 102 of the note-taking app, as shown... Figure 1 As shown in Figure (b), the main interface 102 of the note-taking app includes a new note control 103. In response to the user clicking the new note control 103, the note-taking app creates a new note and displays it as shown in Figure (b). Figure 1 The note-taking interface 104 is shown in Figure (c). The note-taking interface 104 includes a recording control 105. In response to the user clicking the recording control 105, the note-taking app starts recording and displays the following: Figure 1The recording interface 106 is shown in Figure (d). The note-taking app has a voice-to-text function. During recording, when the voice-to-text function is enabled, the voice-to-text control 107 is highlighted, as shown in Figure (d). Figure 1 As shown in Figure (d). Additionally, the note-taking app also has a speaker display function; when this function is enabled, the speaker display control 108 is highlighted. Figure 1 As shown in Figure (d) above. With both the speech-to-text function and the speaker display function enabled, the note-taking app performs blind source separation processing on the recorded audio, identifies each individual speech signal in the recording, and performs voiceprint recognition on each individual speech signal. This allows for separate text conversion of the speech signals from different speakers, and the text is displayed in separate areas, such as... Figure 1 Interface 109 is shown in Figure (e).

[0067] certainly, Figure 1 The scenario shown is only one application scenario of the blind source separation method. The blind source separation method can also be applied to other application scenarios, such as voice wake-up based on voiceprint recognition, speech recognition, etc., which will not be listed here.

[0068] It should be noted that the above example describes blind source separation of real-time acquired audio signals. In other embodiments, the blind source separation method can also be used to perform blind source separation on non-real-time acquired audio signals, that is, to perform blind source separation on pre-recorded audio signals. For example, in a recorder app, note-taking app, or instant messaging app, a blind source separation control can be preset for the recorded audio signal. The user can use this control to trigger blind source separation processing on the recorded audio signal.

[0069] In summary, the embodiments of this application do not limit the application scenarios for blind source separation.

[0070] In related technologies, blind source separation generally includes the following two methods: 1) Blind source separation method based on statistical signals; 2) Blind source separation method based on discriminative neural networks.

[0071] 1) Blind source separation method based on statistical signals.

[0072] The main idea of ​​this method is to extract the spectral and location information of the sound source based on an acoustic model, and then separate the acquired multi-channel signal based on this spectral and location information. For example, Figure 2 This is a schematic diagram illustrating the working principle of blind source separation methods based on statistical signals in related technologies. For example... Figure 2As shown, the method includes the following steps: a) Performing a feature transformation on the mixed signal to obtain a mixed signal in the transform domain. b) Performing phase estimation and spectral estimation on the transform domain signal to obtain phase information and spectral information. The phase information represents the location information of the sound source. c) Filtering the mixed signal using a separation filter based on the phase and spectral information to obtain a multi-channel signal. d) Separating the multi-channel signal to obtain the source signal in the transform domain (also called the separated signal). e) Performing an inverse feature transformation on the source signal in the transform domain to obtain the source signal in the original domain.

[0073] Blind source separation methods based on statistical signals have the following problems: a) Phase estimation is easily affected by phase direction and post-mixing, leading to inaccurate estimation and thus inaccurate separation results. b) Phase and spectrum estimation require an array of multiple microphones, making the method dependent on multiple microphones and only suitable for scenarios with multiple microphones (multi-microphone pickup). It is not suitable for scenarios with a single microphone (single-microphone pickup), resulting in poor applicability. c) Insufficient accuracy in spectrum estimation leads to inaccurate separation results.

[0074] Another typical method for blind source separation based on statistical signals is Independent Vector Analysis (IVA). The principle of IVA is based on the independence assumption, which assumes that the mixed signal is a linear combination of multiple independent source signals, and that these source signals are statistically independent. Besides the problems mentioned above, this method is also sensitive to model assumptions. If the signals do not meet the independence assumption, the separation effect will decrease, and separation residue or over-separation may occur.

[0075] 2) Blind source separation method based on discriminative neural network.

[0076] The main idea of ​​this method is to use a deep neural network model to estimate a mask or mapping function applied to the mixed signal in the transform domain to separate the speech signal. For example, Figure 3 This is a schematic diagram illustrating the working principle of blind source separation methods based on discriminative neural networks in related technologies. For example... Figure 3 As shown, the method includes the following steps: a) performing feature transformation on the mixed signal to obtain a transform-domain mixed signal. b) inputting the transform-domain mixed signal into a deep neural network model, whereby the deep neural network model estimates a mask or mapping function in the transform domain and applies it to the transform-domain mixed signal to obtain the source signal in the transform domain. d) performing inverse feature transformation on the source signal in the transform domain to obtain the source signal in the original domain.

[0077] The blind source separation method based on discriminative neural networks has the following problems: a) Deep neural networks are prone to damaging the signal during mask estimation or mapping function estimation, resulting in low fidelity of the final separated signal. b) This method is highly dependent on the training dataset and has poor separation ability for signals not present in the training dataset; that is, the method has poor generalization performance and therefore unstable separation results.

[0078] Based on this, embodiments of this application provide a blind source separation method, which separates speech signals by repeatedly performing an anti-diffusion process using a single speech separation model (also known as the first model), which is trained based on a diffusion model. Firstly, this method does not require phase estimation and spectral estimation, thus resulting in more accurate separation. Secondly, it does not rely on the independence assumption; therefore, regardless of whether the source signal satisfies the independence assumption, it can effectively extract the source signal, resulting in good signal separation. Thirdly, this method does not require calculation based on a multi-microphone array, thus eliminating reliance on multi-microphone pickup; audio obtained from a single microphone pickup can also be separated, making it highly applicable. Fourthly, this method uses a diffusion model to separate the source signal, fully utilizing the separation capability of the diffusion model without damaging the signal, improving separation quality, and also enhancing the robustness and fidelity of signal separation.

[0079] Furthermore, this application also provides a model training method. An initial diffusion model is pre-trained to obtain a single-speech separation model. Based on this single-speech separation model, the initial adapter is refined and trained to obtain a target adapter. The single-speech separation model separates speech signals one by one, and the target adapter repairs the mixed residual signal obtained after each separation, reducing the deviation between the mixed residual signal and the ideal signal, minimizing the impact on subsequent signal separation, improving the accuracy of signal separation, and increasing the fidelity of the separated speech signal. In addition, this method trains the model based on three signals: mixed signal, single speech signal, and mixed residual signal. It has no special requirements for the training dataset, and the trained model exhibits strong generalization performance and stable separation results.

[0080] The electronic devices used in the blind source separation method and model training method provided in the embodiments of this application will be described below.

[0081] The blind source separation method provided in this application can be applied to electronic devices that can install applications (APPs), such as mobile phones, tablets, wearable devices, in-vehicle devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0082] For example, Figure 4 This is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may include a processor 110, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, a display screen 194, etc. The sensor module 180 may include a touch sensor 180K, etc.

[0083] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0084] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0085] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0086] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0087] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0088] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 can be located in the processor 110, or some functional modules of the audio module 170 can be located in the processor 110. Specifically, in the embodiments of this application, the audio module 170 can be used for blind source separation of sound signals collected by a microphone.

[0089] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0090] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0091] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0092] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0093] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0094] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0095] Figure 5 This is a software structure block diagram of an electronic device 100 according to an embodiment of this application. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer may include a series of application packages.

[0096] The application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS (not shown in the figure).

[0097] like Figure 5 As shown in the embodiments of this application, the application layer may include a blind source separation module. The blind source separation module is used to implement blind source separation. Optionally, the blind source separation module can be a standalone application or module, which can be invoked when the target APP (e.g., a note-taking APP, a recording APP, etc.) needs to perform blind source separation. Optionally, the blind source separation module can also be set up within some APPs, as a module within the APP. For example, the blind source separation module can be a module within a note-taking APP; another example is that the blind source separation module can be a module within a voice assistant APP.

[0098] Optionally, the blind source separation module may include a single speech separation model, a signal processing unit, and a target adapter.

[0099] The input to the backdiffusion process of the single-speech separation model is an audio signal, and the output is the result of separating the audio signal, called the backdiffusion output signal. Half of the backdiffusion output signal is a single speech signal, and the other half is a mixed residual signal. The signal processing unit is used to split the backdiffusion output signal into two equal-length segments, obtaining a single speech signal and a mixed residual signal. The single speech signal represents the speech source signal separated from the mixed signal by the single-speech separation model, while the mixed residual signal represents the signal remaining after separating the single speech signal from the mixed signal.

[0100] The signal processing unit is also used to determine whether the data of a single voice signal is empty. If the data of a single voice signal is not empty, the remaining mixed signal is input to the target adapter.

[0101] The target adapter is used to repair the mixed residual signal, making it closer to the ideal mixed residual signal and reducing its deviation, thus resulting in more accurate signal separation. The repaired mixed residual signal output by the target adapter is copied and spliced ​​by the signal processing unit and then input again into the single-speech separation model for separation. This process is repeated to separate individual speech signals from the mixed signal.

[0102] As an optional implementation, the application layer may also include a model training module. This module trains the initial diffusion model to obtain the aforementioned single-speech separation model, and trains the initial adapter to obtain the aforementioned target adapter. The specific structure of the model training module can be similar to that of the blind source separation module.

[0103] Optionally, in other embodiments, the electronic device may not include a model training module. Instead, the single-speech separation model and target adapter are trained on other devices, and then imported into the electronic device. This application does not limit this aspect.

[0104] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0105] like Figure 5 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0106] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0107] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, phone calls made and received, browsing history and bookmarks, phone books, and more.

[0108] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0109] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0110] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0111] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0112] The Android runtime consists of core libraries and a virtual machine. The Android runtime is responsible for scheduling and managing the Android system.

[0113] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0114] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0115] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0116] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0117] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0118] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0119] A 2D graphics engine is a graphics engine for 2D drawing.

[0120] The kernel layer is the layer between hardware and software. It contains at least display drivers, camera drivers, audio drivers, and sensor drivers. The audio driver can include audio input drivers and audio output drivers. The audio input driver is also called the microphone driver. The microphone driver is used to identify and drive the microphone, transmitting the microphone input signal to the upper layer. The microphone resides in the hardware layer of the electronic device. Figure 5 Not shown in the image.

[0121] The following embodiments of this application will be used to illustrate having Figure 4 and Figure 5 Taking the electronic device with the structure shown as an example, the blind source separation method provided in this application embodiment will be specifically described in conjunction with the accompanying drawings and application scenarios.

[0122] To facilitate understanding, we will first introduce the diffusion model.

[0123] The diffusion model is a type of deep neural network (DNN).

[0124] Alternatively, the diffusion model framework can adopt a denoising diffusion probabilistic model (DDPM), a score-based generative model (SGM), or a stochastic differential equation (SDE), etc.

[0125] Optionally, the model structure of the diffusion model can be a convolutional neural network, such as a U-NET, a gradient-based noise conditional score network (NCSN), or an upgraded version of NCSN (NCSN++).

[0126] A diffusion model is a generative model that, given independent and identically distributed sample data, learns to approximate an unknown data distribution. The sample data originates from location data distributions. The use of diffusion models involves forward diffusion and reverse diffusion processes.

[0127] Forward diffusion can also be called forward diffusion or diffusion process. Backward diffusion can also be called inference process, reverse diffusion process, abstraction process, or denoising process. In forward diffusion, Gaussian noise is added to the data step by step in an orderly manner to generate a Markov chain of the data. In backward diffusion, noise is removed from a noisy data point step by step to generate a Markov chain of the noisy data.

[0128] A Markov chain consists of a series of states and a series of change probabilities. Here, a state refers to data with different noise levels, and a change probability refers to the probability of changing from the current state to the next state, which is implemented using a change matrix.

[0129] For example, such as Figure 6 As shown in Figure (a), for data x0, Gaussian noise is gradually added during the forward diffusion process. The data obtained at step t-1 is x. t-1 In the t-th step of the T steps (total steps) of the forward diffusion process, data x is... t-1 Add a small amount of Gaussian noise to obtain data x. t Data x t Let represent the data obtained after t steps. Here, t = 1, 2, ..., T, where T is a positive integer. That is, for data x0, after T steps, the data obtained in each step are x1, x2, ..., xt, respectively. TIn the forward diffusion process, the parameter t can represent the number of steps to add Gaussian noise, or the number of iterations.

[0130] like Figure 6 As shown in Figure (a), the reverse diffusion process is the opposite of the forward diffusion process, gradually removing noise. In the reverse diffusion process, the parameter t can be understood as the remaining number of iterations. For data x... T After T steps, the data obtained in each step are x. T x T-1 , ..., x1.

[0131] like Figure 6 As shown in Figure (b), taking image data as an example, during the forward diffusion process, Gaussian noise is gradually added to the image data, causing the image to gradually blur. During the reverse diffusion process, the noise gradually decreases, and the image gradually becomes clearer until the original image data is restored.

[0132] like Figure 6 As shown in Figure (c), taking audio data as an example, during the forward diffusion process, Gaussian noise is gradually added to the clean audio data, and the noise in the audio data gradually increases. During the reverse diffusion process, the noise in the audio gradually decreases until the original clean audio data is restored.

[0133] The forward diffusion process can be represented by formula (1):

[0134] q(x t |x t-1 ,y) (1)

[0135] Where q(x) t |x t-1 ,y) represents the expression of data x0 after the t-th iteration, and y represents the conditional input.

[0136] It can be understood that the data x0 after the tth iteration can be directly calculated from the data x0, as shown in the following formula (2):

[0137] q(x t |x t-1 ,y)=(x t |x0,y)=N c (x t ,μ(x0,y,t),σ(t) 2 I) (2)

[0138] Where, N c Let μ(x0,y,t) represent the Gaussian distribution, μ(x0,y,t) represent the mean of the Gaussian distribution, and σ(t) represent the standard deviation of the Gaussian distribution. 2 Let I represent the variance of the Gaussian distribution, and let I represent the identity matrix.

[0139] The mean value μ(x0,y,t) of the Gaussian distribution can be determined based on the condition y, the parameter t used to represent the number of iterations, and the normalization constant γ. The mean value μ(x0,y,t) of the Gaussian distribution can be expressed as formula (3):

[0140] μ(x0,y,t)=e -γt x0+(1-e -γt )y(3)

[0141] The standard deviation σ(t) of the Gaussian distribution can be determined based on the parameter t used to represent the number of iterations, the normalization constant γ, and the preset maximum variance σ. max and the preset minimum variance σ min Confirmed. The square of the standard deviation σ(t) of a Gaussian distribution is the variance of the Gaussian distribution. 2 This can be expressed as formula (4):

[0142]

[0143] According to data x t The expression at time t can determine the data x. t .

[0144] The forward diffusion process can be understood as the training process of the diffusion model. The initial diffusion model is used to process training samples with parameters t = 1, 2, ..., T to obtain denoised training data. Then, the denoised training data is compared with the data x. t-1 The differences between the data are used to adjust the parameters of the initial diffusion model. The training samples for the initial diffusion model include data x. t The diffusion model is the initial diffusion model after parameter adjustment.

[0145] The training data for denoising can be the output of the initial diffusion model, or it can be data x. t It is obtained by removing noise from the output representation of the initial diffusion model.

[0146] The training data for denoising is data x. t Based on the noise removal from the output representation of the initial diffusion model, and then based on the training denoised data and data x... t-1 The difference between the parameters of the initial diffusion model and the data x can be understood as the difference between the output of the initial diffusion model and the data x. t relative data x t-1 The parameters of the initial diffusion model are adjusted based on the differences between the added noise.

[0147] Based on the output of the initial diffusion model and the data x when t = 1, 2, ..., T respectively... t-1The difference between them is used to adjust the parameters of the initial diffusion model, which can be expressed as formula (5):

[0148]

[0149] Among them, s θ This represents the output of the initial diffusion model. Represents data x t-1 Compared to data x t The noise added, arg θ Let || be a preset constant, and || denote the norm. It represents the expectation of a plurality of t for the case t = 1, 2, ..., T. Norms are commonly used to measure the length or size of each vector in a vector space (or matrix).

[0150] The reverse diffusion process can be understood as the reasoning process of the diffusion model. The reverse diffusion process can be expressed as formula (6):

[0151] p θ (x t-1 |x t ,y)(6)

[0152] Based on the above formulas for representing the forward and reverse diffusion processes, the data x0 can be represented as q(x0), and the data x obtained after T iterations can be... T Represented as p θ (x T ), where p θ (x T ) = N c (0, I).

[0153] As a high-quality generative model, the diffusion model generates high-quality speech signals with a good listening experience. Moreover, the diffusion model has good robustness.

[0154] The blind source separation method and model training method provided in the embodiments of this application will be described next. The model training method is the training process of the single-speech separation model and the target adapter, and the blind source separation method is the usage process of the single-speech separation model and the target adapter. For ease of distinction, for some terms involved in the model training process, such as mixed signal, mixed residual signal, and single speech signal, the word "sample" can be added before, after, or in the middle of the name to indicate that it is data from the model training process. It should be understood that these sample data can be essentially the same as the non-sample data corresponding to the model usage process.

[0155] First, let's introduce the model training process.

[0156] The model training process in this embodiment includes two stages: pretraining and refinement training. Pretraining is mainly used to train the single-speech separation model, while refinement training is mainly used to train the target adapter based on the trained single-speech separation model. These will be described separately below.

[0157] 1. Pre-training

[0158] For example, Figure 7 This is a schematic diagram illustrating the principle of training a single-speech separation model as provided in an embodiment of this application. Figure 8 Please refer to the flowchart of an example model training method provided in this application embodiment. Figure 7 and Figure 8 In the pre-training phase, the method includes the following steps S101 to S105. The entity performing these steps can be... Figure 5 The model training module is described in detail in the steps.

[0159] S101. Construct a speech sample library, which includes multiple speech sample groups. Each speech sample group includes: a mixed sample signal (also known as a mixed sample), a target single sample signal (also known as a target single sample), and a mixed residual sample signal (also known as a mixed residual sample).

[0160] In the same group of speech samples, the mixed sample signal is a mixture of k single sample signals (i.e., source signals in the samples), where k is an integer greater than or equal to 2; the target single sample signal is one of the k single sample signals; and the mixed residual sample signal is a mixture of k-1 single sample signals excluding the target single sample signal, i.e., the signal remaining after separating the single sample signals from the mixed sample signal. Optionally, the mixed residual sample signal can be obtained by subtracting the target single sample signal from the mixed sample signal.

[0161] In this embodiment, each segment of single-sample signal is a speech signal; therefore, the mixed-sample signal, the target single-sample signal, and the mixed-remaining-sample signal are also speech signals. The mixed-sample signal can be represented as mix samplesignal, the target single-sample signal can be represented as goal singlesample signal, and the mixed-remaining-sample signal can be represented as mix-single sample signal.

[0162] Optionally, all k single-sample signals can be clean speech signals; therefore, a single-sample signal can also be called a single clean signal or clean speech. Thus, the mixed-sample signal and the mixed residual sample signal are also clean speech signals. The single-speech separation model trained based on the clean mixed-sample signal, the target single-sample signal, and the mixed residual sample signal also produces a clean speech signal, improving the quality and listening experience of the separation results and enhancing the user experience.

[0163] Optionally, the mixed sample signal, the target single sample signal, and the mixed residual sample signal have equal lengths, and their lengths can be represented as len. The dimensions of the three signals can be represented as [len, 1]. It should be noted that in this embodiment, the length of the audio signal refers to the duration of the signal.

[0164] Optionally, when constructing the speech sample library, the number k of individual sample signals constituting the mixed signal in each speech sample group is randomized, each segment of individual sample signal is randomized, and the target individual sample signal is randomized. In this way, the signals in each speech sample group in the speech sample library are randomized, thereby giving the single speech separation model trained based on these speech signals stronger robustness and generalization performance.

[0165] Here, some sample speech data are provided as an example. Figure 9 This is a schematic diagram of an example speech sample group provided in an embodiment of this application, as shown below. Figure 9 As shown, suppose we have four clean single-sample signals: single-sample signal 1 (also represented as clean speech1), single-sample signal 2 (also represented as clean speech2), single-sample signal 3 (also represented as clean speech3), and single-sample signal 4 (also represented as clean speech4). Based on these four single-sample signals, multiple sets of speech samples can be constructed.

[0166] Figure 9As shown, when k=4, speech sample groups 1 to 4 can be constructed. In speech sample groups 1 to 4, the mixed sample signal is a mixture of single sample signals 1 to 4 (shown as mixed sample signal 1 in the figure), the target single sample signals are single sample signals 1 to 4 (shown as target single sample signals 1 to target single sample signals 4 in the figure), and the corresponding mixed residual sample signals are mixtures of the other signals from the four single sample signals excluding the target single sample signal (shown as mixed residual sample signals 1 to mixed residual sample signals 4 in the figure). For example, in speech sample group 1, the target single sample signal is single sample signal 1, and the corresponding mixed residual sample signal is a mixture of single sample signals 2, 3, and 4.

[0167] When k=3, it is possible to construct A group of speech samples. For example, Figure 9 Speech sample groups 5 to 10 are partial speech sample groups when k=3. Specifically, in speech sample groups 5 to 7, the mixed sample signal is a mixture of single sample signals 1 to 3 (shown as mixed sample signal 2 in the figure), the target single sample signals are single sample signals 1 to 3 (shown as target single sample signals 1 to 3 in the figure), and the corresponding mixed residual sample signals are mixtures of the other signals from single sample signals 1 to 3, excluding the target single sample signal (shown as mixed residual sample signals 5 to 7 in the figure). For example, in speech sample group 5, the target single sample signal is a single sample signal, and the corresponding mixed residual sample signal is a mixture of single sample signals 2 and 3.

[0168] In speech sample groups 8 to 10, the mixed sample signal is a mixture of single sample signals 2 to 4 (shown as mixed sample signal 3 in the figure), and the target single sample signals are single sample signals 2 to 4 (shown as target single sample signals 2 to 4 in the figure). The corresponding mixed residual sample signals are the mixtures of the other single sample signals from single sample signals 2 to 4, excluding the target single sample signal (shown as mixed residual sample signals 8 to 10 in the figure). Similarly, when k=3, based on single sample signals 1 to 4, 6 more speech sample groups can be constructed, which will not be described in detail here.

[0169] When k=2, it is possible to construct A group of speech samples. For example, Figure 9Speech sample groups 11 to 14 are partial speech sample groups when k=2. In speech sample groups 11 and 12, the mixed sample signal is a mixture of single sample signal 1 and single sample signal 2 (shown as mixed sample signal 4 in the figure). The target single sample signal in speech sample group 11 is single sample signal 1 (shown as target single sample signal 1 in the figure), and the mixed residual sample signal is single sample signal 2 (shown as mixed residual sample signal 11 in the figure). The target single sample signal in speech sample group 12 is single sample signal 2 (shown as target single sample signal 2 in the figure), and the mixed residual sample signal is single sample signal 1 (shown as mixed residual sample signal 12 in the figure).

[0170] In speech sample groups 13 and 14, the mixed sample signal is a mixture of single sample signal 2 and single sample signal 3 (shown as mixed sample signal 4 in the figure). The target single sample signal in speech sample group 13 is single sample signal 2 (shown as target single sample signal 2 in the figure), and the mixed residual sample signal is single sample signal 3 (shown as mixed residual sample signal 13 in the figure). Similarly, in speech sample group 14, the target single sample signal is single sample signal 3 (shown as target single sample signal 3 in the figure), and the mixed residual sample signal is single sample signal 2 (shown as mixed residual sample signal 14 in the figure). And so on. When k=2, based on single sample signals 1 to 4, eight more speech sample groups can be constructed, which will not be described in detail here.

[0171] Thus, based on single sample signals 1 to 4, 4 + 12 + 12 = 28 groups of speech samples can be constructed. It can be understood that, following this method, more groups of speech samples can be constructed by changing or adding single sample signals, thereby forming a speech sample library.

[0172] It should be noted that, Figure 9 This is merely an illustrative example of a speech sample group and does not constitute any limitation on the structure, storage format, etc., of the speech sample group. In practical applications, speech sample groups can be stored in various forms such as tables, arrays, and sets, without limitation. For example, mixed sample signals can be stored in one list, target single sample signals can be stored in another list, and mixed remaining sample signals can be stored in yet another list, establishing a correspondence between each mixed sample signal and the target single sample signal and mixed remaining sample signals.

[0173] Furthermore, this application does not limit the number of speech sample groups. It is understood that the more speech sample groups there are, the easier it is for the model to converge during training, and the higher the accuracy of the obtained model.

[0174] After constructing the speech sample library, for each group of speech samples in the speech sample library, the following steps S102 to S105 can be executed to train the initial diffusion model until the model converges or the number of iterations reaches the preset number, at which point training stops and a single speech separation model is obtained.

[0175] The following explanation uses any speech sample group A as an example. Speech sample group A includes a mixed sample signal a, a target single sample signal a, and a mixed remaining sample signal a.

[0176] S102. For any group of speech samples A, the target single sample signal a and the mixed remaining sample signal a in speech sample group A are concatenated to obtain the positive input signal a.

[0177] In this embodiment, splicing two audio signals refers to splicing the two speech signals in the time domain. That is, in the time domain, the end of the first speech signal is connected to the beginning of the second speech signal to form a new, continuous audio signal. The frequency domain of the two audio signals does not need to be processed. The length of the spliced ​​audio signal is the sum of the lengths of the two audio signals.

[0178] Specifically, in this step, the length of both the target single sample signal a and the mixed residual sample signal a is len, and the dimension is [len, 1]. Concatenating the target single sample signal a with the mixed residual sample signal a yields a positive input signal a with a length of len*2 and a dimension of [len*2, 1].

[0179] It should be noted that the spliced ​​positive input signal 'a' can be either the target single sample signal 'a' in the first half and the mixed remaining sample signal 'a' in the second half, or the mixed remaining sample signal 'a' in the first half and the target single sample signal 'a' in the second half. There is no limitation on this, as long as the structure of all positive input signals is consistent.

[0180] Furthermore, the positions of the target single-sample signal and the mixed residual signal in the forward input signal during pre-training determine the positions of the single signal and the mixed residual signal in the output signal during model usage. Specifically: Case 1: If the target single-sample signal is located in the first half of the forward input signal and the mixed residual signal is located in the second half, then during model usage, the first half of the output signal is a single signal, and the second half is a mixed residual signal. Case 2: If the mixed residual signal is located in the first half of the forward input signal and the target single-sample signal is located in the second half, then during model usage, the first half of the output signal is a mixed residual signal, and the second half is a single signal. This application's embodiments all use Case 1 as an example for illustration.

[0181] S103. Input the positive input signal a into the initial diffusion model. The initial diffusion model performs the positive diffusion process and outputs the positive output signal a.

[0182] Specifically, the positive input signal a is used as input to the initial diffusion model. The initial diffusion model then performs the following positive diffusion process: the positive input signal a is used as the starting point of the positive diffusion process, i.e., the positive input signal a is used as data x0. Referring to the above formulas (1) to (4), Gaussian noise is added to the current speech signal at each step t to simulate the mixing process of the target single sample signal a and the mixed residual sample signal a. As the number of iterations increases, the target single sample signal a gradually mixes with the mixed residual sample signal a. After T iterations, a mixed signal (i.e., data x0) is finally output. T The length of the mixed signal is len*2, and the dimension is [len*2, 1]. In this embodiment, the signal output after the initial diffusion model performs a forward diffusion process is called the forward output signal a. The first half (length len) and the second half (length len) of the forward output signal a are the same, both being a mixed signal obtained by gradually mixing the target single sample signal a predicted by the initial diffusion model with the mixed remaining sample signal a. In this embodiment, it is called the mixed prediction signal a.

[0183] S104. Copy the mixed sample signal a in the speech sample group A, and then splice the two mixed sample signals a to obtain the spliced ​​sample signal (also known as the spliced ​​sample) a.

[0184] In this embodiment of the application, copying a certain segment of signal means copying the signal in the time domain to obtain a segment of signal with the same time length and frequency domain characteristics as the original signal.

[0185] In this step, the mixed sample signal a has a length of len and a dimension of [len, 1]. After copying the mixed sample signal a, a new mixed sample signal a is obtained. The new mixed sample signal a is then concatenated with the original mixed sample signal a to obtain a signal with a length of len*2 and a dimension of [len*2, 1]. This signal is called the concatenated sample signal a.

[0186] S105. Adjust the parameters of the initial diffusion model based on the loss between the positive output signal a and the spliced ​​sample signal a.

[0187] Optionally, a loss function for the initial diffusion model can be preset. The loss function measures the loss (i.e., difference) between the output signal of the initial diffusion model and the target label during the diffusion process. Specifically, in this step, the target label is the spliced ​​sample signal a. The positive output signal a and the spliced ​​sample signal a are substituted into the preset loss function to calculate the loss value. Based on the magnitude of the loss value, the parameters of the initial diffusion model are adjusted according to formula (5), such as the weights of the neural network, to minimize the loss. The process of adjusting the parameters of the initial diffusion model to minimize the loss is also the process of the model's parameters gradually approaching the target value. This process can be understood as the model learning its ability to separate a single speech signal.

[0188] As an alternative implementation, steps S104 and S105 can also be replaced by: splitting the positive output signal a into two equal-length mixed prediction signals a, using the mixed sample signal a in the speech sample group A as the target label, calculating the loss between the mixed prediction signal a and the mixed sample signal a, and adjusting the parameters of the initial diffusion model based on the loss.

[0189] In general, steps S103 to S105 involve taking the positive input signal a as the starting point, taking the result of copying and splicing the mixed sample signal a (i.e., the spliced ​​sample signal a) as the ending point (i.e., the target label for training), performing a positive diffusion process, training the initial diffusion model, and enabling the model to learn how to gradually separate a single speech signal from the mixed signal.

[0190] It is understandable that, based on each group of speech samples in the speech sample library, steps S102 to S105 are repeated until the loss is less than a preset threshold, i.e., the model converges, or the number of training iterations reaches a preset number, at which point training stops. The resulting diffusion model is called a single-speech separation model.

[0191] In summary, during model training, the target single-sample signal and the mixed residual signal are concatenated (which can be understood as a separated signal) to form the target domain signal of the diffusion model, or target distribution. The diffusion model performs a forward diffusion process, outputting a mixed prediction signal, called the mixed distribution. In other words, the initial diffusion model performs a forward diffusion process to transform the target distribution into a mixed distribution and the separated signal into a mixed signal. This process utilizes the target domain modeling capability of the diffusion model, enabling the model to learn the ability to transform the mixed signal into a separated signal, that is, to learn the ability to separate a single speech signal from the mixed distribution, thus obtaining a single-speech separation model with single-speech separation capability. Therefore, in use, the mixed signal is input into the single-speech separation model, which performs a reverse diffusion process to separate the single speech signal from the mixed signal and output the mixed residual signal. The mixed residual signal is then input into the single-speech separation model again for repeated separation, thus achieving blind source separation.

[0192] 2. Improve training

[0193] It is understandable that a pre-trained single-speech separation model can achieve blind source separation. However, considering that the separation effect of a single-speech separation model is not perfect, the resulting mixed residual signal will deviate from the ideal mixed residual signal (or the real mixed residual signal), and this deviation will iteratively accumulate and affect the subsequent separation effect. Based on this, in this embodiment, an adapter is further designed and trained based on the single-speech separation model. The target adapter obtained through training repairs the mixed residual signal, reduces the deviation of the mixed residual signal, thereby reducing the accumulation of deviation and improving the speech separation effect.

[0194] The adapter can be trained based on mixed sample signals and single sample signals. Optionally, it can continue to use signals from the speech sample library in the above embodiments for training, or it can rebuild a speech sample library. In this embodiment, the adapter is trained by continuing to sample signals from the above speech sample library as an example.

[0195] For example, Figure 10 This is a schematic diagram illustrating the principle of an example training target adapter provided in an embodiment of this application. Figure 11 Please refer to the flowchart of another model training method provided in this application embodiment. Figure 10 and Figure 11 In the improved training phase, the method includes the following steps S201 to S208. The entity performing these steps can be... Figure 5 The model training module is described in detail in the steps.

[0196] It is understandable that during adapter training, the more signals used, the easier it is for the model to converge, and the higher the accuracy of the trained target adapter. In this embodiment, mixed sample signals can be obtained from the speech sample library in a certain order or randomly, and steps S201 to S208 can be executed to train the adapter based on the obtained mixed sample signals. Then, the next mixed sample signal is obtained from the speech sample library, and steps S201 to S208 are repeated to continue training the adapter based on the next mixed sample signal. This process is repeated until all mixed sample signals in the speech sample library are traversed, or the model converges, or the number of training iterations reaches a preset threshold. In the following embodiment, the method of training an adapter based on any mixed sample signal i in the speech sample library is used as an example. The method includes:

[0197] S201. Copy the mixed sample signal i, and then splice the two mixed sample signals i together to obtain the spliced ​​sample signal i.

[0198] The implementation process of this step can refer to step S104 in the above embodiment, and will not be repeated here.

[0199] S202. Input the spliced ​​sample signal i into the single speech separation model. The single speech separation model performs a back diffusion process and outputs the separation prediction signal i.

[0200] The back-diffusion process of the single-speech separation model starts with the audio signal to be separated and ends (outputs) with an audio signal containing a single speech signal and a mixed residual signal. In this embodiment, during the improved training phase, the back-diffusion process of the single-speech separation model starts with the spliced ​​sample signal and outputs the result of separating the spliced ​​sample, i.e., an audio signal containing a single speech signal and a mixed residual signal. For ease of distinction, the output of the single-speech separation model during the improved training phase is called the separation prediction signal, the single speech signal in the separation prediction signal is called the single prediction signal, and the mixed residual signal in the separation prediction signal is called the mixed residual prediction signal. During the model usage phase, the output of the single-speech separation model is called the back-diffusion output signal.

[0201] As described in the above embodiment, the length of the spliced ​​sample signal i is len*2, and the dimension is [len*2, 1]. Therefore, the length of the separated prediction signal i is also len*2, and the dimension is also [len*2, 1]. The first half of the separated prediction signal i is the single prediction signal i, and the dimension can be represented as [0:len, 1]. The second half of the separated prediction signal i is the mixed residual prediction i, and the dimension can be represented as [len+1:len*2, 1].

[0202] S203. The separated prediction signal i is split into two speech signals of equal length to obtain a single prediction signal i and a mixed residual prediction signal i.

[0203] S204. Determine whether the data of the single prediction signal i is empty; if it is empty, obtain the next mixed sample signal m and use the next mixed sample signal m as the mixed sample signal i, and return to step S201; if it is not empty, execute step S205.

[0204] It can be understood that the mixed sample signal i is data in the database, and the mixed sample signal i is a mixture of k segments of single sample signals, where k is greater than or equal to 2. Therefore, the data of the single prediction signal i obtained after the mixed sample signal i is first separated by the single speech separation model is not empty. However, in this embodiment, the output mixed remaining prediction signal will be input into the single speech separation model again for iterative separation. Therefore, when the number of separations is equal to k+1, the single prediction signal i output by the single speech separation model is empty, indicating that the separation of the mixed sample signal i has been completed and separation needs to be stopped. Based on this, in this embodiment, after each signal separation by the single speech separation model, it is determined whether the obtained single prediction signal i is empty. If it is empty, it means that the mixed sample signal i has been separated, so the training process based on the mixed sample signal i ends, the next mixed sample signal m is obtained from the speech sample library, the next mixed sample signal m is used as the mixed sample signal i, and the process returns to step S201 to train the adapter based on the next mixed sample signal. If it is not empty, the process continues to execute step S205 and subsequent steps to continue training the adapter and continue separating signals.

[0205] It should be noted that when the number of separations equals k, the single prediction signal i is not empty, but the remaining prediction signals i are empty. In this case, the steps of training the adapter and separating the signals are still performed, but the signals input to the initial adapter and the single speech separation model are empty data.

[0206] Of course, in other embodiments, it is also possible to detect whether the data of the mixed residual prediction signal i is empty. If it is empty, the next mixed sample signal m is obtained and used as the mixed sample signal i, and the process returns to step S201; if it is not empty, step S205 is executed. In this way, after the number of separations equals k, the separation of the mixed residual prediction signal i is stopped, reducing the algorithm process and improving the algorithm's running efficiency.

[0207] S205. Calculate the similarity between the single predicted signal i and each target single sample signal in the speech sample library, and determine the target single sample signal x that is most similar to the single predicted signal i based on the similarity.

[0208] Optionally, the similarity can be calculated using the cosine similarity method. Specifically, the mel-frequency cepstral coefficients (MFCC) feature vector of the single predicted signal i is extracted to convert the audio signal into a feature vector; similarly, the MFCC feature vector of the target single sample signal is extracted; then, the cosine value between the two MFCC feature vectors is calculated to obtain the similarity between the target single sample signal and the single predicted signal i. The similarity value ranges from [-1, 1], and the closer the similarity is to 1, the more similar the two are. In this case, the one with the highest similarity to the single predicted signal i can be taken as the target single sample signal most similar to the single predicted signal i. In this embodiment, the target single sample signal most similar to the single predicted signal i is denoted as the target single sample signal x.

[0209] Of course, other similarity algorithms can also be used to calculate the similarity and determine the range of similarity values ​​corresponding to the similarity algorithm to identify the target single sample signal x.

[0210] S206. Determine the corresponding mixed residual target signal x based on the mixed sample signal i and the target single sample signal x.

[0211] The mixed residual target signal x represents the signal remaining after removing the target single sample signal x from the mixed sample signal i. Optionally, the mixed residual target signal x can be obtained by subtracting the target single sample signal x from the mixed sample signal i. Of course, other methods can also be used to determine the mixed residual target signal x, such as those based on short-time Fourier transform or wavelet transform, etc., and there is no limitation on this.

[0212] The target single-sample signal x and the mixed residual target signal x can be understood as the output of the single-speech separation model after separating the mixed sample signal i, under ideal conditions. Therefore, the initial adapter, trained using the mixed residual target signal as the training target, can learn to correct the biased signal actually output by the single-speech separation model to the ideal signal by mixing the residual predicted signal, thus reducing the bias of the mixed residual signal.

[0213] S207. Input the mixed residual prediction signal i into the initial adapter. The initial adapter repairs the mixed residual prediction signal i to obtain the repaired mixed residual prediction signal i.

[0214] Optionally, the initial adapter can be implemented based on a neural network (NN), and the network structure includes, but is not limited to, DNN, convolutional neural network (CNN), long short-term memory network (LSTM), etc.

[0215] S208. Adjust the parameters of the initial adapter based on the loss between the repaired hybrid residual prediction signal i and the hybrid residual target signal x.

[0216] Optionally, an initial adapter loss function can be pre-set. Optionally, the initial adapter loss function can be a weighted combination of one or more of the following: amplitude loss, phase loss, and multi-resolution short-time Fourier transform loss (STFT loss).

[0217] This completes one training iteration of the initial adapter. In other words, training the initial adapter involves using the mixed residual prediction signal *i* output by the single speech separation model as input and the mixed residual target signal *x* as the target label, to train the initial adapter to learn the ability to repair the mixed residual prediction signal into a mixed residual signal that infinitely approximates the ideal signal.

[0218] Then, the repaired mixed remaining prediction signal i is used as the mixed sample signal i, and the process returns to step S201 to continue separating other single speech signals in the mixed sample signal i and continue training the initial adapter.

[0219] To elaborate, see Figure 10 The process of separating other individual speech signals from the mixed sample signal i, and the subsequent training process, may include the following steps. For ease of distinction, the relevant signals have been renamed.

[0220] 1) Copy the repaired mixed residual prediction sample signal i, then concatenate the two mixed residual prediction sample signals i, and input the concatenation result into the single speech separation model.

[0221] 2) The single speech separation model performs a back diffusion process and outputs a separation prediction signal j.

[0222] 3) The separate prediction signal j is split into two speech signals of equal length to obtain a single prediction signal i and a mixed residual prediction signal j.

[0223] 4) Determine whether the data of the single prediction signal j is empty; if it is empty, obtain the next mixed sample signal m and use the next mixed sample signal m as the mixed sample signal i, and return to step S201; if it is not empty, calculate the similarity between the single prediction signal j and each target single sample signal in the speech sample library, and determine the target single sample signal y that is most similar to the single prediction signal j in the speech sample library based on the similarity.

[0224] 5) Determine the corresponding mixed residual target signal y based on the repaired mixed residual prediction sample signal i and the target single sample signal x.

[0225] 6) Input the mixed residual prediction signal j into the initial adapter. The initial adapter repairs the mixed residual prediction signal j to obtain the repaired mixed residual prediction signal j.

[0226] It should be noted that the initial adapter here is the adapter with adjusted parameters from the previous training round.

[0227] 7) Adjust the parameters of the initial adapter based on the loss between the repaired mixed residual prediction signal j and the mixed residual target signal y.

[0228] This process is repeated until, after k+1 separations, the single predicted signal output by the single speech separation model is detected to be empty, at which point the loop ends.

[0229] It should be noted that when performing the k-th separation based on the mixed sample signal i, the signal input to the single speech separation model actually contains only one type of single sample signal. In this case, the data of the mixed residual prediction signal output by the single speech separation model is empty, so the data of the signal input to the initial adapter is also empty, and the data of the repaired mixed residual prediction signal output by the initial adapter is also empty. However, in this case, the data of the mixed residual target signal determined based on the mixed sample signal and the single prediction signal is also almost empty, that is, the target label of the initial adapter is almost empty. Therefore, the parameters of the initial adapter will be fine-tuned or not adjusted at all. It can be seen that even when the mixed residual prediction signal is empty, it will not affect the adjustment of the parameters of the initial adapter, and will not affect the accuracy of the final target adapter.

[0230] Repeat steps S201 to S208 above to repeatedly train the initial adapter based on the separation of different mixed sample signals until the model converges or the number of training iterations reaches a preset threshold. The resulting adapter is called the target adapter. The target adapter has the ability to repair the mixed residual signal, and can repair the mixed residual signal to be infinitely close to the ideal mixed residual signal.

[0231] The model training method provided in this application, in the pre-training stage, trains an initial diffusion model to obtain a robust single-speech separation model with high-fidelity output. In the improvement training stage, based on the single-speech separation model, a target adapter with high robustness and accurate output is trained. The combined use of the single-speech separation model and the target adapter can improve the separation effect of blind source separation, reduce the deviation between the mixed residual signal and the signal under ideal conditions, reduce the accumulation of separation deviation in iterative cycles, and improve the accuracy of signal separation.

[0232] Next, the process of blind source separation using a single speech separation model and a target adapter will be explained.

[0233] For example, Figure 12 This is a schematic diagram illustrating the backdiffusion principle of an example single-speech separation model provided in an embodiment of this application. Figure 13 This is a flowchart illustrating an example of a blind source separation method provided in an embodiment of this application. Please refer to [the flowchart / document] as well. Figure 12 and Figure 13 ,by Figure 1 Taking the application scenario shown as an example, the method includes:

[0234] S301, In response to the user entering a recording command in the note-taking interface of the note-taking app, the note-taking app at the application layer sends a recording command to the microphone at the hardware layer of the electronic device.

[0235] Specifically, the recording command is used to indicate the start of recording. Optional, such as... Figure 1 As shown in Figure (c), users can input a recording start command by clicking the recording control 105 in the note interface 104.

[0236] S302, The microphone responds to the recording command and captures the recording input by the user (also known as the seventh audio signal).

[0237] S303: The microphone sends the recording to the Notes app.

[0238] It's understandable that the recording captured by the microphone is essentially an audio stream, and the microphone can send the audio stream segment by segment to the note-taking app.

[0239] S304. When the speech-to-text function and the speaker display function are both enabled, the Notes App will use the recording as a mixed signal (also known as the first audio signal) and input it into the single speech separation model (also known as the first model) in the blind source separation module.

[0240] A mixed signal is the audio signal that is to be blindly separated. A mixed signal can be represented as a mix signal.

[0241] It is understandable that if the note-taking app confirms that neither the speech-to-text function nor the speaker display function is enabled, step S304 and subsequent steps can be skipped, and the recording can be saved directly. If the speech-to-text function is enabled but the speaker display function is disabled, step S304 and subsequent steps can also be skipped, and the recording can be saved, with the speech-to-text process executed on the recording. If the note-taking app confirms that both the speech-to-text function and the speaker display function are enabled, step S304 and subsequent steps are executed to perform blind source separation.

[0242] It should be noted that in this step, the mixed signal is essentially the recording. This is used to indicate that the recording was made with both the speech-to-text function and the speaker display function enabled by the note-taking app, and also to facilitate the explanation of the solution in conjunction with the model. Of course, in some embodiments, the note-taking app may first process the recording to generate the mixed signal, and then input the generated mixed signal into the blind source separation module.

[0243] Furthermore, the term "mixed signal" here is merely a signal name used to distinguish signals, not to limit the signal to necessarily being a mixed signal. In essence, the signal to be separated can be a single signal, i.e., containing only one source signal, or a mixed signal, i.e., containing multiple source signals.

[0244] S305, the single-speech separation model performs a reverse diffusion process and outputs a reverse output signal (also known as a second audio signal).

[0245] The length of the inverse output signal is the same as the length of the input signal (i.e., the mixed signal) from the single-speech separation model. The first half of the inverse output signal is the single speech signal, and the second half is the mixed residual signal. The single speech signal is the speech source signal separated from the mixed signal by the single-speech separation model. The single speech signal can be represented as a singlespeech signal, or simply a single signal. The mixed residual signal is the signal remaining after removing the single speech signal from the mixed signal inferred by the single-speech separation model. The mixed residual signal can be represented as a mix-singlesignal.

[0246] S306, the single-voice separation model sends the reverse output signal to the signal processing unit.

[0247] S307 The signal processing unit splits the inverted output signal into two equal-length speech signals to obtain a single speech signal (also known as the third audio signal) and a mixed residual signal (also known as the fourth audio signal).

[0248] Among them, the single speech signal is the first half of the inverted output signal, and the mixed residual signal is the second half of the inverted output signal.

[0249] S308. The signal processing unit determines whether the data of a single voice signal is empty; if it is empty, the process ends; if it is not empty, step S309 is executed.

[0250] Specifically, if this separation is the first separation of a mixed signal, and the data for a single speech signal is empty, it means that there is no speech signal in the mixed signal, thus ending the blind source separation process. In this case, there is no need for speech-to-text processing, voiceprint recognition, or subsequent display on the interface.

[0251] If this separation is not the first separation of the mixed signal, and the data of the single speech signal is empty, it means that the separation of the mixed signal has been completed, and thus the blind source separation process ends.

[0252] If the data of a single speech signal is not empty, then the single speech signal is the speech source signal separated from the mixed signal. Step S311 and subsequent steps are executed to process the separated single speech signal by speech-to-text conversion, voiceprint recognition and display, and to continue to perform subsequent separation processing.

[0253] S309, the signal processing unit sends a single voice signal to the note-taking app.

[0254] The S310 and Notes app process a single voice signal into text.

[0255] S311 The Notes App performs voiceprint recognition on a single voice signal and determines the speaker corresponding to the single voice signal based on the voiceprint recognition result.

[0256] S312. The Notes App displays the text in the display area corresponding to the speaker in the interface.

[0257] See Figure 1 For example, the text obtained by converting a single voice signal to text is "A design scheme for a note lock needs to be designed". Furthermore, based on the voiceprint recognition result, if the speaker corresponding to the single voice signal is speaker 2, then the text will be displayed in the corresponding display area for speaker 2. Figure 1 The area below the "Speaker 2" control 110 in (e) of the diagram.

[0258] S313. Input the mixed residual signal into the target adapter.

[0259] S314. The target adapter repairs the mixed residual signal to obtain the repaired mixed residual signal (also known as the fifth audio signal).

[0260] S315, the target adapter sends the repaired mixed residual signal to the signal processing unit.

[0261] S316 The signal processing unit copies the repaired mixed residual signal and splices the two repaired mixed residual signals to obtain the spliced ​​signal (also known as the sixth audio signal).

[0262] S317 The signal processing unit uses the spliced ​​signal as a mixed signal and inputs it into the single speech separation model.

[0263] The single speech separation model uses the spliced ​​signal as a mixed signal and repeats the above steps S305 to S317.

[0264] It should be noted that steps S309 to S312 and steps S313 to S317 can be executed in a specific order or simultaneously.

[0265] Furthermore, the processes related to the single-speech separation model and the target adapter described above are essentially similar to the model training process. The difference lies in that this embodiment involves the usage of the single-speech separation model and the target adapter, thus eliminating the need to calculate loss or adjust model parameters. Processes similar to or identical to the training process can be found in the above embodiment and will not be repeated here.

[0266] The blind source separation method provided in this application has at least the following beneficial effects:

[0267] 1) This method uses a single-speech separation model to cyclically separate the mixed signal to obtain individual source signals. The single-speech separation model is trained based on a diffusion model, thus exhibiting strong robustness, and the separated source signals have high fidelity, high quality, and good listening experience.

[0268] 2) This method does not require phase estimation and spectral estimation, thus avoiding the influence of phase direction and post-mixing, and improving the accuracy and quality of the separation results.

[0269] 3) This method relies on the independence assumption. Regardless of whether the source signal satisfies the independence assumption, it can effectively extract the source signal, resulting in good signal separation. This method does not require calculation based on a multi-microphone array, and therefore does not rely on multi-microphone pickup. Audio obtained from single-microphone pickup can also be separated, making it highly applicable.

[0270] 4) This method uses a diffusion model to separate the source signal, making full use of the separation capability of the diffusion model to improve the separation quality, as well as the robustness and fidelity of the signal separation.

[0271] 5) This method has strong generalization ability and stable separation effect.

[0272] 6) In the cyclic separation process, this method repairs the mixed residual signal obtained from the previous separation through the target adapter, reduces the deviation between the mixed residual signal and the signal under ideal conditions, reduces the accumulation of deviation, thereby reducing the impact on subsequent signal separation, improving the accuracy of signal separation, and improving the fidelity of the separated speech signal.

[0273] The foregoing has detailed examples of the blind source separation method and model training method provided in the embodiments of this application. It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0274] This application embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into a separate functional module, such as a detection unit, a processing unit, a display unit, etc., or two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0275] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0276] The electronic device provided in this embodiment is used to execute the above-described blind source separation method and model training method, and thus can achieve the same effect as the above-described implementation method.

[0277] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0278] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a digital signal processor (DSP), and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0279] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device involved in this embodiment can be a device having... Figure 4 The device with the structure shown.

[0280] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the blind source separation method and model training method of any of the above embodiments.

[0281] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the blind source separation method and model training method in the above embodiments.

[0282] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to execute the blind source separation method and model training method in the above-described method embodiments.

[0283] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0284] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0285] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0286] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0287] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0288] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0289] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A blind source separation method, wherein the method is executed by an electronic device, characterized in that, The method includes: Acquire the first audio signal; The first audio signal is input into the first model; the first model is a model trained by inputting the forward input signal into the initial diffusion model to perform the forward diffusion process and using the first spliced ​​sample as the target label. The forward input signal is obtained by splicing the first target single sample and the first mixed sample. The first mixed sample is a mixed signal of k single speech signals, where k is an integer greater than or equal to 2. The first target single sample is one of the k single speech signals. The first model performs a reverse diffusion process to separate the first audio signal and output a second audio signal; The single speech signal in the first audio signal is determined based on the second audio signal.

2. The method according to claim 1, characterized in that, Determining a single speech signal in the first audio signal based on the second audio signal includes: The second audio signal is split into a third audio signal and a fourth audio signal; the third audio signal and the fourth audio signal have the same duration, the third audio signal is used to store data of a single speech signal separated from the first audio signal, and the fourth audio signal is used to store data of the remaining audio signal after the third audio signal is separated from the first audio signal; If the data of the third audio signal is not empty, then the third audio signal is determined to be a single voice signal in the first audio signal.

3. The method according to claim 2, characterized in that, The method further includes: If the data of the third audio signal is not empty, the target adapter is used to repair the fourth audio signal to obtain the fifth audio signal; The fifth audio signal is copied to obtain another fifth audio signal; The two fifth audio signals are spliced ​​together to obtain the sixth audio signal; The sixth audio signal is used as the first audio signal, and the execution step is returned: the first audio signal is input into the first model.

4. The method according to claim 3, characterized in that, The target adapter is obtained by training the initial adapter based on the first model.

5. The method according to claim 2, characterized in that, The third audio signal is the first half of the second audio signal, and the fourth audio signal is the second half of the second audio signal; Alternatively, the third audio signal may be the latter half of the second audio signal, and the fourth audio signal may be the first half of the second audio signal.

6. The method according to any one of claims 1 to 5, characterized in that, The acquisition of the first audio signal includes: Receive a first instruction input by the user through a first application, the first instruction being used to instruct recording; In response to the first instruction, record the seventh audio signal; When the first function of the first application is enabled, the seventh audio signal is used as the first audio signal; the first function is a function implemented based on a single voice signal.

7. The method according to claim 6, characterized in that, The first application is a note-taking application, and the first function includes the ability to display text in the speech according to the speaker.

8. A model training method, said method being performed by an electronic device, characterized in that, The method includes: Multiple sets of speech sample groups are acquired, wherein the first speech sample group includes a first mixed sample, a first target single sample, and a first mixed residual sample with equal time length; the first speech sample group is any one of the multiple sets of speech sample groups, the first mixed sample is a mixed signal of k single speech signals, where k is an integer greater than or equal to 2, the first target single sample is one of the k single speech signals, and the first mixed residual sample is the speech signal remaining after removing the first target single sample from the first mixed sample; Perform the first training process; The first training process includes: splicing the first target single sample and the first mixed sample to obtain a positive input signal; copying the first mixed sample to obtain another segment of the first mixed sample; splicing the two segments of the first mixed sample to obtain a first spliced ​​sample; inputting the positive input signal into an initial diffusion model, using the first spliced ​​sample as the target label, and performing a positive diffusion process through the initial diffusion model to train the initial diffusion model; The other speech sample groups in the multiple speech sample groups are respectively used as the first speech sample group, and the first training process is repeated until the initial diffusion model converges or the number of training times reaches a first preset threshold, so as to obtain the first model.

9. The method according to claim 8, characterized in that, In the positive input signal, the first target single sample is located in the first half, and the first mixed sample is located in the second half; Alternatively, in the positive input signal, the first mixed sample is located in the first half, and the first target single sample is located in the second half.

10. The method according to claim 8 or 9, characterized in that, After obtaining the first model, the method further includes: Perform the second training process; The second training process includes: copying the second mixed sample to obtain another segment of the second mixed sample; the second mixed sample is any one of the mixed samples in the multiple groups of speech samples; splicing the two segments of the second mixed sample to obtain a second spliced ​​sample; inputting the second spliced ​​sample into the first model; the first model performs a backdiffusion process to separate the second spliced ​​sample and output a separation prediction signal; and training the initial adapter based on the separation prediction signal. The other mixed samples in the multiple groups of speech samples are used as the second mixed samples, and the second training process is repeated until the initial adapter converges or the number of training times reaches the second preset threshold, so as to obtain the target adapter.

11. The method according to claim 10, characterized in that, In the positive input signal, the first target single sample is located in the first half, and the first mixed sample is located in the second half; the training of the initial adapter based on the separation prediction signal includes: The separated prediction signal is split into a single prediction signal and a mixed residual prediction signal; the single prediction signal and the mixed residual prediction signal have equal time lengths, the single prediction signal is the first half of the separated prediction signal, and the mixed residual prediction signal is the second half of the separated prediction signal. The single prediction signal is used to store the data of a single speech signal separated from the separated prediction signal, and the mixed residual prediction signal is used to store the data of the audio signal remaining after the single prediction signal is separated from the separated prediction signal. From the multiple sets of speech samples, determine the target single sample that is most similar to the single predicted signal; Remove the most similar single target sample from the second mixed sample to obtain the mixed remaining target signal; The mixed residual prediction signal is input into the initial adapter, and the mixed residual target signal is used as the target label to train the initial adapter.

12. The method according to claim 11, characterized in that, The step of determining the single target sample most similar to the single predicted signal from the multiple groups of speech samples includes: If the data of the single predicted signal is not empty, then the target single sample most similar to the single predicted signal is determined from the multiple groups of speech samples. The second training process also includes: If the data of the single predicted signal is not empty, the signal output by the initial adapter is used as the second mixed sample, and the execution step is returned: the second mixed sample is copied to obtain another segment of the second mixed sample.

13. The method according to claim 11 or 12, characterized in that, The step of determining the single target sample most similar to the single predicted signal from the multiple groups of speech samples includes: Calculate the similarity between the single predicted signal and the target single sample in each of the speech sample groups; The most similar target single sample is determined based on the similarity.

14. An electronic device, characterized in that, The electronic device includes: one or more processors, and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

15. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 13.

Images