A multi-sound source separation system and method based on a generative adversarial network

Abstract

The application discloses a multi-sound source separation system and method based on a generative adversarial network, which comprises a sound signal acquisition and processing all-in-one machine, a server platform, and a multi-sound source separation client. The sound signal acquisition and processing all-in-one machine is used for collecting original audio data emitted by multiple sound sources and processing the original audio data to determine the number of sound sources. The server platform determines whether to process the original audio data according to the determination result of the number of sound sources. If the number of sound sources is one, the original audio data is not processed. If the number of sound sources is multiple, the original audio data is subjected to cross-correlation to determine the position of each sound source, and then the multiple sound source signals are separated through adaptive beamforming. The speech signals after beamforming are optimized by using a generative adversarial network, and the output image data of the generative adversarial network is restored to audio data. The multi-sound source separation client acquires the separation result of the server platform on the multiple sound source signals. The application can effectively separate the speech signals of multiple sound sources, and the speech signals can obtain a higher speech recognition accuracy and a lower word error rate after being recognized.

Description

Technical Field

[0001] This invention belongs to the field of intelligent speech perception, specifically relating to a multi-source sound separation system and method based on generative adversarial networks. Background Technology

[0002] With the rapid development of the mobile internet, more and more smart devices are utilizing speech recognition technology to perceive and understand human voice signals, effectively supporting human-computer interaction between users and these devices. However, in real-world applications, multiple users often exist within the effective perception range of a voice terminal device. These users, acting as multiple "sound sources," typically emit voice signals simultaneously. The signals from these multiple sources superimpose and interfere with each other, significantly reducing the speech recognition rate. Therefore, separating speech from multiple sound sources is crucial for ensuring the performance of speech recognition.

[0003] Current speech separation methods have several shortcomings, primarily in three areas: First, the directional deployment of microphones for multiple users results in high deployment and hardware costs. Second, the use of independent component analysis (ICA) to separate mixed signals is only suitable for ideal scenarios where signal sources are independent and do not follow a Gaussian distribution. However, in real-world environments, due to multipath effects, signals are not independent, significantly reducing speech separation performance. Third, the use of spatial filters requires prior knowledge of the sound source locations. However, due to the superposition and interference of speech signals, as well as the effects of multipath effects, traditional methods cannot accurately estimate the angle of each sound source, greatly limiting their applicability.

[0004] Therefore, based on the above considerations, it is necessary to propose a new multi-source speech separation system that utilizes generative adversarial networks to optimize adaptive beamforming speech signals, effectively separates speech signals from multiple users, improves the accuracy of speech recognition, and enables users to better interact with computers. Summary of the Invention

[0005] To address the shortcomings of the existing technologies, the present invention aims to provide a multi-source speech separation system method based on generative adversarial networks, thereby solving the problems of low localization accuracy, low speech recognition rate, and high cost of existing multi-source speech separation technologies.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention discloses a multi-source sound separation system based on generative adversarial networks, comprising: an integrated sound signal acquisition and processing unit, a server platform, and a multi-source sound separation client; wherein,

[0008] The integrated sound signal acquisition and processing machine is used to acquire raw audio data emitted by multiple sound sources, process the received raw audio data, determine the number of sound sources, and send the determination result and raw audio data to the server platform.

[0009] The server platform acquires data sent by the integrated sound signal acquisition and processing unit in real time, and decides whether to process the raw audio data based on the number of sound sources. If there is a single sound source, no processing is performed; if there are multiple sound sources, the raw audio data is cross-correlated to determine the location of each sound source. Then, multiple sound source signals are separated by adaptive beamforming, and the beamformed speech signal is optimized using a generative adversarial network. The output image data of the generative adversarial network is restored to audio data, and the restored audio data is stored.

[0010] The multi-source separation client obtains and displays the separation results of multiple sound source signals from the server platform.

[0011] Furthermore, the integrated sound signal acquisition and processing unit includes: a microphone array, edge devices, and a communication module;

[0012] Microphone arrays are used to collect raw audio data from multiple sound sources;

[0013] Edge devices process the acquired raw audio data to determine the number of sound sources;

[0014] The communication module sends the judgment results and raw audio data from the edge device to the server platform.

[0015] Furthermore, the microphone array includes multiple omnidirectional microphones and a microcontroller, with a recording frequency range of 20Hz-20kHz and a maximum sampling rate of 48kHz; the microphone array connects to edge devices to receive voice signals from multiple sound sources, using USB, serial port or Ethernet port for connection.

[0016] Furthermore, the edge device is a small artificial intelligence computer that extracts features from the collected raw audio data. Based on the extracted features, it uses a support vector machine to classify the raw audio data to determine whether it is a single sound source or multiple sound sources, and sends the result data to the server platform through a communication module.

[0017] Furthermore, the server platform uses a generative adversarial network to optimize the beamformed speech signal as follows:

[0018] (1) Divide the beamformed audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments. Seconds are treated as a segment, denoted as _____. , , … ,in Indicates the direction number; converts audio data into... The window is divided into segments of seconds, denoted as . ;

[0019] (2) Perform a short-time Fourier transform on the audio data processed in step (1) to obtain the frequency array. The time array t and the complex matrix containing spectral information Its model is denoted as ;

[0020] (3) Modulus of a complex matrix Perform a linear mapping from numerical values ​​to RGB values, where RGB values ​​are denoted as... ,use mold Mapped to the range of 0-255, where This represents the maximum threshold value of the modulus. If the value is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255, ensuring that the RGB value is between 0 and 255;

[0021] (4) Audio data , , … … and After processing in steps (2) and (3), the corresponding single-channel images are obtained respectively. , , … … The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained.

[0022] (5) Map the output image from RGB values ​​to numerical values. The RGB values ​​of the output image are converted into the modulus of the short-time Fourier transform result. The phase of the beamformed audio signal is used to fill the phase of the output image, and then an inverse short-time Fourier transform (ISTFT) is performed to obtain the restored audio signal, denoted as . , , … .

[0023] Furthermore, the specific process of training the generative adversarial network in step (4) is as follows:

[0024] 41) Follow steps (1)-(3) to process the original audio data, The beamforming audio data in each direction and the ground truth audio data of the sound source are processed to obtain a single-channel image. , , , … , This is a single-channel image obtained by converting the Ground Truth audio data of the sound source. and , , … Merged into multi-channel images ,Will and Paired inputs are fed into a generative adversarial network;

[0025] 42) Fix the parameters of the generator, train the discriminator, and given a single-channel image sample, train the discriminator to be able to determine whether it is a real sample or a fake sample generated by the generator.

[0026] 43) Fix the parameters of the discriminator, train the generator, given a multi-channel image, the generator outputs a single-channel image, the discriminator gives a score to the generator's output, and the generator updates its parameters based on the score;

[0027] 44) Repeat the above process until the specified number of iterations T is reached.

[0028] The present invention provides a multi-source sound separation method based on generative adversarial networks, which, based on the above system, includes the following steps:

[0029] 1) The microphone array collects voice signals from multiple sound sources. The edge device processes the raw audio data collected by the microphone array, determines the number of sound sources, and sends the determination result and raw audio data to the server platform.

[0030] 2) The server platform acquires data sent by edge devices in real time and decides whether to process the data based on the number of sound sources. If there is a single sound source, no processing is performed. If there are multiple sound sources, the original audio data is cross-correlated to determine the location of each sound source. Then, multiple sound source signals are separated by adaptive beamforming. Generative adversarial networks are used to optimize the beamformed speech signal. The output image data of the generative adversarial network is restored to audio data, and the restored audio data is stored.

[0031] 3) The multi-source separation client obtains the separation results of multiple sound source signals from the server platform and displays them.

[0032] Further, the generative adversarial network in step 2) includes a generator and a discriminator; the original mixed audio data is transformed by a short-time Fourier transform to obtain a single-channel time-frequency map of the original mixed audio; the beamforming results of the microphone array in the row direction, column direction, and 45° diagonal direction are transformed by a short-time Fourier transform to obtain multiple single-channel time-frequency maps of beamforming signals; the obtained single-channel time-frequency maps are fused into multi-channel data, which is used as the input of the generator; the generator outputs a generated signal time-frequency map; the discriminator is used to determine whether the output generated signal time-frequency map is real, that is, the discriminator judges the signal time-frequency map generated by the generator as fake, and judges the real image as real.

[0033] Furthermore, the objective function used by the generative adversarial network in step 2) is based on the objective function of cGANs with the addition of L1 Loss;

[0034] The objective function of cGANs for:

[0035] ;

[0036] in, It is random noise. Let be the image to be observed, y be the real image, G be the generator, and D be the discriminator. For mathematical expectation; cGANs learn a mapping: ;

[0037] The generator shares information between its input and output, and incorporates L1 Loss to recover the low-frequency components of the image, as detailed below:

[0038] ;

[0039] in, For L1 Loss, It is random noise. Let y be the image to be observed, and y be the actual image.

[0040] Overall objective function for:

[0041] ;

[0042] in, Let cGANs be the objective function. is the regularization coefficient.

[0043] Furthermore, the method for optimizing the beamformed speech signal using a generative adversarial network in step 2) is as follows:

[0044] 21) Divide the beamformed audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments. Seconds are treated as a segment, denoted as _____. , , … ,in Indicates the direction number; converts audio data into... The window is divided into segments of seconds, denoted as . ;

[0045] 22) Perform a short-time Fourier transform on the audio data processed in step 21) to obtain the frequency array. The time array t and the complex matrix containing spectral information Its model is denoted as ;

[0046] 23) Modulus of a complex matrix Perform a linear mapping from numerical values ​​to RGB values, where RGB values ​​are denoted as... ,use mold Mapped to the range of 0-255, where This represents the maximum threshold value of the modulus. If the value is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255, ensuring that the RGB value is between 0 and 255;

[0047] 24) Audio data , , … … and After processing in steps 22) and 23), the corresponding single-channel images are obtained respectively. , , … … The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained.

[0048] 25) Map the output image from RGB values ​​to numerical values. The RGB values ​​of the output image are converted into the modulus of the short-time Fourier transform result. The phase of the beamformed audio signal is used to fill the phase of the output image, and then an inverse short-time Fourier transform (ISTFT) is performed to obtain the restored audio signal, denoted as . , , … .

[0049] The beneficial effects of this invention are:

[0050] 1. High-precision multi-source separation: In scenarios where multiple people are speaking, it can effectively separate the speech signals from multiple sources. After the speech signals are recognized, a high speech recognition accuracy and a low word error rate can be obtained.

[0051] 2. High reliability and high robustness: Because the results of adaptive beamforming are optimized by using generative adversarial networks, it can effectively adapt to echoes and reverberation caused by indoor multipath effects, and is not sensitive to environmental factors such as the size of the conference room and indoor or outdoor environments.

[0052] 3. Reduce computational overhead: By performing the operation of determining the number of sound sources at the edge, the computational pressure on the server is reduced, the computational speed is improved, and users can obtain the processing results from the server in real time.

[0053] 4. Low equipment complexity and cost: Using ordinary microphone arrays, microcontrollers and other equipment, the cost is lower than that of technologies such as directional microphone deployment;

[0054] 5. High expandability: Using common hardware such as ordinary microphones and microcomputers, it can not only be used for multi-source separation, but also add other functions such as sound source localization, voice noise reduction, and voice enhancement. Attached Figure Description

[0055] Figure 1 This is a system architecture diagram of the present invention.

[0056] Figure 2 A schematic diagram for generating adversarial networks.

[0057] Figure 3 A flowchart for optimizing beamforming effects. Detailed Implementation

[0058] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0059] Reference Figure 1 As shown, the present invention provides a multi-source sound separation system based on generative adversarial networks, comprising: an integrated sound signal acquisition and processing unit, a server platform, and a multi-source sound separation client.

[0060] The integrated sound signal acquisition and processing machine is used to acquire raw audio data emitted by multiple sound sources, process the received raw audio data, determine the number of sound sources, and send the determination result and raw audio data to the server platform.

[0061] The integrated sound signal acquisition and processing unit includes: a microphone array, edge devices, and a communication module;

[0062] A microphone array is used to acquire raw audio data from multiple sound sources. The microphone array includes multiple omnidirectional microphones and a microcontroller. Its recording frequency range is 20Hz-20kHz, and its sampling rate is up to 48kHz. The microphone array connects to edge devices to receive voice signals from multiple sound sources and uses USB, serial port, or network port for connection.

[0063] The edge device processes the acquired raw audio data and determines the number of sound sources. The edge device is a small artificial intelligence computer that extracts features from the acquired raw audio data. Based on the extracted features, it uses a support vector machine (SVM) to classify the raw audio data to determine whether it is a single sound source or multiple sound sources, and sends the result data to the server platform through a communication module.

[0064] The communication module sends the judgment results and raw audio data from the edge device to the server platform.

[0065] The server platform acquires data sent by the integrated sound signal acquisition and processing unit in real time, and decides whether to process the raw audio data based on the number of sound sources. If there is a single sound source, no processing is performed; if there are multiple sound sources, the raw audio data is cross-correlated to determine the location of each sound source. Then, multiple sound source signals are separated by adaptive beamforming, and the beamformed speech signal is optimized using a generative adversarial network. The output image data of the generative adversarial network is restored to audio data, and the restored audio data is stored.

[0066] The method by which the server platform optimizes the beamformed speech signal using a generative adversarial network is as follows:

[0067] (1) Divide the beamformed audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments. Seconds are treated as a segment, denoted as _____. , , … ,in Indicates the direction number; converts audio data into... The window is divided into segments of seconds, denoted as . ;

[0068] (2) Perform a short-time Fourier transform (STFT) on the audio data processed in step (1) to obtain the frequency array. The time array t and the complex matrix containing spectral information Its model is denoted as ;

[0069] (3) Modulus of a complex matrix Perform a linear mapping from numerical values ​​to RGB values, where RGB values ​​are denoted as... ,use mold Mapped to the range of 0-255, where This represents the maximum threshold value of the modulus. If the value is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255, ensuring that the RGB value is between 0 and 255;

[0070] (4) Audio data , , … … and After processing in steps (2) and (3), the corresponding single-channel images are obtained respectively. , , … … The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained.

[0071] (5) Map the output image from RGB values ​​to numerical values. The RGB values ​​of the output image are converted into the modulus of the short-time Fourier transform result. The phase of the beamformed audio signal is used to fill the phase of the output image, and then an inverse short-time Fourier transform (ISTFT) is performed to obtain the restored audio signal, denoted as . , , … .

[0072] The specific process of training the generative adversarial network in step (4) is as follows:

[0073] 41) Follow steps (1)-(3) to process the original audio data, The beamforming audio data in each direction and the ground truth audio data of the sound source are processed to obtain a single-channel image. , , , … , This is a single-channel image obtained by converting the Ground Truth audio data of the sound source. and , , … Merged into multi-channel images ,Will and Paired inputs are fed into a generative adversarial network;

[0074] 42) Fix the parameters of the generator, train the discriminator, and given a single-channel image sample, train the discriminator to be able to determine whether it is a real sample or a fake sample generated by the generator.

[0075] 43) Fix the parameters of the discriminator, train the generator, given a multi-channel image, the generator outputs a single-channel image, the discriminator gives a score to the generator's output, and the generator updates its parameters based on the score;

[0076] 44) Repeat the above process until the specified number of iterations T is reached.

[0077] The multi-source separation client obtains and displays the separation results of multiple sound source signals from the server platform.

[0078] The present invention provides a multi-source sound separation method based on generative adversarial networks, which, based on the above system, includes the following steps:

[0079] 1) The microphone array collects voice signals from multiple sound sources. The edge device processes the raw audio data collected by the microphone array, determines the number of sound sources, and sends the determination result and raw audio data to the server platform.

[0080] 2) The server platform acquires data sent by edge devices in real time and decides whether to process the data based on the number of sound sources. If there is a single sound source, no processing is performed. If there are multiple sound sources, the original audio data is cross-correlated to determine the location of each sound source. Then, multiple sound source signals are separated by adaptive beamforming. Generative adversarial networks are used to optimize the beamformed speech signal. The output image data of the generative adversarial network is restored to audio data, and the restored audio data is stored.

[0081] Reference Figure 2As shown, the generative adversarial network in step 2) includes a generator and a discriminator. The original mixed audio data is transformed by a short-time Fourier transform to obtain a single-channel time-frequency map of the original mixed audio. The beamforming results of the microphone array in the row direction, column direction, and 45° diagonal direction are transformed by a short-time Fourier transform to obtain multiple single-channel time-frequency maps of beamforming signals. The obtained single-channel time-frequency maps are fused into multi-channel data, which is used as the input of the generator. The generator outputs a generated signal time-frequency map. The discriminator is used to determine whether the output generated signal time-frequency map is real. That is, the discriminator judges the signal time-frequency map generated by the generator as fake and the real image (Ground Truth time-frequency map) as real.

[0082] For example, the generator uses a U-Net structure, which connects the i-th layer and the ni-th layer, where n is the total number of layers. This connection is made because the i-th and ni-th layers have the same image size and carry similar information. The discriminator uses PatchGAN, which divides the image into multiple fixed-size patches, judges the authenticity of each patch, and finally takes the average value as the discriminator's final output.

[0083] The objective function used by the generative adversarial network in step 2) is based on the objective function of cGANs with the addition of L1 Loss;

[0084] The objective function of cGANs for:

[0085] ;

[0086] in, It is random noise. Let be the image to be observed, y be the real image, G be the generator, and D be the discriminator. For mathematical expectation; cGANs learn a mapping: ;

[0087] The generator shares information between its input and output. (To ensure similarity between the input and output images,) L1 Loss is added to recover the low-frequency components of the image, as detailed below:

[0088] ;

[0089] in, For L1 Loss, It is random noise. y represents the image to be observed, and y represents the actual image.

[0090] Overall objective function for:

[0091] ;

[0092] in, Let cGANs be the objective function. is the regularization coefficient.

[0093] Reference Figure 3 As shown, the method for optimizing the beamformed speech signal using a generative adversarial network in step 2) is as follows:

[0094] 21) Divide the beamformed audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments. Seconds are treated as a segment, denoted as _____. , , … ,in Indicates the direction number; converts audio data into... The window is divided into segments of seconds, denoted as . ;

[0095] 22) Perform a short-time Fourier transform (STFT) on the audio data processed in step 21) to obtain the frequency array. The time array t and the complex matrix containing spectral information Its model is denoted as ;

[0096] 23) Modulus of a complex matrix Perform a linear mapping from numerical values ​​to RGB values, where RGB values ​​are denoted as... ,use mold Mapped to the range of 0-255, where This represents the maximum threshold value of the modulus. If the value is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255, ensuring that the RGB value is between 0 and 255;

[0097] 24) Audio data , , … … and After processing in steps 22) and 23), the corresponding single-channel images are obtained respectively. , , … … The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained.

[0098] 25) Map the output image from RGB values ​​to numerical values. The RGB values ​​of the output image are converted into the modulus of the short-time Fourier transform result. The phase of the beamformed audio signal is used to fill the phase of the output image, and then an inverse short-time Fourier transform (ISTFT) is performed to obtain the restored audio signal, denoted as . , , … .

[0099] 3) The multi-source separation client obtains the separation results of multiple sound source signals from the server platform and displays them.

[0100] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered within the scope of protection of this invention.

Claims

1. A multi-source sound separation system based on generative adversarial networks, characterized in that, include: Audio signal acquisition and processing all-in-one machine, server platform and multi-source separation client; The integrated sound signal acquisition and processing machine is used to acquire raw audio data emitted by multiple sound sources, process the received raw audio data, determine the number of sound sources, and send the determination result and raw audio data to the server platform. The server platform acquires data sent by the integrated sound signal acquisition and processing unit in real time, and decides whether to process the raw audio data based on the result of judging the number of sound sources. If there is a single sound source, no processing is performed; if there are multiple sound sources, the original audio data is cross-correlated to determine the location of each sound source, and then multiple sound source signals are separated by adaptive beamforming. Generative adversarial network is used to optimize the beamformed speech signal, and the output image data of the generative adversarial network is restored to audio data and the restored audio data is stored. The multi-source separation client obtains and displays the separation results of multiple sound source signals from the server platform.

2. The multi-source sound separation system based on generative adversarial networks according to claim 1, characterized in that, The integrated sound signal acquisition and processing unit includes: a microphone array, edge devices, and a communication module; Microphone arrays are used to collect raw audio data from multiple sound sources; Edge devices process the acquired raw audio data to determine the number of sound sources; The communication module sends the judgment results and raw audio data from the edge device to the server platform.

3. The multi-source sound separation system based on generative adversarial networks according to claim 2, characterized in that, The edge device is a small artificial intelligence computer that extracts features from the collected raw audio data. Based on the extracted features, it uses a support vector machine to classify the raw audio data to determine whether it is a single sound source or multiple sound sources, and sends the result data to the server platform through a communication module.

4. The multi-source sound separation system based on generative adversarial networks according to claim 1, characterized in that, The server platform uses a generative adversarial network to optimize the beamformed speech signal as follows: (1) Divide the beamforming audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments, each segment being t. w The second is recorded as a segment, denoted as F1, F2, F3, ..., F... n Where n represents the number of directions; the audio data is divided into t... w The window is divided into segments in seconds, denoted as F. ori ; (2) Perform a short-time Fourier transform on the audio data processed in step (1) to obtain the frequency array f, the time array t, and the complex matrix Z containing spectral information, whose modulus is denoted as M. (3) Perform a linear mapping from numerical values ​​to RGB values ​​on the modulus M of the complex matrix. The RGB values ​​are denoted as R. Use R = 255 * M / max_M to map the modulus M to the range of 0-255, where max_M represents the maximum threshold of the modulus. If M is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255 to ensure that the RGB value is between 0-255. (4) Audio data F1, F2, F3, ..., F i F n and F ori After processing in steps (2) and (3), the corresponding single-channel images I1, I2, I3, ..., I are obtained respectively. i ... I n and I ori The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained. (5) Map the output image from RGB values ​​to numerical values ​​M = R / 255*max_M, convert the RGB values ​​of the output image into the modulus of the short-time Fourier transform result, use the phase of the beamformed audio signal to fill the phase of the output image, and perform an inverse short-time Fourier transform to obtain the restored audio signal, denoted as F′1, F′2, F′3, ..., F′ n .

5. The multi-source sound separation system based on generative adversarial networks according to claim 4, characterized in that, The specific process of training the generative adversarial network in step (4) is as follows: (41) Process the original audio data, the beamforming audio data in n directions, and the ground truth audio data of the sound source according to steps (1)-(3) to obtain a single-channel image I. ori I1, I2, I3, ..., I n and I gt I gt This is a single-channel image obtained by converting the Ground Truth audio data of the sound source, and I... ori and I1, I2, I3, ..., I n Fusion into multi-channel image I multi , will I multi and I gt Paired inputs are fed into a generative adversarial network; (42) Fix the parameters of the generator, train the discriminator, and given a single-channel image sample, train the discriminator to determine whether it is a real sample or a fake sample generated by the generator. (43) Fix the parameters of the discriminator, train the generator, given a multi-channel image, the generator outputs a single-channel image, the discriminator gives a score to the generator's output, and the generator updates its parameters based on the score; (44) Repeat the above process until the specified number of iterations T is reached.

6. A multi-source sound separation method based on generative adversarial networks, based on the system described in any one of claims 1-5, characterized in that, The steps include the following: 1) The microphone array collects voice signals from multiple sound sources. The edge device processes the raw audio data collected by the microphone array, determines the number of sound sources, and sends the determination result and the raw audio data to the server platform. 2) The server platform acquires data sent by edge devices in real time and decides whether to process the data based on the number of sound sources. If there is a single sound source, no processing is performed; if there are multiple sound sources, the original audio data is cross-correlated to determine the location of each sound source, and then multiple sound source signals are separated by adaptive beamforming. Generative adversarial network is used to optimize the beamformed speech signal, and the output image data of the generative adversarial network is restored to audio data and the restored audio data is stored. 3) The multi-source separation client obtains the separation results of multiple sound source signals from the server platform and displays them.

7. The multi-source sound separation method based on generative adversarial networks according to claim 6, characterized in that, The generative adversarial network in step 2) includes a generator and a discriminator. The original mixed audio data is transformed by a short-time Fourier transform to obtain a single-channel time-frequency map of the original mixed audio. The beamforming results of the microphone array in the row direction, column direction, and 45° diagonal direction are transformed by a short-time Fourier transform to obtain multiple single-channel time-frequency maps of beamforming signals. The obtained single-channel time-frequency maps are fused into multi-channel data, which is used as the input of the generator. The generator outputs a generated signal time-frequency map. The discriminator is used to determine whether the output generated signal time-frequency map is real. That is, the discriminator judges the signal time-frequency map generated by the generator as fake and the real image as real.

8. The multi-source sound separation method based on generative adversarial networks according to claim 6, characterized in that, The objective function used by the generative adversarial network in step 2) is based on the objective function of cGANs with the addition of L1 Loss; The objective function of cGANs for: Where z represents random noise, x represents the image to be observed, y represents the real image, G is the generator, and D is the discriminator. For mathematical expectation; cGANs learn a mapping: G:{x, z}→y; The generator shares information between its input and output, and incorporates L1 Loss to recover the low-frequency components of the image, as detailed below: in, Let z be L1 Loss, z be random noise, x be the image to be observed, and y be the real image; The overall objective function G * for: in, Let λ be the objective function of cGANs, and λ be the regularization coefficient.

9. The multi-source sound separation method based on generative adversarial networks according to claim 6, characterized in that, The method for optimizing the beamformed speech signal using a generative adversarial network in step 2) is as follows: 21) Divide the beamformed audio data of the microphone array in the row direction, column direction, and 45° diagonal direction into segments, each segment containing t. w The second is recorded as a segment, denoted as F1, F2, F3, ..., F... n Where n represents the number of directions; the audio data is divided into t... w The window is divided into segments in seconds, denoted as F. ori ; 22) Perform a short-time Fourier transform on the audio data processed in step 21) to obtain the frequency array f, the time array t, and the complex matrix Z containing spectral information, whose modulus is denoted as M. 23) Perform a linear mapping from numerical values ​​to RGB values ​​for the modulus M of the complex matrix. Let the RGB values ​​be R. Use R = 255 * M / max_M to map the modulus M to the range of 0-255, where max_M represents the maximum threshold of the modulus. If M is greater than the maximum threshold, that is, the calculated RGB value exceeds 255, then the RGB value is regarded as 255, ensuring that the RGB value is between 0-255. 24) Audio data F1, F2, F3, ..., F i F n and F ori After processing in steps 22) and 23), the corresponding single-channel images I1, I2, I3, ..., I are obtained respectively. i ... I n and I ori The obtained single-channel images are fused into multi-channel data and input into a trained generative adversarial network. After network inference, the output image is obtained. 25) Map the output image from RGB values ​​to numerical values ​​M = R / 255*max_M, converting the RGB values ​​of the output image into the modulus of the short-time Fourier transform result. Use the phase of the beamformed audio signal to fill the phase of the output image, and perform an inverse short-time Fourier transform to obtain the restored audio signal, denoted as F′1, F′2, F′3, ..., F′ n .

Images