Single-channel human-computer interaction speech separation method, device, computer equipment and medium

Abstract

The present application relates to a single-channel human-computer interaction speech separation method, device, computer equipment and storage medium. The method comprises: training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network; performing speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, and obtaining the machine speech and real person speech separated from the target human-computer interaction speech. The embodiment of the present application can significantly reduce the training data required for deep learning and shorten the training time without losing the separation accuracy of the monophonic human-computer interaction speech, while also improving the speech separation speed.

Description

Technical Field

[0001] The present application relates to the technical field of single-channel speech separation, and in particular to a single-channel human-computer interaction speech separation method, device, computer equipment and storage medium. Background Art

[0002] The single-channel speech separation problem is also known as the "cocktail party" problem in the speech industry. It aims to separate the target sound or extract background music in a noisy environment. The difficulty lies in the number of input sound sources and the speech targets to be separated. For example, the speech separation of a two-person conversation and the speech separation of a three-person conversation, and even the speech separation at a multi-person party are all scenarios of different difficulty, and the technical solutions used are also different.

[0003] From the perspective of traditional technology, methods such as spectral subtraction and Wiener filtering estimate the power spectrum of noise. Wiener filtering is optimal in the scenario of separating pure speech with minimum mean square error. Under the assumption of a prior distribution of speech and noise, the spectral coefficients of speech can be inferred given the noise. These traditional methods require that the noise is stable or changes slowly. Under these conditions, these traditional methods can indeed achieve good results. Speech separation based on deep learning uses supervised training of deep neural networks, which can also achieve good results.

[0004] However, the a priori conditions assumed by the above traditional methods are difficult to meet in reality, especially in the case of low signal-to-noise ratio, which leads to a serious reduction in the speech separation performance of traditional methods in many scenarios, greatly reducing the practicality of industrial implementation. The general deep learning method requires a large amount of data and a long training time, usually at least 1,000 hours. Summary of the invention

[0005] In response to the above-mentioned deficiencies or shortcomings, the present application provides a single-channel human-computer interaction speech separation method, apparatus, computer equipment and storage medium. The embodiments of the present application can significantly reduce the training data required for deep learning and shorten the training time without losing the separation accuracy of monophonic human-computer interaction speech, while also improving the speech separation speed.

[0006] The present application provides a single-channel human-computer interaction speech separation method according to a first aspect. In one embodiment, the method includes:

[0007] Training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network;

[0008] Performing speech separation processing on the target human-machine interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, so as to obtain the machine speech and the real person speech separated from the target human-machine interaction speech;

[0009] The process of each training through the sequential recurrent neural network includes:

[0010] Get the human-computer interaction voice used for this training;

[0011] Performing short-time Fourier transform on the acquired human-computer interaction speech to obtain the spectrum of the human-computer interaction speech;

[0012] Performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum;

[0013] Inputting the spectrum of the human-computer interaction speech into the time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network;

[0014] Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain the second machine speech spectrum and the second real person speech spectrum;

[0015] Calculate a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum respectively;

[0016] Determine whether a training stop condition is met according to the first error and the second error;

[0017] If not, updating the network parameters of the time series recurrent neural network according to the first error and the second error, and performing the next training with the updated time series recurrent neural network;

[0018] If satisfied, stop training, and use the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0019] In one embodiment, a speech separation process is performed on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network to obtain the machine speech and the real person speech separated from the target human-computer interaction speech, including:

[0020] Obtaining the target human-computer interaction voice to be separated;

[0021] Perform short-time Fourier transform on the target human-computer interaction speech to obtain the frequency spectrum of the target human-computer interaction speech;

[0022] The first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network are respectively subjected to matrix decomposition with the spectrum of the target human-computer interaction speech to obtain the machine speech spectrum and the real person speech spectrum;

[0023] The machine speech spectrum and the real person speech spectrum are respectively subjected to inverse Fourier transform to obtain the machine speech and the real person speech.

[0024] In one embodiment, the spectrum of the human-computer interaction speech is subjected to cepstral mean variance normalization and spectrum clustering to obtain a first machine speech spectrum and a first real person speech spectrum, including:

[0025] Perform cepstral mean variance normalization processing on the spectrum of human-computer interaction speech to obtain the processed spectrum;

[0026] The processed frequency spectrum is subjected to frequency spectrum clustering, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the processing result of the frequency spectrum clustering.

[0027] In one embodiment, performing spectrum clustering on the processed spectrum, and obtaining a first machine speech spectrum and a first real person speech spectrum according to the spectrum clustering processing result, includes:

[0028] Using a first clustering algorithm to divide spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters;

[0029] Using a second clustering algorithm, a cluster is separated from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster;

[0030] Obtaining a first machine speech spectrum according to the machine speech cluster;

[0031] A first real person speech spectrum is obtained according to the real person speech cluster.

[0032] In one embodiment, the temporal recurrent neural network is a long short-term memory recurrent neural network; the first clustering algorithm refers to a K-means clustering algorithm; and the second clustering algorithm refers to a db-scan clustering algorithm.

[0033] In one embodiment, the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training are respectively subjected to matrix decomposition with the spectrum of the human-computer interaction speech to obtain a second machine speech spectrum and a second real person speech spectrum, including:

[0034] Perform a dot multiplication of the first time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-machine interaction speech to obtain a second machine speech spectrum;

[0035] The second time-frequency mask output by the temporal recurrent neural network in this training is point-multiplied with the spectrum of the human-computer interaction speech to obtain the second real person speech spectrum.

[0036] In one embodiment, the first error is used to represent the matching degree between the first machine speech spectrum and the second machine speech spectrum; the second error is used to represent the matching degree between the first real person speech spectrum and the second real person speech spectrum.

[0037] According to a second aspect, the present application provides a single-channel human-computer interaction speech separation device. In one embodiment, the device includes:

[0038] A time-frequency mask training module, used for training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network;

[0039] A speech separation module is used to perform speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, so as to obtain the machine speech and the real person speech separated from the target human-computer interaction speech;

[0040] The time-frequency mask training module is used for each training through the time series recurrent neural network, specifically for:

[0041] Get the human-computer interaction voice used for this training;

[0042] Performing short-time Fourier transform on the acquired human-computer interaction speech to obtain the spectrum of the human-computer interaction speech;

[0043] Performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum;

[0044] Inputting the spectrum of the human-computer interaction speech into the time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network;

[0045] Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain the second machine speech spectrum and the second real person speech spectrum;

[0046] Calculate a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum respectively;

[0047] Determine whether a training stop condition is met according to the first error and the second error;

[0048] If not, updating the network parameters of the time series recurrent neural network according to the first error and the second error, and performing the next training with the updated time series recurrent neural network;

[0049] If satisfied, stop training, and use the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0050] According to a third aspect, the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above method embodiments when executing the computer program.

[0051] According to a fourth aspect, the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of an embodiment of any of the above methods are implemented.

[0052] In an embodiment of the present application, a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum are trained by a time-series recurrent neural network; the target human-computer interaction speech to be separated is subjected to speech separation processing according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, and the machine speech and the real person speech separated from the target human-computer interaction speech are obtained. When training by the time-series recurrent neural network, the spectrum of the human-computer interaction speech is subjected to cepstral mean variance normalization and spectrum clustering, and the machine speech spectrum and the real person speech spectrum can be quickly obtained. Since the spectrum obtained by cepstral mean variance normalization and spectrum clustering is of high quality, it can be directly used for error analysis of the output of the time-series recurrent neural network, so that only the human-computer interaction speech needs to be prepared during training without preparing the corresponding clean machine speech and clean real person speech for error analysis, thereby significantly reducing the training data required for deep learning and shortening the training time without losing the separation accuracy of the monophonic human-computer interaction speech, and at the same time improving the speech separation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A schematic flow chart of a single-channel human-computer interaction speech separation method in one embodiment;

[0054] Figure 2 A schematic diagram of the overall process of speech separation in one embodiment;

[0055] Figure 3 A schematic diagram of a process of training a temporal recurrent neural network in one embodiment;

[0056] Figure 4 A schematic diagram of a process for performing speech separation on a target human-computer interaction speech in one embodiment;

[0057] Figure 5 It is a structural block diagram of a single-channel human-computer interaction speech separation device in one embodiment;

[0058] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0060] This application provides a single-channel human-computer interaction speech separation method for scenarios where AI (Artificial Intelligence) devices (such as smart speakers, smart phones, and other devices that can interact with real people by voice) interact with people. In one embodiment, the single-channel human-computer interaction speech separation method includes: Figure 1 The steps shown are described below.

[0061] S110: Training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a temporal recurrent neural network.

[0062] The time series recurrent neural network is a LSTM (Long-Short Term Memory) recurrent neural network. Among them, LSTM is selected in this embodiment because the long-short term memory capability of LSTM can well capture the time series characteristics, which is very helpful for the permutation invariance of speech, and can well train the time-frequency mask (also known as mask, masking, mask, etc., which is called mask in English) of each speech timbre.

[0063] Specifically, this embodiment prepares multiple copies of human-computer interaction speech as training samples (in order to obtain better time-frequency masks, more human-computer interaction speech can be prepared as training samples), and performs iterative training through a temporal recurrent neural network and training samples.

[0064] S120: Performing speech separation processing on the target human-machine interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the temporal recurrent neural network to obtain machine speech and real person speech separated from the target human-machine interaction speech.

[0065] The target human-computer interaction speech refers to the human-computer interaction speech to be separated. The human-computer interaction speech is a mixed speech of machine speech and real person speech. It should be noted that each interaction in the above human-computer interaction is usually performed by a real person and an AI device. Of course, in the entire human-computer interaction process, multiple real people can interact with the AI ​​device.

[0066] In this embodiment, the process of training the first time-frequency mask and the second time-frequency mask by using the time-series recurrent neural network and performing speech separation using the trained first time-frequency mask and the second time-frequency mask can be referred to in Figure 2 .

[0067] Among them, the process of each training is carried out through the time series recurrent neural network, such as Figure 3 As shown, the following steps are included:

[0068] S210: Obtain human-computer interaction voice for this training.

[0069] In this embodiment, the human-computer interaction speech for training does not need to be manually labeled.

[0070] S220: Performing short-time Fourier transform (STFT) on the acquired human-computer interaction speech to obtain a frequency spectrum of the human-computer interaction speech.

[0071] Among them, the human-computer interaction speech is time domain information. Before performing speech separation processing on it, it is first transformed from time domain information to frequency domain information, that is, the spectrum of the human-computer interaction speech, through short-time Fourier transform.

[0072] Before performing short-time Fourier transform, the human-computer interaction speech can be framed and windowed. For example, taking 20ms sampling points as a frame signal (this embodiment does not impose specific restrictions on this, and 32ms sampling points can also be taken as a frame signal), the human-computer interaction speech is divided into J frames, wherein, if the sampling rate is 8kHz, one frame is 256 sampling points, if the sampling frequency is 16kHz, one frame is 512 sampling points, if the length is less than 20ms, the sampling points are first padded to 256 or 512; then each frame signal is windowed, and a Hamming window or a Hanning window can be used as a windowing function; finally, a short-time Fourier transform is performed on each frame of the windowed signal to obtain the spectrum of the human-computer interaction speech. It should be noted that the spectrum amplitude spectrum and phase spectrum of the human-computer interaction speech can be obtained by performing a short-time Fourier transform. Unless otherwise specified below, the spectrum of the human-computer interaction speech refers to the spectrum amplitude of the human-computer interaction speech. The phase spectrum of human-computer interaction also needs to be retained. When the machine speech spectrum or the real person speech spectrum is subsequently inverse Fourier transformed, the machine speech spectrum or the real person speech spectrum needs to be transformed into time domain information in combination with the phase spectrum of human-computer interaction. The short-time Fourier transform and inverse Fourier transform selected in this embodiment are both commonly used technologies and are not described in detail here.

[0073] S230: Perform cepstral mean variance normalization (hereinafter referred to as CMVN) and spectrum clustering on the spectrum of the human-machine interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum.

[0074] S240: Input the frequency spectrum of the human-computer interaction speech into the time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network.

[0075] The temporal recurrent neural network in this embodiment is used to train two time-frequency masks, namely, a first time-frequency mask and a second time-frequency mask.

[0076] The time-frequency mask can be understood as a filter, which is used to filter specific frequency domain information from the frequency domain information of human-computer interaction speech. For example, the first time-frequency mask is trained to filter the spectrum information of machine speech (i.e., machine speech spectrum), and the second time-frequency mask is trained to filter the spectrum information of real person speech (i.e., real person speech spectrum).

[0077] S250: Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain a second machine speech spectrum and a second real person speech spectrum.

[0078] S260: Calculate a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum.

[0079] The first error is used to represent the matching degree between the first machine speech spectrum and the second machine speech spectrum; the second error is used to represent the matching degree between the first real person speech spectrum and the second real person speech spectrum.

[0080] In this embodiment, the first machine speech spectrum and the first real person speech spectrum are real machine speech spectrum and real person speech spectrum, while the second machine speech spectrum and the second real person speech spectrum are predicted machine speech spectrum and real person speech spectrum. In order to make the predicted machine speech spectrum and real person speech spectrum closer to the real machine speech spectrum and real person speech spectrum, in each training, the first error between the first machine speech spectrum and the second machine speech spectrum and the second error between the first real person speech spectrum and the second real person speech spectrum are calculated, so as to judge whether it is necessary to continue training the temporal recurrent neural network based on the calculated first error and second error.

[0081] S270: Determine whether a training stop condition is met based on the first error and the second error.

[0082] The first error and the second error are summed to obtain a total error, and then the total error is used to determine whether a preset stop training condition is met. The stop training condition can be flexibly adjusted according to the specific application scenario, and this embodiment does not specifically limit this. For example, it can be that the number of training times reaches a preset threshold, or it can be determined based on the total loss of this training and the total loss of previous trainings whether the loss of the temporal recurrent neural network is reduced to a minimum value. If it is reduced to a minimum value, then it is determined that this training meets the stop training condition.

[0083] S280: If not satisfied, update the network parameters of the temporal recurrent neural network according to the first error and the second error, and perform the next training with the updated temporal recurrent neural network.

[0084] The first error and the second error are summed to obtain the total error, and then back propagation is performed based on the total error to optimize the network parameters of the time series recurrent neural network. The next training is then performed using the time series recurrent neural network with updated network parameters. For the next training process, please refer to Figure 3 The process shown will not be repeated here.

[0085] S290: If satisfied, stop training, and use the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0086] This embodiment aims at the characteristics of human-computer interaction scenarios, that is, the acoustic features and speech of machine speech are often regular and often interact with real-person speech (also called human voice), and the two have little overlap. A global CMVN is added to the mask-based neural network to enhance robustness (especially in low signal-to-noise ratio conditions, it can significantly improve performance). In this way, better acoustic features of machine speech can be quickly extracted from human-computer interaction speech through a simple deep clustering. Then, deep clustering is performed again on non-machine speech to easily extract the acoustic features of real-person speech. If the previous neural network solution is adopted, in addition to preparing mixed speech for input into the neural network, it is also necessary to prepare clean speech corresponding to the mixed speech for error analysis (specifically including clean machine speech and clean real person speech). However, this embodiment adds global CMVN and deep clustering to the mask-based neural network, so that high-quality acoustic features of machine speech and acoustic features of real person speech can be extracted. Therefore, the extracted acoustic features can be directly used for error analysis of the output of the neural network. In this way, only mixed speech for input into the neural network is needed for training, which can greatly reduce the training data and training time required for training without losing accuracy. After the first time-frequency mask and the second time-frequency mask are trained, the target human-computer interaction speech can be directly separated according to the trained first time-frequency mask and the second time-frequency mask. Compared with general separation algorithms, such as independent component analysis (ICA), sparse principal component analysis (Spars PCA), non-negative matrix decomposition (NMF), etc., this embodiment has a faster speech separation speed. For example, when a general separation algorithm is used to separate human-computer interaction speech, the separation time of a speech with a duration of 1 minute is about 10 seconds, while this embodiment only takes about 6 seconds, which increases the speed by 40%.

[0087] In one embodiment, Figure 4 As shown, the above-mentioned method performs speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time series recurrent neural network to obtain the machine speech and the real person speech separated from the target human-computer interaction speech, including:

[0088] S310: Acquire the target human-computer interaction speech to be separated;

[0089] S320: Performing short-time Fourier transform on the target human-computer interaction speech to obtain a frequency spectrum of the target human-computer interaction speech;

[0090] S330: performing matrix decomposition on the first time-frequency mask and the second time-frequency mask trained by the time series recurrent neural network and the frequency spectrum of the target human-machine interaction speech, respectively, to obtain a machine speech spectrum and a real person speech spectrum;

[0091] S340: Perform inverse Fourier transform on the machine speech spectrum and the real person speech spectrum respectively to obtain the machine speech and the real person speech.

[0092] In this example, see Figure 2 After the first time-frequency mask and the second time-frequency mask are trained, for the target human-computer interaction speech to be separated, after performing a short-time Fourier transform on it to obtain the frequency domain information (i.e., the spectrum) of the target human-computer interaction speech, the trained first time-frequency mask and the second time-frequency mask can be used to perform speech separation processing on it, without the need for subsequent operations such as CMVN, spectrum clustering, etc., which can speed up the speech separation speed.

[0093] Specifically, the trained first time-frequency mask and the spectrum of the target human-computer interaction speech are matrix-decomposed to obtain the predicted machine speech spectrum, and the trained second time-frequency mask and the spectrum of the target human-computer interaction speech are matrix-decomposed to obtain the predicted real person speech spectrum. Then, the predicted machine speech spectrum and the real person speech spectrum are inverse Fourier transformed to transform the predicted machine speech spectrum into time domain information (i.e., machine speech) and the predicted real person speech spectrum into time domain information (i.e., real person speech).

[0094] In one embodiment, the above-mentioned performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum includes: performing cepstral mean variance normalization processing on the spectrum of the human-computer interaction speech to obtain a processed spectrum; performing spectrum clustering on the processed spectrum, and obtaining the first machine speech spectrum and the first real person speech spectrum according to the processing results of the spectrum clustering.

[0095] The processed spectrum is subjected to spectrum clustering, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the spectrum clustering processing result, including:

[0096] Using a first clustering algorithm to divide spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters;

[0097] Using a second clustering algorithm, a cluster is separated from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster;

[0098] Obtaining a first machine speech spectrum according to the machine speech cluster;

[0099] A first real person speech spectrum is obtained according to the real person speech cluster.

[0100] In the human-computer interaction scenario, since the timbre of the voice robot is fixed in advance, the acoustic features of machine speech are often regular, while human voice is often irregular. Based on this, the K-means clustering algorithm is selected to make a preliminary cluster of the processed spectrum. K-means can well distinguish machine speech from real speech. The algorithm specifically divides the time-frequency units included in the processed spectrum into two clusters, one cluster is regular (i.e., the above-mentioned machine speech cluster) and the other cluster is irregular (i.e., the above-mentioned non-machine speech cluster). Since human voice is often irregular and has noise, in the human-computer interaction scenario, a third human voice may also appear. Therefore, the db-scan clustering algorithm is selected to process these irregular signal data. Specifically, db-scan is used to continue clustering the time-frequency units corresponding to the above-mentioned non-machine speech cluster and divide them into a cluster. The separated cluster is the real voice cluster. The spectrum obtained by the time-frequency units corresponding to the machine speech cluster is the first machine speech spectrum, and the spectrum obtained by the time-frequency units corresponding to the real voice cluster is the first real voice spectrum. This embodiment performs spectrum clustering through a combination of K-means and db-scan, which can make the speech separation accuracy far exceed that of ordinary single-channel separation.

[0101] It should be noted that the K-means and db-scan used in the above-mentioned spectral clustering process are only examples. During implementation, other clustering algorithms suitable for distinguishing regular features, i.e., the first clustering algorithm (including the above-mentioned K-means) can be selected according to specific needs to replace the first clustering, and other clustering algorithms suitable for distinguishing irregular features, i.e., the second clustering algorithm (including the above-mentioned db-scan) can be selected to perform the second clustering. In this way, the effect of making the speech separation accuracy exceed that of ordinary single-channel separation can be achieved.

[0102] In one embodiment, the above-mentioned first time-frequency mask and the second time-frequency mask output by the timing recurrent neural network in this training are respectively matrix-decomposed with the spectrum of the human-computer interaction speech to obtain the second machine speech spectrum and the second real person speech spectrum, including: performing a dot product of the first time-frequency mask output by the timing recurrent neural network in this training and the spectrum of the human-computer interaction speech to obtain the second machine speech spectrum; performing a dot product of the second time-frequency mask output by the timing recurrent neural network in this training and the spectrum of the human-computer interaction speech to obtain the second real person speech spectrum.

[0103] Among them, the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network are both binary matrices, and the frequency spectrum of the human-computer interaction speech is also a binary matrix. Therefore, when performing the above-mentioned matrix decomposition, the two matrices (such as the first time-frequency mask and the frequency spectrum of the human-computer interaction speech, the second time-frequency mask and the frequency spectrum of the human-computer interaction speech) are specifically multiplied to obtain the corresponding speech spectrum (such as the second machine speech spectrum, the second real person speech spectrum).

[0104] Figure 1 , Figure 3 and Figure 4 FIG. 1 is a flow chart of a single-channel human-computer interaction speech separation method in one embodiment. It should be understood that although Figure 1 , Figure 3 and Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 , Figure 3 and Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0105] Based on the same inventive concept, the present application also provides a single-channel human-computer interaction speech separation device. Figure 5 As shown, the single-channel human-computer interaction speech separation device includes the following modules:

[0106] A time-frequency mask training module 110, used for training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network;

[0107] The speech separation module 120 is used to perform speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, so as to obtain the machine speech and the real person speech separated from the target human-computer interaction speech;

[0108] The time-frequency mask training module 110 is specifically used to:

[0109] Get the human-computer interaction voice used for this training;

[0110] Performing short-time Fourier transform on the acquired human-computer interaction speech to obtain the spectrum of the human-computer interaction speech;

[0111] Performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum;

[0112] Inputting the spectrum of the human-computer interaction speech into the time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network;

[0113] Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain the second machine speech spectrum and the second real person speech spectrum;

[0114] respectively calculating a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum;

[0115] Determine whether a training stop condition is met according to the first error and the second error;

[0116] If not, updating the network parameters of the time series recurrent neural network according to the first error and the second error, and performing the next training with the updated time series recurrent neural network;

[0117] If satisfied, stop training, and use the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0118] In one embodiment, the speech separation module is specifically used to:

[0119] Obtaining the target human-computer interaction voice to be separated;

[0120] Perform short-time Fourier transform on the target human-computer interaction speech to obtain the frequency spectrum of the target human-computer interaction speech;

[0121] The first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network are respectively subjected to matrix decomposition with the spectrum of the target human-computer interaction speech to obtain the machine speech spectrum and the real person speech spectrum;

[0122] The machine speech spectrum and the real person speech spectrum are respectively subjected to inverse Fourier transform to obtain the machine speech and the real person speech.

[0123] In one embodiment, the time-frequency mask training module is used to perform cepstrum mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain the first machine speech spectrum and the first real person speech spectrum, specifically for:

[0124] Perform cepstral mean variance normalization processing on the spectrum of human-computer interaction speech to obtain the processed spectrum;

[0125] The processed frequency spectrum is subjected to frequency spectrum clustering, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the processing result of the frequency spectrum clustering.

[0126] In one embodiment, the time-frequency mask training module is used to perform spectrum clustering on the processed spectrum, and obtain the first machine speech spectrum and the first real person speech spectrum according to the spectrum clustering processing result, specifically for:

[0127] Using a first clustering algorithm to divide spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters;

[0128] Using a second clustering algorithm, a cluster is separated from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster;

[0129] Obtaining a first machine speech spectrum according to the machine speech cluster;

[0130] A first real person speech spectrum is obtained according to the real person speech cluster.

[0131] In one embodiment, the temporal recurrent neural network is a long short-term memory recurrent neural network; the first clustering algorithm refers to a K-means clustering algorithm; and the second clustering algorithm refers to a db-scan clustering algorithm.

[0132] In one embodiment, the time-frequency mask training module is used to perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech to obtain the second machine speech spectrum and the second real person speech spectrum, specifically for:

[0133] Perform a dot multiplication of the first time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-machine interaction speech to obtain a second machine speech spectrum;

[0134] The second time-frequency mask output by the temporal recurrent neural network in this training is point-multiplied with the spectrum of the human-computer interaction speech to obtain the second real person speech spectrum.

[0135] In one embodiment, the first error is used to represent the matching degree between the first machine speech spectrum and the second machine speech spectrum; the second error is used to represent the matching degree between the first real person speech spectrum and the second real person speech spectrum.

[0136] For the specific definition of the single-channel human-computer interaction voice separation device, please refer to the definition of the single-channel human-computer interaction voice separation method above, which will not be repeated here. Each module in the above-mentioned single-channel human-computer interaction voice separation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0137] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 6 shown.

[0138] The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as a first time-frequency mask and a second time-frequency mask. The specific stored data can also refer to the definition in the above method embodiment. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a single-channel human-computer interaction speech separation method is implemented.

[0139] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0140] This embodiment further provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0141] A first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum are trained through a time-series recurrent neural network; speech separation processing is performed on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained through the time-series recurrent neural network to obtain the machine speech and the real person speech separated from the target human-computer interaction speech; the process of each training through the time-series recurrent neural network includes: obtaining the human-computer interaction speech used for this training; performing short-time Fourier transform on the obtained human-computer interaction speech to obtain the spectrum of the human-computer interaction speech; performing inverse spectrum mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum; inputting the spectrum of the human-computer interaction speech into the time-series recurrent neural network to obtain the first a time-frequency mask and a second time-frequency mask; performing matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain a second machine speech spectrum and a second real person speech spectrum; respectively calculating a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum; judging whether the training stop condition is met according to the first error and the second error; if not, updating the network parameters of the time series recurrent neural network according to the first error and the second error, and performing the next training through the updated time series recurrent neural network; if satisfied, stopping the training, and using the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0142] In one embodiment, the processor executes a computer program to perform speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time series recurrent neural network, and obtains the machine speech and the real person speech separated from the target human-computer interaction speech, and further implements the following steps:

[0143] The target human-computer interaction speech to be separated is obtained; the target human-computer interaction speech is short-time Fourier transformed to obtain the spectrum of the target human-computer interaction speech; the first time-frequency mask and the second time-frequency mask trained by the time series recurrent neural network are matrix-decomposed with the spectrum of the target human-computer interaction speech to obtain the machine speech spectrum and the real person speech spectrum; the machine speech spectrum and the real person speech spectrum are inverse Fourier transformed to obtain the machine speech and the real person speech.

[0144] In one embodiment, the processor executes a computer program to perform cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech, and when obtaining the first machine speech spectrum and the first real person speech spectrum, the following steps are also implemented:

[0145] The frequency spectrum of the human-computer interaction speech is subjected to cepstral mean variance normalization processing to obtain a processed frequency spectrum; the processed frequency spectrum is subjected to frequency spectrum clustering, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the processing result of the frequency spectrum clustering.

[0146] In one embodiment, the processor executes a computer program to perform spectrum clustering on the processed spectrum, and when obtaining the first machine voice spectrum and the first real person voice spectrum according to the spectrum clustering processing result, the following steps are further implemented:

[0147] A first clustering algorithm is used to divide the spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters; a second clustering algorithm is used to separate a cluster from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster; a first machine speech spectrum is obtained according to the machine speech cluster; and a first human speech spectrum is obtained according to the human speech cluster.

[0148] In one embodiment, the processor executes a computer program to perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain the second machine speech spectrum and the second real person speech spectrum, and further implements the following steps:

[0149] The first time-frequency mask output by the timing recurrent neural network in this training is dot-multiplied with the spectrum of the human-computer interaction speech to obtain the second machine speech spectrum; the second time-frequency mask output by the timing recurrent neural network in this training is dot-multiplied with the spectrum of the human-computer interaction speech to obtain the second real person speech spectrum.

[0150] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0151] A first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum are trained through a time-series recurrent neural network; speech separation processing is performed on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained through the time-series recurrent neural network to obtain the machine speech and the real person speech separated from the target human-computer interaction speech; the process of each training through the time-series recurrent neural network includes: obtaining the human-computer interaction speech used for this training; performing short-time Fourier transform on the obtained human-computer interaction speech to obtain the spectrum of the human-computer interaction speech; performing inverse spectrum mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum; inputting the spectrum of the human-computer interaction speech into the time-series recurrent neural network to obtain the first a time-frequency mask and a second time-frequency mask; performing matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain a second machine speech spectrum and a second real person speech spectrum; respectively calculating a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum; judging whether the training stop condition is met according to the first error and the second error; if not, updating the network parameters of the time series recurrent neural network according to the first error and the second error, and performing the next training through the updated time series recurrent neural network; if satisfied, stopping the training, and using the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training as the trained first time-frequency mask and the second time-frequency mask.

[0152] In one embodiment, the computer program is executed by a processor, and when performing speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, and obtaining the machine speech and the real person speech separated from the target human-computer interaction speech, the following steps are also implemented:

[0153] The target human-computer interaction speech to be separated is obtained; the target human-computer interaction speech is short-time Fourier transformed to obtain the spectrum of the target human-computer interaction speech; the first time-frequency mask and the second time-frequency mask trained by the time series recurrent neural network are matrix-decomposed with the spectrum of the target human-computer interaction speech to obtain the machine speech spectrum and the real person speech spectrum; the machine speech spectrum and the real person speech spectrum are inverse Fourier transformed to obtain the machine speech and the real person speech.

[0154] In one embodiment, the computer program is executed by a processor to perform cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum, and further implement the following steps:

[0155] The frequency spectrum of the human-computer interaction speech is subjected to cepstral mean variance normalization processing to obtain a processed frequency spectrum; the processed frequency spectrum is subjected to frequency spectrum clustering, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the processing result of the frequency spectrum clustering.

[0156] In one embodiment, the computer program is executed by a processor to perform spectrum clustering on the processed spectrum, and when obtaining the first machine speech spectrum and the first real person speech spectrum according to the spectrum clustering processing result, the following steps are further implemented:

[0157] A first clustering algorithm is used to divide the spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters; a second clustering algorithm is used to separate a cluster from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster; a first machine speech spectrum is obtained according to the machine speech cluster; and a first human speech spectrum is obtained according to the human speech cluster.

[0158] In one embodiment, the computer program is executed by a processor to perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain the second machine speech spectrum and the second real person speech spectrum, and further implement the following steps:

[0159] The first time-frequency mask output by the timing recurrent neural network in this training is dot-multiplied with the spectrum of the human-computer interaction speech to obtain the second machine speech spectrum; the second time-frequency mask output by the timing recurrent neural network in this training is dot-multiplied with the spectrum of the human-computer interaction speech to obtain the second real person speech spectrum.

[0160] Those skilled in the art can understand that all or part of the processes in the above method embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0161] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0162] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A single-channel human-computer interaction speech separation method, characterized in that: The method comprises: Training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network; Performing speech separation processing on the target human-machine interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, to obtain machine speech and real person speech separated from the target human-machine interaction speech; The process of each training by the temporal recurrent neural network includes: Get the human-computer interaction voice used for this training; Performing short-time Fourier transform on the acquired human-computer interaction speech to obtain a frequency spectrum of the human-computer interaction speech; Performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum; Inputting the frequency spectrum of the human-computer interaction speech into a time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network; Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain a second machine speech spectrum and a second real person speech spectrum; respectively calculating a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum; Determining whether a training stop condition is met according to the first error and the second error; If not, updating the network parameters of the temporal recurrent neural network according to the first error and the second error, and performing the next training with the updated temporal recurrent neural network; If satisfied, the training is stopped, and the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training are used as the trained first time-frequency mask and the second time-frequency mask.

2. The method according to claim 1, characterized in that The method of performing speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network to obtain the machine speech and the real person speech separated from the target human-computer interaction speech includes: Acquire the target human-computer interaction speech to be separated; Performing a short-time Fourier transform on the target human-computer interaction speech to obtain a frequency spectrum of the target human-computer interaction speech; Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network and the frequency spectrum of the target human-computer interaction speech, respectively, to obtain a machine speech spectrum and a real person speech spectrum; The machine speech spectrum and the real person speech spectrum are respectively subjected to inverse Fourier transformation to obtain the machine speech and the real person speech.

3. The method according to claim 1, characterized in that The performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-machine interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum includes: Performing cepstral mean variance normalization processing on the spectrum of the human-computer interaction speech to obtain a processed spectrum; Spectrum clustering is performed on the processed spectrum, and a first machine speech spectrum and a first real person speech spectrum are obtained according to the spectrum clustering processing result.

4. The method according to claim 3, characterized in that The performing spectrum clustering on the processed spectrum and obtaining a first machine speech spectrum and a first real person speech spectrum according to the spectrum clustering processing result includes: Using a first clustering algorithm to divide the spectrum units included in the processed spectrum into machine speech clusters and non-machine speech clusters; Using a second clustering algorithm to separate a cluster from the spectrum units corresponding to the non-machine speech cluster as a human speech cluster; Obtaining a first machine speech spectrum according to the machine speech cluster; A first real person speech spectrum is obtained according to the real person speech cluster.

5. The method according to claim 4, characterized in that The temporal recurrent neural network is a long short-term memory recurrent neural network; the first clustering algorithm refers to the K-means clustering algorithm; and the second clustering algorithm refers to the db-scan clustering algorithm.

6. The method according to claim 1, characterized in that The first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training are respectively subjected to matrix decomposition with the spectrum of the human-computer interaction speech to obtain a second machine speech spectrum and a second real person speech spectrum, including: Performing a dot multiplication of the first time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-machine interaction speech to obtain a second machine speech spectrum; The second time-frequency mask output by the timing recurrent neural network in this training is dot-multiplied with the spectrum of the human-computer interaction speech to obtain a second real person speech spectrum.

7. The method according to claim 1, characterized in that The first error is used to indicate the degree of matching between the first machine speech spectrum and the second machine speech spectrum; the second error is used to indicate the degree of matching between the first real person speech spectrum and the second real person speech spectrum.

8. A single-channel human-computer interaction speech separation device, characterized in that: The device comprises: A time-frequency mask training module, used for training a first time-frequency mask for extracting a machine speech spectrum and a second time-frequency mask for extracting a real person speech spectrum through a time-series recurrent neural network; A speech separation module, used for performing speech separation processing on the target human-computer interaction speech to be separated according to the first time-frequency mask and the second time-frequency mask trained by the time-series recurrent neural network, to obtain machine speech and real person speech separated from the target human-computer interaction speech; The time-frequency mask training module, when used to perform each training through the time series recurrent neural network, is specifically used to: Get the human-computer interaction voice used for this training; Performing short-time Fourier transform on the acquired human-computer interaction speech to obtain a frequency spectrum of the human-computer interaction speech; Performing cepstral mean variance normalization and spectrum clustering on the spectrum of the human-computer interaction speech to obtain a first machine speech spectrum and a first real person speech spectrum; Inputting the frequency spectrum of the human-computer interaction speech into a time series recurrent neural network to obtain a first time-frequency mask and a second time-frequency mask output by the time series recurrent neural network; Perform matrix decomposition on the first time-frequency mask and the second time-frequency mask output by the time series recurrent neural network in this training and the spectrum of the human-computer interaction speech, respectively, to obtain a second machine speech spectrum and a second real person speech spectrum; respectively calculating a first error between the first machine speech spectrum and the second machine speech spectrum, and a second error between the first real person speech spectrum and the second real person speech spectrum; Determining whether a training stop condition is met according to the first error and the second error; If not, updating the network parameters of the temporal recurrent neural network according to the first error and the second error, and performing the next training with the updated temporal recurrent neural network; If satisfied, the training is stopped, and the first time-frequency mask and the second time-frequency mask output by the temporal recurrent neural network in this training are used as the trained first time-frequency mask and the second time-frequency mask.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images