Voice noise reduction method and device, electronic equipment, storage medium and program product

Through the combination of spectrum Mel filter coefficients and depth filter coefficients, combined with convolutional recurrent neural network, the instability problem of traditional speech noise reduction methods in complex noise environments is solved, and the auditory perception quality and understanding of speech are improved. degree, especially under non-stationary noise and low signal-to-noise ratio conditions.

CN118918916APending Publication Date: 2024-11-08BEIJING UNIV OF POSTS & TELECOMM +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410988954.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional speech noise reduction methods based on statistical characteristics are unstable in the old and new environments, and are difficult to adapt to complex and variable noise environments. Overestimating the noise volume will lead to speech distortion, especially in the case of non-stationary noise and low signal-to-noise ratios. .

Method used

By determining the spectrum Mel filter coefficients of noisy speech data, extracting the depth filter coefficients of particle size, combining the time frequency domain convolutional recurrent neural network to extract the embedded characteristics and timing characteristics of speech, the speech noise reduction is achieved and the speech hearing is improved Perceived quality and intelligibility.

Benefits of technology

It effectively improves the noise reduction effect of speech in complex noise environments, reduces the occurrence of speech distortion, improves the clarity and recognition rate of speech, and is more stable in the condition of non-stationary noise and low signal-to-noise ratio.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918916A_ABST
    Figure CN118918916A_ABST
Patent Text Reader

Abstract

The invention provides a voice noise reduction method and device, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining noisy voice data, obtaining a frequency spectrum Mel filter coefficient based on the noisy voice data, carrying out the extraction of the characteristics of the frequency spectrum Mel filter coefficient, obtaining a depth filter coefficient of granularity, and obtaining the depth filter coefficient of the granularity; and determining noisy speech spectrum information of the noisy speech data, and obtaining noise-reduced speech data based on the noisy speech spectrum information and the depth filter coefficient of the granularity. According to the invention, the embedded feature information of the voice is extracted through the convolutional network, and the time sequence feature information of the voice is extracted through the loop network, so that voice noise reduction is realized, and the perception quality and intelligibility of voice auditory sense are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of speech processing technology, and in particular to a speech noise reduction method, device, electronic device, storage medium and program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Traditional speech enhancement technology mainly relies on traditional digital signal processing methods based on statistical characteristics to perform speech noise reduction. Traditional digital signal processing methods based on statistical characteristics suppress noise by analyzing and processing the statistical characteristics of audio signals, thereby improving the quality and intelligibility of speech.

[0004] However, traditional digital signal processing methods based on statistical characteristics are unstable in both new and old environments and are difficult to adapt to complex and changing noise environments. Once the amount of noise is overestimated, speech distortion will occur. Summary of the invention

[0005] In view of this, the purpose of the present disclosure is to propose a speech noise reduction method, device, electronic device, storage medium and program product, which at least to a certain extent solve one of the technical problems in the related art.

[0006] Based on the above purpose, the first aspect of the exemplary embodiment of the present disclosure provides a speech noise reduction method, which is applied to a server, and the method includes:

[0007] Determine noisy speech data, and obtain spectral Mel filter coefficients based on the noisy speech data;

[0008] Extracting the spectral Mel filter coefficient features to obtain granular depth filter coefficients;

[0009] Noisy speech spectrum information of the noisy speech data is determined, and noise-reduced speech data is obtained based on the noisy speech spectrum information and the deep filter coefficients of the granularity.

[0010] Based on the same inventive concept, the second aspect of the exemplary embodiment of the present disclosure provides a speech noise reduction device, including:

[0011] A spectrum coefficient determination module is configured to determine noisy speech data and obtain spectrum Mel filter coefficients based on the noisy speech data;

[0012] A granularity coefficient determination module is configured to extract the spectral Mel filter coefficient features to obtain a granularity depth filter coefficient;

[0013] The noise reduction speech determination module is configured to determine the noisy speech spectrum information of the noisy speech data, and obtain the noise reduction speech data based on the noisy speech spectrum information and the deep filter coefficient of the granularity.

[0014] Based on the same inventive concept, a third aspect of the exemplary embodiment of the present disclosure provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0015] Based on the same inventive concept, a fourth aspect of the exemplary embodiments of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0016] Based on the same inventive concept, a fifth aspect of the exemplary embodiments of the present disclosure provides a computer program product, including computer program instructions. When the computer program instructions are executed on a computer, the computer executes the method described in the first aspect.

[0017] From the above, it can be seen that the embodiments of the present disclosure provide a method, device, electronic device, storage medium and program product for speech noise reduction, the method comprising: determining noisy speech data, obtaining spectral Mel filter coefficients based on the noisy speech data, extracting the features of the spectral Mel filter coefficients to obtain granular depth filter coefficients, determining noisy speech spectrum information of the noisy speech data, and obtaining noise-reduced speech data based on the noisy speech spectrum information and the granular depth filter coefficients. The present disclosure extracts embedded feature information of speech through a convolutional network, and extracts temporal feature information of speech through a recurrent network to achieve speech noise reduction, thereby improving the auditory perceptual quality and intelligibility of speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A schematic diagram of an application scenario of the speech noise reduction method provided by an exemplary embodiment of the present disclosure;

[0020] Figure 2 A flow chart of a speech noise reduction method provided by an exemplary embodiment of the present disclosure;

[0021] Figure 3 A structural schematic diagram of a speech noise reduction device provided by an exemplary embodiment of the present disclosure;

[0022] Figure 4 A schematic diagram of the hardware structure of an electronic device according to an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of a time-frequency domain convolutional recurrent neural network model of a speech denoising method provided by an exemplary embodiment of the present disclosure;

[0024] Figure 6 A schematic diagram of a one-to-one mapping of common filter coefficients of a speech noise reduction method provided by an exemplary embodiment of the present disclosure;

[0025] Figure 7 A schematic diagram of a many-to-one mapping of deep filter coefficients of a speech noise reduction method provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] It is understandable that before using the technical solutions disclosed in the embodiments of this application, the type, scope of use, usage scenarios, etc. of the personal information involved in this application should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0027] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present application according to the prompt message.

[0028] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0029] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation method of the present application. Other methods that meet the relevant laws and regulations may also be applied to the implementation method of the present application.

[0030] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, the principles and spirit of the present disclosure will be described with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0032] It should be understood herein that any number of elements in the drawings is for illustration rather than limitation, and any naming is only for distinction rather than having any limiting meaning.

[0033] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not represent any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connecting" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. The article "one" or "a" before an element does not exclude the existence of multiple such elements.

[0034] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0035] As described in the background technology, in the speech noise reduction scheme in the related art, the traditional digital signal processing method based on statistical characteristics is unstable in new and old environments and difficult to adapt to complex and changing noise environments. Once the noise amount is overestimated, speech distortion will occur, specifically:

[0036] Traditional noise reduction techniques work well for stationary noise, but are less effective for non-stationary, sudden sounds. In addition, as the signal-to-noise ratio decreases, the effectiveness of noise reduction methods also decreases. If noise is suppressed too much during noise reduction, useful information in the speech signal may be mistakenly eliminated, causing speech distortion. This distortion is a deformation of the audio and can be caused by using faulty equipment, overloading of sound in a specific frequency range, or excessive volume.

[0037] In order to solve the above problems, the present disclosure provides a speech noise reduction method, device, electronic device, storage medium and program product solution, which specifically include:

[0038] Determine noisy speech data, obtain spectral Mel filter coefficients based on the noisy speech data, extract features of the spectral Mel filter coefficients to obtain granular deep filter coefficients, determine noisy speech spectrum information of the noisy speech data, and obtain denoised speech data based on the noisy speech spectrum information and the granular deep filter coefficients.

[0039] After introducing the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

[0040] refer to Figure 1 , which is a schematic diagram of an application scenario of the speech noise reduction method provided by an exemplary embodiment of the present disclosure.

[0041] This application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, the server 102, and the data storage system 103 may be connected via a wired or wireless communication network to achieve data interaction.

[0042] The terminal device 101 may be an electronic device with data transmission and multimedia input / output functions close to the user side, including but not limited to a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device, a personal digital assistant (PDA) or other electronic devices capable of realizing the above functions. The electronic device may include a processor and a display screen with a touch input function, the display screen is used to present a graphical user interface, the graphical user interface can display an application interface, and the processor is used to process application data, generate a graphical user interface, and control the display of the graphical user interface on the display screen.

[0043] Server 102 and data storage system 103 can both be independent physical servers, or a server cluster or distributed system composed of multiple physical servers. They can also be cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0044] In some exemplary embodiments, the speech noise reduction method may be run on the terminal device 101 or the server 102 .

[0045] When the speech noise reduction method is run on the server 102 , the server 102 is used to provide a speech noise reduction service to a user of the terminal device 101 .

[0046] The server 102 determines the noisy speech data, and the server 102 obtains the spectral Mel filter coefficients based on the noisy speech data;

[0047] The server 102 extracts the spectral Mel filter coefficient features to obtain a granular depth filter coefficient;

[0048] The server 102 determines noisy speech spectrum information of the noisy speech data, and the server 102 obtains noise-reduced speech data based on the noisy speech spectrum information and the deep filter coefficients of the granularity.

[0049] A large amount of training data is stored in the data storage system 103, each of which includes a noisy speech and a clean speech. The server 102 can train the time-frequency domain convolutional recurrent network model based on a large amount of training data, so that the time-frequency domain convolutional recurrent network model can optimize the processing of noisy speech. The source of the training data includes but is not limited to an existing database, data crawled from the Internet, or data uploaded by the user when using the client. When the sample information output by the time-frequency domain convolutional recurrent network model meets certain requirements and the difference between the label, the server 102 can provide users with a higher quality noise reduction speech service based on the time-frequency domain convolutional recurrent network model.

[0050] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principle of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario.

[0051] refer to Figure 2 , a speech noise reduction method is applied to a server, the method comprising the following steps:

[0052] Step S210: determine noisy speech data, and obtain spectral Mel filter coefficients based on the noisy speech data.

[0053] In specific implementation, noisy speech data:

[0054] Noisy speech data refers to a signal in which noise signals are mixed into speech signals. Specifically, noisy speech signals are composed of original speech signals and noise signals. Noisy speech data is very common in practical applications. For example, in recording, communication, and audio processing, speech signals are inevitably interfered by various noises, which affects the clarity and recognition rate of speech.

[0055] In specific implementation, the method of determining the noisy speech data includes but is not limited to at least one of the following:

[0056] Recording can be done in a real environment using a recording device, through the collection of open source speech datasets, or through generation by audio software.

[0057] In the above exemplary embodiment, the determination of noisy speech data is introduced. The following describes a method for obtaining spectral Mel filter coefficients based on the noisy speech data:

[0058] In a specific implementation, based on the noisy speech data, a spectrum Mel filter coefficient is obtained, including:

[0059] First, the low-frequency band spectrum of the noisy speech data is obtained, and the time domain waveform is converted into a time spectrum using a short-time Fourier transform. The specific operation is to divide the speech into 20 ms frames and then add a window to each frame waveform to prevent spectrum leakage, and then use a fast Fourier transform to convert each frame of the time domain waveform into a spectrum, and then intercept the frequency point corresponding to 0 to 5 kHz as the low-frequency band spectrum;

[0060] As a specific embodiment, after dividing the speech into 20 ms frames, each frame waveform is windowed to prevent spectrum leakage. Different time frame parameters can be set according to different situations to prevent spectrum leakage by windowing each frame waveform.

[0061] Secondly, the full-band Mel filter bank features of the noisy speech data are obtained. The Mel frequency is closer to the hearing mechanism of the human ear than the normal frequency mechanism. It grows very fast in the low frequency range, but very slowly in the high frequency range. Its corresponding relationship with the normal frequency is as follows:

[0062]

[0063] Where f is the normal frequency and m is the corresponding Mel frequency. The energy spectrum of each frame is passed through a set of Mel-scale triangular filter banks to obtain the Mel filter bank features, in which a triangular bandpass filter is used to smooth the spectrum to eliminate harmonics and highlight the speech resonance peaks. At the same time, the original spectrum can be extracted to reduce the amount of calculation.

[0064] Step S220: extract the spectral Mel filter coefficient features to obtain granular depth filter coefficients.

[0065] In this exemplary embodiment, the spectral Mel filter coefficients include: low-band spectral Mel filter coefficients and full-band spectral Mel filter coefficients.

[0066] In a specific implementation, the low-band spectrum Mel filter coefficients include:

[0067] Mel filter coefficients of the 0-5KHz low-frequency band spectrum of noisy speech.

[0068] In specific implementation, the full-band spectrum Mel filter coefficients include:

[0069] Mel filter coefficients for the full-band spectrum of noisy speech from 0 to 48 kHz.

[0070] In the above exemplary embodiment, the spectral Mel filter coefficients are introduced. The following describes a method for extracting the features of the spectral Mel filter coefficients to obtain the granular depth filter coefficients:

[0071] In this exemplary embodiment, the spectral Mel filter coefficient features are extracted to obtain the granular depth filter coefficients, including:

[0072] The granularity of the deep filter coefficients includes: fine-grained deep filter coefficients and coarse-grained Mel filter coefficients. The low-band spectrum Mel filter coefficient features are extracted to obtain first speech feature extraction information. The time series features of the first speech feature extraction information are extracted to obtain first high-dimensional embedding feature information. Based on the first high-dimensional embedding feature information, the fine-grained deep filter coefficients are obtained. The full-band spectrum Mel filter coefficient features are extracted to obtain second speech feature extraction information. The time series features of the second speech feature extraction information are extracted to obtain second high-dimensional embedding feature information. Based on the second high-dimensional embedding feature information, the coarse-grained Mel filter coefficients are obtained. Based on the fine-grained deep filter coefficients and the coarse-grained Mel filter coefficients, the granularity of the deep filter coefficients are obtained.

[0073] In a specific implementation, extracting the low-frequency band spectrum Mel filter coefficient feature to obtain first speech feature extraction information includes:

[0074] refer to Figure 5 The time-frequency domain convolutional recurrent neural network model includes: an encoder and a decoder. The low-frequency band spectrum submodule of the encoder includes: three convolution blocks and two gated recurrent units. The convolution block is composed of a two-dimensional convolution layer, a batch normalization layer and an activation function (Relu) layer. The convolution layer can be regarded as a fully connected layer with parameter sharing and local connection. It can extract features while greatly reducing the computational complexity. The two-dimensional convolution used here enables the convolution layer to have a receptive field in the time domain and frequency domain, and learn the features of different levels of speech. The batch normalization layer normalizes the data. First, the mean and variance of each batch of data are calculated, the batch mean is subtracted from the data and divided by the standard deviation of the batch, and then the two parameters are linearly multiplied and added to output. These two parameters are obtained during training. Relu is used as the activation function, and the calculation formula is as follows:

[0075] ReLU(x)=max(0,x).

[0076] The 0-5KHz low-frequency band spectrum Mel filter coefficients of the noisy speech are input into the encoder low-frequency band spectrum submodule, and the speech features are extracted through three convolution blocks to obtain the first speech feature extraction information.

[0077] In a specific implementation, extracting time series features from the first speech feature extraction information to obtain first high-dimensional embedding feature information includes:

[0078] refer to Figure 5 ,In the low-frequency band spectrum submodule of the time-frequency domain convolutional recurrent neural network model encoder, the first speech feature extraction information is ,extracted through a gated recurrent unit to extract the ,temporal feature information, and finally the encoder outputs the first high-dimensional ,embedded feature information of the speech.

[0079] In a specific implementation, obtaining the fine-grained depth filter coefficient based on the first high-dimensional embedded feature information includes:

[0080] refer to Figure 5 The low-band decoder submodule of the time-frequency domain convolutional recurrent neural network model decoder includes: two gated recurrent units and a linear layer. The first high-dimensional embedded feature information is input into the decoder submodule, and after two gated recurrent units and a linear layer of the low-band decoder, the fine-grained deep filter coefficients based on the frequency point are finally output. The deep filter coefficients output by the low-band decoder are a variant of the general filter coefficients. Common filter coefficients are element-wise multiplication (element-wise multiplication) with the original signal time-frequency point, that is, one-to-one mapping (one-to-one mapping) reference Figure 6 , and the depth filter coefficients can use the area around the current time-frequency point to achieve many-to-one mapping (many-to-one mapping) reference Figure 7 , emphasizing the correlation of adjacent time-frequency regions to generate more accurate filter coefficients, that is, changing the original mask with an output dimension of 1 (a mark used to block or ignore certain data) to a filter with an output dimension of (2L+1,2I+1) (a kernel used to perform convolution operations on input data, whose output dimension can be changed by adjusting parameters). Specifically, assuming that the current time-frequency point is Y(t,f), if the deep filter is expanded along the time axis (Time) with an order of L and along the frequency axis (Frequency) with an order of I, then the deep filter coefficient estimated by the neural network for Y(t,f) is H (t,f) (L+l,I+i), the estimated value calculation formula after filtering this point is as follows:

[0081]

[0082] In a specific implementation, the full-band spectrum Mel filter coefficient feature is extracted to obtain the second speech feature extraction information, including:

[0083] refer to Figure 5 In the full-band submodule of the time-frequency domain convolutional recurrent neural network model encoder, it includes: three convolution blocks and two gated recurrent units. The convolution block is composed of a two-dimensional convolution layer, a batch normalization layer and an activation function (Relu) layer. The convolution layer can be regarded as a fully connected layer with parameter sharing and local connection, which can extract features while greatly reducing the computational complexity. The two-dimensional convolution used here enables the convolution layer to have a receptive field in the time domain and frequency domain, and learn the features of different levels of speech. The batch normalization layer normalizes the data. First, the mean and variance of each batch of data are calculated, the batch mean is subtracted from the data and divided by the standard deviation of the batch, and then the two parameters are linearly multiplied and added to output. These two parameters are obtained during training. Relu is used as the activation function, and the calculation formula is as follows:

[0084] ReLU(x)=max(0,x).

[0085] The 0-48KHz full-band spectrum Mel filter coefficients of the noisy speech are input into the full-band submodule of the encoder, and the speech features are extracted through three convolution blocks to obtain the second speech feature extraction information.

[0086] In a specific implementation, extracting the time series features of the second speech feature extraction information to obtain the second high-dimensional embedding feature information includes:

[0087] refer to Figure 5 ,In the full-band submodule of the time-frequency domain convolutional recurrent neural network model encoder, the second speech feature extraction information extracts the timing feature information through a gated recurrent unit, and finally the encoder outputs the second high-dimensional embedded feature information of the speech.

[0088] In a specific implementation, obtaining the coarse-grained Mel filter coefficient based on the second high-dimensional embedded feature information includes:

[0089] refer to Figure 5 The full-band decoder submodule of the time-frequency domain convolutional recurrent neural network model decoder includes: three deconvolution blocks and two gated recurrent units. The second high-dimensional embedded feature information is input into the full-band decoder submodule. After passing through the three deconvolution blocks and two gated recurrent units of the full-band decoder, the deconvolution layer simultaneously receives the convolution layer output in the encoder and the previous deconvolution layer as input to realize skip connection, and finally outputs the coarse-grained Mel filter coefficients based on the Mel filter.

[0090] In the above exemplary embodiment, it is introduced that the spectral Mel filter coefficient feature is extracted to obtain the granularity depth filter coefficient. The following is an introduction to obtain the granularity depth filter coefficient based on the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient:

[0091] In this exemplary embodiment, the manner of obtaining the depth filter coefficient of the granularity based on the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient includes but is not limited to at least one of the following:

[0092] The fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are added to obtain the depth filter coefficient of the granularity, or the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are concatenated to obtain the depth filter coefficient of the granularity, or the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are processed based on a neural network to obtain the depth filter coefficient of the granularity.

[0093] In a specific implementation, the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are added to obtain the depth filter coefficient of the granularity, including:

[0094] The fine-grained depth filter coefficients are directly added to the coarse-grained Mel filter coefficients to obtain a new feature vector.

[0095] In a specific implementation, the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients are concatenated to obtain the depth filter coefficients of the granularity, including:

[0096] Concatenate the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients together to form a longer feature vector;

[0097] As a specific embodiment, it is assumed that in the Mel Frequency Cepstral Coefficient (MFCC), the first 12 Mel filter coefficients are usually selected and the energy of a current frame is concatenated, which is 13 coefficients in total.

[0098] In a specific implementation, the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients are processed based on a neural network to obtain the depth filter coefficients of the granularity, including:

[0099] Through the neural network, the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients are taken as input, and a new, comprehensive feature vector is generated through network learning and optimization.

[0100] Step S230: determine the noisy speech spectrum information of the noisy speech data, and obtain the noise-reduced speech data based on the noisy speech spectrum information and the deep filter coefficient of the granularity.

[0101] Next, the noisy speech spectrum information of the noisy speech data will be determined:

[0102] In a specific implementation, the method of determining the noisy speech spectrum information of the noisy speech data includes but is not limited to at least one of the following:

[0103] Use the Fast Fourier Transform (FFT) method to perform spectrum analysis, process the noisy speech data through filters, or analyze the noisy speech data through convolutional neural networks.

[0104] In the above exemplary embodiment, the noisy speech spectrum information of the noisy speech data is determined. The following describes a method for obtaining the noise-reduced speech data based on the noisy speech spectrum information and the deep filter coefficients of the granularity:

[0105] In this exemplary embodiment, based on the noisy speech spectrum information and the deep filter coefficient of the granularity, the noise-reduced speech data is obtained, including:

[0106] The noisy speech spectrum information of the noisy speech data is determined, noise-reduced speech spectrum information is obtained based on the noisy speech spectrum information and the deep filter coefficient of the granularity, and the noise-reduced speech spectrum information is transformed to obtain noise-reduced speech data.

[0107] In a specific implementation, based on the noisy speech spectrum information and the deep filter coefficient of the granularity, the noise-reduced speech spectrum information is obtained, including:

[0108] The generated full-band Mel filter coefficients are linearly interpolated to match the dimension of the noisy speech spectrum information, and then the granular depth filter coefficients are applied to the noisy speech spectrum information and multiplied to obtain the noise-reduced speech spectrum.

[0109] In a specific implementation, the noise reduction speech spectrum information is transformed to obtain noise reduction speech data, including:

[0110] The denoised speech is obtained by using inverse short-time Fourier transform on the denoised speech spectrum information.

[0111] In the above exemplary embodiment, the noise-reduced speech data is obtained based on the noisy speech spectrum information and the deep filter coefficients of the granularity. The following is a method of inputting the noise-reduced speech data into a pre-trained time-frequency domain convolutional recurrent network model for evaluation:

[0112] In this exemplary embodiment, the noise reduction speech data is input into a pre-trained time-frequency domain convolutional recurrent network model for evaluation, and the time-frequency domain convolutional recurrent network model is trained by the following method, including:

[0113] Determine noise data and a speech data set, obtain a noise data set based on the noise data and the speech data set, determine noisy speech sample information of the noise data set and clean speech label information of the noise data set corresponding to the noisy speech sample, input the noisy speech sample information into the time-frequency domain convolutional recurrent network model, obtain denoised speech prediction information output by the model, determine the difference between the clean speech label information and the denoised speech prediction information, and based on the difference, update the model parameters through a loss function until the difference between the clean speech label information and the denoised speech prediction information is minimized, thereby obtaining the pre-trained time-frequency domain convolutional recurrent network model.

[0114] In a specific implementation, determining noise data and a speech data set, and obtaining the noise data set based on the noise data and the speech data set includes:

[0115] Open source noise data and speech data sets were used, and then the common data enhancement methods in the field of deep learning were adopted. Different noises, different signal-to-noise ratios, and different reverberation impulse responses were added to the speech to generate noisy speech, and finally a training data set of noisy speech was obtained.

[0116] In a specific implementation, until the difference between the clean speech label information and the noise reduction speech prediction information is minimized, it includes:

[0117] PESQ, STOI and SISNR indicators are used to verify the model effect. The PESQ (Perceptual evaluation of speech quality) indicator is called objective speech quality evaluation. The higher the indicator value, the higher the speech perception. STOI (Short-Time Objective Intelligibility) is called short-time objective intelligibility. It is one of the important indicators for measuring speech intelligibility. The higher the indicator value, the higher the speech intelligibility. STOI has a strong correlation of 95% with noise and time-frequency noisy speech, especially for the noise reduction algorithm of frequency domain processing. SISNR (Scale-invariant signal to noise ratio) is the scale-invariant signal-to-noise ratio, which is used to measure the relative size of speech and noise in the signal. The higher the value, the more speech components and the better the noise reduction effect. The combination of the above three indicators is a very popular speech quality evaluation method today. Referring to Table 1, it can be seen from the three indicators that the speech quality of noisy speech is greatly improved after being processed by the model.

[0118] Table 1 Evaluation index table of noise reduction speech

[0119] PESQ STOI SISNR Noisy speech 1.97 0.920 8.42 Noise reduction voice 2.51 0.922 14.0

[0120] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.

[0121] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a speech noise reduction device.

[0123] refer to Figure 3 , the speech noise reduction device comprises:

[0124] The spectrum coefficient determination module 310 is configured to determine the noisy speech data, and obtain the spectrum Mel filter coefficients based on the noisy speech data;

[0125] The granularity coefficient determination module 320 is configured to extract the spectral Mel filter coefficient features to obtain the granularity depth filter coefficients;

[0126] The noise reduction speech determination module 330 is configured to determine the noisy speech spectrum information of the noisy speech data, and obtain the noise reduction speech data based on the noisy speech spectrum information and the granular deep filter coefficients.

[0127] In this exemplary embodiment, the spectrum coefficient determination module 310 is specifically configured as follows:

[0128] Noisy speech data is determined, and spectral Mel filter coefficients are obtained based on the noisy speech data.

[0129] In this exemplary embodiment, the granularity coefficient determination module 320 is specifically configured as follows:

[0130] The spectral Mel filter coefficients include: low-band spectral Mel filter coefficients and full-band spectral Mel filter coefficients, the granularity of the deep filter coefficients includes: fine-grained deep filter coefficients and coarse-grained Mel filter coefficients, the low-band spectral Mel filter coefficient features are extracted to obtain first speech feature extraction information, the time series features of the first speech feature extraction information are extracted to obtain first high-dimensional embedding feature information, the fine-grained deep filter coefficients are obtained based on the first high-dimensional embedding feature information, the full-band spectral Mel filter coefficient features are extracted to obtain second speech feature extraction information, the The second speech feature extraction information is extracted based on the time series features to obtain the second high-dimensional embedded feature information, and the coarse-grained Mel filter coefficient is obtained based on the second high-dimensional embedded feature information, and the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are added to obtain the depth filter coefficient of the granularity, or the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are concatenated to obtain the depth filter coefficient of the granularity, or the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient are processed based on a neural network to obtain the depth filter coefficient of the granularity.

[0131] In this exemplary embodiment, the noise reduction speech determination module 330 is specifically configured as follows:

[0132] The noisy speech spectrum information of the noisy speech data is determined, noise-reduced speech spectrum information is obtained based on the noisy speech spectrum information and the deep filter coefficient of the granularity, and the noise-reduced speech spectrum information is transformed to obtain noise-reduced speech data.

[0133] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0134] The device of the above embodiment is used to implement the corresponding speech noise reduction method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0135] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech noise reduction method described in any of the above embodiments when executing the program.

[0136] Figure 4A more specific schematic diagram of the hardware structure of an electronic device provided in this embodiment is shown, and the device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 in the device.

[0137] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0138] The memory 1020 may be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 may store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0139] The input / output interface 1030 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0140] The communication interface 1040 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0141] The bus 1050 includes a path that transmits information between the various components of the device (eg, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0142] It should be noted that, although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0143] The electronic device of the above embodiment is used to implement the corresponding speech noise reduction method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0144] Based on the same inventive concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the speech noise reduction method described in any of the above embodiments.

[0145] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0146] The above-mentioned non-transitory computer-readable storage medium can be any available medium or data storage device that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0147] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the speech noise reduction method described in any embodiment in the above exemplary method part, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0148] Based on the same inventive concept, corresponding to the speech noise reduction method described in any of the above embodiments, the present disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer so that the computer and / or the processor execute the speech noise reduction method. Corresponding to the execution subject corresponding to each step in each embodiment of the speech noise reduction method, the processor that executes the corresponding step may belong to the corresponding execution subject.

[0149] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the speech noise reduction method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0150] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." In addition, in some embodiments, the present disclosure may also be implemented in the form of a computer program product in one or more computer-readable media, which contains computer-readable program code.

[0151] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive examples) of computer-readable storage media may include, for example: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device.

[0152] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0153] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0154] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] It should be understood that each box in the flowchart and / or block diagram and the combination of boxes in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine, and these computer program instructions are executed by a computer or other programmable data processing device to produce a device that implements the functions / operations specified in the boxes in the flowchart and / or block diagram.

[0156] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce a product that includes an instruction device that implements the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0157] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby enabling the instructions executed on the computer or other programmable device to provide a process for implementing the functions / operations specified in the blocks in the flowchart and / or block diagram.

[0158] In addition, although the operations of the disclosed method are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0159] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0161] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0162] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the provided drawings. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform to be implemented in the embodiments of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). In the case of elaborating specific details (e.g., circuits) to describe exemplary embodiments of the present application, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0163] Although the present application has been described in conjunction with specific embodiments of the present application, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0164] The embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of the present application.

[0165] Although the spirit and principle of the present disclosure have been described with reference to several specific embodiments, it should be understood that the present disclosure is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present disclosure is intended to cover various modifications and equivalent arrangements included in the spirit and scope of the attached claims. The scope of the attached claims conforms to the broadest interpretation, thereby including all such modifications and equivalent structures and functions.

Claims

1. A speech noise reduction method, characterized in that: include: Determine noisy speech data, and obtain spectral Mel filter coefficients based on the noisy speech data; Extracting the spectral Mel filter coefficient features to obtain granular depth filter coefficients; Noisy speech spectrum information of the noisy speech data is determined, and noise-reduced speech data is obtained based on the noisy speech spectrum information and the deep filter coefficients of the granularity.

2. The method according to claim 1, characterized in that The spectral Mel filter coefficients include: low-band spectral Mel filter coefficients and full-band spectral Mel filter coefficients; The granular depth filter coefficients include: fine-grained depth filter coefficients and coarse-grained Mel filter coefficients; The extracting the spectral Mel filter coefficient feature to obtain the granular depth filter coefficient includes: Extracting the low-frequency band spectral Mel filter coefficient features to obtain first speech feature extraction information; Extracting time series features of the first speech feature extraction information to obtain first high-dimensional embedding feature information; Based on the first high-dimensional embedded feature information, obtaining the fine-grained deep filter coefficients; Extracting the full-band spectrum Mel filter coefficient features to obtain second speech feature extraction information; Extracting the time series features of the second speech feature extraction information to obtain second high-dimensional embedding feature information; Based on the second high-dimensional embedded feature information, obtaining the coarse-grained Mel filter coefficients; Based on the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients, the depth filter coefficients of the granularity are obtained.

3. The method according to claim 2, characterized in that The obtaining of the depth filter coefficient of the granularity based on the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient comprises: Adding the fine-grained depth filter coefficient and the coarse-grained Mel filter coefficient to obtain the depth filter coefficient of the granularity; or, Concatenate the fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients to obtain the depth filter coefficients of the granularity; or, The fine-grained depth filter coefficients and the coarse-grained Mel filter coefficients are processed based on a neural network to obtain the depth filter coefficients of the granularity.

4. The method according to claim 1, characterized in that: The step of determining the noisy speech spectrum information of the noisy speech data and obtaining the noise-reduced speech data based on the noisy speech spectrum information and the deep filter coefficient of the granularity includes: Determine the noisy speech spectrum information of the noisy speech data, and obtain the noise-reduced speech spectrum information based on the noisy speech spectrum information and the deep filter coefficient of the granularity; The noise reduction speech spectrum information is transformed to obtain noise reduction speech data.

5. The method according to claim 1, characterized in that After determining the noisy speech spectrum information of the noisy speech data and obtaining the denoised speech data based on the noisy speech spectrum information and the deep filter coefficients of the granularity, the method further includes: inputting the denoised speech data into a pre-trained time-frequency domain convolutional recurrent network model for evaluation.

6. The method according to claim 5, characterized in that The step of inputting the noise reduction speech data into a pre-trained time-frequency domain convolutional recurrent network model for evaluation comprises training the time-frequency domain convolutional recurrent network model by the following method: Determine noise data and a speech data set, and obtain the noise data set based on the noise data and the speech data set; Determine noisy speech sample information of the noisy data set and clean speech label information of the noisy data set corresponding to the noisy speech sample; Inputting the noisy speech sample information into the time-frequency domain convolutional recurrent network model to obtain noise-reduced speech prediction information output by the model; Determining a difference between the clean speech tag information and the noise-reduced speech prediction information; Based on the difference, the model parameters are updated through a loss function until the difference between the clean speech label information and the denoised speech prediction information is minimized, thereby obtaining the pre-trained time-frequency domain convolutional recurrent network model.

7. A speech noise reduction device, characterized in that: include: A spectrum coefficient determination module is configured to determine noisy speech data and obtain spectrum Mel filter coefficients based on the noisy speech data; A granularity coefficient determination module is configured to extract the spectral Mel filter coefficient features to obtain a granularity depth filter coefficient; The noise reduction speech determination module is configured to determine the noisy speech spectrum information of the noisy speech data, and obtain the noise reduction speech data based on the noisy speech spectrum information and the deep filter coefficient of the granularity.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer program instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.