Speech classification methods, devices, electronic equipment and storage media

Abstract

This invention discloses a speech classification method, apparatus, electronic device, and storage medium. The invention can acquire speech to be identified; perform time-frequency analysis on the speech to be identified to obtain a spectrogram; use an identification model to identify the type of the spectrogram, obtaining a type score; when the type score falls within a first preset range, the speech type is determined to be a natural human voice; when the type score falls within a second preset range, the speech type is determined to be a spliced / forged voice. In this invention, the spectrogram of the speech to be identified can be input into the identification model to identify its type, thereby determining whether the speech to be identified is a natural human voice or a spliced / forged voice. Therefore, this solution can improve the accuracy of speech classification.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically to a speech classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, when communication networks provide information retrieval and network services, they must ensure that the information of service recipients is not threatened by eavesdropping, theft, or tampering to meet people's most basic security needs. There are instances of artificially generated and spliced ​​speech, such as maliciously deleting, adding to, or completely splicing together someone else's voice into another sentence, which poses a significant security risk.

[0003] Existing speech classification methods have been proposed to address this potential problem. However, since these existing technologies cannot effectively distinguish between spliced ​​speech, there is an urgent need for a new speech classification method to improve the accuracy of speech classification. Summary of the Invention

[0004] This invention provides a speech classification method, apparatus, electronic device, and storage medium that can improve the accuracy of speech classification.

[0005] This invention provides a speech classification method, including: Obtain the speech to be identified; Time-frequency analysis is performed on the speech to be identified to obtain its spectrogram. A discrimination model is used to identify the type of the spectrogram and obtain the type score of the spectrogram. When the type score falls within the first preset range, the speech type of the speech to be identified is determined to be the natural human voice type. When the type score falls within the second preset range, the speech type of the speech to be identified is determined to be a splicing forgery type.

[0006] This invention also provides a voice classification device, comprising: The acquisition unit is used to acquire the speech to be identified; The time-frequency unit is used to perform time-frequency analysis on the speech to be identified and obtain the spectrogram of the speech to be identified. The scoring unit is used to perform type identification on the spectrogram using a discrimination model to obtain a type score for the spectrogram. The natural human voice unit is used to determine the voice type of the speech to be identified as natural human voice when the type score belongs to a first preset range. The splicing forgery unit is used to determine that the speech type of the speech to be identified is a splicing forgery type when the type score belongs to a second preset range.

[0007] In some embodiments, the scoring unit further includes: Acquisition subunit, used to acquire natural human speech; The segmentation subunit is used to segment natural human speech to obtain speech segments. The splicing subunit is used to randomly splice speech segments based on natural human voice speech to obtain spliced ​​speech; The time-frequency subunit is used to perform time-frequency analysis on natural human speech and spliced ​​speech respectively, to obtain the spectrogram of natural human speech and the spectrogram of spliced ​​speech. The training subunit is used to train the preset model using the spectrograms of natural human speech and spliced ​​speech as training samples until the preset model converges, thus obtaining the discrimination model.

[0008] In some embodiments, the speech segment of natural human voice includes a speech segment of a first length and a speech segment of a second length, and the segmented subunit includes: The first segmentation submodule is used to segment the natural human voice speech according to a first length to obtain a speech segment of the first length of the natural human voice speech. The second segmentation submodule is used to segment the natural human voice speech according to the second length to obtain speech segments of the second length of the natural human voice speech.

[0009] In some embodiments, the segmented subunit includes: The acquisition submodule is used to acquire a first segment set and a second segment set. The first segment set includes a speech segment of a first length of natural human voice, and the second segment set includes a speech segment of a second length of natural human voice. The first selection submodule is used to randomly select a speech segment from the first segment set as the first speech segment; The second selection submodule is used to randomly select a speech segment from the second segment set as the second speech segment; The splicing submodule is used to splice the first and second speech segments to obtain the spliced ​​speech.

[0010] In some embodiments, the first selection submodule is specifically used for: Randomly select N audio segments from the first segment set as the first audio segment, where N is a positive integer; The second selection submodule is specifically used for: Randomly select N speech segments from the second segment set as the second speech segment.

[0011] In some embodiments, the splicing submodule is used for: The order of the first and second audio segments is shuffled. The scrambled first and second speech segments are spliced ​​together end to end, and the audio at the splicing boundary is smoothed to obtain the spliced ​​speech.

[0012] In some embodiments, the discrimination model includes multiple feature extraction blocks with identical structures, and a scoring unit, including: The previous sub-unit is used to determine the local features of the previous time step, where the first local feature of the previous time step is the spectrogram; The current sub-unit is used to extract features from the current time step and determine the local features of the current time step based on the local features of the previous time step. The previous time step is the time step before the current time step. The resulting molecular unit is used to calculate the type score of the spectrogram based on spectrogram features, which include local features at the current time.

[0013] In some embodiments, the feature extraction block includes a first function, a second function, and an attention network. The attention network is used to assign weights to features at each location in the spectrogram. The current subunit includes: The first extraction submodule is used to perform feature extraction on the local features of the previous time step using the first function at the current time step to obtain the first local features; The attention submodule is used to assign attention weights to the current attention network as the first local feature, and obtain the weighted first local feature. The second extraction submodule is used to perform feature extraction on the weighted first local features using the second function at the current time, so as to obtain the local features at the current time.

[0014] In some embodiments, the attention network is used to assign weights to the features at each channel position in the spectrogram, and the attention submodule is used for: The first local feature is split into channels to obtain the channel sub-features corresponding to each channel of the first local feature; The channel sub-features of the channel are scored to obtain the channel score; Based on the channel score, the channel sub-features of the channel are weighted to obtain the weighted sub-features of the channel; The first local feature after weighting is obtained, which includes the weighted sub-features of each channel.

[0015] In some embodiments, the attention network is used to assign weights to features at each spatial location in the spectrogram. Specifically, when the attention submodule is used to obtain the weighted first local features, which include weighted sub-features for each channel, it is used to: Spatial decomposition is performed on the channel weighted features to obtain spatial sub-features corresponding to each space. The channel weighted features include the channel weighted sub-features. The spatial sub-features of each space are concatenated to obtain the concatenated features; The spatial sub-features of the space are scored based on the splicing features to obtain the spatial score of the space; Based on the spatial score, the spatial sub-features of the space are weighted to obtain the weighted sub-features of the space. The first local feature is obtained after weighting, which includes the weighted sub-features of each space.

[0016] In some embodiments, the current subunit is used for: Max pooling is applied to the local features from the previous time step to obtain the pooled local features from the previous time step. The feature extraction block at the current time step is used to determine the local features at the current time step based on the local features of the previous time step after pooling.

[0017] In some embodiments, the time-frequency unit is used for: Multiple local sampled audios are obtained by sliding sampling on the speech to be identified using a preset sampling window with a preset step size; Perform time-frequency conversion on each local sampled audio to obtain the spectrogram of each local sampled audio; The spectrogram is obtained, which includes the spectrum of each local sampled audio.

[0018] This invention also provides an electronic device, including a memory storing multiple instructions; the processor loads instructions from the memory to execute steps in any of the speech classification methods provided in this invention.

[0019] This invention also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the speech classification methods provided in this invention.

[0020] The embodiments of the present invention can acquire the speech to be identified; perform time-frequency analysis on the speech to be identified to obtain a spectrogram of the speech to be identified; use an identification model to identify the type of the spectrogram to obtain a type score of the spectrogram; when the type score is within a first preset range, determine that the speech type of the speech to be identified is a natural human voice type; when the type score is within a second preset range, determine that the speech type of the speech to be identified is a spliced ​​forgery type.

[0021] Because artificially spliced ​​speech exhibits a visually distinct fragmented appearance in its spectrogram, this invention utilizes spectrograms to effectively identify fabricated speech. Specifically, the spectrogram of the speech to be identified is input into a discrimination model to determine its type, thereby classifying it as either a natural human voice or a fabricated one. Therefore, this solution can improve the accuracy of speech classification. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1a This is a schematic diagram of a scenario for the speech classification method provided in an embodiment of the present invention; Figure 1b This is a flowchart illustrating the speech classification method provided in an embodiment of the present invention; Figure 2a This is the spectrogram of the spliced ​​and forged speech of the speech classification method provided in the embodiments of the present invention; Figure 2b This is a schematic diagram of random segmentation and splicing of the speech classification method provided in this embodiment of the invention; Figure 2c This is a schematic diagram of the structure of the discrimination model in the speech classification method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the structure of the speech classification device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a smartphone provided in an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] This invention provides a speech classification method, apparatus, electronic device, and storage medium.

[0026] Specifically, the voice classification device can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet, smart Bluetooth device, laptop, or personal computer (PC); the server can be a single server or a server cluster consisting of multiple servers.

[0027] In some embodiments, the voice classification device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the voice classification method of the present invention.

[0028] In some embodiments, the server may also be implemented as a terminal.

[0029] For example, refer to Figure 1a The electronic device can be a smartphone, which can acquire the speech to be identified; perform time-frequency analysis on the speech to be identified to obtain a spectrogram of the speech to be identified; use an identification model to identify the type of the spectrogram to obtain a type score of the spectrogram; when the type score is within a first preset range, determine that the speech type of the speech to be identified is a natural human voice type; when the type score is within a second preset range, determine that the speech type of the speech to be identified is a spliced ​​forgery type.

[0030] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0031] Artificial intelligence (AI) is a technology that uses digital computers to simulate human perception of the environment, acquire knowledge, and use that knowledge. This technology can enable machines to have functions similar to human perception, reasoning, and decision-making. AI technology mainly includes several major areas such as computer vision, speech processing, natural language processing, machine learning, and deep learning.

[0032] Computer vision (CV) is a technology that uses computers to perform operations such as recognition, measurement, and further processing of target images, replacing the human eye. Computer vision technology typically includes image processing, image recognition, image semantic understanding, image retrieval, virtual reality, augmented reality, and simultaneous localization and mapping (SLAM), such as image processing techniques like image coloring and image outline extraction.

[0033] In this embodiment, a speech classification method based on image recognition is provided, such as... Figure 1b As shown, the specific process of this speech classification method can be as follows: 101. Obtain the speech to be identified.

[0034] There are various methods for acquiring the speech to be identified, such as using an onboard microphone system to collect the speech; obtaining it from a speech database via a network; or directly reading the speech if it is stored in local memory, etc.

[0035] 102. Perform time-frequency analysis on the speech to be identified to obtain the spectrogram of the speech to be identified.

[0036] Time-frequency analysis, or joint time-frequency analysis (JTFA), describes the energy density or intensity of a signal at different times and frequencies simultaneously by designing a joint function of time and frequency, i.e., a spectrogram.

[0037] The spectrogram provides joint distribution information in the time and frequency domains, clearly describing the relationship between signal frequency and time.

[0038] In the spectrogram, the horizontal axis represents time, and the vertical axis represents frequency. Several waves exist on the vertical axis at the same time, which overlap to form the final wave.

[0039] Sound signals are one-dimensional signals; intuitively, only time-domain information is visible, not frequency-domain information. Fourier transform can transform the signal to the frequency domain, yielding a sound spectrum. However, a spectrum only describes a specific point in time and cannot represent a period of time. Therefore, by overlaying multiple consecutive spectrums onto a single graph, a sound spectrogram can be obtained.

[0040] refer to Figure 2a Since splicing and forging speech involves randomly piecing together speech segments, and each speech segment covers a different frequency band with different energy values, the spectrogram of spliced ​​and forged speech will show more obvious energy value jumps in certain frequency bands compared to the spectrogram of natural human voice, resulting in a fragmented and discontinuous visual appearance in its spectrogram.

[0041] Therefore, in this embodiment, using spectrograms as sound features to determine the speech type of the speech to be identified can improve the accuracy of speech classification.

[0042] In some embodiments, the spectrogram of the speech to be identified can be calculated using the Short-time Fourier Transform (STFT).

[0043] STFT involves multiplying the speech x(t) to be identified by a time-finite window function w(t) before performing a Fourier transform, shifting the window function along the time axis of the speech to be identified, and performing segment-by-segment spectral analysis of the speech to be identified, thereby obtaining a set of local spectrograms.

[0044] The short-time Fourier transform of the speech x(t) to be identified at time t is defined as:

[0045] in, For window functions, This is the offset.

[0046] For example, in some embodiments, step 102 includes the following steps: Multiple local sampled audios are obtained by sliding sampling on the speech to be identified using a preset sampling window with a preset step size; Perform time-frequency conversion on each local sampled audio to obtain the spectrogram of each local sampled audio; The spectrogram is obtained, which includes the spectrum of each local sampled audio.

[0047] The sampling window is the window function.

[0048] In some embodiments, the window function can be a Hamming window with a frame length of 25 milliseconds and a frame shift of 10 milliseconds.

[0049] 103. Use a discrimination model to identify the type of the spectrogram and obtain the type score of the spectrogram.

[0050] The discrimination model can be any type of artificial neural network (ANN), such as convolutional neural networks (CNN), deep residual networks (ResNet), and so on.

[0051] The identification model identifies the type of the spectrogram, obtains the probability distribution of the type of the spectrogram, and thus obtains the type score of the spectrogram.

[0052] In some embodiments, the discrimination model needs to be pre-trained before step 103, therefore the following steps are included before step 103: (1) Acquire natural human speech; (2) Based on the natural human voice speech, segmentation processing is performed to obtain speech segments of natural human voice speech; (3) Randomly splice speech segments based on natural human voice speech to obtain spliced ​​speech; (4) Perform time-frequency analysis on natural human speech and spliced ​​speech respectively to obtain the spectrogram of natural human speech and the spectrogram of spliced ​​speech; (5) Use the spectrograms of natural human voice speech and the spectrograms of spliced ​​speech as training samples to train the preset model until the preset model converges and the discrimination model is obtained.

[0053] After obtaining the spectrograms of natural human speech and spliced ​​speech, the types of these spectrograms can be labeled to obtain training samples. Specifically, the spectrogram of natural human speech can be labeled as "natural human speech type," and the spectrogram of spliced ​​speech can be labeled as "spoofed splicing type."

[0054] The method for step (4) performing time-frequency analysis on natural human voice speech and spliced ​​speech respectively to obtain the spectrogram of natural human voice speech and the spectrogram of spliced ​​speech can be referred to step 102, and will not be repeated here.

[0055] To improve the randomness of segmentation, natural human speech can be divided into multiple segments according to different durations. Therefore, in some embodiments, step "(2) performing segmentation processing based on natural human speech to obtain speech segments of natural human speech" includes the following steps: The natural human voice is segmented according to the first length to obtain a speech segment of the first length. The natural human voice is segmented according to the second length to obtain the second length of the natural human voice speech segment.

[0056] The natural human voice speech segment includes a speech segment of first length and a speech segment of second length.

[0057] The first length and the second length can be preset. For example, the first length can be 0.2 seconds and the second length can be 0.4 seconds.

[0058] It should be noted that in addition to the first and second lengths, multiple durations such as the third, fourth, and fifth lengths can be set according to actual needs.

[0059] To improve the randomness of splicing, the aforementioned speech segments divided into multiple segments according to duration can be stored in sets corresponding to those durations. During splicing, speech segments are randomly selected from each set for splicing. Therefore, in some embodiments, step "(3) performing random splicing processing based on speech segments to obtain spliced ​​speech" includes the following steps: A. Obtain a first segment set and a second segment set, wherein the first segment set includes a speech segment of a first length of natural human voice, and the second segment set includes a speech segment of a second length of natural human voice; B. Randomly select a speech segment from the first segment set as the first speech segment; C. Randomly select a speech segment from the second segment set as the second speech segment; D. By splicing the first and second speech segments together, the spliced ​​speech is obtained.

[0060] Specifically, in some embodiments, step "B. Randomly select a speech segment from the first segment set as the first speech segment" can be to randomly select N speech segments from the first segment set as the first speech segment.

[0061] Specifically, in some embodiments, step "C. Randomly select speech segments from the second segment set as the second speech segment" can be to randomly select N speech segments from the second segment set as the second speech segment.

[0062] Where N is a positive integer.

[0063] It should be noted that in the above embodiments, in addition to setting the first and second lengths, multiple durations such as the third, fourth, and fifth lengths can also be set according to actual needs. Therefore, multiple sets corresponding to these lengths can also be set. For example, the third segment set includes speech segments of the third length of natural human voice, the fourth segment set includes speech segments of the fourth length of natural human voice, the fifth segment set includes speech segments of the fifth length of natural human voice, and so on.

[0064] Similarly, during the splicing process, speech segments can be randomly selected from multiple sets such as the third, fourth, and fifth segments, and then spliced ​​together with the first and second speech segments to obtain the spliced ​​speech.

[0065] To further enhance the randomness of the splicing and make the spliced ​​speech more realistic, in some embodiments, step "D. Splicing the first speech segment and the second speech segment to obtain the spliced ​​speech" includes the following steps: The order of the first and second audio segments is shuffled. The first and second audio segments are shuffled and then spliced ​​together end to end. The audio at the splicing boundary is then smoothed to obtain the spliced ​​audio.

[0066] For example, refer to Figure 2bIt can cut multiple natural human voices into corresponding speech segments in three stages with trimming durations of 0.2 seconds, 0.4 seconds, and 0.6 seconds. Then, it can place these segments into corresponding segment sets, randomly select three speech segments from each of the three segment sets, shuffle the order of these nine speech segments, splice the beginning and end of the nine shuffled speech segments together, and smooth the audio at the splicing boundary.

[0067] Furthermore, in some embodiments, the specific parameters for model training are set as follows: The gradient descent method was used to optimize the model, with a momentum of 0.9, a weight decay of 1e-4, a learning rate of 5e-5, a batch size of 32, and a large margin softmax loss function, where the margin was set to 4.

[0068] The preparation of training data has been introduced above. The following section will introduce how to use a discrimination model to classify speech to be discriminated.

[0069] In some embodiments, the discrimination model includes multiple structurally identical feature extraction blocks, which can be used to extract image features of each local region in the spectrogram. The following are the steps performed by each feature extraction block: (1) Determine the local features of the previous time step, wherein the first local feature of the previous time step is the spectrogram; (2) Use the feature extraction block of the current time to determine the local features of the current time based on the local features of the previous time. The previous time is the time before the current time. (3) Calculate the type score of the spectrogram based on the spectrogram features, which include the local features at the current time.

[0070] Repeat the above steps until all feature extraction blocks have calculated the local features at the current time. Finally, the type score of the spectrogram can be calculated based on the local features calculated at the current time for the last feature extraction block.

[0071] In some embodiments, the feature extraction block includes a first function, a second function, and an attention network. The attention network is used to assign weights to the features at each position in the spectrogram. Step "(2) Using the feature extraction block at the current time, determine the local features at the current time based on the local features at the previous time" includes the following steps: A. Use the first function at the current time step to perform feature extraction on the local features of the previous time step to obtain the first local features; B. Using the attention network at the current moment as the first local feature, attention weights are assigned to obtain the weighted first local feature; C. Use the second function at the current time to perform feature extraction on the weighted first local features to obtain the local features at the current time.

[0072] Both the first and second functions can be Max-Feature-Map (MFM, an activation function). In CNNs, MFM can be used for local feature selection, meaning that MFM can select different filters at different locations in the image to learn the optimal features.

[0073] Among them, attention networks can effectively focus on the fragmented and discontinuous positions that often appear in the spectrogram of spliced ​​and forged speech, thus enabling better classification.

[0074] In some embodiments, the attention network can be used to assign weights to the features at each channel position in the spectrogram. Step "B. Use the attention network at the current time to assign attention weights to the first local features to obtain the weighted first local features" includes the following steps: a. Perform channel splitting on the first local feature to obtain the channel sub-features corresponding to each channel of the first local feature; b. Score the channel sub-features of the channel to obtain the channel score; c. Based on the channel score, the channel sub-features of the channel are weighted to obtain the weighted sub-features of the channel; d. Obtain the weighted first local features, which include the weighted sub-features of each channel.

[0075] For example, in some embodiments, the attention network can utilize the dependencies between feature channels to assign different attention weights (i.e., channel scores) to each channel, thereby allowing the neural network to focus on certain feature channels, that is, to enhance feature channels that are useful for speech authentication and suppress feature channels that are not very useful.

[0076] For example, in some embodiments, the attention network can first perform average pooling on the feature map [C×H×W] of the first local feature in the channel dimension, then use two fully connected layers to compress and restore the channel dimension, realize the weight allocation in the channel dimension, and then multiply it with the map value of the corresponding channel of the original feature map to obtain the feature map containing the channel attention mechanism.

[0077] Where C is the channel, H is the pixel row in channel C, and W is the pixel column in channel C.

[0078] In addition to introducing an attention mechanism to the channel dimension as mentioned above, an attention mechanism can also be introduced to the spatial dimension at the same time.

[0079] Therefore, in some embodiments, the attention network is used to assign weights to the features at each spatial location in the spectrogram. Step "d. Obtain the weighted first local features, which include the weighted sub-features of each channel" includes the following steps: Spatial decomposition is performed on the channel weighted features to obtain spatial sub-features corresponding to each space. The channel weighted features include the channel weighted sub-features. The spatial sub-features of each space are concatenated to obtain the concatenated features; The spatial sub-features of the space are scored based on the splicing features to obtain the spatial score of the space; Based on the spatial score, the spatial sub-features of the space are weighted to obtain the weighted sub-features of the space. The first local feature is obtained after weighting, which includes the weighted sub-features of each space.

[0080] For example, in some embodiments, the spatial attention network can use average pooling and max pooling in the spatial dimension to generate channel weight tensors C×1×1, both of which are input into a shared multilayer perceptron. The two sets of weights generated are summed to form the channel attention assignment. For spatial attention, the original feature map is also processed by average pooling and max pooling in the channel dimension to generate channel weight tensors 1×H×W, respectively. Then, the two tensors are concatenated and convolved through a standard convolutional layer to form the spatial attention assignment. The kernel size of the convolutional layer is 7×7. Similar to the channel attention network, the attention weight values ​​generated by the spatial attention network are multiplied with the corresponding elements of the original feature map to obtain a feature map containing the channel attention mechanism.

[0081] In some embodiments, step "C. Using the feature extraction block at the current time, determine the local features at the current time based on the local features at the previous time" includes the following steps: Max pooling is applied to the local features from the previous time step to obtain the pooled local features from the previous time step. The feature extraction block at the current time step is used to determine the local features at the current time step based on the local features of the previous time step after pooling.

[0082] For example, refer to Figure 2c The network structure of the discrimination model can include four feature extraction blocks with the same structure. Each block consists of a convolutional layer with a 1×1 kernel, an MFM layer, a BN (Batch Normalization) layer, a convolutional layer with a 3×3 kernel, and an MFM layer.

[0083] In addition, the discrimination model can also include a combination of multiple independent convolutional parts in each convolutional layer, and the maximum element of the layer output is calculated by using the MFM activation function; the kernel size of the max pooling layer is 2×2 and the stride is 2×2, which is used to reduce the time and frequency dimensions; the fully connected 1 can contain a low-dimensional high-level speech representation, and then the type score is calculated by using the fully connected 2 with softmax activation.

[0084] according to Figure 2c It is evident that the discrimination model of this network structure is more lightweight, making it suitable for electronic devices with lower computing power, such as portable devices. Specifically for smartphones, the discrimination model of this solution can achieve efficient and low-power speech classification when integrated into smartphones.

[0085] 104. When the type score falls within the first preset range, the speech type of the speech to be identified is determined to be the natural human voice type.

[0086] The discrimination model can score the speech segment by segment. A suitable preset range is selected based on accuracy and false alarm rate. For example, when the type score falls within the first preset range, the speech segment is determined to be of the natural human voice type.

[0087] 105. When the type score falls within the second preset range, the speech type of the speech to be identified is determined to be a splicing forgery type.

[0088] For example, when the type score falls within the second preset range, the speech type of the speech segment is determined to be a splicing forgery type.

[0089] In some embodiments, if any splicing forgery type speech segment exists in the speech to be identified, the entire speech to be identified is judged to be splicing forgery type.

[0090] This invention introduces channel attention network and spatial attention network into the discrimination model, and combines the characteristics of spectrograms to enhance the discrimination model's ability to distinguish between real and fake speech, thereby improving the accuracy of speech authentication.

[0091] The speech classification scheme provided in this invention can be applied to various human voice recognition scenarios. It can accurately and quickly distinguish between natural human voices and maliciously forged artificially spliced ​​speech, thereby achieving user security protection.

[0092] For example, this invention can be applied to scenarios such as voice fraud prevention, voiceprint unlocking on mobile devices, voice payment in electronic payments, and voice control in smart cars, etc.

[0093] For example, in the case of voice fraud prevention, in some embodiments, the present invention can identify whether the voice of the caller is a real human voice or a maliciously forged artificial voice when the user is making a voice communication; when the identification result is artificial voice, the user can be reminded on the user's voice communication interface, for example, by displaying a fraud warning message on the voice communication interface: the caller may be involved in fraud.

[0094] Specifically, for example, scammers may request money transfers from users through instant messaging platforms such as WeChat and QQ. If the user requests a voice reply, the scammer will send a pre-forged, artificially synthesized voice message impersonating a friend or relative to gain the user's trust and ultimately defraud them of money. This invention can acquire and identify the scammer's voice message. If it is identified as a maliciously forged, artificially synthesized voice message, a fraud warning message can be displayed on the instant messaging platform.

[0095] Specifically, for example, scammers might forge a human voice, supposedly belonging to a user's relative or friend, and then play this forged voice over the phone to request a bank transfer. This invention can identify the voice in real time; if it is identified as maliciously forged human voice, a fraud warning message can be displayed on the user's call interface.

[0096] For example, taking the voiceprint unlocking scenario on a mobile device as an example, in some embodiments, the present invention can first identify whether the collected unlocking voice is a real human voice or a maliciously forged artificially spliced ​​voice before the mobile device unlocks with its voiceprint; when the identification result is a real human voice, the mobile device then performs voiceprint unlocking; when the identification result is artificially spliced ​​voice, the unlocking voice is not responded to.

[0097] For example, taking the voice payment scenario of electronic payment as an example, in some embodiments, the present invention can first identify whether the payment voice is a real human voice or a maliciously forged artificially spliced ​​voice; when the identification result is a real human voice, the payment terminal makes electronic payment according to the payment voice; when the identification result is an artificially spliced ​​voice, the payment can be locked for a period of time to ensure the user's property security.

[0098] Since the voice classification method provided in this embodiment of the invention only requires a small amount of computing resources to accurately identify whether the voice is a real human voice or a maliciously forged artificially spliced ​​voice, this solution has a wide range of applications and can be installed on electronic devices with low computing power, such as mobile phones. Moreover, due to its fast identification speed and high efficiency, this invention can be well applied in real-time voice communication scenarios, further ensuring the timeliness of anti-fraud and other warnings.

[0099] As can be seen from the above, the embodiments of the present invention can acquire the speech to be identified; perform time-frequency analysis on the speech to be identified to obtain a spectrogram of the speech to be identified; use an identification model to identify the type of the spectrogram to obtain a type score of the spectrogram; when the type score belongs to a first preset range, the speech type of the speech to be identified is determined to be a natural human voice; when the type score belongs to a second preset range, the speech type of the speech to be identified is determined to be a spliced ​​and forged type. Therefore, this solution can effectively identify spliced ​​and forged fake speech using spectrograms, determining whether the speech to be identified is a natural human voice or a spliced ​​and forged one. Thus, this solution can improve the accuracy of speech classification.

[0100] To better implement the above methods, this embodiment of the invention also provides a voice classification device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0101] For example, in this embodiment, the method of the present invention will be described in detail by taking the voice classification device specifically integrated into a smartphone as an example.

[0102] For example, such as Figure 3 As shown, the speech classification device may include an acquisition unit 301, a time-frequency unit 302, a scoring unit 303, a natural human voice unit 304, and a splicing and forgery unit 305, as follows: (a) Acquiring Unit 301: The acquisition unit 301 can be used to acquire the speech to be identified.

[0103] (ii) Time-frequency unit 302: The time-frequency unit 302 can be used to perform time-frequency analysis on the speech to be identified and obtain the spectrogram of the speech to be identified.

[0104] In some embodiments, the time-frequency unit 302 can be used for: Multiple local sampled audios are obtained by sliding sampling on the speech to be identified using a preset sampling window with a preset step size; Perform time-frequency conversion on each local sampled audio to obtain the spectrogram of each local sampled audio; The spectrogram is obtained, which can include the spectrogram of each local sampled audio.

[0105] (III) Scoring Unit 303: The scoring unit 303 can be used to perform type identification on the spectrogram using a discrimination model to obtain a type score for the spectrogram.

[0106] In some embodiments, the scoring unit 303 may further include an acquisition subunit, a segmentation subunit, a splicing subunit, a time-frequency subunit, and a training subunit, as follows: (1) Obtaining sub-units: The acquisition subunit can be used to acquire natural human speech.

[0107] (2) Segmented sub-units Segmentation subunits can be used to segment natural human speech to obtain speech segments.

[0108] In some embodiments, a speech segment of natural human voice may include a speech segment of a first length and a speech segment of a second length, and the segmentation subunit may include a first segmentation submodule and a second segmentation submodule, wherein: The first segmentation submodule can be used to segment natural human speech according to a first length to obtain speech segments of the first length of natural human speech; The second segmentation submodule can be used to segment natural human speech according to a second length to obtain speech segments of the second length.

[0109] In some embodiments, the segmentation subunit may include an acquisition submodule, a first selection submodule, a second selection submodule, and a splicing submodule, wherein: The acquisition submodule can be used to acquire a first segment set and a second segment set. The first segment set may include a speech segment of a first length of natural human voice, and the second segment set may include a speech segment of a second length of natural human voice. The first selection submodule can be used to randomly select a speech segment from the first segment set as the first speech segment; The second selection submodule can be used to randomly select a speech segment from the second segment set as the second speech segment; The splicing submodule can be used to splice the first speech segment and the second speech segment to obtain spliced ​​speech.

[0110] In some embodiments, the first selection submodule can specifically be used for: Randomly select N audio segments from the first segment set as the first audio segment, where N is a positive integer; The second selection submodule can be specifically used for: Randomly select N speech segments from the second segment set as the second speech segment.

[0111] In some embodiments, the splicing submodule can be used for: The order of the first and second audio segments is shuffled. The scrambled first and second speech segments are spliced ​​together end to end, and the audio at the splicing boundary is smoothed to obtain the spliced ​​speech.

[0112] (3) Splicing sub-units The splicing subunit can be used to randomly splice speech segments based on natural human voice speech to obtain spliced ​​speech.

[0113] (4) Time-frequency subunit The time-frequency subunit can be used to perform time-frequency analysis on natural human speech and spliced ​​speech respectively, to obtain the spectrogram of natural human speech and the spectrogram of spliced ​​speech.

[0114] (5) Training sub-unit The training subunit can be used to train a preset model using spectrograms of natural human speech and spliced ​​speech as training samples until the preset model converges, thus obtaining the discrimination model.

[0115] In some embodiments, the discrimination model may include multiple feature extraction blocks with the same structure, and the scoring unit 303 may include the previous sub-unit, the current sub-unit, and the scoring unit, as follows: (6) Previous subunit: The previous sub-unit can be used to determine the local features of the previous time step, where the first local feature of the previous time step is the spectrogram.

[0116] (7) Current subunit: The current sub-unit can be used to extract features from the current time step and determine the local features of the current time step based on the local features of the previous time step, where the previous time step is the time step before the current time step.

[0117] In some embodiments, the current subunit can be used for: Max pooling is applied to the local features from the previous time step to obtain the pooled local features from the previous time step. The feature extraction block at the current time step is used to determine the local features at the current time step based on the local features of the previous time step after pooling.

[0118] In some embodiments, the feature extraction block may include a first function, a second function, and an attention network. The attention network can be used to assign weights to features at each location in the spectrogram. The current sub-unit may include a first extraction sub-module, an attention sub-module, and a second extraction sub-module, as follows: A. First Extraction Submodule: The first extraction submodule can be used to perform feature extraction on the local features of the previous time step using the first function at the current time step to obtain the first local features.

[0119] B. Attention Submodule: The attention submodule can be used to assign attention weights to the current attention network as the first local feature, and obtain the weighted first local feature.

[0120] In some embodiments, the attention network can be used to assign weights to features at each channel location in the spectrogram, and the attention submodule can be used for: The first local feature is split into channels to obtain the channel sub-features corresponding to each channel of the first local feature; The channel sub-features of the channel are scored to obtain the channel score; Based on the channel score, the channel sub-features of the channel are weighted to obtain the weighted sub-features of the channel; The first local feature after weighting is obtained, which may include the weighted sub-features of each channel.

[0121] In some embodiments, the attention network can be used to assign weights to features at each spatial location in the spectrogram. Specifically, when the attention submodule is used to obtain a weighted first local feature, which may include weighted sub-features for each channel, it can be used for: Spatial decomposition is performed on the channel weighted features to obtain spatial sub-features corresponding to each space of the channel weighted features. The channel weighted features can include the channel weighted sub-features. The spatial sub-features of each space are concatenated to obtain the concatenated features; The spatial sub-features of the space are scored based on the splicing features to obtain the spatial score of the space; Based on the spatial score, the spatial sub-features of the space are weighted to obtain the weighted sub-features of the space. The first local feature is obtained after weighting, which may include the weighted sub-features of each space.

[0122] C. Second Extraction Submodule: The second extraction submodule can be used to perform feature extraction on the weighted first local features using the second function at the current time, so as to obtain the local features at the current time.

[0123] (8) Obtain the molecular unit: The resulting molecular unit can be used to calculate the type score of a spectrogram based on spectrogram features, which can include local features at the current time.

[0124] (iv) Natural Human Voice Unit 304: The natural human voice unit 304 can be used to determine the voice type of the speech to be identified as natural human voice when the type score belongs to a first preset range.

[0125] (v) Forged Unit 305: The splicing and forgery unit 305 can be used to determine that the speech type of the speech to be identified is a splicing and forgery type when the type score belongs to a second preset range.

[0126] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0127] As can be seen from the above, the speech classification device in this embodiment acquires the speech to be identified by the acquisition unit; performs time-frequency analysis on the speech to be identified by the time-frequency unit to obtain the spectrogram of the speech to be identified; uses a scoring unit to perform type identification on the spectrogram using a discrimination model to obtain the type score of the spectrogram; when the type score belongs to a first preset range, the natural human voice unit determines that the speech type of the speech to be identified is the natural human voice type; when the type score belongs to a second preset range, the splicing and forgery unit determines that the speech type of the speech to be identified is the splicing and forgery type.

[0128] Therefore, the embodiments of the present invention can improve the accuracy of speech classification.

[0129] This invention also provides an electronic device, which can be a terminal, a server, or other similar devices. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0130] In some embodiments, the voice classification device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the voice classification method of the present invention.

[0131] Because this solution is highly efficient and consumes less computing resources, it is very suitable for use in portable devices such as smartphones.

[0132] For example, such as Figure 4 As shown, it illustrates a structural schematic diagram of a smartphone according to an embodiment of the present invention, specifically: The smartphone may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, an input module 404, and a communication module 405. Those skilled in the art will understand that... Figure 4The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the smartphone, connecting various parts of the smartphone via various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, thereby performing overall detection of the smartphone. In some embodiments, the processor 401 may include one or more processing cores; in some embodiments, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 401.

[0133] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the smartphone, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0134] The smartphone also includes a power supply 403 that supplies power to the various components. In some embodiments, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0135] The smartphone may also include an input module 404, which can be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0136] The smartphone may also include a communication module 405. In some embodiments, the communication module 405 may include a wireless module, through which the smartphone can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 405 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0137] Although not shown, smartphones may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the smartphone loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows: Obtain the speech to be identified; Time-frequency analysis is performed on the speech to be identified to obtain its spectrogram. A discrimination model is used to identify the type of the spectrogram and obtain the type score of the spectrogram. When the type score falls within the first preset range, the speech type of the speech to be identified is determined to be the natural human voice type. When the type score falls within the second preset range, the speech type of the speech to be identified is determined to be a splicing forgery type.

[0138] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0139] As can be seen from the above, this solution can improve the accuracy of speech classification.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0141] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the speech classification methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps: Obtain the speech to be identified; Time-frequency analysis is performed on the speech to be identified to obtain its spectrogram. A discrimination model is used to identify the type of the spectrogram and obtain the type score of the spectrogram. When the type score falls within the first preset range, the speech type of the speech to be identified is determined to be the natural human voice type. When the type score falls within the second preset range, the speech type of the speech to be identified is determined to be a splicing forgery type.

[0142] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0143] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations of the information security or voice classification aspects provided in the above embodiments.

[0144] Since the instructions stored in the storage medium can execute the steps in any of the speech classification methods provided in the embodiments of the present invention, the beneficial effects that any of the speech classification methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0145] The foregoing has provided a detailed description of a speech classification method, apparatus, electronic device, and computer-readable storage medium provided by embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A speech classification method, characterized in that, include: Obtain the speech to be identified; Time-frequency analysis is performed on the speech to be identified to obtain the spectrogram of the speech to be identified; A discrimination model is used to identify the type of the spectrogram, resulting in a type score for the spectrogram. This includes: determining local features from the previous time step, where the previous time step is the time preceding the current time step, and the first local feature from the previous time step is the spectrogram; performing feature extraction on the local features from the previous time step using a first function at the current time step to obtain a first local feature; assigning attention weights to the first local feature using an attention network at the current time step to obtain a weighted first local feature; performing feature extraction on the weighted first local feature using a second function at the current time step to obtain the local feature at the current time step; and calculating the type score of the spectrogram based on the spectrogram features, where the spectrogram features include the local features from the current time step. The step of using the attention network at the current time to assign attention weights to the first local feature to obtain the weighted first local feature includes: performing average pooling on the feature map of the first local feature along the channel dimension to obtain the pooling result; using two fully connected layers to compress and restore the pooling result along the channel dimension to generate the weight corresponding to each channel; and multiplying the weight by the map value of the corresponding channel of the feature map to obtain the weighted first local feature. When the type score falls within a first preset range, the speech type of the speech to be identified is determined to be a natural human voice type. When the type score falls within the second preset range, the speech type of the speech to be identified is determined to be a splicing forgery type.

2. The speech classification method as described in claim 1, characterized in that, Before using a discrimination model to perform type identification on the spectrogram and obtain the type score of the speech to be identified, the method further includes: Acquire natural human speech; The natural human voice is segmented to obtain speech segments. The speech segments based on the natural human voice are randomly spliced ​​together to obtain spliced ​​speech; Time-frequency analysis was performed on the natural human voice speech and the spliced ​​speech speech respectively to obtain the spectrogram of the natural human voice speech and the spectrogram of the spliced ​​speech speech. The spectrograms of the natural human voice and the spliced ​​speech are used as training samples to train a preset model until the preset model converges, thus obtaining the discrimination model.

3. The speech classification method as described in claim 2, characterized in that, The natural human voice speech segment includes a speech segment of a first length and a speech segment of a second length. The segmentation process based on the natural human voice speech to obtain the speech segment includes: The natural human voice is segmented according to a first length to obtain a speech segment of the first length. The natural human voice is segmented according to a second length to obtain a speech segment of the second length.

4. The speech classification method as described in claim 3, characterized in that, The random splicing process based on the speech segments to obtain spliced ​​speech includes: Obtain a first segment set and a second segment set, wherein the first segment set includes a speech segment of the natural human voice of a first length, and the second segment set includes a speech segment of the natural human voice of a second length; A speech segment is randomly selected from the first segment set as the first speech segment; Randomly select a speech segment from the second segment set as the second speech segment; By splicing the first and second speech segments together, a spliced ​​speech is obtained.

5. The speech classification method as described in claim 4, characterized in that, The step of randomly selecting a speech segment from the first segment set as the first speech segment includes: Randomly select N speech segments from the first segment set as the first speech segment, where N is a positive integer; The step of randomly selecting a speech segment from the second segment set as the second speech segment includes: Randomly select N speech segments from the second set of segments as the second speech segments.

6. The speech classification method as described in claim 4, characterized in that, The process of splicing the first and second speech segments to obtain spliced ​​speech includes: The order of the first and second audio segments is shuffled. The scrambled first and second speech segments are spliced ​​together end to end, and the audio at the splicing boundary is smoothed to obtain the spliced ​​speech.

7. The speech classification method as described in claim 1, characterized in that, The attention network is used to assign weights to the features at each channel position in the spectrogram. The step of using the attention network at the current time to assign attention weights to the first local features, resulting in weighted first local features, includes: The first local feature is subjected to channel splitting to obtain channel sub-features corresponding to each channel of the first local feature; The channel sub-features of the channel are scored to obtain the channel score; Based on the channel score of the channel, the channel sub-features of the channel are weighted to obtain the weighted sub-features of the channel; The first local feature after weighting is obtained, which includes the weighted sub-features of each channel.

8. The speech classification method as described in claim 7, characterized in that, The attention network is used to assign weights to the features at each spatial location in the spectrogram, resulting in a weighted first local feature. This weighted first local feature includes weighted sub-features for each channel, including: The channel weighted features are spatially decomposed to obtain spatial sub-features corresponding to each space of the channel weighted features, wherein the channel weighted features include the weighted sub-features of the channels; The spatial sub-features of each space are spliced ​​together to obtain the spliced ​​features; The spatial sub-features of the space are scored based on the splicing features to obtain the spatial score of the space. Based on the spatial score of the space, the spatial sub-features of the space are weighted to obtain the weighted sub-features of the space; The first local feature after weighting is obtained, which includes the weighted sub-features of each space.

9. The speech classification method as described in claim 1, characterized in that, The step of performing time-frequency analysis on the speech to be identified to obtain the spectrogram of the speech to be identified includes: Multiple local sampled audios are obtained by sliding sampling on the speech to be identified using a preset sampling window with a preset step size; Perform time-frequency conversion on each of the local sampled audios to obtain the spectrogram of each of the local sampled audios; A spectrogram is obtained, the spectrogram comprising the spectrum of each of the local sampled audio frequencies.

10. A speech classification device, characterized in that, include: The acquisition unit is used to acquire the speech to be identified; The time-frequency unit is used to perform time-frequency analysis on the speech to be identified and obtain the spectrogram of the speech to be identified. The scoring unit is used to perform type identification on the spectrogram using a discrimination model to obtain a type score for the spectrogram. The scoring unit is specifically used for: determining the local features of the previous time step, wherein the previous time step is the time before the current time step, and the first local feature of the previous time step is the spectrogram; performing feature extraction on the local features of the previous time step using a first function of the current time step to obtain a first local feature; assigning attention weights to the first local feature using an attention network of the current time step to obtain a weighted first local feature; performing feature extraction on the weighted first local feature using a second function of the current time step to obtain the local feature of the current time step; and calculating the type score of the spectrogram based on the spectrogram features, wherein the spectrogram features include the local features of the current time step. The step of using the attention network at the current time to assign attention weights to the first local feature to obtain the weighted first local feature includes: performing average pooling on the feature map of the first local feature along the channel dimension to obtain the pooling result; using two fully connected layers to compress and restore the pooling result along the channel dimension to generate the weight corresponding to each channel; and multiplying the weight by the map value of the corresponding channel of the feature map to obtain the weighted first local feature. The natural human voice unit is used to determine the voice type of the speech to be identified as natural human voice when the type score belongs to a first preset range. The splicing forgery unit is used to determine that the speech type of the speech to be identified is a splicing forgery type when the type score belongs to a second preset range.

11. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the speech classification method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the speech classification method according to any one of claims 1 to 9.

Images