Voice classification method and related device and equipment

Abstract

The present application discloses a speech classification method and related devices and equipment, wherein the speech classification method includes: obtaining the speech to be classified, extracting features of the speech to be classified, and obtaining speech features of the speech to be classified; mapping the speech features into a three-dimensional spectrogram; and classifying the speech to be classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified. The above scheme can improve the accuracy and reliability of speech classification.

Description

Technical Field

[0001] The present application relates to the technical field of speech classification, and in particular to a speech classification method and related devices and equipment. Background Art

[0002] In real life, speech can often reflect the state of the object making the sound. For example, by knocking on a water tank, the water tank makes a sound, and the sound emitted by the water tank can be used to determine whether the inner wall of the water tank is free of cracks; on the production line, the sound emitted by the products to be classified can be classified to determine the quality and state of the products to be classified.

[0003] Traditional speech classification is often done by manual on-site identification of the objects to be classified. However, in real life, there is a large gap in experience between different people, and there may be completely different judgment results for the speech emitted by the objects to be classified. Moreover, for the same person, his or her performance level in different time periods cannot be stable and can easily fluctuate due to external influences.

[0004] Therefore, there is an urgent need for a speech classification method to improve the accuracy of speech classification. Summary of the invention

[0005] The present application provides a speech classification method and related devices and equipment to solve the problem of unstable speech classification accuracy existing in the prior art.

[0006] The present application provides a speech classification method, comprising: acquiring speech to be classified, performing feature extraction on the speech to be classified, and obtaining speech features of the speech to be classified; mapping the speech features into a three-dimensional spectrogram; and classifying the speech to be classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

[0007] Among them, the steps of acquiring the speech to be classified, performing feature extraction on the speech to be classified, and obtaining the speech features of the speech to be classified include: acquiring the speech to be classified, performing Fourier transform on the speech to be classified, and obtaining the frequency domain signal of the speech to be classified; performing feature extraction on the frequency domain signal to obtain the two-dimensional speech features of the speech to be classified; the step of mapping the speech features into a three-dimensional spectrogram includes: performing matrix operations on the two-dimensional speech features to obtain the third-dimensional speech features; mapping the two-dimensional speech features and the third-dimensional speech features into a three-dimensional spectrogram.

[0008] Among them, the step of extracting features from the frequency domain signal to obtain two-dimensional speech features of the speech to be classified includes: extracting features from the frequency domain signal to obtain two-dimensional speech features of each channel of the frequency domain signal; performing matrix operations on the two-dimensional speech features to obtain third-dimensional speech features, including: calculating the mean of the two-dimensional speech features of each channel; and obtaining the third-dimensional speech features based on the mean.

[0009] Among them, the step of extracting features from the frequency domain signal to obtain two-dimensional speech features of each channel of the frequency domain signal includes: filtering the speech to be classified through a Mel triangle filter group; performing logarithmic operation and discrete cosine transform on the filtered speech to be classified to obtain speech features of the speech to be classified.

[0010] Among them, the step of acquiring the speech to be classified, extracting features of the speech to be classified, and obtaining the speech features of the speech to be classified also includes: acquiring the voiceprint signal of the product to be classified, and determining the voiceprint signal as the speech to be classified; using the three-dimensional spectrogram to classify the speech to be classified, and obtaining the classification result of the speech to be classified also includes: determining the quality category of the product to be classified based on the classification result.

[0011] Among them, the step of obtaining the voiceprint signal of the product to be classified also includes: in a silent environment, controlling the microphone through the edge smart box to collect the voiceprint signal of the product to be classified on the production line.

[0012] Among them, after the step of determining the quality category of the products to be classified based on the classification results, it also includes: determining and controlling the transportation direction of the products to be classified on the production line based on the quality category through the edge intelligent box.

[0013] Among them, the step of using the three-dimensional spectrogram to classify the speech to be classified and obtaining the classification result of the speech to be classified includes: using the three-dimensional spectrogram to classify the speech to be classified through a speech classification model to obtain the classification result of the speech to be classified.

[0014] Among them, the step of using the three-dimensional spectrogram to classify the speech to be classified through the speech classification model to obtain the classification result of the speech to be classified also includes: inputting the three-dimensional spectrogram into the speech classification model, and sequentially performing convolution and pooling processing through the first convolution layer, the second convolution layer, the first maximum pooling layer, two third convolution layers, two fourth convolution layers, the second maximum pooling layer, two fifth convolution layers, and the average pooling layer of the speech classification model; classifying the three-dimensional spectrogram after convolution and pooling processing through the fully connected layer to obtain the classification result.

[0015] Among them, the speech classification method also includes: obtaining sample speech and standard classification results of the sample speech; wherein the standard classification results of the sample speech include all classification types; performing feature extraction on the sample speech to obtain sample speech features of the sample speech; mapping the sample speech features into a sample three-dimensional spectrogram; inputting the sample three-dimensional spectrogram into the initial model for classification to obtain a predicted classification result of the sample speech; calculating the loss between the standard classification result and the predicted classification result through a loss function; and obtaining a speech classification model in response to the loss satisfying a preset condition.

[0016] Among them, the speech classification method also includes: receiving manual sampling of classification results; in response to the number of classification results that are different from the sampling results exceeding a preset number, retraining the speech classification model based on the to-be-classified speech corresponding to the preset number of classification results.

[0017] The present application also provides a speech classification device, including: an acquisition module, used to acquire the speech to be classified, perform feature extraction on the speech to be classified, and obtain the speech features of the speech to be classified; a mapping module, used to map the speech features into a three-dimensional spectrogram; and a classification module, used to classify the speech to be classified using the three-dimensional spectrogram, and obtain the classification result of the speech to be classified.

[0018] The present application also provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement any of the above-mentioned speech classification methods.

[0019] The present application also provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a processor, any of the above-mentioned speech classification methods is implemented.

[0020] The above scheme, this application obtains the speech features of the speech to be classified by first extracting features of the speech to be classified, then maps the speech features into a three-dimensional spectrogram, and finally uses the three-dimensional spectrogram to classify the speech to be classified to obtain the classification result of the speech to be classified. The accuracy, intuitiveness and speed of the three-dimensional spectrogram can be used to enhance the speech features, reduce the classification steps and difficulty, thereby improving the efficiency and effect of classification, and finally improving the classification accuracy and reliability of the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an embodiment of the speech classification method of the present application;

[0022] Figure 2 It is a flowchart of another embodiment of the speech classification method of the present application;

[0023] Figure 3 yes Figure 2 A schematic diagram of the structure of a speech classification model in an embodiment;

[0024] Figure 4 It is a structural schematic diagram of an implementation scheme of a speech classification system corresponding to the speech classification method of the present application;

[0025] Figure 5 It is a structural diagram of an embodiment of a speech classification device of the present application;

[0026] Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0027] Figure 7 A schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0028] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0029] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0030] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of the associated objects. There can be three kinds of relationships. For example, A and / or B can be: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally means that the associated objects before and after are in an "or" relationship. In addition, "more" in this article means two or more than two.

[0031] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the speech classification method of the present application.

[0032] Specifically, the following steps may be included:

[0033] Step S11: Acquire the speech to be classified, perform feature extraction on the speech to be classified, and obtain speech features of the speech to be classified.

[0034] The speech to be classified is obtained, and features are extracted from the speech to be classified to obtain speech features of the speech to be classified. The speech to be classified may include speech emitted by the object to be classified, speech with actual semantics that needs to be classified, or other speech that needs to be classified.

[0035] In a specific application scenario, a deep neural network can be used to extract features of the speech to be classified, thereby obtaining speech features of the speech to be classified. The deep neural network may include a convolutional neural network, a recurrent neural network, etc., which are not specifically limited here.

[0036] In another specific application scenario, the speech to be classified can also be feature extracted by Mel-Frequency Cepstral Coefficients (MFCC), linear prediction coefficients (LPC), linear prediction cepstral coefficients (LPCC), line spectrum frequency (LSF), discrete wavelet transform (DWT), or perceptual linear prediction (PLP) to obtain speech features corresponding to the speech to be classified. This embodiment does not limit the method for extracting speech features.

[0037] Step S12: Mapping speech features into a three-dimensional spectrogram.

[0038] After obtaining the speech features of the speech to be classified, the speech features are mapped into a three-dimensional spectrogram. Among them, the three-dimensional spectrogram is a speech data graph that uses a two-dimensional plane to express three-dimensional information. The three-dimensional spectrogram can intuitively, accurately and quickly display specific speech features.

[0039] In a specific application scenario, three-dimensional speech features can be obtained, a first image can be generated through the first-dimensional speech features and the second-dimensional speech features, and a second image can be generated based on the third-dimensional speech features and the second-dimensional speech features, and finally the first image and the second image are synthesized in 3D and mapped to a three-dimensional spectrogram. In a specific application scenario, when the three-dimensional speech features can include time parameters, frequency parameters, and energy parameters, the first image can be generated through the time parameters and energy parameters, and the second image can be generated based on the frequency parameters and the energy parameters, and finally the first image and the second image are synthesized in 3D and mapped to a three-dimensional spectrogram.

[0040] In another specific application scenario, the speech features of the speech to be classified may also be subjected to matrix operations to determine three-dimensional data of the speech features, and then mapped into a three-dimensional spectrogram based on the three-dimensional data of the speech features.

[0041] Step S13: classify the speech to be classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

[0042] After obtaining the three-dimensional spectrogram, the speech to be classified is classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

[0043] In a specific application scenario, the trained speech classification model can be used to classify the speech to be classified based on the feature data in the three-dimensional spectrogram to obtain the classification result of the speech to be classified. The structure of the speech classification model can include the structure of deep neural networks such as convolutional neural networks and recurrent neural networks.

[0044] In another specific application scenario, the speech to be classified can be classified based on the three-dimensional spectrogram using the Softmax classification method to obtain a classification result of the speech to be classified.

[0045] In another specific application scenario, the speech to be classified may also be classified by a classifier based on the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

[0046] The classification result of the speech to be classified in this embodiment may include a binary classification result or a multi-classification classification result.

[0047] Since the three-dimensional spectrogram can display specific speech features more accurately, intuitively and quickly, classification based on speech features in the form of three-dimensional spectrogram can enhance speech features, reduce classification steps and difficulty, and thus improve classification efficiency and effect, making the classification results of the speech to be classified have a certain degree of accuracy and reliability.

[0048] Through the above steps, the method for speech classification of this embodiment first extracts features of the speech to be classified to obtain speech features of the speech to be classified, then maps the speech features into a three-dimensional spectrogram, and finally uses the three-dimensional spectrogram to classify the speech to be classified to obtain a classification result of the speech to be classified. The accuracy, intuitiveness, and speed of the three-dimensional spectrogram can be used to enhance speech features, reduce classification steps and difficulty, and thus improve classification efficiency and effect, and finally improve speech classification accuracy and reliability.

[0049] See also Figure 2 , Figure 2 This is a flow chart of another embodiment of the speech classification method of the present application. Specifically, it may include the following steps:

[0050] Step S21: Acquire the speech to be classified, perform Fourier transform on the speech to be classified to obtain the frequency domain signal of the speech to be classified, perform feature extraction on the frequency domain signal to obtain the two-dimensional speech feature of the speech to be classified.

[0051] The voice to be classified is obtained. In a specific application scenario, the voiceprint signal of the product to be classified can be obtained to determine the voiceprint signal as the voice to be classified. In a specific application scenario, the edge smart box can also be used to control the microphone to collect the voiceprint signal of the product to be classified on the production line in a silent environment. The microphone can include a high-fidelity industrial microphone, for example, a microphone with a sampling rate of 48k or 96k, so as to sample the ultrasonic waves in some of the voices to be classified, so that the collected voiceprint signal can fully characterize all the features of the voice to be classified.

[0052] Among them, the silent environment of this embodiment can be a silent area set on the production line, and the products to be classified are transported to the silent area through the production line, so that the voiceprint signal of the products to be classified is collected by at least one pickup set in the silent area.

[0053] After obtaining the speech to be classified, Fourier transform is performed on the speech to be classified to convert the speech to be classified in the time domain into the frequency domain, thereby obtaining the frequency domain signal of the speech to be classified. The frequency domain signal is a two-dimensional signal including time and frequency. Feature extraction is then performed on the frequency domain signal to obtain the two-dimensional speech features of the speech to be classified.

[0054] Before Fourier transforming the speech to be classified, the speech to be classified may be pre-emphasized, framed and windowed to maximize the target speech emitted by the product to be classified in the speech to be classified, so as to achieve the purpose of optimizing feature extraction. Pre-emphasis may include removing noise from the speech to be classified through a high-pass filter.

[0055] In a specific application scenario, the method for extracting features from a frequency domain signal includes: first filtering the frequency domain signal of the speech to be classified through a Mel triangle filter group, thereby smoothing the frequency domain signal and eliminating the effect of harmonics, highlighting the resonance peak of the target speech emitted by the product to be classified in the speech to be classified, thereby highlighting the characteristics of the target speech. Then, a logarithmic operation is performed on the filtered frequency domain signal to further highlight the characteristics of the target speech and obtain the speech features of the speech to be classified. The speech features of this application scenario may include FBank features. Among them, the Fbank feature processes the audio in a manner similar to the human ear. The Fbank feature retains more original speech data and can improve the performance of speech recognition and classification.

[0056] In a specific application scenario, the frequency domain signal of the speech to be classified can also be filtered through a group of Mel triangle filter groups to smooth the frequency domain signal and eliminate the effect of harmonics, highlighting the resonance peak of the target speech emitted by the product to be classified in the speech to be classified, thereby highlighting the characteristics of the target speech. Then, the frequency domain signal after filtering is sequentially subjected to logarithmic operation and discrete cosine transform (DCT) to further highlight the characteristics of the target speech and obtain the speech characteristics of the speech to be classified. The speech characteristics of this application scenario can include MFCC features.

[0057] After obtaining the frequency domain signal, the feature extraction method can be used to extract the features of the frequency domain signal to obtain the two-dimensional speech features of each channel of the frequency domain signal. The channel may include a pickup channel of a microphone. When there are multiple microphones for collecting the speech to be classified, the corresponding channels are also multiple.

[0058] Step S22: performing matrix operations on the two-dimensional speech features to obtain third-dimensional speech features, and mapping the two-dimensional speech features and the third-dimensional speech features into a three-dimensional spectrogram.

[0059] After obtaining the speech features of the speech to be classified, matrix operations are performed on the two-dimensional speech features to obtain the third-dimensional speech features, and the two-dimensional speech features and the third-dimensional speech features are mapped into a three-dimensional spectrogram.

[0060] In a specific application scenario, the mean of the two-dimensional speech features of each channel can be calculated, and then the third-dimensional speech features of each channel can be obtained based on the mean. Thus, the two-dimensional speech features and the third-dimensional speech features are mapped into a three-dimensional spectrogram. Specifically, the speech features of each channel are mapped to the three-dimensional spectrogram, and the three-dimensional coordinates of the speech features of each channel on the unit spectrogram are obtained, thereby obtaining a certain number of channel features. For example, 64 channel features can be obtained, which are expressed as 64 groups of 3*3 matrices.

[0061] Step S23: using the three-dimensional spectrogram to classify the speech to be classified through the speech classification model to obtain the classification result of the speech to be classified.

[0062] The three-dimensional spectrogram converted from speech features is used to classify the speech to be classified through a speech classification model to obtain a classification result of the speech to be classified.

[0063] In a specific application scenario, before inputting the three-dimensional spectrogram into a speech classification model, the data of the entire three-dimensional spectrogram can be divided into batches, and multiple batch polarities can be classified at the same time to process the data of the entire three-dimensional spectrogram in batches, thereby improving classification efficiency.

[0064] In a specific application scenario, the three-dimensional spectrogram converted from speech features can be input into a speech classification model, and then processed by convolution and pooling in sequence through the first convolution layer, the second convolution layer, the first maximum pooling layer, two third convolution layers, two fourth convolution layers, the second maximum pooling layer, two fifth convolution layers, and the average pooling layer of the speech classification model; the three-dimensional spectrogram after convolution and pooling is classified through a fully connected layer to obtain a classification result of the speech to be classified.

[0065] In a specific application scenario, the three-dimensional spectrogram converted from speech features can be input into a speech classification model, and convolution and pooling processing is performed in sequence through the first convolution layer, the second convolution layer, the first maximum pooling layer, two third convolution layers, two fourth convolution layers, the second maximum pooling layer, two fifth convolution layers, and the average pooling layer of the speech classification model; the three-dimensional spectrogram after convolution and pooling processing is classified through a fully connected layer to obtain a classification result of the speech to be classified. In a specific application scenario, the fully connected layer can also include a classifier to further classify the three-dimensional spectrogram after convolution and pooling processing through the fully connected layer through the classifier.

[0066] The quality categories corresponding to the classification results in this embodiment include three classification results: qualified, excellent and unqualified.

[0067] See also Figure 3 , Figure 3 yes Figure 2 A schematic diagram of the structure of an embodiment of a speech classification model in an embodiment.

[0068] The speech model 30 of this embodiment includes a first convolutional layer 31, a second convolutional layer 32, a first maximum pooling layer 33, two third convolutional layers 34, two fourth convolutional layers 35, a second maximum pooling layer 36, two fifth convolutional layers 37, an average pooling layer 38 and a fully connected layer 39 which are cascaded in sequence.

[0069] Among them, the first convolution layer 31 includes a 3*64 convolution layer with a convolution kernel of 9*3, the second convolution layer 32 includes a 64*64 convolution layer with a convolution kernel of 7*3, the third convolution layer 34 includes a 64*64 convolution layer with a convolution kernel of 7*3, and the fourth convolution layer 35 includes a 64*64 convolution layer with a convolution kernel of 3*3.

[0070] In a specific application scenario, the three-dimensional coordinates of the speech features of each channel on the unit spectrogram, that is, the 64-channel features, can be input into the speech classification model in sequence, and then processed by convolution and pooling, namely, a 3*64 convolution layer with a convolution kernel of 9*3, a 64*64 convolution layer with a convolution kernel of 7*3, a maximum pooling layer, four 64*64 convolution layers with two 7*3 and two 3*3 convolution kernels, a maximum pooling layer, two 64*64 convolution layers with a convolution kernel of 3*3, and an average pooling layer; the three-dimensional spectrogram after convolution and pooling is classified through a fully connected layer to obtain a three-category classification result.

[0071] The speech classification model of this embodiment is obtained by obtaining sample speech and standard classification results of sample speech, then extracting features of the sample speech, obtaining sample speech features of the sample speech, mapping the sample speech features into a sample three-dimensional spectrogram, and then inputting the sample three-dimensional spectrogram into the initial model for classification, obtaining the predicted classification results of the sample speech, and calculating the loss between the predicted classification results and the standard classification results based on the loss function, and finally responding to the loss meeting the preset conditions. The standard classification results of the sample speech include all classification types, that is, the results of the three-class quality categories of qualified, excellent and unqualified; and the standard classification results of the sample speech can be obtained by manual annotation. During training, 80% of all the sample speech obtained can be used as a training set and 20% as a verification set. When the loss function converges, the converged model is verified by the sample speech in the verification set. After the classification accuracy of the converged model for the sample speech in the verification set meets the preset accuracy, a speech classification model that can be used in practice is obtained.

[0072] Among them, obtaining sample speech, extracting features from sample speech, mapping sample speech features into sample three-dimensional spectrograms, and the model structure of the initial model in the speech classification model training process are the same as the corresponding steps in the speech classification method in the aforementioned embodiment. Please refer to the previous text and will not be repeated here.

[0073] The loss function in the speech classification model in this embodiment may include a cross entropy function, a square error loss function, a logarithmic loss function, and other damage functions. In a specific application scenario, when the loss function is a cross entropy function, the loss between the standard classification result and the predicted classification result is calculated based on the following formula:

[0074]

[0075] Among them, L is the loss value, M is the number of classification categories, that is, the number of classifications, c is any category, i is any sample speech, N is the number of training generations, y icis a sign function (0 or 1), if the true category of sample i is equal to c, it takes 1, otherwise it takes 0, p ic is the predicted probability that sample i belongs to category c.

[0076] During the model training process, the preset conditions may include: the loss value between the standard classification result calculation and the predicted classification result is less than a set value or the loss value between the standard classification result calculation and the predicted classification result is within a set range or other conditions.

[0077] In a specific application scenario, observe the comprehensive error of sample errors. When the error difference between two adjacent training generations N is less than the first set value, record the comprehensive average error. According to the empirical value, the final cross entropy result sample error comprehensive average value of each batch is less than the second set value, which means that the preset condition is met, the training is completed, and the model converges. Among them, the range of the first set value is 10 -4 ~10 -6 , for example: 10 -4 , 10 -5 or 10 -6 The second setting value ranges from 1.00 to 1.50, for example, 1.00, 1.15, 1.30, 1.33 or 1.50, etc., which can be set based on actual conditions and are not limited here. When the first setting value is 10 -5 When the second setting value is 1.33, the classification result of the trained speech classification model is closer to the training label.

[0078] The speech to be classified is classified using a three-dimensional spectrogram. After the classification result of the speech to be classified is obtained, the quality category of the product to be classified is determined based on the classification result. Finally, the transportation direction of the product to be classified on the production line is determined and controlled based on the classification result by the edge smart box. Specifically, when the classification result determines that the quality category of the product to be classified is excellent, the edge smart box controls the transportation direction of the product to be classified on the production line so that it is transmitted to the next production line link of the excellent product. When the classification result determines that the quality category of the product to be classified is qualified, the edge smart box controls the transportation direction of the product to be classified on the production line so that it is transmitted to the next production line link of the qualified product; when the classification result determines that the quality category of the product to be classified is unqualified, the edge smart box controls the transportation direction of the product to be classified on the production line so that it is transmitted to the next production line link of the unqualified product.

[0079] In a specific application scenario, after the voice classification model classifies the products to be classified on the production line, it can receive manual sampling of the classification results, and compare the classification results obtained by the voice classification model with the results of the manual sampling. When the number of classification results that are different from the sampling results exceeds a preset number, the voice classification model is retrained based on the voices to be classified corresponding to the preset number of classification results, so as to iteratively update the voice classification model to further improve the classification accuracy and reliability of the voice classification model. The preset number can be set based on actual needs, for example: 20, 50, 100, etc., and is not limited here.

[0080] Through the above steps, the speech classification method of this embodiment automates and unmanns the classification of products to be classified on the production line through a pre-trained speech classification model, improves the efficiency, accuracy and stability of product classification, and saves human resources. In addition, this embodiment combines the classification of products to be classified with the edge smart box, and uses the output results of the speech classification model to control the production line to achieve automatic product classification, which can further improve the intelligence and automation of speech classification and improve product classification efficiency. This embodiment maps speech features into a three-dimensional spectrogram, and uses the three-dimensional spectrogram to classify the speech to be classified through the speech classification model, thereby combining the speech classification model with image feature processing for model calculation, strengthening speech features, reducing classification steps and difficulty, and thus improving classification efficiency and effect. This embodiment also receives manual sampling of classification results, and iteratively updates the applied speech classification model based on the sampling results, thereby realizing the full process of production line sound recording, real-time calculation, data upload, model iteration, and model update, realizing continuous automatic optimization of the model, and reducing human investment in data collection and version update.

[0081] See also Figure 4 , Figure 4 It is a structural diagram of an implementation scheme of a speech classification system corresponding to the speech classification method of the present application.

[0082] The speech classification system 40 of this embodiment includes at least one edge smart box 42 and at least one microphone 41. Each edge smart box 42 corresponds to at least one microphone 41, and the edge smart box 42 is communicatively connected with the corresponding at least one microphone 41. The edge smart boxes 42 can be connected through a local area network to improve the deployment performance of the entire speech classification system 40.

[0083] The speech classification system 40 of this embodiment can be applied to the quality classification of products before they are shipped out of the workshop.

[0084] Specifically, the speech classification system 40 of this embodiment can be applied to multiple production lines in workshop production, wherein each production line is respectively provided with an edge intelligence box 42 and at least one microphone 41. The more microphones 41 there are on each production line, the clearer and more obvious the features of the speech to be classified obtained by the microphone 41 are.

[0085] In a specific application scenario, the sound emitted by the motor of the product to be classified on the corresponding production line can be obtained through the microphone 41 to obtain the voice to be classified. The microphone 41 transmits the voice to be classified to the corresponding edge intelligence box 42, and then the edge intelligence box 42 determines whether there is motor noise in the voice to be classified based on the voice to be classified, so as to perform quality classification on the product to be classified corresponding to the voice to be classified.

[0086] In another specific application scenario, the sound of the equipment operation of the product to be classified on the corresponding production line can be obtained through the microphone 41 to obtain the voice to be classified. The microphone 41 transmits the voice to be classified to the corresponding edge smart box 42, and then the edge smart box 42 determines whether the voice to be classified meets the quiet standard of normal use by the user based on the voice to be classified, so as to classify the quiet level of the product to be classified corresponding to the voice to be classified.

[0087] At least one microphone 41 on the production line is set in a quiet area on the production line. In a specific application scenario, the voice classification system 40 may also include a sensor, which is in communication with the edge smart box 42. When the sensor detects that the product to be classified is transmitted to the quiet area, it notifies the edge smart box 42 to control the corresponding at least one microphone 41 to start picking up sound, thereby obtaining the voice to be classified, thereby realizing automatic sound collection of the voice classification system 40.

[0088] After the edge smart box 42 classifies the products to be classified corresponding to the voice to be classified, it can control the transportation flow of the products to be classified on the corresponding production line based on the classification result, so as to actually classify the products to be classified. The edge smart box 42 in this embodiment may include but is not limited to: a microcomputer, a server, and in addition, the edge smart box 42 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0089] In a specific application scenario, the quality categories of the products to be classified include excellent, qualified, and unqualified. The transportation flow directions of each production line include three, namely the first transportation line, the second transportation line, and the third transportation line. When the edge smart box 42 determines that the quality category of the product to be classified is excellent based on the classification result, the edge smart box 42 opens the first transportation line, closes the second transportation line and the third transportation line, so that the excellent products to be classified are transmitted to the first transportation line for transmission; when the edge smart box 42 determines that the quality category of the product to be classified is qualified based on the classification result, the edge smart box 42 opens the second transportation line, closes the first transportation line and the third transportation line, so that the qualified products to be classified are transmitted to the second transportation line for transmission; when the edge smart box 42 determines that the quality category of the product to be classified is unqualified based on the classification result, the edge smart box 42 opens the third transportation line, closes the second transportation line and the first transportation line, so that the unqualified products to be classified are transmitted to the third transportation line for transmission.

[0090] The speech classification system 40 of this embodiment is used to implement the speech classification method of any of the above embodiments.

[0091] In other embodiments, the voice classification model is respectively configured on each edge smart box 42 in multiple production lines in the workshop production. Each edge smart box 42 obtains the voice to be classified transmitted by the corresponding at least one microphone 41, and classifies the voice to be classified through the configured voice classification model to obtain the classification result of the voice to be classified. The edge smart box 42 determines the quality category of the product to be classified based on the classification result, and then uses the quality category of the product to be classified to control the transportation flow of the production line, so that the product to be classified is transported to the corresponding location. Thereby, the automatic classification of the voice classification system 40 is realized.

[0092] Among them, since the speech to be classified emitted by different types of products to be classified has some different characteristics, the speech classification model configured by the edge intelligent box 42 on each production line can be set based on the type of products to be classified transported or prepared on the production line. That is, in actual engineering applications, speech classification models can be established according to different types of equipment products to be classified, and the classification of the speech to be classified of the products to be classified can be determined by the speech classification model calculation.

[0093] In other embodiments, the speech classification system 40 further includes a switch 44, a tagging server 43, and a cloud server 45. The tagging server 43 is communicatively connected to the switch 44, the switch 44 is communicatively connected to the cloud server 45, and the switch 44 is also communicatively connected to at least one edge smart box 42 respectively.

[0094] The voice classification system 40 of this embodiment collects the voice of the products to be classified on the production line through the microphone 41, and then transmits the voice to be classified to the corresponding edge smart box 42, so as to classify the voice to be classified through the voice classification model in the edge smart box 42 to obtain the classification result. The edge smart box 42 further controls the transmission direction of the products to be classified on the production line based on the classification result, and the edge smart box 42 can also transmit the classification result to the switch 44, and the switch 44 can transmit the classification result to the cloud server 45 for backup storage.

[0095] The labeling server 43 can obtain the classification results from the switch 44, so as to perform a random check on the classification results manually. The labeling server 43 is a server that can provide a function of receiving manual labeling.

[0096] In a specific application scenario, the speech whose manual sampling results are different from the classification results of the speech classification model currently configured on the edge smart box 42 is used as a new training set. When the number of samples in the new training set reaches a certain number, the model training process is automatically triggered by the annotation server 43 to retrain the existing speech classification model. After the training is completed, a new speech classification model is obtained, and the new speech classification model is sent to the corresponding edge smart box 42 through the switch 44 by the annotation server 43 to realize the iterative update of the model.

[0097] This implementation method proposes a voice quality inspection solution for unmanned equipment on the production line. Through automated sound collection on the production line and combined with an end-to-end deep learning model, it provides a classification judgment on product quality and provides a basis for automated processing.

[0098] See also Figure 5 , Figure 5 It is a structural diagram of an embodiment of a speech classification device of the present application.

[0099] The speech classification device 50 of this embodiment includes: an acquisition module 51, a mapping module 52 and a classification module 53. The acquisition module 51 is used to acquire the speech to be classified, perform feature extraction on the speech to be classified, and obtain the speech features of the speech to be classified; the mapping module 52 is used to map the speech features into a three-dimensional spectrogram; the classification module 53 is used to classify the speech to be classified using the three-dimensional spectrogram and obtain the classification result of the speech to be classified.

[0100] The acquisition module 51 is also used to acquire the speech to be classified, perform Fourier transform on the speech to be classified, and obtain the frequency domain signal of the speech to be classified; perform feature extraction on the frequency domain signal to obtain the two-dimensional speech features of the speech to be classified. The mapping module 52 is also used to perform matrix operations on the two-dimensional speech features to obtain the third-dimensional speech features; and map the two-dimensional speech features and the third-dimensional speech features into a three-dimensional spectrogram.

[0101] The acquisition module 51 is also used to extract features from the frequency domain signal to obtain two-dimensional speech features of each channel of the frequency domain signal; the mapping module 52 is also used to calculate the mean of the two-dimensional speech features of each channel; and the third-dimensional speech features are obtained based on the mean.

[0102] The acquisition module 51 is also used to filter the speech to be classified through a Mel triangle filter bank; perform logarithmic operation and discrete cosine transform on the speech to be classified after the filtering process to obtain the speech features of the speech to be classified.

[0103] The acquisition module 51 is further used to acquire the voiceprint signal of the product to be classified, and determine the voiceprint signal as the speech to be classified; the classification module 53 is further used to determine the quality category of the product to be classified based on the classification result.

[0104] The acquisition module 51 is also used to control the microphone to collect the voiceprint signals of the products to be classified on the production line through the edge intelligence box in a silent environment.

[0105] The classification module 53 is also used to determine the transportation direction of the products to be classified on the production line based on the quality category through the edge intelligence box.

[0106] The classification module 53 is also used to classify the speech to be classified by using the three-dimensional spectrogram through the speech classification model to obtain the classification result of the speech to be classified.

[0107] The acquisition module 51 is also used to input the three-dimensional spectrogram into the speech classification model, and perform convolution and pooling processing through the first convolution layer, the second convolution layer, the first maximum pooling layer, the two third convolution layers, the two fourth convolution layers, the second maximum pooling layer, the two fifth convolution layers, and the average pooling layer of the speech classification model in sequence; the three-dimensional spectrogram after the convolution and pooling processing is classified through the fully connected layer to obtain the classification result.

[0108] The acquisition module 51 is also used to obtain the sample speech and the standard classification result of the sample speech; wherein the standard classification result of the sample speech includes all classification types; perform feature extraction on the sample speech to obtain the sample speech features of the sample speech; map the sample speech features into a sample three-dimensional spectrogram; input the sample three-dimensional spectrogram into the initial model for classification to obtain the predicted classification result of the sample speech; calculate the loss between the standard classification result and the predicted classification result through a loss function; and obtain a speech classification model in response to the loss satisfying a preset condition.

[0109] The classification module 53 is also used to receive manual sampling of classification results; in response to the number of classification results that are different from the sampling results exceeding a preset number, the speech classification model is retrained based on the to-be-classified speech corresponding to the preset number of classification results.

[0110] The above scheme can improve the accuracy and reliability of speech classification.

[0111] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps of any of the above-mentioned speech classification method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0112] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned speech classification method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0113] The above scheme can improve the accuracy and reliability of speech classification.

[0114] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 701 that can be executed by a processor, and the program instructions 701 are used to implement the steps of any of the above-mentioned speech classification method embodiments.

[0115] The above scheme can improve the accuracy and reliability of speech classification.

[0116] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0117] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A speech classification method, characterized in that: The speech classification method comprises: Acquire the speech to be classified, perform feature extraction on the speech to be classified, and obtain the speech feature of the speech to be classified; wherein, acquire the speech to be classified, perform Fourier transform on the speech to be classified, and obtain the frequency domain signal of the speech to be classified; perform feature extraction on the frequency domain signal, and obtain the two-dimensional speech feature of the speech to be classified; Mapping the speech features into a three-dimensional spectrogram; wherein, performing a matrix operation on the two-dimensional speech features to obtain a third-dimensional speech feature; mapping the two-dimensional speech features and the third-dimensional speech features into the three-dimensional spectrogram; The speech to be classified is classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

2. The speech classification method according to claim 1, characterized in that: The step of extracting features from the frequency domain signal to obtain two-dimensional speech features of the speech to be classified comprises: Performing feature extraction on the frequency domain signal to obtain two-dimensional speech features of each channel of the frequency domain signal; The step of performing matrix operation on the two-dimensional speech features to obtain the third-dimensional speech features comprises: Calculate and obtain the mean of the two-dimensional speech features of each channel; A third dimension of speech features is obtained based on the mean value.

3. The speech classification method according to claim 2, characterized in that: The step of extracting features from the frequency domain signal to obtain two-dimensional speech features of each channel of the frequency domain signal comprises: Performing filtering processing on the speech to be classified by using a Mel triangle filter bank; Logarithmic operation and discrete cosine transformation are performed on the speech to be classified after filtering to obtain speech features of the speech to be classified.

4. The speech classification method according to any one of claims 1 to 3, characterized in that: The step of acquiring the speech to be classified, extracting features of the speech to be classified, and obtaining speech features of the speech to be classified also includes: Acquire a voiceprint signal of the product to be classified, and determine the voiceprint signal as the voice to be classified; The step of classifying the speech to be classified by using the three-dimensional spectrogram to obtain the classification result of the speech to be classified also includes: The quality category of the product to be classified is determined based on the classification result.

5. The speech classification method according to claim 4, characterized in that: The step of obtaining the voiceprint signal of the product to be classified also includes: In a silent environment, the edge intelligent box controls the microphone to collect the voiceprint signals of the products to be classified on the production line.

6. The speech classification method according to claim 4, characterized in that: After the step of determining the quality category of the product to be classified based on the classification result, the following step is further included: The edge intelligence box determines and controls the transportation direction of the products to be classified on the production line based on the quality category.

7. The speech classification method according to claim 1, characterized in that: The step of classifying the speech to be classified by using the three-dimensional spectrogram to obtain the classification result of the speech to be classified includes: The speech to be classified is classified by using the three-dimensional spectrogram through a speech classification model to obtain a classification result of the speech to be classified.

8. The speech classification method according to claim 7, characterized in that: The step of using the three-dimensional spectrogram to classify the speech to be classified through a speech classification model to obtain a classification result of the speech to be classified also includes: Inputting the three-dimensional spectrogram into the speech classification model, and sequentially performing convolution and pooling processing through the first convolution layer, the second convolution layer, the first maximum pooling layer, the two third convolution layers, the two fourth convolution layers, the second maximum pooling layer, the two fifth convolution layers, and the average pooling layer of the speech classification model; The three-dimensional spectrogram after convolution and pooling processing is classified through a fully connected layer to obtain the classification result.

9. The speech classification method according to claim 7, characterized in that: The speech classification method also includes: Acquire a sample speech and a standard classification result of the sample speech; wherein the standard classification result of the sample speech includes all classification types; Extracting features from the sample speech to obtain sample speech features of the sample speech; Mapping the sample speech features into a sample three-dimensional spectrogram; Inputting the sample three-dimensional spectrogram into the initial model for classification to obtain a predicted classification result of the sample speech; Calculating the loss between the standard classification result and the predicted classification result based on the loss function; In response to the loss satisfying a preset condition, the speech classification model is obtained.

10. The speech classification method according to claim 7, characterized in that: The speech classification method also includes: Receiving manual sampling of the classification results; In response to the number of differences between the classification results and the results of the random inspection exceeding a preset number, the speech classification model is retrained based on the to-be-classified speech corresponding to the preset number of classification results.

11. A speech classification device, characterized in that: The speech classification device comprises: An acquisition module is used to acquire the speech to be classified, perform feature extraction on the speech to be classified, and obtain the speech features of the speech to be classified; wherein, after acquiring the speech to be classified, Fourier transform is performed on the speech to be classified to obtain the frequency domain signal of the speech to be classified; feature extraction is performed on the frequency domain signal to obtain the two-dimensional speech features of the speech to be classified; A mapping module, used for mapping the speech features into a three-dimensional spectrogram; wherein a matrix operation is performed on the two-dimensional speech features to obtain a third-dimensional speech feature; and the two-dimensional speech features and the third-dimensional speech features are mapped into the three-dimensional spectrogram; The classification module is used to classify the speech to be classified using the three-dimensional spectrogram to obtain a classification result of the speech to be classified.

12. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the speech classification method as claimed in any one of claims 1 to 10.

13. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the speech classification method according to any one of claims 1 to 10 is implemented.

Images