Voice processing method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a voice processing method, which comprises the following steps: carrying out framing processing on a to-be-processed voice sequence to obtain voice frames; performing voice activity detection on the voice frame to obtain a first confidence coefficient corresponding to the voice frame, the first confidence coefficient being used for indicating a credibility degree that the voice frame has an effective voice activity; determining a target voice frame in the voice frames based on the first confidence coefficient; and performing voice-to-text processing on the target voice frame to obtain a voice processing result of the to-be-processed voice sequence. According to the invention, the problem of low accuracy of follow-up ASR, semantic understanding and multi-modal fusion caused by the fact that an effective audio clip cannot be accurately judged in current children voice dialogue and article recognition products in a noisy environment in an existing voice processing method is solved.

Description

Voice processing methods, devices, electronic equipment and storage media Technical Field

[0001] This application belongs to the field of artificial intelligence technology, and in particular relates to a speech processing method, device, electronic device and storage medium. Background Technology

[0002] In children's voice dialogue and object recognition products, accurately identifying "valid audio segments" (i.e., children's actual vocalizations, valid commands, or emotional speech segments) is a prerequisite for ensuring the accuracy of subsequent ASR, semantic understanding, and multimodal fusion. However, real-world environments often contain complex noise (indoor television, family conversations, traffic noise), echoes, overlapping multiple speakers, and short, intermittent vocalizations, resulting in high false positive and false negative rates for traditional VADs based on energy thresholds or simple statistical features. Meanwhile, if speech enhancement (noise suppression, echo cancellation, speech gain) is designed independently of VAD, it can cause delays and erroneous triggering (e.g., enhancement is only performed after noise is misidentified as speech, wasting computation and leading to ASR errors). Therefore, a speech processing method that achieves low latency, high recall, and high accuracy is needed to address the problem that existing speech processing methods cannot accurately identify "valid audio segments" in noisy environments in current children's voice dialogue and object recognition products, leading to low accuracy in subsequent ASR, semantic understanding, and multimodal fusion. Summary of the Invention

[0003] This application provides a speech processing method that addresses the problem that existing speech processing methods, in noisy environments, cannot accurately determine "valid audio segments" in current children's voice dialogue and object recognition products, leading to low accuracy in subsequent ASR, semantic understanding, and multimodal fusion. The method involves segmenting the speech sequence to be processed into frames, performing speech activity detection on these frames to obtain a first confidence level, and identifying the target speech frame within the frames based on this first confidence level. The target speech frame is then converted to text, yielding the speech processing result for the speech sequence to be processed. This method solves the problem that existing speech processing methods, in noisy environments, cannot accurately determine "valid audio segments" in current children's voice dialogue and object recognition products, leading to low accuracy in subsequent ASR, semantic understanding, and multimodal fusion.

[0004] In a first aspect, embodiments of this application provide a speech processing method, the method comprising the following steps: performing frame-by-frame processing on a speech sequence to be processed to obtain speech frames; performing speech activity detection on the speech frames to obtain a first confidence level corresponding to the speech frames, the first confidence level being used to indicate the degree of credibility of valid speech activity in the speech frames; determining a target speech frame in the speech frames based on the first confidence level; and performing speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

[0005] Optionally, the step of performing speech activity detection on the speech frame to obtain the first confidence level corresponding to the speech frame includes: performing pre-enhancement processing on the speech frame to obtain a pre-enhancement speech frame; and performing speech activity detection on the pre-enhancement speech frame to obtain the first confidence level corresponding to the speech frame.

[0006] Optionally, determining the target speech frame in the speech frames based on the first confidence level includes: if the first confidence level is greater than a preset first confidence threshold, then the speech frame is determined as the target speech frame; if the first confidence level is less than a preset second confidence threshold, then the speech frame is discarded, where the second confidence threshold is less than the first confidence threshold; if the first confidence level is less than or equal to the first confidence threshold and greater than or equal to the second confidence threshold, then the speech frame undergoes spectral reduction processing to obtain a spectral-reduced speech frame; speech activity detection is performed on the spectral-reduced speech frame to obtain a second confidence level corresponding to the spectral-reduced speech frame; and based on the second confidence level, whether the spectral-reduced speech frame is the target speech frame is determined.

[0007] Optionally, the step of performing spectral reduction processing on the speech frame to obtain a spectral-reduced speech frame includes: performing voiceprint feature extraction processing on the speech frame to obtain target voiceprint features of the speech frame; comparing the target voiceprint features with the base voiceprint features in the voiceprint database to determine the target spectral density, where each base voiceprint feature corresponds to a base spectral density; and performing spectral reduction processing on the speech frame based on the target spectral density to obtain a spectral-reduced speech frame.

[0008] Optionally, determining whether the spectrally reduced speech frame is a target speech frame based on the second confidence level includes: if the second confidence level is greater than the first confidence level threshold, then determining the speech frame as a target speech frame; if the second confidence level is less than the first confidence level threshold, then determining whether the speech frame is a speech frame to be addressed; if the speech frame is a speech frame to be addressed, then adding the voiceprint features and the spectral density of the speech frame to be addressed to the voiceprint database.

[0009] Optionally, before comparing the target voiceprint features with the base voiceprint features in the voiceprint database to determine the target spectrogram, the method further includes: acquiring noise audio files of different types of noise, and performing frame-by-frame processing on the noise audio files to obtain noise frames; extracting multi-dimensional features from the noise frames to obtain multi-dimensional features of the noise frames; performing clustering processing on the multi-dimensional features to obtain clustering results; establishing voiceprint feature templates and spectrogram templates for each noise category based on the clustering results; and constructing a voiceprint database based on the voiceprint feature templates and the spectrogram templates.

[0010] Optionally, the step of performing speech activity detection on the speech frame to obtain the first confidence level corresponding to the speech frame includes: acquiring effective speech activity samples of the target population; adjusting the pre-trained speech activity detector based on the effective speech activity samples to obtain an adjusted speech activity detector; and performing speech activity detection on the speech frame using the adjusted speech activity detector to obtain the first confidence level corresponding to the speech frame.

[0011] Secondly, embodiments of this application provide a speech processing apparatus, comprising: a first processing module for performing frame-based processing on a speech sequence to be processed to obtain speech frames; a second processing module for performing speech activity detection on the speech frames to obtain a first confidence level corresponding to the speech frames, the first confidence level being used to indicate the degree of credibility of valid speech activity in the speech frames; a determination module for determining a target speech frame in the speech frames based on the first confidence level; and a third processing module for performing speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

[0012] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the voice processing method provided in the embodiments of the present invention.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the speech processing method provided in the embodiments of the present invention.

[0014] The above-mentioned solution of this application has the following beneficial effects: The speech sequence to be processed is segmented into frames to obtain speech frames; speech activity is detected on the speech frames to obtain a first confidence level corresponding to the speech frame, the first confidence level being used to indicate the degree of credibility of valid speech activity in the speech frame; based on the first confidence level, the target speech frame is determined in the speech frames; the target speech frame is processed into speech-to-text to obtain the speech processing result of the speech sequence to be processed. This invention solves the problem that existing speech processing methods, in noisy environments, cannot accurately determine "valid audio segments" in current children's voice dialogue and object recognition products, leading to low accuracy in subsequent ASR, semantic understanding, and multimodal fusion.

[0015] Other beneficial effects of this application will be described in detail in the following detailed description section. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is a flowchart of a speech processing method provided in an embodiment of this application; Figure 2 is a flowchart of a process for constructing a voiceprint database provided in an embodiment of this application; Figure 3 is a schematic diagram of a speech processing device provided in an embodiment of this application; Figure 4 is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0020] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0022] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0024] As shown in Figure 1, Figure 1 is a flowchart of a speech processing method provided by an embodiment of the present invention. The speech processing method includes the following steps: 101. The speech sequence to be processed is divided into frames to obtain speech frames.

[0025] In this embodiment of the invention, the above-described speech processing method can be applied to a speech processing platform, which can be built on or distributed servers. The speech processing platform includes a data interface (for sensors or users to upload data), a knowledge database, and a knowledge database construction program. The data interface can be used to obtain a first feature vector to be searched, and the knowledge database construction program can be used to construct the knowledge database. The knowledge database is specifically used to provide additional association information for the identified data entities, thereby improving the depth of the data recognition system's understanding of the content.

[0026] The aforementioned speech sequence to be processed can be a raw audio signal stream containing potentially valid speech and background environment, acquired in real time by an audio acquisition device (such as a microphone). The speech sequence to be processed can be in WAV data format, PCM data format, etc. For example, it could be a speech sequence to be processed obtained by turning on a television or vacuum cleaner and then using an audio acquisition device (such as a professional microphone) to collect the speech.

[0027] Furthermore, a uniform sampling rate can be applied to the speech sequence to be processed, such as a uniform sampling rate of 16kHz, and the volume of the speech can be adjusted to ensure that the energy of all speech sequences to be processed is within a reasonable range before performing frame-by-frame processing on the speech sequence to be processed.

[0028] The aforementioned frame segmentation process involves dividing the speech sequence to be processed into multiple shorter segments, such as segments with a frame length of 20 ms and a frame shift of 10 ms. Frame segmentation includes pre-emphasis, window functions, and framing. Pre-emphasis can compensate for energy loss in the high-frequency part of the signal. Window functions can reduce amplitude changes at frame boundaries and improve the stability of the analysis; for example, Hamming windows or rectangular windows can be used. Framing can divide the speech sequence to be processed into multiple short segments according to the set frame length and frame shift. For example, if the speech sequence to be processed is a 1-second speech signal with a frame length of 20 milliseconds and a frame shift of 10 milliseconds, the speech sequence to be processed can be pre-emphasized, and then a window function can be used to divide the speech sequence to be processed into multiple 20-millisecond segments. This will result in a sequence containing multiple speech frames, each of which is an independent short segment that can be analyzed and processed separately.

[0029] In one possible implementation, for example, a sliding window of length L (e.g., L=20ms) can be used to segment the speech sequence to be processed at intervals of S (e.g., S=10ms). Each segment of speech data obtained is a speech frame. Since the frame shift S is less than the frame length L, there is partial overlap between adjacent speech frames.

[0030] The aforementioned speech frame can be a short speech signal segment of fixed duration obtained by framing the speech sequence to be processed.

[0031] 102. Perform speech activity detection on the speech frame to obtain the first confidence level corresponding to the speech frame.

[0032] In this embodiment of the invention, the aforementioned voice activity detection can be a process of calculating and outputting a confidence level characterizing the presence of valid voice within a voice frame based on the acoustic features of the voice frame. Voice Activity Detection (VAD) is a technique for identifying voice segments and non-voice segments (such as silence or noise) from audio signals. The core function of VAD is to extract valid voice signals from noisy environments, ensuring that subsequent tasks such as speech recognition, semantic understanding, and speech enhancement can be performed efficiently.

[0033] Specifically, preprocessing operations such as filtering and normalization can be performed on the speech frames to eliminate background noise and device interference, resulting in preprocessed speech frames. Useful information, such as short-time energy, zero crossover rate, and cepstral coefficients, can then be extracted from these preprocessed speech frames to obtain speech features. These features are then processed by a trained speech recognition model to obtain the confidence score corresponding to the speech frame. The trained speech recognition model can be a speech recognition model built based on deep learning or machine learning, such as CNN or RNN. Alternatively, the trained speech recognition model can be obtained by training an untrained speech recognition model using a training dataset. This training dataset includes sample speech data, valid speech annotation data, and invalid speech annotation data. The annotation data can be structured labels added to the original data, enabling the machine learning model to recognize and process both the original and annotated data. Through the annotation data, the model can establish a mapping relationship between input data and correct output labels. Finally, the untrained speech recognition model can also be a speech recognition model built based on deep learning or machine learning, such as CNN or RNN. The training described above can be supervised training. Supervised training uses a set of data with known labels to train the model. By optimizing the model parameters, the model can predict the labels of new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to tune the model's parameters to minimize the difference between the model's output label and the input data. A loss function measures the difference between the model's prediction and the true result; its purpose is to improve prediction accuracy by minimizing the loss function value through adjusting the model parameters. The aforementioned loss function can be the mean squared error loss function, cross-entropy loss function, etc.

[0034] The trained speech recognition model described above can identify the confidence levels corresponding to valid speech and invalid speech within a speech frame. The valid speech can be the speech corresponding to a group of children.

[0035] The aforementioned first confidence level is used to indicate the degree of confidence that a speech frame contains valid speech activity.

[0036] In one possible implementation, for example, there is a speech frame containing silence, speech, and background noise. In the silence part, the speech activity detection algorithm can give a lower confidence score, while in the speech part, the confidence score will be significantly higher. Based on the confidence score, it can be determined which frames are speech and which are silence.

[0037] 103. Based on the first confidence level, the target speech frame is determined in the speech frame.

[0038] In this embodiment of the invention, the first confidence level is used to indicate the degree of confidence that there is valid voice activity in the voice frame.

[0039] The aforementioned target speech frame can be a speech frame containing valid speech content determined from among the speech frames based on a first confidence score. Further, based on the first confidence score corresponding to the speech frame, speech frames with a first confidence score greater than a preset confidence threshold can be designated as target speech frames. The preset confidence threshold can be a confidence threshold pre-set by the system. It is understood that when a speech frame has a first confidence score greater than the preset confidence threshold, the speech frame is determined to be valid speech and is designated as the target speech frame.

[0040] 104. Perform speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

[0041] In this embodiment of the invention, the aforementioned speech-to-text processing can be a process of converting a target speech frame into text data. This can be achieved by using automatic speech recognition (ASR) technology to process the target speech frame into text. ASR is an artificial intelligence technology that can convert human speech signals into text. By analyzing the spectrum, acoustic features, and language models of the speech signal, ASR identifies and decodes the text content within the speech, thereby achieving speech-to-text conversion.

[0042] The above-mentioned speech processing result can be the text form generated after the target speech frame is processed into speech-to-text, which corresponds to the effective speech content in the speech sequence to be processed. For example, if the target speech frame is "The weather is so nice today, let's go to the park", after speech-to-text processing, the speech processing result is "The weather is so nice today, let's go to the park".

[0043] In this embodiment of the invention, the present invention performs frame-by-frame processing on the speech sequence to be processed to obtain speech frames, performs speech activity detection on the speech frames to obtain the first confidence level corresponding to the speech frames, and determines the target speech frame in the speech frames based on the first confidence level. The target speech frame is then processed into speech-to-text, thereby obtaining the speech processing result of the speech sequence to be processed. This solves the problem that existing speech processing methods cannot accurately determine "valid audio segments" in noisy environments in current children's voice dialogue and object recognition products, resulting in low accuracy of subsequent ASR, semantic understanding and multimodal fusion.

[0044] In this embodiment of the invention, the speech sequence to be processed is segmented into frames to obtain speech frames; speech activity detection is performed on the speech frames to obtain a first confidence level corresponding to the speech frame, which indicates the degree of confidence that the speech frame contains valid speech activity; based on the first confidence level, a target speech frame is determined in the speech frames; the target speech frame is then processed into speech-to-text to obtain the speech processing result of the speech sequence to be processed. This invention solves the problem that existing speech processing methods, in noisy environments, cannot accurately determine "valid audio segments" in current children's voice dialogue and object recognition products, leading to low accuracy in subsequent ASR, semantic understanding, and multimodal fusion.

[0045] It is understood that in the specific implementation of this application, data such as voice data, confidence data, and voice frame data are involved. When the embodiments in this application are applied to specific products or technologies, user permission or consent is required. Furthermore, the collection, use, and processing of related data, as well as the training, deployment, and invocation of algorithm models, must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0046] Optionally, in the step of performing speech activity detection on the speech frame to obtain the first confidence level corresponding to the speech frame, the speech frame can be pre-enhanced to obtain a pre-enhanced speech frame; and speech activity detection can be performed on the pre-enhanced speech frame to obtain the first confidence level corresponding to the speech frame.

[0047] In this embodiment of the invention, the aforementioned speech frame can be a short speech signal segment of fixed duration obtained by performing frame-segmentation processing on the speech sequence to be processed.

[0048] The aforementioned pre-enhancement processing can be a lightweight operation applied to speech frames to suppress general background noise and improve the overall signal-to-noise ratio. It can be achieved by extracting speech features using a lightweight convolutional neural network (CNN) to reduce background noise interference. This lightweight CNN can be a CNN architecture that significantly reduces the number of parameters, computational cost, and memory usage through specific design strategies while maintaining high model performance. Pre-enhancement processing aims to extract the clearest and cleanest possible speech content from speech signals affected by noise or reverberation. Pre-enhancement processing can improve voice communication quality, increase speech recognition accuracy, and optimize user experience.

[0049] The aforementioned pre-enhanced speech frame can be a speech frame obtained after pre-enhancing the speech frame.

[0050] The aforementioned speech activity detection can be a process that calculates and outputs a confidence level representing the presence of valid speech in the pre-enhanced speech frame based on its acoustic features. Speech activity detection (VAD) is a technique for identifying speech segments and non-speech segments (such as silence or noise) from audio signals. The core function of VAD is to extract valid speech signals from noisy environments, ensuring the efficient execution of subsequent tasks such as speech recognition, semantic understanding, and speech enhancement.

[0051] The aforementioned first confidence level is used to indicate the degree of confidence that a speech frame contains valid speech activity.

[0052] It should be noted that when performing speech activity detection on the pre-enhanced speech frames, useful information, such as short-time energy, zero crossover rate, and cepstral coefficients, is extracted from the pre-enhanced speech frames to obtain speech features. These speech features are then processed by a trained speech recognition model to obtain the first confidence score corresponding to the speech frame. The trained speech recognition model can be a speech recognition model built based on deep learning or machine learning, such as CNN or RNN. This trained speech recognition model can identify the confidence scores corresponding to valid speech and invalid speech within a speech frame.

[0053] Optionally, in the step of determining the target speech frame in a speech frame based on a first confidence level, if the first confidence level is greater than a preset first confidence level threshold, then the speech frame is determined as the target speech frame; if the first confidence level is less than a preset second confidence level threshold, then the speech frame is discarded, and the second confidence level threshold is less than the first confidence level threshold; if the first confidence level is less than or equal to the first confidence level threshold and greater than or equal to the second confidence level threshold, then the speech frame undergoes spectral reduction processing to obtain a spectral-reduced speech frame; speech activity detection is performed on the spectral-reduced speech frame to obtain a second confidence level corresponding to the spectral-reduced speech frame; based on the second confidence level, it is determined whether the spectral-reduced speech frame is the target speech frame.

[0054] In this embodiment of the invention, the aforementioned preset first confidence threshold can be a first confidence threshold pre-set by the system, or a high confidence threshold. When the first confidence level is greater than the preset first confidence threshold, the speech frame is determined to be valid speech, and the speech frame is identified as the target speech frame.

[0055] The aforementioned preset second confidence threshold can be a pre-set second confidence threshold by the system, or it can be a low confidence threshold. When the first confidence level is less than the preset second confidence threshold, the speech frame is discarded directly. The preset second confidence threshold is less than the preset first confidence threshold.

[0056] The aforementioned spectral reduction processing can be a process that, for speech frames with a first confidence level less than or equal to a first confidence threshold and greater than or equal to a second confidence threshold, subtracts noise spectrum components from the spectrum of the speech frame to further suppress specific noise and improve the speech signal-to-noise ratio.

[0057] The aforementioned speech activity detection can be a process that calculates and outputs a confidence level representing the presence of valid speech in the subtracted speech frame based on the acoustic features of the subtracted speech frame.

[0058] The aforementioned second confidence level can be the confidence level corresponding to the subtracted speech frame.

[0059] It should be noted that if the first confidence level is greater than the preset first confidence level threshold, the speech frame can be identified as the target speech frame; if the first confidence level is less than the preset second confidence level threshold, the speech frame is discarded directly; if the first confidence level is less than or equal to the first confidence level threshold and greater than or equal to the second confidence level threshold, the speech frame is subjected to spectral reduction processing to obtain a spectral-reduced speech frame, and speech activity detection is performed on the spectral-reduced speech frame to obtain the second confidence level corresponding to the spectral-reduced speech frame. Based on the second confidence level, it is determined whether the spectral-reduced speech frame is the target speech frame.

[0060] Optionally, in the step of performing spectral reduction processing on the speech frame to obtain the spectral-reduced speech frame, the speech frame can be processed to extract voiceprint features to obtain the target voiceprint features of the speech frame; the target voiceprint features are compared with the base voiceprint features in the voiceprint database to determine the target spectral density, with each base voiceprint feature corresponding to a base spectral density; based on the target spectral density, the speech frame is processed to obtain the spectral-reduced speech frame.

[0061] In this embodiment of the invention, the above-mentioned voiceprint feature extraction process can be a process of extracting acoustic feature parameters that can characterize the speaker's identity from a speech frame.

[0062] The target voiceprint features of the aforementioned speech frame can be obtained by extracting voiceprint features from the speech frame, and are used to characterize the overall acoustic content of the speech frame as a multidimensional feature vector.

[0063] The aforementioned voiceprint database can be a database that stores standardized voiceprint feature templates and their corresponding average spectrum (phonogram) templates for various known types of environmental noise through pre-collection and analysis. The voiceprint database can be a local voiceprint database or a cloud-based voiceprint database.

[0064] The aforementioned base voiceprint features can be pre-built and stored in a voiceprint database, representing statistically representative standardized voiceprint feature vectors for a specific category of environmental noise (such as television noise or fan noise). Each base voiceprint feature corresponds to a base audio spectrum. The base audio can be the original, source-based environmental noise recording file or data stream used in the voiceprint database.

[0065] The above comparison can be a process of calculating the similarity between the target voiceprint features and each base voiceprint feature in the voiceprint database in order to find the most matching noise category.

[0066] The aforementioned target audio spectrum can be the audio corresponding to the noise category that best matches the target voiceprint features, obtained by comparing the target voiceprint features with the base voiceprint features in the voiceprint database.

[0067] The aforementioned spectral subtraction processing can be a process of subtracting the noise spectrum component from the spectrum of the target audio spectrum based on the target audio spectrum, in order to further suppress specific noise and improve the speech signal-to-noise ratio (SNR). SNR (Signal-to-Noise Ratio) is the ratio of signal power to noise power, a metric for measuring signal clarity; a high SNR indicates good signal quality, while a low SNR indicates more noise.

[0068] It should be noted that voiceprint feature extraction can be performed on the speech frame to obtain the target voiceprint feature of the speech frame. The target voiceprint feature is then compared with the base voiceprint feature in the voiceprint database to determine the target spectral density. Based on the target spectral density, the speech frame is subjected to spectral reduction processing to obtain the spectral-reduced speech frame, which can improve the accuracy of speech recognition.

[0069] Optionally, in the step of determining whether the spectral-subtracted speech frame is the target speech frame based on the second confidence level, if the second confidence level is greater than the first confidence level threshold, the speech frame is determined to be the target speech frame; if the second confidence level is less than the first confidence level threshold, it is determined whether the speech frame is the speech frame to be treated; if the speech frame is the speech frame to be treated, the voiceprint features of the speech frame to be treated and the spectral density of the speech frame to be treated are added to the voiceprint database.

[0070] In this embodiment of the invention, the second confidence level can be the confidence level corresponding to the reduced-frequency speech frame.

[0071] The aforementioned first confidence threshold can be a pre-set first confidence threshold by the system. When the second confidence level corresponding to the down-converted speech frame is greater than the first confidence threshold, the down-converted speech frame is determined to be valid speech, and the down-converted speech frame is identified as the target speech frame.

[0072] The aforementioned speech frames to be treated can be speech frames whose second confidence level is lower than the first confidence threshold after spectral reduction, which cannot be processed and need to be submitted to the cloud server for manual or more complex automated analysis and processing.

[0073] The voiceprint features of the aforementioned speech frames to be governed can be voiceprint feature vectors extracted from the speech frames to be governed.

[0074] The audio of the aforementioned speech frame to be treated can be the complete time-domain digital audio signal corresponding to the speech frame to be treated.

[0075] Furthermore, when the reduced-frequency speech frame is a speech frame to be processed, the speech frame can be uploaded to the cloud for manual processing, and the voiceprint features of the processed speech frame and the spectral density of the speech frame to be processed can be added to the voiceprint database.

[0076] Optionally, before comparing the target voiceprint features with the base voiceprint features in the voiceprint database to determine the target spectral density, it is also possible to obtain noise audio files of different types of noise, and perform frame segmentation on the noise audio files to obtain noise frames; perform multi-dimensional feature extraction on the noise frames to obtain multi-dimensional features of the noise frames; perform clustering on the multi-dimensional features to obtain clustering results; based on the clustering results, establish voiceprint feature templates and spectral density templates for each noise category; and construct a voiceprint database based on the voiceprint feature templates and spectral density templates.

[0077] In this embodiment of the invention, the different types of noise mentioned above can be noises with different sources, characteristics and generation mechanisms, such as traffic noise, industrial noise, and human voices.

[0078] The aforementioned noise audio file can be an audio file containing various noises.

[0079] The above-mentioned frame segmentation process can be a process of dividing a noisy audio file into multiple shorter segments, such as dividing it into multiple segments with a frame length of 20 ms and a frame shift of 10 ms.

[0080] The aforementioned noise frame can be a short-duration speech signal segment of fixed duration obtained by framing the noise audio file.

[0081] The aforementioned multi-dimensional feature extraction can be a process of extracting various types of acoustic feature parameters from noisy frames. It can extract multi-dimensional features such as short-time energy, zero-crossing rate, spectral envelope, spectral centroid, noise spectrum, and feature values ​​extracted using the librosa package and NumPy. The aforementioned short-time energy reflects the energy intensity of an audio frame; the aforementioned zero-crossing rate (ZCR) reflects the activity level of an audio signal; the aforementioned spectral envelope describes the shape of the spectrum, typically represented by Mel-frequency cepstral coefficients (MFCC); the aforementioned spectral centroid reflects the average frequency of the spectrum; the aforementioned spectral contrast is used to distinguish the spectral characteristics of different noise sources; feature extraction based on the librosa package and numpy can be expressed as: import librosa import numpy as np # Extract features mfccs=librosa.feature.mfcc(y=audio, sr=16000, n_mfcc=13) energy = np.sum(audio**2, axis=1) zcr = librosa.feature.zero_crossing_rate(audio) spectral_contrast=librosa.feature.spectral_contrast(y=audio, The `Librosa` package, defined as `sr=16000) spectral_centroid=librosa.feature.spectral_centroid(y=audio, sr=16000)`, is a Python toolkit for audio and music analysis, providing rich functionality for processing and extracting audio features. NumPy (Numerical Python), also mentioned above, is an open-source scientific computing library for the Python language. Licensed under the BSD license, NumPy focuses on efficiently handling multidimensional arrays and matrix operations, providing powerful numerical computation capabilities for Python.

[0082] The aforementioned multidimensional features can be a collection of various types of acoustic feature parameters extracted from a noisy frame, with these different types of acoustic feature parameters collectively forming a multidimensional vector. Multidimensional features are used to comprehensively characterize the acoustic properties of a noisy frame.

[0083] The clustering process described above can be the process of using clustering algorithms to cluster multi-dimensional features. These clustering algorithms aim to group objects in a dataset so that objects within the same group (called a cluster) are similar to each other, while objects in different groups are significantly different. Algorithms such as K-means can be used. The core idea of ​​the K-Means algorithm is to iteratively optimize the allocation of data points into K clusters to minimize the distance from each point to the center of its cluster.

[0084] The aforementioned voiceprint feature template can be a fixed feature vector representing the typical acoustic characteristics of different types of noise. The voiceprint feature template serves as the "standard answer" in the voiceprint database for rapid comparison and identification of noise categories. Specifically, after obtaining the clustering results, for each noise category, the arithmetic mean of the multi-dimensional feature vectors of all noise frames within that category can be calculated. The calculated mean vector is then used as the voiceprint feature template for that noise category.

[0085] The aforementioned spectral template can be a fixed spectral template representing the typical spectral energy distribution of different types of noise. After completing the "clustering process," the noise frames under each noise category have been determined. Specifically, after obtaining the clustering results, for each noise category, the signals of all noise frames can be converted from the time domain to the frequency domain to obtain the amplitude spectrum of each frame. The amplitude spectra of all frames are then averaged point by point to obtain the averaged spectrum, which is then used as the spectral template for the noise category.

[0086] The aforementioned voiceprint database can be constructed based on voiceprint feature templates and phonological templates.

[0087] As shown in Figure 2, Figure 2 is a flowchart of the construction of a voiceprint database provided by an embodiment of the present invention. Specifically, it includes the following steps: Step 201: Noise audio input.

[0088] Among them, the noise audio consists of noise audio files of different types of noise.

[0089] Step 201: Audio preprocessing.

[0090] The process involves resampling the noisy audio and standardizing the sampling rate, for example, to 16kHz; then dividing the standardized noisy audio into frames, for example, with a frame length of 20ms and a frame shift of 10ms; and finally normalizing the speech frames and adjusting the audio volume to ensure that the energy of all noisy frames is within a reasonable range.

[0091] Step 203: Feature extraction.

[0092] The extraction of multi-dimensional features for classification and identification from noisy frames can be achieved through various methods, including short-time energy, zero-crossing rate, spectral envelope, spectral centroid, noise spectrogram, and feature extraction based on the librosa package and NumPy. The short-time energy reflects the energy intensity of the audio frame; the zero-crossing rate (ZCR) reflects the activity level of the audio signal; the spectral envelope describes the shape of the spectrum, typically represented by Mel-frequency cepstral coefficients (MFCC); the spectral centroid reflects the average frequency of the spectrum; and the noise spectrogram distinguishes the spectral characteristics of different noise sources.

[0093] Step 204: Cluster analysis.

[0094] This process uses clustering algorithms (such as K-means) to cluster multi-dimensional features, classifying noisy frames into different categories to obtain clustering results. Categories can include television, vacuum cleaners, traffic, etc.

[0095] Step 205: Construct a voiceprint database.

[0096] Based on the clustering results, a voiceprint feature template and a spectrogram template are created for each noise category. A voiceprint database is then constructed using these templates. The voiceprint database can be implemented using a Redis cache. This Redis cache can be a high-performance key-value in-memory database used to store and quickly retrieve the voiceprint feature templates and spectrogram templates for high-frequency noise.

[0097] Optionally, in the step of detecting speech activity on a speech frame and obtaining the first confidence level corresponding to the speech frame, effective speech activity samples of the target population can be obtained; based on the effective speech activity samples, the pre-trained speech activity detector can be adjusted to obtain the adjusted speech activity detector; and the speech activity detection on the speech frame can be performed using the adjusted speech activity detector to obtain the first confidence level corresponding to the speech frame.

[0098] In this embodiment of the invention, the effective speech activity samples of the target population can be effective speech activity samples collected from the target population. The target population can be children. The effective speech activity samples of the target population include effective speech activity samples and confidence labeling data of the effective speech activity samples.

[0099] The pre-trained speech activity detector mentioned above can be a speech activity detector built based on deep learning or machine learning, such as CNN, RNN, etc. The core task of the speech activity detector is to analyze the audio stream and automatically distinguish which parts contain human speech and which parts are silence, background noise, or other non-speech sounds.

[0100] The aforementioned adjustments can be supervised training and model parameter tuning. Supervised training uses a set of effective speech activity samples from a target population with known labels to train a pre-trained speech activity detector. By optimizing the speech activity detector parameters, the detector can predict labels for new data or make decisions based on the characteristics of existing data. During training, a minimum loss function can be used to tune the speech activity detector parameters to minimize the difference between the output label and the input data. This loss function measures the difference between the speech activity detector's prediction and the actual result, aiming to improve prediction accuracy by minimizing the loss function value through parameter tuning. This loss function can be a mean squared error loss function, a cross-entropy loss function, etc.

[0101] The adjusted speech activity detector described above is able to identify the confidence level of valid speech in a speech frame.

[0102] The aforementioned speech activity detection can be a process of representing the confidence level of a speech frame containing valid speech using an adjusted speech activity detector. The core function of speech activity detection is to extract valid speech signals from a noisy environment, ensuring that subsequent tasks such as speech recognition, semantic understanding, and speech enhancement can be performed efficiently.

[0103] The aforementioned first confidence level is used to indicate the degree of confidence that a speech frame contains valid speech activity.

[0104] In this embodiment of the invention, the present invention performs frame-by-frame processing on the speech sequence to be processed to obtain speech frames, performs speech activity detection on the speech frames to obtain the first confidence level corresponding to the speech frames, and determines the target speech frame in the speech frames based on the first confidence level. The target speech frame is then processed into speech-to-text, thereby obtaining the speech processing result of the speech sequence to be processed. This solves the problem that existing speech processing methods cannot accurately determine "valid audio segments" in noisy environments, which leads to low accuracy in subsequent ASR, semantic understanding and multimodal fusion in current children's voice dialogue and object recognition products.

[0105] As shown in Figure 3, this embodiment of the invention provides a speech processing device, which includes: a first processing module 301, used to perform frame-segmentation processing on the speech sequence to be processed to obtain speech frames; a second processing module 302, used to perform speech activity detection on the speech frames to obtain a first confidence level corresponding to the speech frames, wherein the first confidence level is used to indicate the degree of credibility of the existence of valid speech activity in the speech frames; a determination module 303, used to determine a target speech frame in the speech frames based on the first confidence level; and a third processing module 304, used to perform speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

[0106] Optionally, the second processing module 302 is further configured to perform pre-enhancement processing on the speech frame to obtain a pre-enhancement speech frame; and to perform speech activity detection on the pre-enhancement speech frame to obtain a first confidence level corresponding to the speech frame.

[0107] Optionally, the determining module 303 is further configured to: determine the speech frame as the target speech frame if the first confidence level is greater than a preset first confidence threshold; discard the speech frame if the first confidence level is less than a preset second confidence threshold, wherein the second confidence threshold is less than the first confidence threshold; perform spectral reduction processing on the speech frame to obtain a spectral-reduced speech frame if the first confidence level is less than or equal to the first confidence threshold and greater than or equal to the second confidence threshold; perform speech activity detection on the spectral-reduced speech frame to obtain a second confidence level corresponding to the spectral-reduced speech frame; and determine whether the spectral-reduced speech frame is the target speech frame based on the second confidence level.

[0108] Optionally, the determining module 303 is further configured to perform voiceprint feature extraction processing on the speech frame to obtain the target voiceprint feature of the speech frame; compare the target voiceprint feature with the base voiceprint feature in the voiceprint database to determine the target spectral density, where each base voiceprint feature corresponds to a base spectral density; and perform spectral reduction processing on the speech frame based on the target spectral density to obtain the spectral-reduced speech frame.

[0109] Optionally, the determining module 303 is further configured to: if the second confidence level is greater than the first confidence level threshold, determine the speech frame as the target speech frame; if the second confidence level is less than the first confidence level threshold, determine whether the speech frame is a speech frame to be addressed; if the speech frame is a speech frame to be addressed, add the voiceprint features of the speech frame to be addressed and the spectral density of the speech frame to be addressed to the voiceprint database.

[0110] Optionally, the device is further configured to acquire noise audio files of different types of noise, and perform frame segmentation processing on the noise audio files to obtain noise frames; extract multi-dimensional features from the noise frames to obtain multi-dimensional features of the noise frames; perform clustering processing on the multi-dimensional features to obtain clustering results; based on the clustering results, establish a voiceprint feature template and a spectrogram template for each noise category; and construct a voiceprint database based on the voiceprint feature template and the spectrogram template.

[0111] Optionally, the second processing module 302 is further configured to acquire effective speech activity samples of the target population; adjust the pre-trained speech activity detector based on the effective speech activity samples to obtain an adjusted speech activity detector; and perform speech activity detection on the speech frame using the adjusted speech activity detector to obtain the first confidence level corresponding to the speech frame.

[0112] As shown in Figure 4, this embodiment of the invention also provides an electronic device, including a processor, which can execute any of the above-mentioned voice processing methods.

[0113] Specifically, it includes a processor 401 and a memory 402, as well as a computer program stored in the memory 402 and capable of running on the processor 401 to execute a speech processing method. The processor 401 runs the speech processing method computer program stored in the memory 402, performing the following steps: dividing the speech sequence to be processed into frames to obtain speech frames; performing speech activity detection on the speech frames to obtain a first confidence level corresponding to the speech frame, the first confidence level indicating the degree of confidence that the speech frame contains valid speech activity; based on the first confidence level, identifying a target speech frame in the speech frames; and performing speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

[0114] Optionally, the step of performing voice activity detection on the voice frame and obtaining a first confidence level corresponding to the voice frame, executed by the processor 401, includes: performing pre-enhancement processing on the voice frame to obtain a pre-enhancement voice frame; and performing voice activity detection on the pre-enhancement voice frame to obtain a first confidence level corresponding to the voice frame.

[0115] Optionally, the step of determining the target speech frame in the speech frame based on the first confidence level, executed by the processor 401, includes: if the first confidence level is greater than a preset first confidence threshold, then the speech frame is determined as the target speech frame; if the first confidence level is less than a preset second confidence threshold, then the speech frame is discarded, wherein the second confidence threshold is less than the first confidence threshold; if the first confidence level is less than or equal to the first confidence threshold and greater than or equal to the second confidence threshold, then the speech frame is subjected to spectral reduction processing to obtain a spectral-reduced speech frame; speech activity detection is performed on the spectral-reduced speech frame to obtain a second confidence level corresponding to the spectral-reduced speech frame; and based on the second confidence level, whether the spectral-reduced speech frame is the target speech frame is determined.

[0116] Optionally, the process of processor 401 performing spectral reduction processing on the speech frame to obtain a spectral-reduced speech frame includes: performing voiceprint feature extraction processing on the speech frame to obtain target voiceprint features of the speech frame; comparing the target voiceprint features with base voiceprint features in the voiceprint database to determine the target spectral density, where each base voiceprint feature corresponds to a base spectral density; and performing spectral reduction processing on the speech frame based on the target spectral density to obtain a spectral-reduced speech frame.

[0117] Optionally, the step of determining whether the spectral-subtracted speech frame is a target speech frame based on the second confidence level, performed by the processor 401, includes: if the second confidence level is greater than the first confidence level threshold, then determining the speech frame as a target speech frame; if the second confidence level is less than the first confidence level threshold, then determining whether the speech frame is a speech frame to be treated; if the speech frame is a speech frame to be treated, then adding the voiceprint features of the speech frame to be treated and the spectral density of the speech frame to be treated to the voiceprint database.

[0118] Optionally, before comparing the target voiceprint features with the base voiceprint features in the voiceprint database to determine the target spectrogram, the method executed by the processor 401 further includes: acquiring noise audio files of different types of noise, and performing frame-segmentation processing on the noise audio files to obtain noise frames; extracting multi-dimensional features from the noise frames to obtain multi-dimensional features of the noise frames; performing clustering processing on the multi-dimensional features to obtain clustering results; establishing voiceprint feature templates and spectrogram templates for each noise category based on the clustering results; and constructing a voiceprint database based on the voiceprint feature templates and the spectrogram templates.

[0119] Optionally, the step of performing speech activity detection on the speech frame and obtaining a first confidence level corresponding to the speech frame, executed by the processor 401, includes: acquiring effective speech activity samples of the target population; adjusting a pre-trained speech activity detector based on the effective speech activity samples to obtain an adjusted speech activity detector; and performing speech activity detection on the speech frame using the adjusted speech activity detector to obtain a first confidence level corresponding to the speech frame.

[0120] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the deduplication method for road inspection defects provided in this invention and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0121] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A speech processing method, characterized in that, The method includes the following steps: performing frame segmentation on the speech sequence to be processed to obtain speech frames; performing speech activity detection on the speech frames to obtain a first confidence level corresponding to the speech frames, wherein the first confidence level is used to indicate the degree of credibility of the existence of valid speech activity in the speech frames; determining a target speech frame in the speech frames based on the first confidence level; and performing speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

2. The speech processing method as described in claim 1, characterized in that, The step of performing speech activity detection on the speech frame to obtain a first confidence level corresponding to the speech frame includes: performing pre-enhancement processing on the speech frame to obtain a pre-enhancement speech frame; and performing speech activity detection on the pre-enhancement speech frame to obtain a first confidence level corresponding to the speech frame.

3. The speech processing method as described in claim 2, characterized in that, The step of determining the target speech frame in the speech frame based on the first confidence level includes: if the first confidence level is greater than a preset first confidence threshold, then the speech frame is determined as the target speech frame; if the first confidence level is less than a preset second confidence threshold, then the speech frame is discarded, where the second confidence threshold is less than the first confidence threshold; if the first confidence level is less than or equal to the first confidence threshold and greater than or equal to the second confidence threshold, then the speech frame undergoes spectral reduction processing to obtain a spectral-reduced speech frame; speech activity detection is performed on the spectral-reduced speech frame to obtain a second confidence level corresponding to the spectral-reduced speech frame; and based on the second confidence level, whether the spectral-reduced speech frame is the target speech frame is determined.

4. The speech processing method as described in claim 3, characterized in that, The step of performing spectral reduction processing on the speech frame to obtain a spectral-reduced speech frame includes: performing voiceprint feature extraction processing on the speech frame to obtain the target voiceprint feature of the speech frame; comparing the target voiceprint feature with the base voiceprint feature in the voiceprint database to determine the target spectral density, where each base voiceprint feature corresponds to a base spectral density; and performing spectral reduction processing on the speech frame based on the target spectral density to obtain the spectral-reduced speech frame.

5. The speech processing method as described in claim 4, characterized in that, The step of determining whether the spectral-subtracted speech frame is a target speech frame based on the second confidence level includes: if the second confidence level is greater than the first confidence level threshold, then the speech frame is determined to be a target speech frame; if the second confidence level is less than the first confidence level threshold, then it is determined whether the speech frame is a speech frame to be addressed; if the speech frame is a speech frame to be addressed, then the voiceprint features of the speech frame to be addressed and the spectral density of the speech frame to be addressed are added to the voiceprint database.

6. The speech processing method as described in claim 4, characterized in that, Before comparing the target voiceprint features with the base voiceprint features in the voiceprint database to determine the target spectral density, the method further includes: acquiring noise audio files of different types of noise, and performing frame-by-frame processing on the noise audio files to obtain noise frames; extracting multi-dimensional features from the noise frames to obtain multi-dimensional features of the noise frames; performing clustering processing on the multi-dimensional features to obtain clustering results; establishing voiceprint feature templates and spectral density templates for each noise category based on the clustering results; and constructing a voiceprint database based on the voiceprint feature templates and the spectral density templates.

7. The speech processing method according to any one of claims 1 to 6, characterized in that, The step of detecting speech activity on the speech frame to obtain a first confidence level corresponding to the speech frame includes: acquiring effective speech activity samples of the target population; adjusting the pre-trained speech activity detector based on the effective speech activity samples to obtain an adjusted speech activity detector; and detecting speech activity on the speech frame using the adjusted speech activity detector to obtain a first confidence level corresponding to the speech frame.

8. A voice processing device, characterized in that, The speech processing device includes: a first processing module for segmenting the speech sequence to be processed into frames to obtain speech frames; a second processing module for detecting speech activity in the speech frames to obtain a first confidence level corresponding to the speech frames, wherein the first confidence level is used to indicate the degree of confidence that there is valid speech activity in the speech frames; a determination module for determining a target speech frame in the speech frames based on the first confidence level; and a third processing module for performing speech-to-text processing on the target speech frame to obtain the speech processing result of the speech sequence to be processed.

9. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of the speech processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the speech processing method as described in any one of claims 1 to 7.

Images