A speech endpoint detection method and module suitable for in-vehicle speech recognition system
Through the PR Net network model combined with driving condition data, voice and noise characteristics are extracted, which solves the problem of low speech recognition accuracy in high noise environments in traditional methods, and achieves more accurate speech endpoint detection and recognition.
Patent Information
- Application Number
- CN202110677109.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-18
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-06-18
AI Technical Summary
Traditional speech endpoint detection methods have low recognition accuracy in high noise environments, especially in vehicle-mounted environments, where artificial acoustic features are difficult to describe noisy speech, resulting in limited recognition capabilities of speech recognition systems.
The PR Net network model is used to preprocess the voice audio data and noisy audio data. The model is trained through feature extraction and classification tasks, combined with driving condition data, and the speech frame detection is performed using convolutional models and multi-classifiers, and the prediction results are output to mark Real-time audio data.
It improves the accuracy of voice endpoint detection in the on-board environment, has good noise immunity, and can detect different vehicle speeds and window opening conditions, solving the problems of difficulty in selecting voice features and slow recognition speed.
Smart Images

Figure CN113920985B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition pre-processing, and in particular to a speech endpoint detection method and module suitable for an in-vehicle speech recognition system. Background Art
[0002] With the rapid development of automobile intelligence, voice recognition and control systems have gradually become standard equipment in automobiles. Noises under different working conditions in driving scenarios, including road noise, tire noise, engine noise, wind noise, air conditioning noise, etc., have high energy and a high degree of overlap between their frequency ranges and the frequency range of speech. They have a strong masking effect on speech, which seriously affects the recognition ability of the speech recognition system.
[0003] Speech endpoint detection is a necessary step in the speech recognition process. Accurate detection and judgment of the starting point of the speech part in the signal can greatly improve the recognition rate of the speech recognition system. The extraction of artificial acoustic features in traditional detection methods is difficult to describe noisy speech in driving scenes, which affects the endpoint detection capability of speech signals in vehicle-mounted speech systems.
[0004] Most commonly used speech endpoint detection methods are based on the extraction of artificial acoustic features, including time domain features and frequency domain features, and finally determine the starting and ending points of speech by setting threshold values. A series of speech endpoint detection methods based on time domain features include short-time energy, zero-crossing rate, correlation function and other features. After fast Fourier transforming the signal, the speech endpoint can be identified based on the signal frequency domain features. For example, speech endpoint detection algorithms based on information entropy, frequency band variance, spectral distance, etc. These artificially defined acoustic feature parameters are small in number and it is difficult to fully describe the data characteristics. They only have good detection performance under high signal-to-noise ratio. Under low signal-to-noise ratio, artificial acoustic features cannot be well calculated, the detection accuracy fluctuates greatly, and the detection performance is greatly reduced. Especially in driving conditions, the lack of scene feature fusion makes it difficult to effectively improve the recognition accuracy of the vehicle-mounted speech recognition system under high noise. Summary of the invention
[0005] In order to overcome the shortcomings of the prior art, the present invention provides a speech endpoint detection method and module suitable for an in-vehicle speech recognition system. The technical solution is as follows:
[0006] In one aspect, the present invention provides a method for detecting speech endpoints in a vehicle speech recognition system, comprising the following steps:
[0007] S1, preprocessing the collected voice audio data, the noise audio data during driving, and the driving condition data corresponding to the noise audio data to obtain multiple groups of sample data to be trained;
[0008] S2, using the PR Net network model to perform feature extraction and classification tasks on several groups of sample data, and output prediction results respectively through full connection operations;
[0009] S3, respectively performing loss function calculation on the prediction result and the label result in the corresponding sample data, and continuously adjusting the parameter weights of the PR Net network model according to the calculation results;
[0010] S4, executing S2-S3 for multiple cycles to perform training, when the number of cycles of the PR Net network model training is equal to the number set in advance, or the calculation result reaches a preset accuracy range, the training is stopped to obtain the optimal PR Net network model;
[0011] S5, inputting the pre-processed real-time audio data and corresponding driving condition data during the actual driving process into the optimal PR Net network model, and the optimal PR Net network model outputs the detection result;
[0012] S6. Marking the speech-containing frames in the real-time audio data according to the detection result;
[0013] S7. Output the marked real-time audio data to the downstream speech recognition system.
[0014] Furthermore, in step S1, processing the collected data includes the following steps:
[0015] S101, extracting a segment of speech audio data, a segment of noise audio data, and driving condition data corresponding to the noise audio data;
[0016] S102, performing voice endpoint detection on the voice audio data after framing, so as to obtain voice audio data with a voice detection mark;
[0017] S103, fusing the speech audio data with the speech detection mark and the noise audio data to obtain noisy speech with the speech endpoint detection mark;
[0018] S104, dividing the noisy speech into a frame of n milliseconds to obtain time-domain noisy speech frames, and taking m frames of data as a training data unit;
[0019] S105, performing frequency domain conversion on the time domain noisy speech frame to obtain noisy speech spectrum data, so that the training data unit is converted into two-dimensional frequency domain data;
[0020] S106, calculating the spectral entropy of the two-dimensional frequency domain data;
[0021] S107, correspondingly binding the driving condition data in step S101, the label result data of the noisy speech in step S102, the two-dimensional frequency domain data in step S105, and the spectral entropy in step S106 into a set of sample data to be trained;
[0022] S108. Repeat steps S101-S107 multiple times to obtain multiple groups of sample data to be trained.
[0023] Furthermore, the PR Net network model extracts features from the two-dimensional frequency domain data, and then fully connects the extracted feature results with the corresponding spectral entropy and the driving condition data, and uses multiple classifiers to classify the fully connected results to obtain classification results, and fully connects the classification results to output prediction results.
[0024] Furthermore, in step S2, the data processing in the PR Net network model includes the following steps:
[0025] S201, performing a convolution operation on the two-dimensional frequency domain data;
[0026] S202, extracting features from the convolution result through a convolution model and performing deep connection;
[0027] S203, after performing a convolution operation on the result after the deep connection, a convolution image is obtained, the convolution image is sequentially subjected to convolution and pooling operations through the first branch to obtain a feature value, the feature value is fully connected with the corresponding spectral entropy and driving condition data, and then the result after the full connection operation is obtained through an activation function to obtain a binary classification result, the convolution image is input into the convolution model through the second branch to extract features, and a deep connection is performed;
[0028] S204, looping through step S203 once or multiple times, wherein the result after deep connection in step S203 in the next loop is the result of deep connection after the convolution image in step S203 in the previous loop is input into the convolution model through the second branch to extract features, so as to obtain a plurality of classification results;
[0029] S205: Perform a full connection operation on all classification results to output prediction results.
[0030] Furthermore, the convolution model includes a first convolution model and a second convolution model, the first convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×3, and the second convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×1 respectively;
[0031] The convolution model data processing comprises the following steps:
[0032] S2001, extracting features from the convolution result through a first convolution model to obtain a first result, and performing a deep connection on the first result;
[0033] S2002. After performing a convolution operation on the first result after deep connection, extract features through a second convolution model to obtain a second result, and perform deep connection on the second result.
[0034] Furthermore, the PR Net network model sets the number of model training times to x before training, monitors the loss of the validation set every y times during the training process, and uses the early stopping method to monitor the update of the PR Net model.
[0035] Furthermore, the prediction result, label result and detection result are all represented by an element in {0,1}, where 0 indicates that there is no speech frame and 1 indicates that there is a speech frame.
[0036] Furthermore, the driving condition data includes one or more of vehicle speed data, window status data and air conditioning status data.
[0037] On the other hand, the present invention further provides a speech endpoint detection module suitable for an in-vehicle speech recognition system, comprising a speech frame determination unit and a speech frame marking unit, wherein the speech frame determination unit is used to detect speech-containing frames in real-time audio data, and the speech frame marking unit is used to mark speech-containing frames in the real-time audio data according to the detection result of the speech frame determination unit;
[0038] The speech frame judgment unit adopts a PR Net network model, which includes a main classifier, an auxiliary classifier and a convolution model. The main classifier is used to distinguish the two-dimensional frequency domain data in the input data. The convolution model uses convolution kernels of different sizes to perform parallel convolution operations and is used to extract feature information of the two-dimensional frequency domain data. The auxiliary classifier is used to classify the feature information.
[0039] Furthermore, the convolution model includes a first convolution model and a second convolution model, the first convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×3 respectively, and the second convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×1 respectively.
[0040] The beneficial effects brought by the technical solution provided by the present invention are as follows:
[0041] (1) It has good noise immunity and can distinguish between noise data and noisy speech in vehicle environments;
[0042] (2) It can detect voice endpoints for different vehicle speeds and different window opening conditions;
[0043] (3) Increase the coupling of features in noisy speech under driving conditions, and effectively solve the problems of low accuracy of speech endpoint detection, slow recognition speed of some methods, and difficulty in selecting speech features under driving conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0045] Figure 1 It is a schematic diagram of sample data processing flow of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0046] Figure 2 It is a schematic diagram of the training process of a PR Net network model of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0047] Figure 3 It is a schematic diagram of the endpoint detection process of the PR Net network model of the speech endpoint detection method applicable to the in-vehicle speech recognition system provided by an embodiment of the present invention;
[0048] Figure 4 is a schematic diagram of an in-vehicle application of a voice endpoint detection method applicable to an in-vehicle voice recognition system provided by an embodiment of the present invention;
[0049] Figure 5 It is a schematic diagram of a speech audio signal in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0050] Figure 6 It is a schematic diagram of a noise audio signal in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0051] Figure 7 It is a schematic diagram of a speech audio signal with a speech detection mark in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0052] Figure 8 It is a schematic diagram of a noisy speech audio signal with a speech detection mark in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0053] Fig. 9It is a schematic diagram of an audio signal containing a speech frame in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0054] Fig.10 It is a schematic diagram of a two-dimensional spectrum containing a speech frame in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0055] Fig.11 It is a schematic diagram of the structure of a PRNet network model of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0056] Fig.12 It is a schematic diagram of the structure of a convolution kernel of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0057] Fig.13 It is a structural schematic diagram of a first convolutional model of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0058] Fig.14 It is a structural schematic diagram of a second convolutional model of a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0059] Fig.15 It is a schematic diagram of a real-time noisy speech audio signal in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention;
[0060] Fig.16 It is a schematic diagram of resonance peak characteristics of a frequency domain image in a speech endpoint detection method applicable to an in-vehicle speech recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0061] In order to enable those skilled in the art to better understand the scheme of the present invention, and to more clearly understand the purpose, technical scheme and advantages of the present invention, the technical scheme in the embodiment of the present invention is clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be noted that the implementation methods not shown or described in the accompanying drawings are forms known to those skilled in the art. In addition, although demonstrations of parameters containing specific values may be provided herein, it should be understood that the parameters do not need to be exactly equal to the corresponding values, but may be approximated to the corresponding values within an acceptable error tolerance or design constraint. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present invention. In addition, the terms "including" and "having" in the specification and claims of the present invention and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, device, product or equipment containing a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0062] In one embodiment of the present invention, a speech endpoint detection method applicable to an in-vehicle speech recognition system is provided, comprising the following steps:
[0063] S1. Preprocessing the collected speech audio data, the noise audio data during driving, and the driving condition data corresponding to the noise audio data to obtain multiple groups of sample data to be trained.
[0064] Wherein, in step S1, see Figure 1 , the collected data processing includes the following steps:
[0065] S101, extracting a segment of speech audio data, a segment of noise audio data, and driving condition data corresponding to the noise audio data, wherein the driving condition data includes one or more of vehicle speed data, vehicle window status data, and air conditioning status data;
[0066] S102, performing voice endpoint detection on the voice audio data after framing, so as to obtain voice audio data with a voice detection mark;
[0067] S103, fusing the speech audio data with the speech detection mark and the noise audio data to obtain noisy speech with the speech endpoint detection mark;
[0068] S104, dividing the noisy speech into a frame of n milliseconds to obtain time-domain noisy speech frames, and taking m frames of data as a training data unit;
[0069] S105, performing frequency domain conversion on the time domain noisy speech frame to obtain noisy speech spectrum data, so that the training data unit is converted into two-dimensional frequency domain data;
[0070] S106, calculating the spectral entropy of the two-dimensional frequency domain data;
[0071] S107, the driving condition data in step S101, the label result data of the noisy speech in step S102, the two-dimensional frequency domain data in step S105 and the spectral entropy in step S106 are correspondingly bound into a set of sample data to be trained, wherein the label result is represented by an element in {0,1}, 0 indicates no speech frame, and 1 indicates speech frame;
[0072] S108. Repeat steps S101-S107 multiple times to obtain multiple groups of sample data to be trained.
[0073] S2. Use the PR Net network model to perform feature extraction and classification tasks on several groups of sample data, and output prediction results through full connection operations.
[0074] Among them, in step S2, the PR Net network model extracts features from the two-dimensional frequency domain data, and then fully connects the extracted feature results with the corresponding spectral entropy and the driving condition data, and uses multiple classifiers to classify the fully connected results to obtain classification results, and the classification results are fully connected to output prediction results.
[0075] Specifically, see Figure 2 , the data processing in the PR Net network model includes the following steps:
[0076] S201, performing a convolution operation on the two-dimensional frequency domain data;
[0077] S202, extracting features from the convolution result through a convolution model and performing deep connection;
[0078] The convolution model includes a first convolution model (denoted as Pr-CNN-a) and a second convolution model (denoted as Pr-CNN-b), the first convolution model uses convolution kernels of 3×1, 7×1 and 15×3, and the second convolution model uses convolution kernels of 3×1, 7×1 and 15×1 respectively;
[0079] The convolution model data processing comprises the following steps:
[0080] S2001, extracting features from the convolution result through a first convolution model to obtain a first result, and performing a deep connection on the first result;
[0081] S2002. After performing a convolution operation on the first result after deep connection, extract features through a second convolution model to obtain a second result, and perform deep connection on the second result.
[0082] S203, after performing a convolution operation on the result after the deep connection, a convolution image is obtained, the convolution image is sequentially subjected to convolution and pooling operations through the first branch to obtain a feature value, the feature value is fully connected with the corresponding spectral entropy and driving condition data, and then the result after the full connection operation is obtained through an activation function to obtain a binary classification result, the convolution image is input into the convolution model through the second branch to extract features, and a deep connection is performed;
[0083] S204, looping through step S203 once or multiple times, wherein the result after deep connection in step S203 in the next loop is the result of extracting features from the convolution image in step S203 in the previous loop through the convolution model and performing deep connection, so as to obtain a number of classification results;
[0084] S205: Perform a full connection operation on all classification results to output prediction results.
[0085] The prediction result is represented by an element in {0,1}, where 0 indicates that there is no speech frame and 1 indicates that there is a speech frame.
[0086] S3, respectively performing loss function calculation on the prediction result and the label result in the corresponding sample data, and continuously adjusting the parameter weights of the PR Net network model according to the calculation results.
[0087] S4. Execute S2-S3 for multiple cycles to perform training. When the number of cycles of the PR Net network model training is equal to the number set in advance, or the calculation result reaches a preset accuracy range, stop training to obtain the optimal PR Net network model.
[0088] The PR Net network model sets the number of model training times to x before training, monitors the loss of the validation set every y times during training, and uses the early stopping method to monitor the update of the PR Net model.
[0089] It should be noted that in one cycle, one or more groups of sample data are taken as a batch and output using the same PR Net network model. Then, the loss calculations are performed separately and the parameter weights are adjusted continuously. The PR Net network model in the next cycle is the PR Net network model after the parameter weights in the previous cycle are updated.
[0090] S5. See Figure 3 and Figure 4 , the pre-processed real-time audio data and the corresponding driving condition data during the actual driving process are input into the optimal PR Net network model, and the optimal PR Net network model outputs the detection result.
[0091] S6. Mark the speech-containing frames in the real-time audio data according to the detection result.
[0092] S7. Output the marked real-time audio data to the downstream speech recognition system.
[0093] The PR Net network model makes full use of the advantages of traditional signal processing and deep learning methods. Traditional signal processing has a high accuracy rate for speech endpoint detection when the signal-to-noise ratio is high, but the accuracy rate is greatly reduced when the signal-to-noise ratio is low. The deep learning method can extract speech features under strong noise by learning a large amount of sample data, and realize speech endpoint detection under low signal-to-noise ratio. Therefore, in the preparation of training data, the traditional short-time energy and short-time average zero-crossing rate methods are used to mark the endpoints of pure speech. When training sample data, the PR Net network model is used to perform feature extraction and classification tasks on multiple parameters, and the multiple parameters include driving condition data and spectral entropy information. The driving condition data corresponds to the background noise under the driving condition. When performing a full connection operation, the generalization performance of the network for real-time data can be reflected. The irrelevance of noise and the relevance of speech data can be well reflected in the spectral entropy data of the frequency domain image. The noise data is more chaotic, and the speech data is more concentrated. The spectral entropy of each frame is calculated before inputting the model, and the spectral entropy sum is calculated within a given minimum voiced interval, and the calculated spectral entropy sum is used as an input parameter of the network.
[0094] In one embodiment, the PR Net network model structure uses three classifiers (including a main classifier, two auxiliary classifiers) and two convolution models (respectively denoted as Pr-CNN-a and Pr-CNN-b) to perform three cycles to predict the results. The classifiers are used mainly to prevent problems such as gradient disappearance as the network level deepens. During network training, the PRNet network model first extracts features from the spectral information of the speech, and then performs a full connection operation (denoted as FC) on the extracted feature results, spectral entropy, and driving condition data. The three classifiers classify the fully connected results respectively, and finally perform a full connection operation on the three classified results and output the final results.
[0095] The specific steps are as follows:
[0096] Step 1: Preliminary data preparation
[0097] Step 1.1 collects voice audio data, noise audio data during driving, and driving condition data;
[0098] Among them, the speech audio data refers to pure speech including human speech; the noise audio data during driving refers to the noise audio data collected corresponding to a certain vehicle speed and window and air conditioning status; the driving condition data includes vehicle speed data, window status data and air conditioning status data;
[0099] The vehicle speed data is the vehicle speed corresponding to the background noise, and the value is an integer, such as 80 represents 80km / h; the air-conditioning data represents the gear information of the air-conditioning windshield under the background noise, and the data contains [0,1,2,3,4], which respectively represent [closed, 1st gear on, 2nd gear on, 3rd gear on, 4th gear on]; the window data represents the window opening and closing status information under the background noise, and the data contains [0,0.5,1], which respectively represent the window [closed, half-open, fully open] status.
[0100] Step 1.2 performs upsampling or downsampling operations on the collected audio data, unifying the sampling rates of the speech audio data and the noise audio data, for example, using a sampling rate of 16 kHz;
[0101] Step 1.3 See Figure 5 and Figure 6 , sequentially extracting a segment of speech audio data, randomly extracting a segment of noise audio data, and the driving condition data corresponding to the noise;
[0102] Step 1.4 divides the speech audio data into frames and performs speech endpoint detection. Endpoint detection can be performed using short-time energy, short-time zero-crossing rate, double threshold method, etc. Figure 7 , to obtain speech audio data with speech detection mark;
[0103] Step 1.5: Fuse the speech audio data with speech detection markers and the noisy audio data, see Figure 8 , get the noisy speech with speech endpoint detection mark;
[0104] Step 1.6: Divide the noisy speech obtained in step 1.5 into 25ms frames. Fig. 9 , frame length H = 400 sampling points (under the condition of sampling rate of 16kHz), frame shift S is 1 / 4 of the frame length, and the minimum voiced sound interval is 15 frames, that is, every 15 frames is a training data unit;
[0105] Step 1.7: Convert the time-domain noisy speech frame obtained in step 1.6 into the frequency domain to obtain the noisy speech spectrum data. Fig.10 , from step 1.6, we know that after converting a training data unit of every 15 frames into the frequency domain, it becomes a two-dimensional spectrum graph of size 201×15;
[0106] Step 1.8 calculates the spectral entropy of the two-dimensional spectrum graph in step 1.7. The calculation process is as follows:
[0107] After the time domain speech signal x(t) is windowed, framed and transformed by FFT, the frequency component f of the kth spectral line is k The energy spectrum is Y i (k), then the normalized spectral probability density function of each frequency component is defined as:
[0108]
[0109] Where: P i (k) is the kth frequency component f of the i-th frame k The corresponding probability density, N is the FFT length;
[0110] The spectral entropy H(i) of the i-th frame is expressed as:
[0111]
[0112] Set the minimum voiced interval M frames, accumulate the spectral entropy of M frames, and obtain the spectral entropy within the minimum voiced interval t time, that is, the spectral entropy sum C(t):
[0113]
[0114] Step 1.9 binds the driving condition data of step 1.3, the label result data of step 1.4, the two-dimensional spectrum diagram of step 1.7 and the spectral entropy calculated in step 1.8 into a sample data to be trained.
[0115] Step 1.10 Repeat the above steps 1.3-1.9 to generate a large amount of labeled sample data to be trained.
[0116] Step 2: Design the PR Net network model
[0117] Step 2.1 See Fig.11 , perform data processing on each input data to be trained, and divide the data into two-dimensional frequency domain data and other data;
[0118] Step 2.2 See Fig.12 , perform convolution operation on the two-dimensional frequency domain data separated in step 2.1;
[0119] Step 2.3 performs parallel convolution operation on the result after convolution in step 2.2, see Fig.13 , extract features through the Pr-CNN-a(1) structure, and perform deep connection on the output results;
[0120] Step 2.4 applies a 3×3 convolution operation to the result of the convolution in step 2.3, so that the original 201×15 image becomes a 199×13 image, see Fig.14, and then after the Pr-CNN-b(1) operation, the output results are deeply connected;
[0121] Step 2.5: Apply a 5×5 convolution operation to the result after convolution in step 2.4, so that the original 199×13 image becomes a 195×9 image. After the convolution, two branches (i.e., branch 1 and branch 2) are established. Branch 1 obtains a feature value after convolution and pooling operations in sequence. This value is fully connected with other data, and then the fully connected result is passed through a sigmoid activation layer to obtain a first classification result of a binary classification, i.e., the probability of the segment containing speech or not containing speech. Branch 2 is then deeply connected to the output result after the Pr-CNN-a(2) operation.
[0122] Step 2.6: Apply a 3×3 convolution operation to the result of the convolution of branch 2 in step 2.5 to convert the original 195×9 image into a 193×7 image. After the Pr-CNN-b(2) operation, the output result is deeply connected.
[0123] Step 2.7: Apply a 5×5 convolution operation to the result after convolution in step 2.6 to change the original 193×7 image into a 189×3 image. After convolution, two branches (i.e., branch 3 and branch 4) are established. Branch 3 obtains a feature value after convolution and pooling operations in sequence. This value is fully connected with other data, and then the fully connected result is passed through a sigmoid activation layer to obtain a second classification result of binary classification. Branch 4 is then deeply connected to the output result after Pr-CNN-a(3) operation.
[0124] Step 2.8: A 3×3 convolution operation is performed on the result of the convolution of branch 4 in step 2.7 to convert the original 189×3 image into a 187×1 image. After the Pr-CNN-b(3) operation, the output result is deeply connected.
[0125] Step 2.9 applies average pooling and full connection operations to the result after convolution in step 2.8, and then passes through a sigmoid activation layer to obtain a binary third classification result. The third classification result is fully connected with the first classification result in step 2.5 and the second classification result in step 2.7, and the result is output.
[0126] Step 3: PR Net network model training phase
[0127] Step 3.1: Process each sample data to be trained, separate the label from the training data, and send the separated data without label to the network for training;
[0128] Step 3.2 calculates the Loss function by combining the prediction result obtained in step 3.1 with the label result of the actual data, and continuously adjusts the parameter weight to optimize the network;
[0129] Among them, the representation of the output label result is {0,1}, and the Loss expression is as follows:
[0130]
[0131] In the formula, y is the label result and y^ is the probability result.
[0132] Before training, the number of model training is set to n. During the training process, the loss of the validation set is monitored every m times, and the early stopping method is used to monitor the update of the model. When the number of cycles of deep neural network model training is equal to the number of n set in advance, or the loss stops decreasing for several cycles, the model training is stopped. At this time, the model has reached the optimal state. The optimization process is the process of back propagation, and the weight parameters are continuously adjusted by reversely calculating the partial derivative according to the calculated loss.
[0133] Step 3.3 saves the network model with adjusted parameters.
[0134] Step 4: PR Net network model application phase
[0135] Step 4.1 During driving, the vehicle-mounted sound pickup device obtains real-time audio data, and the relevant sensors obtain driving condition data;
[0136] Step 4.2 See Fig.15 The voice front-end processing system performs operations such as framing, frequency domain conversion, and spectral entropy calculation on the audio data obtained in step 4.1 in sequence to obtain the voice data to be detected;
[0137] Step 4.3: Input the multi-input speech data to be detected obtained in step 4.2 into the PR Net network model for detection. The model gives a detection result, where 0 represents no speech frame and 1 represents a speech frame.
[0138] Step 4.4 See Figure 8 , according to the detection result obtained in step 4.3, the noisy speech is marked to obtain the marked noisy speech.
[0139] In one embodiment of the present invention, a speech endpoint detection module suitable for an in-vehicle speech recognition system is provided, comprising a speech frame judgment unit and a speech frame marking unit, wherein the speech frame judgment unit is used to detect speech-containing frames in real-time audio data, and the speech frame marking unit is used to mark the speech-containing frames in the real-time audio data according to the detection result of the speech frame judgment unit.
[0140] Since the traditional convolutional neural network uses a square convolution kernel to extract image features, but from the observation of the speech spectrogram, the minimum voiced interval does not have the properties of a square image, and the speech spectrogram is obtained by Fourier transforming the continuous time-domain speech signal frame by frame, the information in the vertical direction can better reflect the speech characteristics, such as the resonance peak information, and has correlation, it is considered to modify the receptive field of the image into a rectangular shape, and establish a PR Net network model (full name Parallel rectangular convolution neural network). The spectrum image of the signal is classified by combining spectral entropy and the PR Net network model, and the speech frame and the noise frame are distinguished by extracting the spectrum feature information. The PR Net network model is a network model with dimensional differences, and a multi-parameter input speech endpoint detection method is used to fuse the artificial acoustic features with the features extracted by the network model, and the driving conditions (vehicle speed, window status, air conditioning status) are introduced into the model training process, which can improve the speech signal endpoint detection capability of the vehicle speech recognition system in the low signal-to-noise ratio situation of the driving scene, thereby improving the recognition rate of the vehicle speech recognition system.
[0141] Therefore, the speech frame judgment unit adopts the PR Net network model for data detection, and the PR Net network model includes a main classifier, an auxiliary classifier and a convolution model. The main classifier is used to distinguish the two-dimensional frequency domain data in the input data, and the convolution model uses convolution kernels of different sizes to perform parallel convolution operations and to extract feature information of the two-dimensional frequency domain data. The auxiliary classifier is used to classify the feature information.
[0142] Among them, the convolution model includes a first convolution model and a second convolution model. Both model structures use convolution kernels of different sizes to perform parallel convolution operations, with the purpose of extracting feature information of different scales on the spectrum image. The first convolution model uses convolution kernels of 3×1, 7×1 and 15×3 sizes, respectively, and the second convolution model uses convolution kernels of 3×1, 7×1 and 15×1 sizes, respectively. The convolution kernel of 15×3 can extract the overall features of the spectrum, such as resonance peak information, the convolution kernel of 3×1 can extract the details of the spectrum, such as high-frequency voiceless information, and the convolution kernel of 7×1 detects inter-frame information to prevent omission of features. The results after parallel convolution are deeply connected, that is, the extracted features of different scales are merged. The convolution kernel is a vertical rectangle. The purpose is that the vertical strip-shaped receptive field can more easily extract the frequency domain image information of speech, such as Fig.16As shown, the shape of the resonance peak in the frequency domain image is more like a vertical strip. Since the resonance peak is an important feature that reflects the resonance characteristics of the vocal tract, it represents the most direct source of pronunciation information, and people use the resonance peak information in speech perception. Therefore, the resonance peak is a very important feature parameter in speech signal processing, and has been widely used as the main feature of speech recognition and the basic information of speech coding transmission. The resonance peak information is contained in the frequency envelope, so the key to extracting the resonance peak parameters is to estimate the spectrum envelope of natural speech. It is generally believed that the maximum value in the spectrum envelope is the resonance peak. In the frequency domain image of speech, the resonance peak appears as a part with a more concentrated frequency distribution in the same time period, that is, it looks like a "vertical strip".
[0143] This embodiment has achieved good experimental results on the in-vehicle voice system. Figure 4 As shown, the trained model results are pre-loaded into the voice endpoint detection module of the vehicle voice system, the vehicle-mounted single-channel microphone picks up real-time audio data, the CAN bus obtains real-time driving condition data, and the obtained data is sent to the VAD module of the vehicle chip for voice endpoint detection, and the detection results are sent to the downstream voice recognition system.
[0144] The speech endpoint detection method and module for the in-vehicle speech recognition system provided by the present invention are combined with the driving condition characteristics of the actual scene, and an endpoint detection model of multi-feature fusion is established. A model framework with dimensional differences is adopted, and a convolution kernel is designed in combination with the spectral image characteristics of the speech, and a feature extraction kernel of the model is designed to better identify the speech features in the signal, thereby improving the accuracy of speech recognition. At present, the recognition effect of the vehicle-mounted speech recognition system is poor in the driving condition with low signal-to-noise ratio. One of the important reasons is that the ability to detect the speech endpoints in the signal is insufficient, and the interference of driving noise causes the algorithm to have misdetection and missed detection. This method has a certain noise resistance and has a good ability to distinguish between noise data and noisy speech in the vehicle environment. It can detect speech endpoints for scenes with different vehicle speeds and different window opening conditions. It still has a good effect at a low signal-to-noise ratio of -10dB. Increasing the coupling of features in noisy speech in driving conditions effectively solves the problems of low accuracy of speech endpoint detection in driving conditions, slow recognition speed of some methods, and difficulty in selecting speech features.
[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A speech endpoint detection method suitable for an in-vehicle speech recognition system, characterized in that: The following steps are involved: S1, preprocessing the collected voice audio data, the noise audio data during driving, and the driving condition data corresponding to the noise audio data to obtain multiple groups of sample data to be trained; S2, using the PR Net network model to perform feature extraction and classification tasks on several groups of sample data, and output prediction results respectively through full connection operations; S3, respectively performing loss function calculation on the prediction result and the label result in the corresponding sample data, and continuously adjusting the parameter weights of the PR Net network model according to the calculation results; S4, executing S2-S3 for multiple cycles to perform training, when the number of cycles of the PR Net network model training is equal to the number set in advance, or the calculation result reaches a preset accuracy range, the training is stopped to obtain the optimal PR Net network model; S5, inputting the pre-processed real-time audio data and corresponding driving condition data during the actual driving process into the optimal PR Net network model, and the optimal PR Net network model outputs the detection result; S6. Marking the speech-containing frames in the real-time audio data according to the detection result; S7. Output the marked real-time audio data to the downstream speech recognition system.
2. The speech endpoint detection method applicable to the in-vehicle speech recognition system according to claim 1, characterized in that: In step S1, the collected data is processed including the following steps: S101, extracting a segment of speech audio data, a segment of noise audio data, and driving condition data corresponding to the noise audio data; S102, performing voice endpoint detection on the voice audio data after framing, so as to obtain voice audio data with a voice detection mark; S103, fusing the speech audio data with the speech detection mark and the noise audio data to obtain noisy speech with the speech endpoint detection mark; S104, dividing the noisy speech into a frame of n milliseconds to obtain time-domain noisy speech frames, and taking m frames of data as a training data unit; S105, performing frequency domain conversion on the time domain noisy speech frame to obtain noisy speech spectrum data, so that the training data unit is converted into two-dimensional frequency domain data; S106, calculating the spectral entropy of the two-dimensional frequency domain data; S107, correspondingly binding the driving condition data in step S101, the label result data of the noisy speech in step S102, the two-dimensional frequency domain data in step S105, and the spectral entropy in step S106 into a set of sample data to be trained; S108. Repeat steps S101-S107 multiple times to obtain multiple groups of sample data to be trained.
3. The speech endpoint detection method applicable to the in-vehicle speech recognition system according to claim 2, characterized in that: The PRNet network model extracts features from the two-dimensional frequency domain data, and then fully connects the extracted feature results with the corresponding spectral entropy and the driving condition data, and uses multiple classifiers to classify the fully connected results to obtain classification results, and fully connects the classification results to output prediction results.
4. The speech endpoint detection method applicable to the in-vehicle speech recognition system according to claim 2, characterized in that: In step S2, the data processing in the PR Net network model includes the following steps: S201, performing a convolution operation on the two-dimensional frequency domain data; S202, extracting features from the convolution result through a convolution model and performing deep connection; S203, after performing a convolution operation on the result after the deep connection, a convolution image is obtained, the convolution image is sequentially subjected to convolution and pooling operations through the first branch to obtain a feature value, the feature value is fully connected with the corresponding spectral entropy and driving condition data, and then the result after the full connection operation is obtained through an activation function to obtain a binary classification result, the convolution image is input into the convolution model through the second branch to extract features, and a deep connection is performed; S204, looping through step S203 once or multiple times, wherein the result after deep connection in the next loop step S203 is the result of deep connection after the convolution image in the previous loop step S203 is input into the convolution model through the second branch to extract features, so as to obtain a plurality of classification results; S205: Perform a full connection operation on all classification results to output prediction results.
5. The speech endpoint detection method applicable to the in-vehicle speech recognition system according to claim 4, characterized in that: The convolution model includes a first convolution model and a second convolution model, the first convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×3, and the second convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×1 respectively; The convolution model data processing comprises the following steps: S2001, extracting features from the convolution result through a first convolution model to obtain a first result, and performing a deep connection on the first result; S2002. After performing a convolution operation on the first result after deep connection, extract features through a second convolution model to obtain a second result, and perform deep connection on the second result.
6. The speech endpoint detection method applicable to the in-vehicle speech recognition system according to claim 1, characterized in that: The PR Net network model sets the number of model training times to x before training, monitors the loss of the validation set every y times during training, and uses the early stopping method to monitor the update of the PR Net model.
7. The method for detecting speech endpoints in a speech recognition system in a vehicle according to claim 2, characterized in that: The prediction result, label result and detection result are all represented by an element in {0,1}, where 0 indicates that there is no speech frame and 1 indicates that there is a speech frame.
8. The method for detecting speech endpoints in a speech recognition system in a vehicle according to claim 1, characterized in that: The driving condition data includes one or more of vehicle speed data, window status data and air conditioning status data.
9. A speech endpoint detection module suitable for an in-vehicle speech recognition system, characterized in that: It includes a speech frame determination unit and a speech frame marking unit, wherein the speech frame determination unit is used to detect speech-containing frames in real-time audio data, and the speech frame marking unit is used to mark the speech-containing frames in the real-time audio data according to the detection result of the speech frame determination unit; The speech frame judgment unit adopts a PR Net network model, which includes a main classifier, an auxiliary classifier and a convolution model. The main classifier is used to distinguish the two-dimensional frequency domain data in the input data. The convolution model uses convolution kernels of different sizes to perform parallel convolution operations and is used to extract feature information of the two-dimensional frequency domain data. The auxiliary classifier is used to classify the feature information.
10. The speech endpoint detection module suitable for an in-vehicle speech recognition system according to claim 9, characterized in that: The convolution model includes a first convolution model and a second convolution model. The first convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×3 respectively, and the second convolution model uses convolution kernels of sizes 3×1, 7×1 and 15×1 respectively.
