Offline voice data processing method and device, equipment and storage medium

Abstract

The invention relates to an offline voice data processing method and device, equipment and a storage medium, and the method comprises the steps: carrying out the diversified collection of voice data through a microphone array; adaptive filtering noise reduction is carried out on the collected voice data, endpoint detection is carried out on the voice data after noise reduction processing so as to determine a voice starting point and a voice ending point in the voice data, and framing windowing is carried out on continuous data in the voice data after endpoint detection, finally, Mel-frequency cepstrum coefficient and high-order feature extraction and fusion are carried out on the data after framing and windowing, and preprocessed voice data are obtained. Training a voice recognition model based on the preprocessed voice data; carrying out model pruning and low-rank decomposition on the trained speech recognition model, carrying out parameter sharing, and finally carrying out retraining on the speech recognition model after parameter sharing; and processing off-line voice data input in real time based on the retrained voice recognition model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of voice data processing, and particularly relates to an offline voice data processing method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence and deep learning technology, speech recognition technology has been widely applied in the fields of smart home, mobile devices, vehicle-mounted systems, etc. However, these speech recognition models are usually large in scale, requiring a large amount of computing resources and storage space, which leads to many challenges when running on resource-constrained devices. The existing cloud computing method has the problems of high delay, high data transmission cost and privacy security, prompting offline voice processing to become an important research direction.

[0003] In view of the above technical problems in the prior art, there is currently no effective solution. SUMMARY

[0004] The present application provides an offline voice data processing method and device, equipment and a storage medium to solve the problem that the speech recognition model is large in scale in the prior art, requiring a large amount of computing resources and storage space.

[0005] In a first aspect, the application provides a method for processing offline speech data, comprising: collecting speech data in a diversified manner using a microphone array, wherein the diversified collection refers to covering different speakers, different language types, different speech speeds and different noise backgrounds; performing adaptive filtering and noise reduction on the collected speech data, and performing endpoint detection on the speech data after noise reduction to determine the speech start point and the speech end point in the speech data, and performing frame segmentation and windowing on the continuous data in the speech data after endpoint detection; finally, performing Mel-frequency cepstral coefficient and high-order feature extraction and fusion on the data after frame segmentation and windowing to obtain preprocessed speech data; training a speech recognition model based on the preprocessed speech data, wherein the speech recognition model combines a convolutional neural network and a long short-term memory network; the preprocessed speech data is input into the convolutional neural network, and the convolutional neural network captures the local correlation of the speech data and outputs a feature map dimension, adjusts the feature map dimension to a sequence format, and inputs it into the long short-term memory network, determines the attention score of the time step output by the long short-term memory network based on the memory mechanism, and performs weighted fusion based on the attention score; performing model pruning on the trained speech recognition model to reduce redundant neurons and weights, performing low-rank decomposition on the speech recognition model after model pruning to reconstruct the speech recognition model, sharing the parameters in the reconstructed speech recognition model, and finally retraining the speech recognition model after parameter sharing; processing real-time input offline speech data based on the retrained speech recognition model.

[0006] In a second aspect, the application provides a device for processing offline speech data, comprising: a collection module configured to collect speech data in a diversified manner using a microphone array, wherein the diversified collection refers to speech data covering different speakers, different language types, different speech speeds and different noise backgrounds; a first processing module configured to perform adaptive filtering noise reduction on the collected speech data, perform endpoint detection on the speech data after noise reduction to determine the speech start point and the speech end point in the speech data, perform frame segmentation and windowing on continuous data in the speech data after endpoint detection, and finally perform Mel frequency cepstrum coefficient and high-order feature extraction and fusion on the data after frame segmentation and windowing to obtain preprocessed speech data; a training module configured to train a speech recognition model based on the preprocessed speech data, wherein the speech recognition model combines a convolutional neural network and a long short-term memory network; the preprocessed speech data is input into the convolutional neural network, the convolutional neural network captures the local correlation of the speech data and outputs a feature map dimension, the feature map dimension is adjusted to a sequence format and input into the long short-term memory network, the long short-term memory network determines an attention score of a time step based on a memory mechanism, and the attention score is weighted and fused; a second processing module configured to reduce redundant neurons and weights by model pruning on the trained speech recognition model, perform low-rank decomposition on the speech recognition model after model pruning to reconstruct the speech recognition model, share parameters in the reconstructed speech recognition model, and finally retrain the speech recognition model after parameter sharing; and a third processing module configured to process real-time input offline speech data based on the retrained speech recognition model.

[0007] In a third aspect, the application provides a device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the method for processing offline speech data according to the first aspect of the application.

[0008] In a fourth aspect, the application further provides a computer storage medium storing computer executable instructions for executing the method for processing offline speech data according to the first aspect of the application.

[0009] Compared with the prior art, the technical solution provided by the embodiment of the present application has the following advantages: through the speech recognition model in the embodiment of the present application, various application scenarios can be adapted, offline processing is supported to reduce the dependence on network connection and cloud computing, user data privacy is effectively protected, the processing efficiency and response speed are significantly improved through reducing the computing resource requirement, that is, model compression, the device battery endurance time is prolonged, the resource-limited environment is adapted, and the dependence on cloud computing is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, those skilled in the art can obtain other drawings from these drawings without any creative effort.

[0012] One or more embodiments are exemplarily illustrated by pictures in the drawings corresponding to the embodiments, and these exemplary illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, unless otherwise specified. The drawings do not constitute a proportional limitation.

[0013] Figure 1 A flowchart of a processing method of offline speech data provided by the embodiment of the present application;

[0014] Figure 2 An optional flowchart of a processing method of offline speech data provided by the embodiment of the present application;

[0015] Figure 3 A flowchart of a processing method of efficient offline speech based on model compression provided by the embodiment of the present application;

[0016] Figure 4 A structural schematic diagram of a processing device of offline speech data provided by the embodiment of the present application;

[0017] Figure 5 A structural schematic diagram of a device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0019] The following disclosure provides many different embodiments, or examples, for implementing different structures of the present application. For the purpose of simplicity, the components and arrangements of the described examples are shown in the following description. It should be understood, however, that they are only examples and are not intended to limit the present application. Furthermore, the present application can be implemented in a variety of environments and applications, other than those described in the following examples. Also, the present application is described, for purposes of example, relating to specific structures and acts of the subject innovation. Those of ordinary skill in the art will recognize that the subject innovation can be practiced with many types of structural combinations and formed of

[0020] In order to solve the problem that the speech recognition model in the prior art is large in size, resulting in the need for a large amount of computing resources and storage space, the present application provides a processing method for offline speech data, as shown in Figure 1 The steps of the method include:

[0021] In step 101, the microphone array is used for diversified collection of speech data, wherein the diversified collection refers to speech data covering different speakers, different language types, different speech speeds and different noise backgrounds;

[0022] In the embodiments of the present application, the microphone array refers to the setting of corresponding microphone types and microphone numbers according to actual needs to collect speech data. In addition, the diversified speech data in the embodiments of the present application covers different speakers, different language types, different speech speeds and different noise backgrounds; wherein the different speakers include speakers of different ages, different genders, different working types and different races. Different speech types include various language families, such as Chinese, English, Portuguese, etc. Different speech speeds refer to different people, because the speech speed of different people is different in different environments or different working scenarios. Different noise backgrounds refer to the environment for collecting speech data, which can be various.

[0023] In step 102, the collected speech data is subjected to adaptive filtering and noise reduction, and the speech data after noise reduction processing is subjected to endpoint detection to determine the speech starting point and the speech ending point in the speech data, and the continuous data in the speech data after endpoint detection is subjected to frame segmentation and windowing, and finally the Mel frequency cepstrum coefficient and high-order feature extraction and fusion are performed on the data after frame segmentation and windowing to obtain preprocessed speech data;

[0024] By the above adaptive filtering and noise reduction of the collected voice data, the noise data in the voice data can be removed in a targeted manner, so that the voice data is cleaner. In addition, endpoint detection of the voice data can accurately identify the start and end points of the voice data, so that the subsequent trained voice recognition model can accurately identify the start and end points of the voice data, and the voice data recognition is more efficient. In addition, the continuous data in the voice data is framed and windowed to make the segmented data still have continuity, which improves the training efficiency of the subsequent voice recognition model. Finally, the mel frequency cepstrum coefficient and high-order feature extraction are performed on the framed and windowed data and fused, and the fused voice data greatly improves the training efficiency of the subsequent voice recognition model, which can make the voice recognition more efficient.

[0025] In step 103, the voice recognition model is trained based on the preprocessed voice data, wherein the voice recognition model fuses a convolutional neural network and a long short-term memory network; the preprocessed voice data is input into the convolutional neural network, and the convolutional neural network captures the local correlation of the voice data and outputs a feature map dimension, which is adjusted to a sequence format and input into the long short-term memory network, and the attention score of the time step output by the long short-term memory network is determined based on the memory mechanism, and the attention score is weighted and fused;

[0026] As can be seen, in the present application, the voice recognition model fuses a convolutional neural network and a long short-term memory network, and through the fusion of the two networks, the voice data recognition efficiency is higher and more accurate.

[0027] In step 104, the trained voice recognition model is pruned to remove redundant neurons and weights, and the pruned voice recognition model is decomposed into low rank to reconstruct the voice recognition model, and the parameters in the reconstructed voice recognition model are shared, and finally the voice recognition model with shared parameters is retrained.

[0028] As can be seen, in the embodiment of the present application, the voice recognition model can be compressed by model pruning, low rank decomposition and parameter sharing, which optimizes the model structure and reduces the computational complexity. The compressed model has wide hardware adaptability and can run efficiently on various embedded devices, mobile devices and Internet of Things terminals, supports offline processing, reduces dependence on network connection and cloud computing, and effectively protects user data privacy. Although the model is compressed, the computational resource demand is reduced and the processing efficiency and response speed are improved.

[0029] In step 105, the offline voice data input in real time is processed based on the retrained voice recognition model.

[0030] As can be seen, through steps 101 to 105 of this application embodiment, diverse acquisition of speech data is carried out using a microphone array, enabling the subsequent speech recognition model to recognize speech data in various application scenarios. Then, adaptive filtering and noise reduction are performed on the acquired speech data, and endpoint detection is performed on the denoised speech data to determine the start and end points of speech. Continuous data in the endpoint-detected speech data is then framed and windowed. Finally, Mel-frequency cepstral coefficients and higher-order features are extracted and fused from the framed and windowed data to obtain preprocessed speech data. This preprocessing of the speech data improves the efficiency of subsequent speech model training and enhances the recognition efficiency of the trained speech recognition model. Subsequently, the speech recognition model is compressed through model pruning, low-rank decomposition, and parameter sharing, optimizing the model structure and reducing computational complexity. Finally, the retrained speech recognition model is used to process real-time input offline speech data. As can be seen, the speech recognition model in this application embodiment can not only adapt to various application scenarios, but also support offline processing to reduce dependence on network connection and cloud computing, effectively protect user data privacy, and significantly improve processing efficiency and response speed by reducing computing resource requirements, i.e. model compression, extending device battery life, adapting to resource-constrained environments, and reducing dependence on cloud computing.

[0031] In optional embodiments of this application, the method of diverse acquisition of voice data using a microphone array involved in step 101 above may further include:

[0032] Step 11: Use a combination array of omnidirectional microphones and cardioid microphones, or a combination array of omnidirectional microphones and supercardioid microphones, to collect voice data in a variety of ways, wherein the number of microphones in the combination array is multiple.

[0033] For step 11, the explanation is as follows, with a specific example:

[0034] Microphone type selection: Use a combination of an omnidirectional microphone (360° pickup, suitable for ambient noise acquisition) and a cardioid / supercardioid microphone (directionally enhances the target sound source). Sensitivity specifications: ≥-38dBV / Pa, signal-to-noise ratio ≥65dB, ensuring the capture of weak speech signals.

[0035] Example configuration:

[0036] Conference Room Scenario: 8 silicon microphones form a 4×2 planar array, spaced 5cm apart, covering a frequency range of 20Hz-20kHz. Spherical array (e.g., 32 microphones): 3D spatial sound field reconstruction for complex noisy environments. Acquisition System Architecture Synchronization Mechanism: Cascaded TLV320ADC3101 chips support 8-channel synchronous ADC sampling (sampling rate 48kHz, 24-bit quantization). Signal Transmission: Data is transmitted to the STM32 main controller via TDM (Time Division Multiplexing) protocol to avoid clock skew caused by multiple CODECs.

[0037] Speaker diversity coverage strategy: Record speakers of different ages (children / adults / elderly), genders, and pronunciation habits. Example: Collect data from 100 speakers (50 men / 50 women, aged 5-70), each reading a standard text (such as a press release) and engaging in free conversation.

[0038] Dialect and Language Variation Strategies: Covering Mandarin, Cantonese, Wu dialects, and mixed Chinese-English speech. Example: Collecting 10 hours of Cantonese dialogue in a dialect region (e.g., Guangdong), incorporating mixed Chinese-English phrases (e.g., "Tomorrow's meeting is canceled"). Frequency-domain blind source separation (BSS) of the microphone array is used to separate the mixed dialect signals. Speech Rate Variation Simulation Strategy: Designing slow (≤2 words / second), medium (3-4 words / second), and fast (≥5 words / second) speech samples. Example: Asking speakers to read the same text (e.g., "AA City, BB District, CC Street") at different speeds, and incorporating impromptu rapid question and answer sessions.

[0039] Background noise introduction: Noise types and examples: Noise scenarios, noise sources, acquisition examples, steady-state noise, air conditioner noise, computer fan noise, continuous recording for 30 minutes in an office environment; non-steady-state noise, keyboard typing, door opening and closing, footsteps, sudden interference in a simulated meeting, high-noise environment, traffic noise, crowd noise, outdoor speech noise reduction technology around subway stations: Beamforming: Delayed summation beamformer enhances the signal-to-noise ratio by 15dB in the target direction. Degree learning method: Train a DNN model to separate noise in real time.

[0040] Specific Implementation Example: Hardware Configuration of the Conference Room Voice Acquisition System: 6-microphone circular array (cardioid), STM32F4 main controller, TLV320ADC3101×4 cascaded to achieve 8-channel synchronous acquisition. Acquisition Process: Environmental Settings: Background noise in the conference room (air conditioning + projector) is controlled at 35dB. Speaker Grouping: 3 groups of people (including dialect speakers) take turns speaking, with the speaking speed increasing from slow to fast. Noise Injection: Sudden noises such as knocking on the table and coughing are artificially added. Data Output: Raw Data: 8-channel PCM audio (48kHz / 24bit). Labeled Data: Sound source direction angle, speaker ID, noise type label.

[0041] In an optional embodiment of this application, the method of adaptively filtering and denoising the acquired speech data and performing endpoint detection on the denoised speech data to determine the start and end points of the speech data in step 102 may further include:

[0042] Step 21: Enhance the high-frequency components of the speech data through a high-pass filter, convert the enhanced speech data into a corresponding input vector, and use the square of the input vector as the power of the input signal.

[0043] Step 22: Determine the power of the input signal as a normalization factor to update the filter weights, and then use the updated filter to perform noise reduction on the speech data.

[0044] Step 23: If the energy of a consecutive first preset number of frames exceeds a first preset threshold but is lower than a second preset threshold, determine that the consecutive first preset number of frames is the starting point of speech.

[0045] Step 24: If the number of consecutive first preset number frames continues to decrease and falls below the third preset threshold, determine the first preset number frame as the voice endpoint.

[0046] As can be seen, in this embodiment, a high-pass filter can be used to enhance high-frequency components to compensate for the effects of the vocal cords and lips. Then, the power of the input signal (i.e., the square of the input vector) is used as a normalization factor to suppress step size fluctuations caused by large signals, thereby achieving noise reduction of the speech data and obtaining cleaner speech data. Endpoint detection is then used to identify the start position of the speech signal, i.e., the moment when the speech signal stands out from background noise or other non-speech signals. Endpoint detection is also used to identify the end position of the speech signal, i.e., the moment when the speech signal gradually weakens and returns to background noise or other non-speech signals. This allows the subsequently trained speech recognition model to accurately identify the start and end points in the speech data, making speech data recognition more efficient.

[0047] In an optional embodiment of this application, the method of performing frame-by-frame windowing on the continuous data in the speech data after endpoint detection, and finally extracting and fusing Mel-frequency cepstral coefficients and higher-order features from the frame-by-frame windowed data, as involved in step 102 above, may further include:

[0048] Step 31: Segment the continuous data in the speech data after endpoint detection, and perform edge smoothing on the segmented data using a window function;

[0049] Step 32: Extract the first second preset number of Mel frequency cepstral coefficients, as well as the corresponding first-order and second-order differences, from the framed and windowed speech data;

[0050] Step 33: Feature fusion is performed on the fundamental frequency and formants of the windowed speech data after frame segmentation, along with a second preset number of Mel frequency cepstral coefficients and the corresponding first-order and second-order differences.

[0051] In this specific example, frame-based windowing can be achieved by dividing the speech signal (sampling rate 8kHz) into frames (e.g., frame length 25ms, frame shift 10ms), and adding a Hamming window to each frame to reduce spectral leakage. A 512-point Fast Fourier Transform (FFT) is performed on each frame, and the power spectrum is obtained by taking the square of the modulus. Then, 40 triangular filters (with center frequencies nonlinearly distributed according to the Mel scale) are applied to simulate human auditory characteristics, making the speech data more realistic and thus improving the recognition efficiency of subsequent speech recognition models. Finally, Mel-frequency cepstral coefficients and higher-order features are extracted and fused from the frame-based windowed data. In a specific example, the first 13 Mel-frequency cepstral coefficients (MFCC) can be taken, and first-order differences (Δ) and second-order differences (ΔΔ) can be added. Higher-order feature fusion: Pitch period: The fundamental frequency F0 is extracted using the autocorrelation method. Formants: F1-F3 are obtained by rooting the linear predictive coding (LPC) coefficients. Feature concatenation: Final feature vector = [MFCC(1-13),ΔMFCC,ΔΔMFCC,F0,F1,F2,F3]. It is evident that the fused speech data significantly improves the training efficiency of subsequent speech recognition models, resulting in higher speech recognition efficiency.

[0052] In an optional embodiment of this application, the method of training the speech recognition model based on the preprocessed speech data involved in step 103 above may further include:

[0053] Step 41: Reconstruct the speech data into a matrix multiplied by the time step and the feature dimension, and input the matrix into the convolution kernel of the convolutional neural network for sliding scanning to capture the local correlation of the speech data, and obtain the output feature map dimension multiplied by the time step and the number of channels.

[0054] Step 42: After adjusting the feature map dimensions to sequence format, input it into the Long Short-Term Memory network and obtain the output time step through the gating mechanism of the Long Short-Term Memory network;

[0055] Step 43: Perform weighted fusion of time steps based on the attention mechanism, and adjust the model parameters in the speech recognition model based on the weighted fusion result.

[0056] In steps 41 to 43 above, the input processing of the convolutional neural network (CNN) involves reconstructing time-series data (such as speech frames and sensor readings) into a matrix of [time step × feature dimension], which is then scanned by a 1D convolutional kernel (e.g., kernel_size = 3). This CNN is used to capture local correlations (such as phoneme waveforms in speech and short-term fluctuations in stock prices), and outputs a feature map with dimensions of [time step × number of channels].

[0057] Input transformation for Long Short-Term Memory (LSTM) networks: The output of a CNN needs to be reshaped into a sequence format (reshape to [batch_size, timesteps, channels]). The LSTM gating mechanism includes: Forget gate: determines the proportion of historical memory retained; Input gate: controls the writing of new information; Output processing: setting return_sequences = True retains the output of all time steps (providing a complete sequence for the Attention layer).

[0058] Dynamic feature weighting: Attention mechanism is used to calculate the attention score of LSTM output at each time step. Weighted fusion: Output vector context = Σ(α_t*h_t) is used to focus on key time points (such as stressed syllables in speech, stock price change points).

[0059] As can be seen, in this embodiment of the application, a large-scale dataset is used for model training, and a distributed training framework is used for model optimization and hyperparameter tuning, which makes the trained speech recognition model more efficient and accurate.

[0060] In optional embodiments of this application, the steps 104 described above, including pruning the trained speech recognition model to remove redundant neurons and weights, performing low-rank decomposition on the pruned speech recognition model to reconstruct it, and sharing parameters in the reconstructed speech recognition model, may further include:

[0061] Step 51: Calculate the sum of the absolute values ​​of the weights of each filter in the speech recognition model layer by layer, and remove filters whose sum of absolute values ​​is lower than the fourth preset threshold in each layer to reduce redundant neurons and weights.

[0062] Step 52: Divide the filter weight matrix in the speech recognition model after subtraction into two sub-matrices to reconstruct the speech recognition model;

[0063] Step 53: Apply the same weight matrix to the input gate and forget gate in the Long Short-Term Memory network of the speech recognition model.

[0064] In this specific example, a baseline model (such as ResNet-18) is trained for speech emotion recognition to identify more important speech data. Then, the L1 norm of the convolutional layer weights is calculated, and channels below a threshold (e.g., 10%) are removed. This method of model pruning reduces the number of model parameters by 60% while increasing inference speed by 2 times (tested on embedded devices).

[0065] Low-Rank Decomposition Example (Fully Connected Layer Decomposition): Principle: Decompose the weight matrix into two smaller matrices (e.g., W = U * V^T). Low-rank decomposition reduces the storage requirements of speech recognition models by 70%, making it suitable for mobile deployment.

[0066] Parameter Sharing: In LSTM layers, the same weight matrix is ​​used for the input gate and the forget gate, which can reduce the number of parameters and improve training speed.

[0067] Retraining & Validation: Quantization-Aware Training (QAT): Simulates 8-bit integer quantization during training to reduce deployment accuracy loss. Then, Adversarial Validation: Injects Gaussian noise and temporal offset to test model robustness.

[0068] In optional implementations of this application's embodiments, the speech recognition model can also be deployed locally, and its performance can be evaluated and optimized. Based on this, such as... Figure 2 As shown, the method in this application embodiment may further include:

[0069] Step 201: Integrate the speech recognition model locally and perform performance evaluation and model optimization on the speech recognition model.

[0070] To achieve an integrated system for real-time speech signal processing and model optimization on local devices, it is necessary to combine multi-algorithm collaborative pipelines, lightweight model design, and dynamic fine-tuning techniques.

[0071] The following is the specific implementation process: First, the system architecture is designed, and a hardware-software collaborative framework is established. Hardware platform selection: Mobile end: The processing chip must support parallel processing of noise reduction, VAD, and feature extraction. Embedded end: NVIDIA Jetson NX (GPU acceleration) or STM32H7 (DSP instruction set) to meet real-time computing requirements. Software stack architecture: graph TD A [microphone input] --> B [audio preprocessing pipeline] --> C [feature extraction engine] --> D [lightweight model inference] --> E [model fine-tuning module] --> noise reduction output C --> feature vector D --> update weights D. Multi-threaded scheduling: Audio acquisition (1ms frame), noise reduction (2ms), VAD (0.5ms), and feature extraction (3ms) run in separate threads, transferring data through a circular buffer. Real-time processing pipeline optimization: Zero-copy data transfer: DMA directly accesses the microphone hardware buffer, avoiding CPU copying overhead. Computational graph fusion: Noise reduction filtering, MFCC calculation, and differential features are merged into a single computational graph, reducing intermediate result storage.

[0072] Secondly, real-time pipelined processing, combined with adaptive denoising (parallel processing) algorithm selection for endpoint detection: GRU-based RNN denoising (model size 82KB), processing latency <5ms, supporting non-stationary noise suppression. Fixed-point acceleration: Q15 format for weight storage, DSP instruction set for accelerated multiply-accumulate operations (e.g., ARM CMSIS-DSP library). Adaptive threshold update: E_{\text{max}} and \mu_{\text{zcr}} are updated every 60 seconds based on background noise. Integrated feature extraction and model inference: MFCC++ feature engine computation optimization: FFT uses Radier algorithm (reducing complex multiplication by 50%). Mel filter bank is pre-compiled into sparse matrix, replacing loops with matrix multiplication. High-order feature fusion: output includes 13-dimensional MFCC + ΔMFCC + ΔΔMFCC + fundamental frequency F0 + formants F1-F3. Double buffering mechanism: Thread 1: Feature extraction → write to buffer A; Thread 2: Read from buffer B → model inference → result output. Buffers are switched every frame to avoid lock contention.

[0073] Performance evaluation and model optimization: The following is a complete explanation of the performance evaluation and optimization process for a local offline voice system, with step-by-step analysis and specific examples:

[0074] 1) Performance Evaluation System Construction: Core indicators for recognition accuracy evaluation: Word Error Rate (WER): Calculates the percentage of insertion, deletion, and substitution errors (Formula: WER = (S + D + I) / N, where S is the number of substitutions, D is the number of deletions, I is the number of insertions, and N is the total number of reference words). Sentence Error Rate (SER): The percentage of complete sentence-level errors, applicable to command control scenarios. Specific scenario indicators: such as the false alarm rate and missed detection rate for wake word detection. Example: Testing of an in-vehicle voice control system in a driving noise (60dB) environment: WER = 5% in a quiet environment, and WER rises to 18% in a noisy environment (the main error is confusion of numerical commands, such as "five" and "two"). Through confusion matrix analysis, it was found that the percentage of numerical command errors reached 70%, requiring targeted optimization of the acoustic model. Key indicators: End-to-end latency: The total time from voice input to result output (target ≤ 200ms). Staged time consumption: noise reduction (≤ 20ms), feature extraction (≤ 30ms), and model inference. Key parameters: Memory usage: Resident memory after model loading (target <100MB). CPU / GPU utilization: Peak load during inference (target <70%). Power consumption: Energy consumption during continuous operation (target <3W for embedded devices).

[0075] 2) Optimization Strategies and Implementation: Model Compression Techniques: Quantization: Convert FP32 weights to INT8, reducing memory usage by 75%. Pruning: Remove neurons with weights below a threshold (e.g., pruning rate of 50%). Knowledge Distillation: Use a large model (teacher) to guide the training of a small model (student). Example: In a medical consultation system, the student model size is only 1 / 5 of the teacher model, with a WER difference of <2%. Operator Fusion: Merge consecutive layers (e.g., Conv+ReLU) to reduce computational overhead. Example: After optimization of the ONNX Runtime, the computational cost in the feature extraction stage is reduced by 30%. Hardware Acceleration: Utilize NPU / DSP to deploy the model.

[0076] 3) User Feedback-Driven Iterative Optimization Feedback Collection Mechanism Log Analysis: Records user-corrected recognition errors (e.g., misrecognizing "turn on the air conditioner" as "turn on the light"). Example: Smart home system statistics show that 25% of errors are concentrated on device control verbs. Proactive Testing: Release a test version and invite users to label problematic sentences. Scenario: Dialect command recognition optimization problem: The error rate for Cantonese user commands is as high as 32%. Optimization Steps: Add 10 hours of Cantonese command data and enhance with mixed noise. Adaptive Training: Add a dialect adaptation layer after the LSTM layer and fine-tune for 2000 steps. Result: The error rate is reduced to 12%, while the Mandarin error rate remains at 5%.

[0077] 4) Performance verification and iterative closed-loop evaluation - optimization closed-loop design: A [Scenario-based testing] --> B [Accuracy / Speed / Resource data] --> C {Does it meet the standard?} --> No --> D [Identify bottlenecks: model / preprocessing / hardware] --> Yes --> E [Deploy the official version] --> F [Select optimization strategy: quantization / pruning / distillation] --> G [Retraining and verification] --> A.

[0078] The present application will be further explained below with reference to specific embodiments. These specific embodiments provide an efficient offline speech processing method based on model compression. The processing flow of this method is as follows: Figure 3 As shown, the specific steps include:

[0079] Step 301, Data Collection: Collect diverse speech data using a high-sensitivity microphone array, covering different speakers, dialects, speech rates, and background noise.

[0080] Step 302, Data Preprocessing: Run a data preprocessing algorithm that combines Mel frequency cepstral coefficients, adaptive filtering and noise reduction, endpoint detection, and high-order feature extraction on a computing device.

[0081] Step 303: On a high-performance computing cluster or GPU server, design a complex speech recognition model using a deep learning framework. This speech recognition model integrates a convolutional neural network (CNN), a long short-term memory network (LSTM), and an attention mechanism. Then, train the model using a large-scale dataset, and optimize and tune the model's hyperparameters using a distributed training framework.

[0082] Step 304, model compression;

[0083] To ensure a fast system response, the following steps are taken with the trained model:

[0084] Model pruning removes redundant neurons and weights and maintains model performance through retraining.

[0085] Low-rank decomposition involves performing low-rank decomposition on the important weight matrices of the model, and then reconstructing the model using the smaller matrices resulting from the decomposition, thereby achieving further compression.

[0086] Parameter sharing introduces a mechanism to reduce the total number of parameters and improve computational efficiency.

[0087] Retraining and validation: The compressed model is retrained on a high-performance computing device to repair the performance loss during compression and maintain high accuracy.

[0088] To address this, methods such as pruning, quantization, knowledge distillation, low-rank decomposition, and parameter sharing significantly reduce the number of parameters and storage requirements of the speech recognition model, lowering computational complexity and improving device energy efficiency and real-time processing performance. The compressed model exhibits broad hardware adaptability, enabling efficient operation on various embedded devices, mobile devices, and IoT terminals. It supports offline processing, reducing reliance on network connectivity and cloud computing, and effectively protecting user data privacy. Simultaneously, the compression technology ensures high accuracy while reducing resource consumption, simplifying model deployment and maintenance processes, lowering system operation and maintenance costs, and making it suitable for various application scenarios such as smart homes and in-vehicle systems, demonstrating significant application value and broad market prospects. Furthermore, model compression technology significantly improves processing efficiency and response speed, extends device battery life, adapts to resource-constrained environments, and reduces reliance on cloud computing. At the same time, compressed models reduce hardware costs and heat generation, enhance user experience and device stability, and are widely applicable to embedded devices, mobile devices, and IoT terminals, demonstrating significant application value and market potential.

[0089] Step 305: Local integration of the speech recognition model;

[0090] Integrating the speech recognition model into the local target device's speech system creates the conditions for further high-precision offline speech recognition. Simultaneously, noise reduction, endpoint detection, and feature extraction algorithms are run in real-time on the local device to process the input speech signal, fine-tune the compression model, and improve offline accuracy.

[0091] Step 306: After building the offline speech system with the local high-performance offline model, comprehensively evaluate the model's recognition accuracy, processing speed, and resource consumption in the target use scenario. Simultaneously, collect user feedback to further optimize the model and system performance, iteratively improving its effectiveness in practical applications.

[0092] It should be noted that the hardware configuration for offline voice processing in this embodiment includes:

[0093] Computing devices: embedded devices (such as NVIDIA Jetson Nano), mobile devices (such as smartphones), dedicated hardware accelerators (such as TPU, NPU), and Internet of Things (IoT) devices (such as smart home controllers).

[0094] Storage devices: high-speed solid-state drives (SSDs), random access memory (RAM).

[0095] Sensor: High-sensitivity microphone array.

[0096] Power Management System: High-efficiency power management module.

[0097] The software architecture for offline voice processing in this application embodiment includes:

[0098] Operating systems: Embedded Linux, Android / iOS, Real-time Operating System (RTOS). Development frameworks and tools: TensorFlow, PyTorch, TensorFlow Lite, ONNX, TensorFlow ModelOptimization Toolkit.

[0099] Application layer software: real-time speech processing applications, model management and update systems, performance monitoring and feedback systems.

[0100] As can be seen, this specific implementation adopts a systematic combination of multiple model compression techniques, namely, the systematic integration of pruning, quantization, and parameter sharing. By comprehensively applying these techniques, the number of model parameters and storage requirements are significantly reduced, effectively optimizing the model's size and computational complexity, while maintaining high accuracy in speech recognition to the greatest extent possible, achieving more efficient offline speech processing. Furthermore, this implementation utilizes knowledge distillation and low-rank decomposition to optimize the model structure, reducing computational complexity, enabling the compressed model to achieve efficient real-time speech processing on resource-constrained devices. By fully utilizing hardware accelerators, inference speed and processing power are further improved, ensuring fast and reliable speech recognition services on various hardware platforms such as embedded devices, mobile devices, and IoT terminals, significantly enhancing the user experience. Furthermore, this implementation proposes an efficient offline speech processing method, enabling the model to run independently on local devices, reducing reliance on network connections and cloud computing. This not only improves system stability and reliability but also effectively protects user data privacy, adapting to application needs in network-limited or no-network environments. By reducing energy consumption and extending device battery life, the feasibility and wide applicability of the model in practical applications are further enhanced.

[0101] Corresponding to the above Figure 1 In addition to the methods described above, this application also provides an offline voice data processing apparatus, such as... Figure 4 As shown, the device includes:

[0102] Acquisition module 402 is used to acquire diverse speech data using a microphone array. Diverse acquisition means that the speech data covers different speakers, different language types, different speech rates and different noise backgrounds.

[0103] The first processing module 404 is used to perform adaptive filtering and noise reduction on the acquired speech data, and to perform endpoint detection on the noise-reduced speech data to determine the speech start and end points in the speech data, and to perform frame-by-frame windowing on the continuous data in the speech data after endpoint detection, and finally to extract and fuse Mel frequency cepstral coefficients and higher-order features from the frame-by-frame windowed data to obtain preprocessed speech data.

[0104] Training module 406 is used to train a speech recognition model based on preprocessed speech data. The speech recognition model integrates a convolutional neural network and a long short-term memory network. The preprocessed speech data is input into the convolutional neural network, which captures the local correlations of the speech data and outputs a feature map dimension. The feature map dimension is then adjusted to a sequence format and input into the long short-term memory network. The attention score of the time step output by the long short-term memory network is determined based on the memory mechanism, and weighted fusion is performed based on the attention score.

[0105] The second processing module 408 is used to reduce redundant neurons and weights in the trained speech recognition model through model pruning, and to reconstruct the speech recognition model by performing low-rank decomposition on the pruned speech recognition model, as well as to share the parameters in the reconstructed speech recognition model, and finally to retrain the speech recognition model after parameter sharing.

[0106] The third processing module 410 is used to process the real-time input offline speech data based on the retrained speech recognition model.

[0107] In an optional embodiment of this application, the first processing module may further include: a first processing unit, configured to enhance the high-frequency components of the speech data through a high-pass filter, convert the enhanced speech data into a corresponding input vector, and use the square of the input vector as the power of the input signal; a second processing unit, configured to determine the power of the input signal as a normalization factor to update the filter weights, and perform noise reduction processing on the speech data through the updated filter; a third processing unit, configured to determine the first preset number of consecutive frames as the speech start point when the energy of the first preset number of consecutive frames exceeds a first preset threshold and is lower than a second preset threshold; and a fourth processing unit, configured to determine the first preset number of frames as the speech end point when the energy of the first preset number of consecutive frames continuously decreases and is lower than a third preset threshold.

[0108] In an optional embodiment of this application, the first processing module in this application embodiment may further include: a fifth processing unit, used to segment the continuous data in the speech data after endpoint detection, and to perform edge smoothing processing on the segmented data through a window function; a sixth processing unit, used to extract the first second preset number of Mel frequency cepstral coefficients, and the corresponding first-order difference and second-order difference from the speech data after frame-segmentation and windowing; and a seventh processing unit, used to perform feature fusion on the second preset number of Mel frequency cepstral coefficients, and the corresponding first-order difference and second-order difference, based on the fundamental frequency and formants in the speech data after frame-segmentation and windowing.

[0109] In an optional embodiment of this application, the training module may further include: an eighth processing unit, configured to reconstruct the speech data into a matrix multiplied by the time step and the feature dimension, and input the matrix into the convolutional kernel of the convolutional neural network for sliding scanning to capture the local correlation of the speech data, thereby obtaining the output feature map dimension multiplied by the number of channels; a ninth processing unit, configured to adjust the feature map dimension into a sequence format and input it into a long short-term memory network, thereby obtaining the output time step through the gating mechanism of the long short-term memory network; and a tenth processing unit, configured to perform weighted fusion of the time steps based on the attention mechanism, and adjust the model parameters in the speech recognition model based on the weighted fusion result.

[0110] In an optional embodiment of this application, the second processing module may further include: an eleventh processing unit, configured to calculate the sum of the absolute values ​​of the weights of each filter in the speech recognition model layer by layer, and remove filters whose sum of absolute values ​​is lower than a fourth preset threshold at each layer to reduce redundant neurons and weights; a twelfth processing unit, configured to divide the filter weight matrix in the speech recognition model after reduction into two sub-matrices to reconstruct the speech recognition model; and a thirteenth processing unit, configured to use the same weight matrix for the input gate and forget gate in the long short-term memory network of the speech recognition model.

[0111] In an optional embodiment of this application, the acquisition module may further include: an acquisition unit, used to acquire voice data in a variety of ways using a combination array of omnidirectional microphones and cardioid microphones, or using a combination array of omnidirectional microphones and supercardioid microphones, wherein the number of microphones in the combination array is multiple.

[0112] In an optional embodiment of this application, the apparatus further includes a fourth processing module, used to integrate the speech recognition model locally and to perform performance evaluation and model optimization on the speech recognition model.

[0113] like Figure 5As shown in the figure, this application provides a device including a processor 511, a communication interface 512, a memory 513, and a communication bus 514, wherein the processor 511, the communication interface 512, and the memory 513 communicate with each other through the communication bus 514.

[0114] Memory 513 is used to store computer programs;

[0115] In one embodiment of this application, when the processor 511 executes the program stored in the memory 513, it implements the offline voice data processing method provided in any of the foregoing method embodiments, and its function is similar, so it will not be described again here.

[0116] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the offline voice data processing method provided in any of the foregoing method embodiments.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0120] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for processing offline voice data, characterized in that, include: The microphone array is used to collect diverse voice data, wherein the diverse collection refers to the voice data covering different speakers, different language types, different speech rates and different noise backgrounds; The acquired speech data is subjected to adaptive filtering and noise reduction. Endpoint detection is performed on the noise-reduced speech data to determine the speech start and end points. Continuous data in the speech data after endpoint detection is segmented and windowed. Finally, Mel frequency cepstral coefficients and higher-order features are extracted and fused from the segmented and windowed data to obtain the preprocessed speech data. A speech recognition model is trained based on preprocessed speech data, wherein the speech recognition model integrates a convolutional neural network and a long short-term memory network. The preprocessed speech data is input into the convolutional neural network, and the convolutional neural network captures the local correlations of the speech data and outputs a feature map dimension. The feature map dimension is then adjusted to a sequence format and input into the long short-term memory network. The attention score of the time step output by the long short-term memory network is determined based on the memory mechanism, and a weighted fusion is performed based on the attention score. The trained speech recognition model is pruned to remove redundant neurons and weights, and the pruned speech recognition model is decomposed into low-rank components to reconstruct the speech recognition model. The parameters in the reconstructed speech recognition model are shared, and finally the speech recognition model with shared parameters is retrained. The real-time input offline speech data is processed based on the retrained speech recognition model.

2. The method according to claim 1, characterized in that, The acquired speech data undergoes adaptive filtering and noise reduction, and endpoint detection is performed on the denoised speech data to determine the start and end points of speech, including: The high-frequency components of the speech data are enhanced by a high-pass filter, and the enhanced speech data is converted into a corresponding input vector. The square of the input vector is used as the power of the input signal. The power of the input signal is determined as a normalization factor to update the filter weights, and the updated filter is used to perform noise reduction on the speech data. If the energy of a consecutive first preset number of frames exceeds a first preset threshold but is lower than a second preset threshold, then the consecutive first preset number of frames are determined as the starting point of the speech. If the number of consecutive first preset number frames continues to decrease and falls below a third preset threshold, the first preset number frame is determined to be the voice endpoint.

3. The method according to claim 1, characterized in that, Frame-by-frame windowing is applied to the continuous speech data after endpoint detection. Finally, Mel-frequency cepstral coefficients and higher-order features are extracted and fused from the frame-by-frame windowed data, including: The continuous data in the speech data after endpoint detection is segmented, and the edge smoothing of the segmented data is performed using a window function. Extract the first second preset number of Mel frequency cepstral coefficients, as well as the corresponding first-order and second-order differences, from the framed and windowed speech data; Feature fusion is performed on the fundamental frequency and formants of the windowed speech data after frame segmentation, along with a second preset number of Mel frequency cepstral coefficients and the corresponding first-order and second-order differences.

4. The method according to claim 1, characterized in that, The speech recognition model is trained based on the preprocessed speech data, including: The speech data is reconstructed into a matrix multiplied by the time step and the feature dimension, and the matrix is ​​input into the convolutional kernel of the convolutional neural network for sliding scanning to capture the local correlation of the speech data, resulting in the output feature map dimension multiplied by the time step and the number of channels. The feature map dimensions are adjusted to a sequence format and then input into a long short-term memory network. The output time step is obtained through the gating mechanism of the long short-term memory network. The time steps are weighted and fused based on an attention mechanism, and the model parameters in the speech recognition model are adjusted based on the weighted fusion result.

5. The method according to claim 1, characterized in that, The trained speech recognition model is pruned to remove redundant neurons and weights, and then low-rank decomposition is performed on the pruned speech recognition model to reconstruct it. Parameters in the reconstructed speech recognition model are shared, including: The absolute sum of the weights of each filter in the speech recognition model is calculated layer by layer, and filters whose absolute sum is lower than a fourth preset threshold are removed in each layer to reduce redundant neurons and weights. The filter weight matrix in the speech recognition model after subtraction is divided into two sub-matrices to reconstruct the speech recognition model. The input gate and forget gate in the long short-term memory network of the speech recognition model use the same weight matrix.

6. The method according to claim 1, characterized in that, Diverse acquisition of voice data using microphone arrays, including: A combination array of omnidirectional microphones and cardioid microphones, or a combination array of omnidirectional microphones and supercardioid microphones, is used to collect voice data in a variety of ways, wherein the number of microphones in the combination array is multiple.

7. The method according to claim 1, characterized in that, The method further includes: The speech recognition model is integrated locally, and its performance is evaluated and optimized.

8. An offline voice data processing device, characterized in that, include: The acquisition module is used to acquire diverse voice data using a microphone array. The diverse acquisition means that the voice data covers different speakers, different language types, different speech rates, and different noise backgrounds. The first processing module is used to perform adaptive filtering and noise reduction on the acquired speech data, and to perform endpoint detection on the noise-reduced speech data to determine the speech start and end points in the speech data. It also performs frame segmentation and windowing on the continuous data in the speech data after endpoint detection, and finally extracts and fuses Mel frequency cepstral coefficients and higher-order features from the frame segmented and windowed data to obtain preprocessed speech data. The training module is used to train a speech recognition model based on preprocessed speech data, wherein the speech recognition model integrates a convolutional neural network and a long short-term memory network. The preprocessed speech data is input into the convolutional neural network, and the convolutional neural network captures the local correlations of the speech data and outputs a feature map dimension. The feature map dimension is then adjusted to a sequence format and input into the long short-term memory network. Based on the memory mechanism, the attention score of the time step output by the long short-term memory network is determined, and a weighted fusion is performed based on the attention score. The second processing module is used to remove redundant neurons and weights from the trained speech recognition model through model pruning, and to reconstruct the speech recognition model by performing low-rank decomposition on the pruned speech recognition model, as well as to share the parameters in the reconstructed speech recognition model, and finally to retrain the speech recognition model after parameter sharing. The third processing module is used to process the real-time input offline speech data based on the retrained speech recognition model.

9. A device, characterized in that, include: At least one communication interface; At least one bus connected to the at least one communication interface; At least one processor connected to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to perform the offline voice data processing method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, The device stores computer-executable instructions for performing the offline voice data processing method according to any one of claims 1 to 7.

Images