Voice packet loss processing method, device, equipment and storage medium based on deep learning and variational mode decomposition

Abstract

The present invention discloses a voice packet loss processing method and apparatus, equipment and storage medium based on deep learning and variational mode decomposition, which relates to the field of communication technology. The method comprises the following steps: extracting multidimensional features of a received voice signal; performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network to generate estimated features of lost frames, and reconstructing the voice signal based on the estimated features to obtain an interpolated and reconstructed voice signal; performing variational mode decomposition on the interpolated and reconstructed voice signal to obtain effective components; reconstructing the effective components to obtain a filtered voice signal, and performing post-processing optimization on the filtered voice signal to obtain an optimized voice signal. By integrating the feature learning capability of deep learning with the adaptive signal processing advantage of variational mode decomposition, the method reduces bandwidth consumption, improves voice communication quality in packet loss scenarios, and meets the requirements of real-time, efficient and high-quality VoIP communication.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a voice packet loss processing method and apparatus, device, and storage medium based on deep learning and variational mode decomposition. Background Art

[0002] In VoIP communications, voice packet loss severely impacts communication quality. The current mainstream packet loss handling technology is redundant transmission, which compensates for packet loss by sending additional voice data. However, this consumes significant network bandwidth, exacerbating network congestion and reducing communication efficiency when the network load is high. For example, in some busy office areas, VoIP calls using redundant transmission can experience noticeable lag.

[0003] Traditional interpolation methods struggle to capture the complex nonlinear characteristics of voice signals. When dealing with packet loss, the recovered voice quality is poor, with distortion, blurring, and other issues. Fixed-parameter filtering methods, on the other hand, cannot adapt to the dynamic changes in network noise. In complex and ever-changing network environments, where the frequency and intensity of noise fluctuate constantly, fixed-parameter filters struggle to effectively remove noise and cannot meet the requirements for high-quality voice communication.

[0004] The above content is only used to assist in understanding the technical solution of the present invention and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of the present invention is to provide a voice packet loss processing method and device, equipment and storage medium based on deep learning and variational mode decomposition, aiming to solve the technical problem that voice message loss seriously affects communication quality.

[0006] To achieve the above objectives, the present invention provides a method for processing voice packet loss based on deep learning and variational mode decomposition, the method comprising the following steps:

[0007] Extracting multidimensional features of the received speech signal;

[0008] Performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network to generate estimated features of lost frames, and reconstructing a speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal;

[0009] performing variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components;

[0010] The effective component is reconstructed to obtain a filtered speech signal, and the filtered speech signal is post-processed and optimized to obtain an optimized speech signal.

[0011] In one embodiment, the step of extracting multi-dimensional features of the received speech signal includes:

[0012] Splitting the received speech signal into a frame sequence of fixed length according to a preset time interval;

[0013] Monitoring the transmission status of the frame sequence through a frame sequence number continuity detection mechanism, and determining the location of voice packet loss when a frame sequence number jump or frame sequence number loss is detected;

[0014] Based on the valid speech frames adjacent to the speech packet loss position, Mel-frequency cepstral coefficients and linear prediction cepstral coefficients are extracted as multi-dimensional speech features.

[0015] In one embodiment, the steps of performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network to generate estimated features of lost frames, and reconstructing a speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal include:

[0016] Build a spatiotemporal convolutional autoencoder network consisting of a temporal attention module, a multi-scale convolution module, and a bottleneck layer;

[0017] Inputting the multidimensional features into the spatiotemporal convolutional autoencoder network to generate estimated features of the lost frame, the estimated features including Mel-frequency cepstral coefficients and linear prediction cepstral coefficients;

[0018] The Mel-frequency cepstral coefficients and the linear prediction cepstral coefficients are converted into a time-domain speech signal through an inverse transform operation to obtain a speech signal after interpolation and reconstruction.

[0019] In one embodiment, the step of performing variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components includes:

[0020] Decomposing the interpolated and reconstructed speech signal into a plurality of eigenmode functions;

[0021] Calculating the center frequency and energy proportion of each of the eigenmode functions;

[0022] The eigenmode functions whose center frequencies are within a preset frequency range and whose energy proportion is greater than a preset energy proportion threshold are retained as valid components.

[0023] In one embodiment, the steps of reconstructing the effective component to obtain a filtered speech signal, and performing post-processing optimization on the filtered speech signal to obtain an optimized speech signal include:

[0024] weighted superposition of the effective components to generate a filtered speech signal;

[0025] performing spectral subtraction processing on the filtered speech signal to obtain an enhanced speech signal;

[0026] The enhanced speech signal is subjected to smooth transition processing to obtain an optimized speech signal.

[0027] In one embodiment, the method further includes:

[0028] Evaluate the optimized speech signal in real time using a speech quality evaluation model to obtain an evaluation result;

[0029] When the evaluation result is lower than a preset quality threshold, a parameter adjustment mechanism is triggered to dynamically adjust spectral subtraction parameters and weighting function parameters according to the evaluation result to optimize speech quality.

[0030] In one embodiment, the step of performing smooth transition processing on the enhanced speech signal to obtain an optimized speech signal includes:

[0031] Design a weighting function that matches the boundary characteristics of the interpolated frame and the original frame;

[0032] The enhanced speech signal is weighted by the weighting function, so that a transition boundary between the interpolated frame and the original frame can achieve a smooth transition, thereby obtaining an optimized speech signal.

[0033] In addition, to achieve the above objectives, the present invention also proposes a voice packet loss processing device based on deep learning and variational mode decomposition, the device comprising:

[0034] A feature acquisition module is used to extract multi-dimensional features of the received speech signal;

[0035] an interpolation and reconstruction module, configured to perform packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network, generate estimated features of lost frames, and reconstruct the speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal;

[0036] A filtering module, configured to perform variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components;

[0037] The optimization module is used to reconstruct the effective component to obtain a filtered speech signal, and perform post-processing optimization on the filtered speech signal to obtain an optimized speech signal.

[0038] In addition, to achieve the above-mentioned objectives, the present invention also proposes a voice packet loss processing device based on deep learning and variational modal decomposition, the device comprising: a memory, a processor, and a voice packet loss processing program based on deep learning and variational modal decomposition stored in the memory and executable on the processor, the voice packet loss processing program based on deep learning and variational modal decomposition being configured to implement the steps of the voice packet loss processing method based on deep learning and variational modal decomposition as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which is stored a voice packet loss processing program based on deep learning and variational mode decomposition. When the voice packet loss processing program based on deep learning and variational mode decomposition is executed by a processor, the steps of the voice packet loss processing method based on deep learning and variational mode decomposition as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the voice packet loss processing method based on deep learning and variational mode decomposition as described above.

[0041] One or more technical solutions proposed in this application have at least the following technical effects:

[0042] Extract the multidimensional features of the received voice signal; use the spatiotemporal convolutional autoencoder network to perform packet loss detection on the multidimensional features, generate estimated features of the lost frame, and reconstruct the voice signal based on the estimated features to obtain an interpolated and reconstructed voice signal; perform variational mode decomposition on the interpolated and reconstructed voice signal to obtain effective components; reconstruct the effective components to obtain a filtered voice signal, and perform post-processing optimization on the filtered voice signal to obtain an optimized voice signal. By integrating the feature learning ability of deep learning with the adaptive signal processing advantages of variational mode decomposition, bandwidth consumption is reduced, and the quality of voice communication in packet loss scenarios is improved, meeting the requirements of real-time, efficient, and high-quality VoIP communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 A flowchart illustrating a first embodiment of a method for processing voice packet loss based on deep learning and variational mode decomposition is provided in this application;

[0046] Figure 2 This is a diagram of the overall system architecture provided in Example 1 of the voice packet loss processing method based on deep learning and variational mode decomposition of this application;

[0047] Figure 3 A flowchart illustrating a second embodiment of a method for processing voice packet loss based on deep learning and variational mode decomposition is provided for this application;

[0048] Figure 4 This is a schematic diagram of the module structure of a voice packet loss processing device based on deep learning and variational mode decomposition according to an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice packet loss processing method based on deep learning and variational mode decomposition in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] It should be noted that the execution entity of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, a voice packet loss processing device based on deep learning and variational mode decomposition, etc. The following uses the voice packet loss processing device based on deep learning and variational mode decomposition as an example to illustrate this embodiment and the following embodiments.

[0054] Based on this, the embodiment of the present application provides a voice packet loss processing method based on deep learning and variational mode decomposition, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the voice packet loss processing method based on deep learning and variational mode decomposition of this application.

[0055] In this embodiment, the voice packet loss processing method based on deep learning and variational mode decomposition includes steps S10 to S40:

[0056] Step S10, extracting multi-dimensional features of the received speech signal;

[0057] It should be noted that multiple features are extracted from the received speech signal so that subsequent processing can more accurately reconstruct and optimize the speech signal.

[0058] In its implementation, the received voice stream is segmented into fixed-length frames at preset 20ms intervals to facilitate subsequent processing. A frame number continuity detection mechanism monitors the voice frame transmission status in real time. Once a frame number jump or loss is detected, voice packet loss is identified and its precise location is recorded. Multi-dimensional voice features, such as Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCCs), are extracted from adjacent valid voice frames, providing rich data for subsequent interpolation of lost frames.

[0059] like Figure 2 As shown in the figure, after voice data is input, it first enters the voice message preprocessing stage. The preprocessed voice data is segmented into voice streams, followed by packet loss detection. If packet loss is detected, a spatiotemporal convolutional autoencoder is triggered to perform interpolation and reconstruction to recover the lost data. Next, feature extraction is performed on the reconstructed voice data. This extraction process involves the encoder (including convolutional layers, pooling layers, and a temporal attention module). The extracted features are input to the VMD adaptive filtering module, which performs VMD decomposition of the voice signal to separate different frequency modes. The decomposed signal undergoes effective signal filtering and reconstruction filtering to purify the voice signal. Furthermore, this process includes spectral subtraction and smoothing to enhance the voice signal and optimize its quality. Finally, the optimized voice signal is input to the perceptual optimization fine-tuning module for voice quality assessment. If the assessment results indicate that the voice quality does not meet the preset standard, the parameter fine-tuning mechanism is triggered to dynamically adjust the relevant parameters based on the assessment results. This entire process is closely linked, ensuring comprehensive optimization of the voice data and ultimately outputting a high-quality voice signal, completing the entire processing flow.

[0060] In a feasible implementation, step S10 includes steps A11 to A13:

[0061] A11: Divide the received voice signal into a fixed-length frame sequence according to a preset time interval;

[0062] It should be noted that the speech signal is segmented into fixed-length frames at preset time intervals. This facilitates segment-by-segment processing of the speech signal, facilitating subsequent feature extraction and analysis. For example, the speech signal is segmented into frames every 20 milliseconds.

[0063] A12: The frame sequence continuity detection mechanism monitors the transmission status of the frame sequence. When a frame number jump or frame number loss is detected, the voice packet loss location is determined.

[0064] It should be noted that the frame sequence continuity detection mechanism monitors the transmission status of the frame sequence. Frame loss may occur during voice signal transmission. This mechanism detects jumps or missing frame numbers, thereby determining the specific location of voice packet loss. Specifically, if the sequence number of the current frame is discontinuous with the sequence number of the previous frame, frame loss is determined.

[0065] A13: Based on the valid speech frames adjacent to the speech packet loss location, Mel-frequency cepstral coefficients and linear prediction cepstral coefficients are extracted as multidimensional speech features.

[0066] It should be noted that based on valid speech frames adjacent to the speech packet loss location, Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Prediction Cepstral Coefficients (LPCCs) are extracted as multidimensional speech features. MFCCs effectively characterize the spectral characteristics of speech signals, while LPCCs, derived through linear predictive coding, characterize the time domain characteristics of speech signals. These features comprehensively describe the characteristics and variations of speech signals. Specifically, for each valid speech frame, the MFCCs and LPCCs are calculated and combined into a multidimensional feature vector.

[0067] Step S20, performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network, generating estimated features of the lost frames, and reconstructing the speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal;

[0068] It should be noted that the deep learning network is used to detect packet loss on the speech signal and attempts to reconstruct the lost speech frames to restore the integrity of the speech signal.

[0069] In the specific implementation, a spatiotemporal convolutional autoencoder (STCAE) network is constructed, consisting of a temporal attention module, a multi-scale convolution module, and a bottleneck layer. The temporal attention module captures the temporal dependencies of the speech signal, highlighting key speech information. The multi-scale convolution module uses convolution kernels of different sizes to extract features at multiple frequency scales. The bottleneck layer compresses the input features into low-dimensional vectors for efficient feature abstraction. The STCAE encoder processes the speech feature vectors of the 4-8 frames preceding and following the lost frame, generating a feature representation that incorporates contextual information and reflects the underlying feature patterns of the lost frame. The STCAE decoder, based on the feature representation generated by the encoder, outputs the estimated MFCC and LPCC features of the lost frame, reconstructing the speech information of the lost frame. An inverse transform is performed to convert the estimated speech features into a time-domain speech signal, completing the interpolation reconstruction of the lost frame.

[0070] It should be understood that in actual applications, network parameters such as the number and size of convolution kernels and the potential dimension can be adjusted to optimize model performance based on the characteristics of the voice data and hardware resources. At the same time, the model should be trained and optimized using an appropriate training dataset and training strategy.

[0071] Understandably, the initialization phase involves collecting a large amount of speech sample data with varying packet loss scenarios to train the constructed STCAE model. Using a mean square error loss function and the Adam optimizer, model parameters are adjusted via backpropagation to enable the model to accurately interpolate and reconstruct lost frames. Experiments are conducted to test the impact of different VMD parameter combinations (such as alpha and K) on speech filtering performance for different speech signal types, determining the optimal parameter configuration for different scenarios and adapting to diverse speech processing needs.

[0072] During real-time processing, the system processes voice packets sequentially according to the above process. Through efficient algorithms and rational module collaboration, it ensures that lost voice packets are interpolated and filtered in a short time, meeting the real-time requirements of VoIP communications.

[0073] In a feasible implementation, step S20 includes steps A21 to A23:

[0074] A21: Construct a spatiotemporal convolutional autoencoder network consisting of a temporal attention module, a multi-scale convolution module, and a bottleneck layer;

[0075] It should be noted that the construction of a spatiotemporal convolutional autoencoder network includes a temporal attention module, a multi-scale convolution module, and a bottleneck layer. The temporal attention module can capture the important features of the speech signal in the time series, the multi-scale convolution module can extract features at different scales, and the bottleneck layer is used to reduce the feature dimension and reduce computational complexity. The temporal attention module calculates the attention weight of each time step in the time series to highlight important time steps; the multi-scale convolution module uses convolution kernels of different sizes to extract features at different scales; and the bottleneck layer reduces the computational complexity by reducing the feature dimension.

[0076] A22: Input the multidimensional features into the spatiotemporal convolutional autoencoder network to generate estimated features of the lost frame. The estimated features include Mel-frequency cepstral coefficients and linear prediction cepstral coefficients.

[0077] It should be noted that multidimensional features are fed into the spatiotemporal convolutional autoencoder network to generate estimated features for the lost frame. The estimated features include MFCCs and LPCCs, which describe the spectral and temporal characteristics of the lost frame. The multidimensional feature vector is fed into the network, which uses the learned feature mapping to generate feature estimates for the lost frame.

[0078] A23: The Mel-frequency cepstral coefficients and the linear prediction cepstral coefficients are converted into a time-domain speech signal through an inverse transform operation to obtain an interpolated and reconstructed speech signal.

[0079] It should be noted that the estimated features are converted into a time-domain speech signal through an inverse transform operation to obtain the interpolated and reconstructed speech signal. The inverse transform operation converts the frequency domain features back into a time-domain signal, thereby reconstructing the lost speech frames and making the speech signal more complete. The estimated MFCC and LPCC are converted into a time-domain signal through the inverse Mel-frequency cepstral transform and the inverse linear prediction cepstral transform, and the reconstructed speech frames are inserted into the original speech signal through interpolation.

[0080] Step S30, performing variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components;

[0081] It should be noted that the reconstructed speech signal is decomposed into multiple eigenmode functions through the variational mode decomposition technology, and the effective components are screened out.

[0082] In its implementation, the interpolated and reconstructed speech signal undergoes variational mode decomposition (VMD), decomposing it into K intrinsic mode functions (IMFs). Each IMF represents a speech component at a different frequency and time scale. Parameters such as the center frequency, bandwidth, and energy distribution of each IMF are accurately calculated to provide a basis for screening valid components. Based on preset rules, IMF components with center frequencies in the primary speech frequency range of 100Hz to 8kHz and an energy contribution greater than 5% are retained. Noise and irrelevant components are removed, preserving the useful speech information. The filtered valid IMF components are then weighted and reconstructed to generate the filtered speech signal, improving speech quality.

[0083] It should be understood that VMD parameters (such as alpha and K) need to be adjusted appropriately based on the characteristics of the speech signal. Experiments can be conducted to test the filtering effects under different parameter combinations and select the optimal parameter configuration to achieve the best noise suppression and speech preservation effects.

[0084] In a feasible implementation, step S30 includes steps A31 to A33:

[0085] A31: Decompose the interpolated and reconstructed speech signal into multiple eigenmode functions;

[0086] It should be noted that the interpolated and reconstructed speech signal is decomposed into multiple intrinsic mode functions (IMFs). IMFs are the manifestations of the speech signal in different modes, reflecting the different characteristics and variations of the speech signal. The variational mode decomposition algorithm decomposes the speech signal into multiple mode functions with different center frequencies and bandwidths.

[0087] A32: Calculate the center frequency and energy proportion of each eigenmode function;

[0088] It should be noted that the center frequency and energy contribution of each eigenmode function are calculated. The center frequency represents the primary frequency component of each mode function, while the energy contribution represents the energy contribution of the mode function to the overall speech signal. Spectral analysis is performed on each eigenmode function, its power spectral density is calculated, and its center frequency and energy contribution are determined.

[0089] A33: Retains the eigenmode functions whose center frequencies are within the preset frequency range and whose energy proportions are greater than the preset energy proportion threshold as valid components.

[0090] It should be noted that eigenmode functions with a center frequency within a preset frequency range and an energy contribution greater than a preset energy contribution threshold are retained as valid components. This step, by setting the frequency range and energy contribution threshold, screens out modal functions that significantly contribute to the speech signal, removes noise and interference components, and compares the center frequency and energy contribution of each eigenmode function with the preset threshold to retain those that meet the criteria.

[0091] Step S40: reconstructing the effective component to obtain a filtered speech signal, and performing post-processing optimization on the filtered speech signal to obtain an optimized speech signal.

[0092] It should be noted that the effective component is reconstructed into a filtered speech signal and post-processed and optimized to improve the quality of the speech signal.

[0093] In its implementation, spectral subtraction is used to process the filtered speech signal, further suppressing residual noise and improving speech clarity. A weighting function is designed to smooth the transition between the interpolated and original frames, reducing discontinuities caused by the interpolation operation and ensuring a more natural and smooth speech transition. A speech quality assessment model (such as PESQ) is introduced to evaluate the processed speech quality in real time. Based on the assessment results, processing parameters are dynamically adjusted to optimize speech quality.

[0094] It should be understood that the present invention also relates to a voice packet loss processing system based on deep learning and variational mode decomposition, comprising:

[0095] Feature extraction and packet loss detection module: responsible for voice stream segmentation, packet loss detection and voice feature extraction, providing data support for subsequent processing.

[0096] Spatiotemporal convolutional autoencoder interpolation module: uses the constructed STCAE network to complete the interpolation reconstruction of lost frames.

[0097] VMD adaptive filtering module: implements VMD decomposition, effective component screening and reconstruction filtering of speech signals.

[0098] Perception optimization and post-processing module: performs noise suppression, smooth transition and quality assessment optimization on the processed speech signal.

[0099] Compared to traditional redundant transmission methods, this invention eliminates the need to transmit redundant data, saving 30%-50% of network bandwidth, reducing network transmission pressure and improving bandwidth utilization. Even with limited network bandwidth, it allows more VoIP calls to proceed smoothly. PESQ testing shows that in common scenarios with a packet loss rate of 10%-20%, the processed voice quality score of this invention improves by 1.2-1.5 points compared to traditional methods, providing users with a clearer and more natural voice communication experience. The deep learning-based interpolation model can learn online, automatically adapting to the voice characteristics of different speakers and changes in the network environment. The adaptive filtering characteristics of variational mode decomposition can effectively cope with complex and changing network noise, ensuring stable system operation under different conditions. The entire processing process utilizes efficient algorithms and architectural designs, with end-to-end processing latency of less than 10ms, meeting the stringent low-latency requirements of real-time VoIP communication and ensuring smooth voice communication.

[0100] This embodiment provides a voice packet loss processing method based on deep learning and variational mode decomposition, which extracts multidimensional features of a received voice signal; uses a spatiotemporal convolutional autoencoder network to perform packet loss detection on the multidimensional features, generates estimated features of lost frames, and reconstructs the voice signal based on the estimated features to obtain an interpolated and reconstructed voice signal; performs variational mode decomposition on the interpolated and reconstructed voice signal to obtain effective components; reconstructs the effective components to obtain a filtered voice signal, and performs post-processing optimization on the filtered voice signal to obtain an optimized voice signal. By integrating the feature learning capability of deep learning with the adaptive signal processing advantages of variational mode decomposition, bandwidth consumption is reduced, the quality of voice communication in packet loss scenarios is improved, and the requirements of real-time, efficient, and high-quality VoIP communication are met.

[0101] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 Step S40 includes steps S401 to S403:

[0102] Step S401, weighted superposition of effective components to generate a filtered speech signal;

[0103] It should be noted that weighted superposition is to perform weighted processing on the effective components and then superimpose them to generate a filtered speech signal. By assigning different weights to different components, important components are highlighted and unimportant components are suppressed, thereby improving the purity of the speech signal.

[0104] Step S402, performing spectral subtraction processing on the filtered speech signal to obtain an enhanced speech signal;

[0105] It should be noted that the spectral subtraction process is to subtract the estimated noise spectrum in the spectral domain to enhance the spectrum of the speech signal, which can effectively suppress noise and improve the clarity of the speech signal.

[0106] Step S403: performing smooth transition processing on the enhanced speech signal to obtain an optimized speech signal.

[0107] It should be noted that the smooth transition processing is to smooth the enhanced speech signal to reduce the mutation in the spectrum, make the speech signal more natural, and ensure the continuity of the spectrum change through time window or frequency window technology.

[0108] In a feasible implementation manner, step S403 includes steps A41 to A42:

[0109] A41: Use the speech quality assessment model to evaluate the optimized speech signal in real time to obtain the evaluation result;

[0110] It should be noted that the speech quality assessment model is used to assess the quality of speech signals in real time, and the assessment results output by the model are used to determine whether the speech signals meet the quality requirements.

[0111] A42: When the evaluation result is lower than the preset quality threshold, the parameter adjustment mechanism is triggered to dynamically adjust the spectral subtraction parameters and weighting function parameters according to the evaluation result to optimize the speech quality.

[0112] It should be noted that when the speech quality does not meet the requirements, the parameters of the spectral subtraction and weighting functions are automatically adjusted.

[0113] Furthermore, a weighting function is designed to match the boundary features of the interpolated frame and the original frame;

[0114] The enhanced speech signal is weighted by a weighting function to achieve a smooth transition between the transition boundary of the interpolated frame and the original frame, thereby obtaining an optimized speech signal.

[0115] In the specific implementation, a weighting function is designed to match the boundary features of the interpolated frame and the original frame to ensure the smoothness of the transition; the weighting function is applied to process the speech signal to achieve a smooth transition between the transition boundary of the interpolated frame and the original frame, thereby obtaining an optimized speech signal.

[0116] This embodiment provides a voice packet loss processing method based on deep learning and variational mode decomposition. This method generates a filtered voice signal by weighted superposition of effective components and enhances and optimizes the signal using spectral subtraction and time-domain smoothing techniques. A real-time voice quality assessment model is introduced to monitor the optimized voice signal and dynamically adjust relevant parameters when voice quality falls short of standards, ensuring that the output voice signal remains high quality. This effectively improves the clarity and naturalness of the voice signal, enhances the system's adaptability to complex voice environments, and ultimately outputs a high-quality voice signal.

[0117] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the voice packet loss processing method based on deep learning and variational mode decomposition of the present application. More forms of simple transformations based on this technical concept are all within the scope of protection of the present application.

[0118] This application also provides a voice packet loss processing device based on deep learning and variational mode decomposition, please refer to Figure 4 , the voice packet loss processing device based on deep learning and variational mode decomposition includes:

[0119] A feature acquisition module 10 is used to extract multi-dimensional features of the received speech signal;

[0120] An interpolation and reconstruction module 20 is configured to detect packet loss using a spatiotemporal convolutional autoencoder network on multidimensional features, generate estimated features of lost frames, and reconstruct the speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal.

[0121] The filtering module 30 is used to perform variational mode decomposition on the interpolated and reconstructed speech signal to obtain effective components;

[0122] The optimization module 40 is used to reconstruct the effective component to obtain a filtered speech signal, and perform post-processing optimization on the filtered speech signal to obtain an optimized speech signal.

[0123] The voice packet loss processing device based on deep learning and variational modal decomposition provided in this application, which employs the voice packet loss processing method based on deep learning and variational modal decomposition in the above-mentioned embodiments, can resolve the technical problem of voice message loss seriously affecting communication quality. Compared with the prior art, the beneficial effects of the voice packet loss processing device based on deep learning and variational modal decomposition provided in this application are the same as those of the voice packet loss processing method based on deep learning and variational modal decomposition provided in the above-mentioned embodiments. Other technical features of the voice packet loss processing device based on deep learning and variational modal decomposition are the same as those disclosed in the above-mentioned embodiments and are not further described here.

[0124] In one embodiment, the feature acquisition module 10 is further configured to divide the received speech signal into a frame sequence of fixed length according to a preset time interval;

[0125] The frame sequence continuity detection mechanism monitors the transmission status of the frame sequence and determines the location of voice packet loss when a frame number jump or frame number loss is detected.

[0126] Based on the valid speech frames adjacent to the speech packet loss position, Mel-frequency cepstral coefficients and linear prediction cepstral coefficients are extracted as multi-dimensional speech features.

[0127] In one embodiment, the interpolation and reconstruction module 20 is further used to construct a spatiotemporal convolutional autoencoder network comprising a temporal attention module, a multi-scale convolution module, and a bottleneck layer;

[0128] The multidimensional features are fed into a spatiotemporal convolutional autoencoder network to generate estimated features of the lost frame. The estimated features include Mel-frequency cepstral coefficients and linear prediction cepstral coefficients.

[0129] The Mel-frequency cepstral coefficients and the linear prediction cepstral coefficients are converted into time-domain speech signals through an inverse transform operation to obtain an interpolated and reconstructed speech signal.

[0130] In one embodiment, the filtering module 30 is further configured to decompose the interpolated and reconstructed speech signal into a plurality of eigenmode functions;

[0131] Calculate the center frequency and energy proportion of each eigenmode function;

[0132] The eigenmode functions whose center frequencies are within a preset frequency range and whose energy proportion is greater than a preset energy proportion threshold are retained as valid components.

[0133] In one embodiment, the optimization module 40 is further configured to perform weighted superposition of the effective components to generate a filtered speech signal;

[0134] Performing spectral subtraction on the filtered speech signal to obtain an enhanced speech signal;

[0135] The enhanced speech signal is subjected to smooth transition processing to obtain an optimized speech signal.

[0136] In one embodiment, the optimization module 40 is further configured to evaluate the optimized speech signal in real time using a speech quality evaluation model to obtain an evaluation result;

[0137] When the evaluation result is lower than the preset quality threshold, the parameter adjustment mechanism is triggered to dynamically adjust the spectral subtraction parameters and weighting function parameters according to the evaluation result to optimize the speech quality.

[0138] In one embodiment, the optimization module 40 is further configured to design a weighting function that matches the boundary features of the interpolated frame and the original frame;

[0139] The enhanced speech signal is weighted by a weighting function to achieve a smooth transition between the transition boundary of the interpolated frame and the original frame, thereby obtaining an optimized speech signal.

[0140] The present application provides a voice packet loss processing device based on deep learning and variational mode decomposition. The voice packet loss processing device based on deep learning and variational mode decomposition includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the voice packet loss processing method based on deep learning and variational mode decomposition in the above-mentioned embodiment 1.

[0141] Reference below Figure 5 , which shows a schematic structural diagram of a voice packet loss processing device based on deep learning and variational mode decomposition suitable for implementing the embodiments of the present application. The voice packet loss processing device based on deep learning and variational mode decomposition in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The voice packet loss processing device based on deep learning and variational modal decomposition shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0142] like Figure 5As shown, the voice packet loss processing device based on deep learning and variational mode decomposition may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in ROM (Read Only Memory) 1002 or programs loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the voice packet loss processing device based on deep learning and variational mode decomposition. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication devices 1009 can allow the voice packet loss processing device based on deep learning and variational mode decomposition to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a voice packet loss processing device based on deep learning and variational mode decomposition with various systems, it should be understood that implementation or presence of all the illustrated systems is not required. More or fewer systems may alternatively be implemented or present.

[0143] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0144] The voice packet loss processing device based on deep learning and variational modal decomposition provided in this application, which employs the voice packet loss processing method based on deep learning and variational modal decomposition in the above-mentioned embodiment, can resolve the technical problem of voice message loss seriously affecting communication quality. Compared with the prior art, the beneficial effects of the voice packet loss processing device based on deep learning and variational modal decomposition provided in this application are the same as those of the voice packet loss processing method based on deep learning and variational modal decomposition provided in the above-mentioned embodiment. Other technical features of the voice packet loss processing device based on deep learning and variational modal decomposition are the same as those disclosed in the above-mentioned embodiment and are not further described here.

[0145] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0146] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0147] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, and the computer-readable program instructions are used to execute the voice packet loss processing method based on deep learning and variational mode decomposition in the above-mentioned embodiment.

[0148] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory or Flash memory), optical fiber, CD-ROM (CD-Read Only Memory), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0149] The above-mentioned computer-readable storage medium can be included in the voice packet loss processing device based on deep learning and variational mode decomposition; or it can exist independently without being assembled into the voice packet loss processing device based on deep learning and variational mode decomposition.

[0150] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a voice packet loss processing device based on deep learning and variational mode decomposition, the voice packet loss processing device based on deep learning and variational mode decomposition: extracts multidimensional features of a received voice signal; uses a spatiotemporal convolutional autoencoder network to perform packet loss detection on the multidimensional features, generates estimated features of lost frames, and reconstructs the voice signal based on the estimated features to obtain an interpolated and reconstructed voice signal; performs variational mode decomposition on the interpolated and reconstructed voice signal to obtain effective components; reconstructs the effective components to obtain a filtered voice signal, and performs post-processing optimization on the filtered voice signal to obtain an optimized voice signal.

[0151] The computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a LAN (Local Area Network) or a WAN (Wide Area Network), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0153] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0154] The computer-readable storage medium provided in this application stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned method for handling voice packet loss based on deep learning and variational mode decomposition. This method can address the technical issue of voice packet loss severely impacting communication quality. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are similar to those of the method for handling voice packet loss based on deep learning and variational mode decomposition provided in the aforementioned embodiments, and are not further elaborated here.

[0155] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned voice packet loss processing method based on deep learning and variational mode decomposition.

[0156] The computer program product provided in this application can address the technical issue of voice packet loss seriously impacting communication quality. Compared to the prior art, the computer program product provided in this application has the same beneficial effects as the voice packet loss processing method based on deep learning and variational mode decomposition provided in the above-mentioned embodiment, and will not be further elaborated here.

[0157] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A voice packet loss processing method based on deep learning and variational mode decomposition, characterized in that: The method comprises: Extracting multidimensional features of the received speech signal; Performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network to generate estimated features of lost frames, and reconstructing a speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal; performing variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components; Reconstructing the effective component to obtain a filtered speech signal, and performing post-processing optimization on the filtered speech signal to obtain an optimized speech signal; The step of performing packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network to generate estimated features of lost frames, and reconstructing a speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal comprises: Build a spatiotemporal convolutional autoencoder network consisting of a temporal attention module, a multi-scale convolution module, and a bottleneck layer; Inputting the multidimensional features into the spatiotemporal convolutional autoencoder network to generate estimated features of the lost frame, the estimated features including Mel-frequency cepstral coefficients and linear prediction cepstral coefficients; Converting the Mel-frequency cepstral coefficients and the linear prediction cepstral coefficients into a time-domain speech signal through an inverse transform operation to obtain an interpolated and reconstructed speech signal; The step of performing variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components includes: Decomposing the interpolated and reconstructed speech signal into a plurality of eigenmode functions; Calculating the center frequency and energy proportion of each of the eigenmode functions; The eigenmode functions whose center frequencies are within a preset frequency range and whose energy proportion is greater than a preset energy proportion threshold are retained as valid components.

2. The method according to claim 1, wherein The step of extracting multidimensional features of the received speech signal comprises: Splitting the received speech signal into a frame sequence of fixed length according to a preset time interval; Monitoring the transmission status of the frame sequence through a frame sequence number continuity detection mechanism, and determining the location of voice packet loss when a frame sequence number jump or frame sequence number loss is detected; Based on the valid speech frames adjacent to the speech packet loss position, Mel-frequency cepstral coefficients and linear prediction cepstral coefficients are extracted as multi-dimensional speech features.

3. The method according to claim 1, wherein The steps of reconstructing the effective component to obtain a filtered speech signal, and performing post-processing optimization on the filtered speech signal to obtain an optimized speech signal include: weighted superposition of the effective components to generate a filtered speech signal; performing spectral subtraction processing on the filtered speech signal to obtain an enhanced speech signal; Performing smooth transition processing on the enhanced speech signal to obtain an optimized speech signal.

4. The method according to claim 3, wherein The method further comprises: Evaluate the optimized speech signal in real time using a speech quality evaluation model to obtain an evaluation result; When the evaluation result is lower than a preset quality threshold, a parameter adjustment mechanism is triggered to dynamically adjust spectral subtraction parameters and weighting function parameters according to the evaluation result to optimize speech quality.

5. The method according to claim 3, wherein The step of performing smooth transition processing on the enhanced voice signal to obtain an optimized voice signal comprises: Design a weighting function that matches the boundary characteristics of the interpolated frame and the original frame; The enhanced speech signal is weighted by the weighting function, so that a transition boundary between the interpolated frame and the original frame can achieve a smooth transition, thereby obtaining an optimized speech signal.

6. A voice packet loss processing device based on deep learning and variational mode decomposition, characterized in that: The device comprises: A feature acquisition module is used to extract multi-dimensional features of the received speech signal; an interpolation and reconstruction module, configured to perform packet loss detection on the multidimensional features using a spatiotemporal convolutional autoencoder network, generate estimated features of lost frames, and reconstruct the speech signal based on the estimated features to obtain an interpolated and reconstructed speech signal; The interpolation and reconstruction module is also used to build a spatiotemporal convolutional autoencoder network consisting of a temporal attention module, a multi-scale convolution module, and a bottleneck layer; Inputting the multidimensional features into the spatiotemporal convolutional autoencoder network to generate estimated features of the lost frame, the estimated features including Mel-frequency cepstral coefficients and linear prediction cepstral coefficients; Converting the Mel-frequency cepstral coefficients and the linear prediction cepstral coefficients into a time-domain speech signal through an inverse transform operation to obtain an interpolated and reconstructed speech signal; A filtering module, configured to perform variational modal decomposition on the interpolated and reconstructed speech signal to obtain effective components; The filtering module is further used to decompose the interpolated and reconstructed speech signal into multiple eigenmode functions; Calculating the center frequency and energy proportion of each of the eigenmode functions; Retain the eigenmode functions whose center frequencies are within a preset frequency range and whose energy proportion is greater than a preset energy proportion threshold as valid components; The optimization module is used to reconstruct the effective component to obtain a filtered speech signal, and perform post-processing optimization on the filtered speech signal to obtain an optimized speech signal.

7. A voice packet loss processing device based on deep learning and variational mode decomposition, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice packet loss processing method based on deep learning and variational mode decomposition as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the voice packet loss processing method based on deep learning and variational mode decomposition as described in any one of claims 1 to 5 are implemented.

Images