Lightweight acoustic feature extraction method for end-side audio and video quality inspection

Abstract

The invention relates to a lightweight acoustic feature extraction method for end-side audio and video quality inspection. The method comprises the following steps: end-side equipment separates audio and video stream data through a double-time-sequence anchor point alignment method, and acquires independent audio data; performing grading preprocessing on the independent audio data to obtain noise-reduced independent audio data; on the basis of the segmented audio, a time domain and frequency domain collaborative extraction algorithm is adopted, multi-dimensional time domain features and frequency features sensitive to audio quality difference are obtained, and a lightweight acoustic feature vector is obtained through an incremental principal component analysis method; based on the lightweight acoustic feature vector, utilizing a dual-threshold matching judgment mode to obtain similarity between features; and on the basis of the feature similarity, through feature dimension deviation verification, obtaining a quality inspection result containing a standard grade and feature anomaly positioning. Lightweight acoustic feature extraction of end side audio and video quality inspection is realized.

Description

Technical Field

[0001] This invention belongs to the field of audio and video processing and quality inspection technology, specifically relating to a lightweight acoustic feature extraction method for end-side audio and video quality inspection. Background Technology

[0002] With the rapid development of scenarios such as intelligent monitoring, remote conferencing, and live-streaming e-commerce, edge devices (such as smart cameras, edge terminals, and portable quality inspection instruments) have become the core carriers for audio and video data acquisition and preliminary quality inspection. The demand for lightweight, real-time, and high-precision acoustic feature extraction technology is becoming increasingly urgent. However, current lightweight acoustic feature extraction technology in edge audio and video quality inspection still has the following areas for improvement: The pain points of the current technology are: (1) The audio quality standards of different edge quality inspection scenarios are significantly different: the monitoring scenario needs to focus on the audio signal-to-noise ratio (to ensure that the voice is clear and distinguishable), the conference scenario needs to focus on the echo suppression effect (to avoid call interference), and the live broadcast scenario needs to focus on the sound quality fidelity (to ensure the audience experience). However, the standard feature library of the existing technology is mostly a fixed template and is not dynamically adjusted according to the characteristics of the scenario - for example, a unified Euclidean distance threshold (such as 0.5) is used for quality inspection judgment, without considering the difference between the large fluctuation of audio features in the monitoring scenario and the high stability requirements of features in the conference scenario, resulting in insufficient quality inspection accuracy in some scenarios; at the same time, the standard feature library needs to be updated manually by uploading new samples, and cannot be automatically iterated based on the actual quality inspection data of the edge, making it difficult to adapt to the newly added in-vehicle audio and video, remote medical audio and video and other scenarios. (2) The edge audio and video acquisition scenarios have problems such as variable environmental noise (such as wind noise and traffic noise in the monitoring scenario, echo and interference from conversations in the conference scenario) and frequent signal fluctuations (such as audio interruption caused by network packet loss in the live broadcast scenario and signal jitter in the in-vehicle scenario). However, existing technologies are not optimized for this type of interference: traditional short-time energy algorithms are sensitive to transient noise and are prone to misidentifying sudden noise as valid signal features; the MFCC algorithm uses a fixed frequency band filter bank, which cannot adapt to the frequency distribution differences in different scenarios (such as the need to focus on the mid and low frequency bands in vehicle scenarios and the need to focus on the high frequency band sound quality in live streaming scenarios), resulting in low feature discrimination and a quality inspection misjudgment rate of over 15%, making it difficult to meet the accurate quality inspection needs in complex end-side environments. (3) Edge devices typically need to run multiple tasks simultaneously, such as audio and video acquisition, network transmission, and local storage. However, existing feature extraction technologies have high memory and storage requirements: traditional PCA algorithms require storing all historical feature data, with single-device storage exceeding 100MB; feature vectors use floating-point storage, with a single 16-dimensional feature vector occupying over 128 bytes, which can easily lead to edge storage overflow during long-term operation; more importantly, the algorithm does not have a dynamic resource adaptation mechanism. When the edge device's memory usage exceeds 80% or the battery level is below 20%, it still maintains a high resource consumption mode, which can easily cause device lag, interruption of quality inspection tasks, and even affect core functions such as audio and video acquisition, resulting in insufficient practicality. When the lightweight acoustic feature extraction technology for front-end audio and video quality inspection has obvious shortcomings in terms of computing power adaptation, robustness, real-time performance, scenario versatility, and resource consumption, there is an urgent need for an improved feature extraction method that can balance lightweight and accuracy, adapt to complex edge environments, and meet real-time quality inspection requirements, in order to fill the existing technological gap and promote the application of edge audio and video quality inspection technology. Summary of the Invention

[0003] To address the aforementioned problems in the existing technology, this invention provides a lightweight acoustic feature extraction method for end-side audio and video quality inspection.

[0004] The objective of this invention can be achieved through the following technical solutions: S1: The end-side device separates audio and video stream data using a dual-time anchor point alignment method. It adds audio sampling point anchors and video frame pixel anchors on the basis of audio and video frame timestamps to collect independent audio data. S2: Perform hierarchical preprocessing on the independent audio data. Based on the real-time computing power of the CPU obtained by the end device through the real-time computing power detection submodule, select an adaptive filtering algorithm according to the real-time computing power to obtain the noise-reduced independent audio data. Then, perform sliding window and fixed segment fusion segmentation processing on the noise-reduced independent audio data to obtain segmented audio without fragmented signals. S3: Based on the segmented audio, a time-domain and frequency-domain collaborative extraction algorithm is adopted. The short-time energy algorithm is used to correct the intra-frame energy gradient and obtain multi-dimensional time-domain features. The Mel frequency cepstral coefficient algorithm is used to calculate the spectral centroid and spectral bandwidth to obtain frequency features that are sensitive to audio quality differences. The multi-dimensional time-domain features and frequency features are subjected to incremental principal component analysis to obtain lightweight acoustic feature vectors. S4: Based on the lightweight acoustic feature vector, a dual-threshold matching judgment mode is used to calculate the Euclidean distance between the feature vector and the quality inspection standard feature library to obtain the similarity between the features; based on the feature similarity, the feature dimension deviation is checked to obtain the quality inspection result containing the compliance level and feature anomaly location.

[0005] As a preferred technical solution of the present invention, the step of separating the pixel anchor points of audio and video frames by dual temporal anchor point alignment includes: using the inherent timestamps of the audio and video streams as base anchor points, and obtaining an initially aligned audio and video stream through linear interpolation; adding sampling point feature anchor points to the audio-specific anchor point module layer based on the audio and video streams, extracting sampling points at fixed intervals, and calibrating the audio stream; and adding pixel feature anchor points to the video stream based on the audio and video streams to obtain a corrected video stream.

[0006] Specifically, the sliding window and fixed segment fusion segmentation method includes: dynamically determining the sliding window parameters based on the sampling rate of the noise-reduced independent audio data; performing sliding detection on the noise-reduced audio data and calculating the average energy of the audio signal within each sliding window; using a dynamic threshold calibration mechanism, comparing the average energy of each sliding window with the average energy of the silent segment; setting sliding windows with an average energy greater than or equal to a fixed value as valid signal intervals, and marking sliding windows with an average energy lower than a fixed value as invalid silent intervals.

[0007] Specifically, the method for selecting an adaptive filtering algorithm based on real-time computing power is as follows: using the lightweight computing power detection submodule built into the end-side device, the computing power status is classified into levels using the fuzzy comprehensive evaluation method; based on the computing power classification results, an adaptive filtering algorithm is applied to the corresponding computing power devices. In the redundant state, an adaptive wavelet threshold filtering algorithm is used to decompose the audio signal into wavelet coefficients of different frequency scales to suppress noise in layers. In the balanced state, a Kalman filtering algorithm is used to update the gain parameter in real time according to the noise intensity of the audio signal to suppress mid- and low-frequency noise. Median filtering is applied to the noise-dense region to suppress the high-frequency noise region. In the loaded state, a lightweight mean and median hybrid filtering algorithm is used to calculate the short-time energy variance of the audio frame and identify the noise-dense region.

[0008] Specifically, the identification of noise-dense regions includes: dividing the preprocessed audio signal into multiple consecutive audio frames within a fixed time window, using an intra-frame energy gradient correction mechanism to reduce burst signal interference, calculating the corrected short-time energy variance of the audio frames, and comparing the short-time energy variance with a preset noise energy variance threshold to obtain the location result of the noise-dense region.

[0009] Specifically, the method for incremental principal component analysis of the multi-dimensional time-domain features and frequency features includes: setting a target dimension for lightweight feature vectors based on the quality inspection requirements of batch audio features, and obtaining principal components that retain the core information of the audio features through a principal component contribution rate threshold; if new audio feature samples are added, dynamically adjusting the audio principal component matrix using an incremental update mechanism, and reflecting the latest audio feature distribution based on the audio principal component matrix; projecting the multi-dimensional audio feature vector after the fusion of the time and frequency domains onto the final principal component matrix to generate lightweight acoustic feature vectors.

[0010] Specifically, the dynamic adjustment of the audio principal component matrix includes: a lightweight update based on the statistical information of newly added audio feature samples and on the reuse of historical principal component parameters, and the resulting principal component matrix is ​​used to reflect the real-time audio feature distribution.

[0011] Specifically, the processing of frequency features sensitive to audio quality differences includes: pre-emphasizing the complete segmented audio using the Mel frequency cepstral coefficient algorithm, obtaining high-frequency signal energy using a pre-emphasis filter, and compensating for high-frequency signal attenuation; applying a Hanning window to suppress spectral leakage, and performing a fast Fourier transform on the windowed frame signal to obtain spectral amplitude values; and using a frequency band-differentiated filter bank design based on the spectral amplitude values ​​to achieve a balance between lightweight audio data and high recognizability.

[0012] Specifically, the balance between lightweighting and high recognizability of the audio data includes: designing a time-domain window function based on the frame signal characteristics of the complete segmented audio, wherein the length and shape of the window function are dynamically adjusted according to the frequency distribution characteristics of the frame signal; a short window is used for frame signals with a high proportion of high-frequency components to improve time resolution, and a long window is used for frame signals with a high proportion of low-frequency components to ensure frequency resolution; simultaneously, a weighted attenuation coefficient is introduced into the window function design to make the attenuation rate of the signal at the window edge match the energy change gradient of the signal within the frame, further suppressing spectral leakage during the fast Fourier transform process; based on the window function processing, in the filter bank design stage, by screening the frequency bands corresponding to the quality inspection and features, the characteristic frequency band information for defect identification is obtained, resulting in a design that meets the lightweighting requirements for end-side audio and video quality inspection computation complexity, and can obtain the characteristic frequency bands and detailed features of the audio signal corresponding to the quality inspection defects.

[0013] Specifically, the dual-threshold matching judgment mode includes: the end device retrieves the standard feature vector set of the corresponding scene from the local quality inspection standard feature library according to the scene tag of the audio and video to be inspected, calculates the Euclidean distance between the lightweight feature vector and the scene standard vector, and generates a first-level threshold judgment; based on the threshold judgment, the qualified candidate features enter the second step of feature dimension deviation verification, and if they do not meet the standard, the core reason is recorded as insufficient overall similarity.

[0014] Specifically, the first-level threshold determination includes: the end-side device retrieves the standard feature vector set corresponding to the scene from the local quality inspection standard feature library based on the scene tag of the audio and video to be inspected, performs real-time update and verification of the retrieved standard feature vector set, calculates the feature deviation degree between the newly added qualified samples and the original set, and performs preliminary screening of the audio and video samples.

[0015] Specifically, the feature dimension deviation verification includes: using a weighted grading mechanism, the feature dimension deviation verification labels the deviation rate, standard value, and candidate value of each abnormal dimension, and locates the original audio features corresponding to the abnormal signal based on the abnormal dimension.

[0016] The beneficial effects of this invention are as follows: It achieves precise alignment of audio and video streams and independent audio data acquisition through a dual-time-anchor alignment method, effectively avoiding signal distortion caused by timestamp asynchrony. Combined with a hierarchical preprocessing and adaptive filtering algorithm selection mechanism, the processing strategy is dynamically adjusted according to the device's real-time computing power, significantly reducing computational resource consumption while ensuring noise reduction effectiveness, thus meeting the stringent requirements of edge devices for low latency and real-time processing.

[0017] A time-domain and frequency-domain collaborative feature extraction algorithm, combined with short-time energy correction and Mel-frequency cepstral coefficient analysis, is employed to retain key frequency domain information sensitive to audio quality differences while compressing high-dimensional features through incremental principal component analysis, generating lightweight acoustic feature vectors. This process reduces storage and computational overhead while ensuring high feature discriminability in quality inspection tasks.

[0018] Based on a dual-threshold matching judgment mode, this method achieves refined evaluation of audio quality through Euclidean distance calculation and feature dimension deviation verification. This method not only outputs the overall compliance level but also accurately locates abnormal feature dimensions and their corresponding original signal segments, improving the interpretability of the quality inspection results and the targeted nature of subsequent processing. The entire method is designed to fully consider the resource constraints and environmental diversity of end-devices. For example, it avoids signal fragmentation through a sliding window and fixed-segment fusion segmentation method; it utilizes an adaptive filtering algorithm to handle different noise scenarios; and it employs an incremental update mechanism to dynamically optimize the feature extraction model, enabling it to continuously adapt to new audio feature distributions without relying on the cloud.

[0019] By employing lightweight algorithm design techniques such as IPCA, dynamic window function adjustment, and filter bank frequency band selection, the system maintains its scalability while reducing computational complexity. It supports real-time updates of the local feature library and rapid adaptation to new scenarios, facilitating deployment and widespread application in resource-constrained IoT and edge computing environments.

[0020] A lightweight acoustic feature extraction method for edge-side audio and video quality inspection achieves efficient acoustic feature extraction and audio and video quality inspection under limited computing power. It combines real-time performance, lightweight design, high accuracy, and adaptability, and is suitable for edge-side application scenarios such as mobile devices and embedded systems. Attached Figure Description

[0021] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating a lightweight acoustic feature extraction method for end-side audio and video quality inspection according to the present invention. Figure 2 This is a structural block diagram of the lightweight algorithm in this invention; Detailed Implementation

[0023] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the specific embodiments, structures, features, and effects of the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments: Please see Figure 1-2 A lightweight acoustic feature extraction method for end-side audio and video quality inspection.

[0024] S1: The end-side device separates audio and video stream data using a dual-time anchor point alignment method. It adds audio sampling point anchors and video frame pixel anchors on the basis of audio and video frame timestamps to collect independent audio data. S2: Perform hierarchical preprocessing on the independent audio data. Based on the real-time computing power of the CPU obtained by the end device through the real-time computing power detection submodule, select an adaptive filtering algorithm according to the real-time computing power to obtain the noise-reduced independent audio data. Then, perform sliding window and fixed segment fusion segmentation processing on the noise-reduced independent audio data to obtain segmented audio without fragmented signals. S3: Based on the segmented audio, a time-domain and frequency-domain collaborative extraction algorithm is adopted. The short-time energy algorithm is used to correct the intra-frame energy gradient and obtain multi-dimensional time-domain features. The Mel frequency cepstral coefficient algorithm is used to calculate the spectral centroid and spectral bandwidth to obtain frequency features that are sensitive to audio quality differences. The multi-dimensional time-domain features and frequency features are subjected to incremental principal component analysis to obtain lightweight acoustic feature vectors. S4: Based on the lightweight acoustic feature vector, a dual-threshold matching judgment mode is used to calculate the Euclidean distance between the feature vector and the quality inspection standard feature library to obtain the similarity between the features; based on the feature similarity, the feature dimension deviation is checked to obtain the quality inspection result containing the compliance level and feature anomaly location.

[0025] As a preferred technical solution of the present invention, the step of separating the pixel anchor points of audio and video frames by dual temporal anchor point alignment includes: using the inherent timestamps of the audio and video streams as base anchor points, and obtaining an initially aligned audio and video stream through linear interpolation; adding sampling point feature anchor points to the audio-specific anchor point module layer based on the audio and video streams, extracting sampling points at fixed intervals, and calibrating the audio stream; and adding pixel feature anchor points to the video stream based on the audio and video streams to obtain a corrected video stream.

[0026] Specifically, the sliding window and fixed segment fusion segmentation method includes: dynamically determining the sliding window parameters based on the sampling rate of the noise-reduced independent audio data; performing sliding detection on the noise-reduced audio data and calculating the average energy of the audio signal within each sliding window; using a dynamic threshold calibration mechanism, comparing the average energy of each sliding window with the average energy of the silent segment; setting sliding windows with an average energy greater than or equal to a fixed value as valid signal intervals, and marking sliding windows with an average energy lower than a fixed value as invalid silent intervals.

[0027] In this embodiment, taking a laboratory hazardous chemical leak monitoring scenario as an example, an independent audio sensor with a 16kHz sampling rate is used to collect ambient sound. Real-time detection of the leaking sound wave is achieved through a segmentation method that combines a sliding window and a fixed segment. The specific dynamic threshold calibration process is as follows: Sampling rate: 16000Hz (i.e., 16000 sample points are collected per second).

[0028] Sliding window parameters: window length 256 sample points (corresponding to a duration of 16ms), step size 64 sample points (adjacent windows overlap by 192 sample points).

[0029] Fixed segment selection: Take the first 200 sliding windows (covering the initial 3.2-second silence segment) as the silence reference segment.

[0030] Energy calculations were performed on the audio data from the first 200 sliding windows (numbered 0-199): The energy formula for each window is as follows: Where N=256, This represents the nth sample value of the i-th window.

[0031] The average energy of the silent segment is: Fixed threshold setting: Dynamic calibration rule: For every 100 new sliding windows, the average energy of the silent segment is recalculated and the value is updated. Real-time detection is performed on subsequent sliding windows (numbered 200-499): Example window 200 detection process: Sample values ​​within the window: [0.15, -0.08, 0.20, ...] (simulated leakage sound waves) Calculate energy: Compare: Then mark the valid signal range (leakage acoustic wave detection trigger).

[0032] Specifically, the method for selecting an adaptive filtering algorithm based on real-time computing power is as follows: using the lightweight computing power detection submodule built into the end-side device, the computing power status is classified into levels using the fuzzy comprehensive evaluation method; based on the computing power classification results, an adaptive filtering algorithm is applied to the corresponding computing power devices. Under high redundancy conditions, an adaptive wavelet threshold filtering algorithm is used to decompose the audio signal into wavelet coefficients of different frequency scales to suppress noise in layers. Under balanced conditions, a Kalman filtering algorithm is used to update the gain parameter in real time according to the noise intensity of the audio signal to suppress mid- and low-frequency noise. Median filtering is applied to the noise-dense regions to suppress high-frequency noise regions. Under high load conditions, a lightweight mean and median hybrid filtering algorithm is used to calculate the short-time energy variance of the audio frame and identify noise-dense regions.

[0033] In this embodiment, the fuzzy comprehensive evaluation method is used as the input: CPU / GPU computing power data monitored in real time by the lightweight computing power detection submodule built into the end device; the output: computing power status classification (high redundancy, balance, high load) classification standard: high redundancy: available computing power > 70% Balance: 30% ≤ available computing power ≤ 70% High load: available computing power < 30%; the adaptive filtering strategy matching dynamically selects the filtering algorithm according to the computing power classification result: (1) High redundancy state to adaptive wavelet threshold filtering: the audio signal is transformed by wavelet and decomposed into wavelet coefficients of different frequency scales; according to the noise energy distribution, the high frequency wavelet coefficients are processed by soft threshold (suppressing random noise); the low frequency wavelet coefficients are compressed by amplitude (suppressing steady-state noise); the signal is reconstructed and the energy characteristics of the speech segment are preserved; multi-scale noise suppression is suitable for end devices with sufficient computing power. The formula of the lightweight mean and median mixed filtering algorithm is as follows: the input signal sequence is x[n], the filter window length is L=2k+1 (odd number), and the output signal is y[n]. The mixed filtering formula is: Where μ[n] is the local mean, defined as the arithmetic mean of the samples within the window, med[n] is the local median, defined as the median value of the samples within the window after sorting, and α∈[0,1] is the dynamic weighting coefficient, which is adaptively adjusted according to the local characteristics of the signal.

[0034] Specifically, the identification of noise-dense regions includes: dividing the preprocessed audio signal into multiple consecutive audio frames within a fixed time window, using an intra-frame energy gradient correction mechanism to reduce burst signal interference, calculating the corrected short-time energy variance of the audio frames, and comparing the short-time energy variance with a preset noise energy variance threshold to obtain the location result of the noise-dense region.

[0035] In this embodiment, a 10-second raw audio signal is acquired with a sampling rate of 16 kHz and a quantization bit depth of 16 bits. The signal is recorded in a quiet environment but contains occasional knocking sounds (sudden interference) and continuous background noise (such as fan noise). A first-order high-pass filter is used to boost the high-frequency components, and the transfer function is: Where z -1 The delay operator is a unit.

[0036] The preprocessed signal is denoted as x[n], with a length of N=160000 sampling points.

[0037] The preprocessed signal is divided into consecutive audio frames, with the following parameters: Frame length: 25 ms (i.e., 0.025 × 16000 = 400 dots) Frame shift: 10 ms (i.e., 0.01 × 16000 = 160 points) Total frames: Where frame_length is the length of a single frame (sampling points), and frame_shift is the frame shift (sampling points). The signal of the m-th frame is represented as: Calculate the raw short-time energy for each frame: Define intra-frame subframes (each subframe consists of 50 points, for a total of 8 subframes), and calculate the energy difference between adjacent subframes: Energy gradient: in, Energy gradient of frame m (the degree of drastic energy change within the frame). Local energy of the k-th subframe of the m-th frame; : Local energy of the m-th frame and the k-1-th subframe.

[0038] like > (tTg is a preset gradient threshold, such as Tg=5×) If the noise energy variance is too high, it is considered to be a sudden interference, and the energy of that frame is suppressed and corrected; otherwise, the original energy is maintained. Similarly, a preset noise energy variance threshold is set, and the short-time energy variance of each frame is compared with the threshold. If the variance is greater than the threshold, the frame is marked as a noise-dense frame.

[0039] Specifically, the method for incremental principal component analysis of the multi-dimensional time-domain features and frequency features includes: setting a target dimension for lightweight feature vectors based on the quality inspection requirements of batch audio features, and obtaining principal components that retain the core information of the audio features through a principal component contribution rate threshold; if new audio feature samples are added, dynamically adjusting the audio principal component matrix using an incremental update mechanism, and reflecting the latest audio feature distribution based on the audio principal component matrix; projecting the multi-dimensional audio feature vector after the fusion of the time and frequency domains onto the final principal component matrix to generate lightweight acoustic feature vectors.

[0040] In this embodiment, the target dimension k of the lightweight feature vector is determined (k is much smaller than the original feature dimension d). The selection criterion is the principal component contribution rate (variance explained rate), where each principal component captures a proportion of the original data variance. Typically, a cumulative contribution rate threshold is set (e.g., 85% or 95%). The algorithm selects a minimum number of k principal components whose cumulative contribution rate exceeds this threshold, ensuring that most of the core information is retained after dimensionality reduction. This achieves a balance between information preservation and dimensionality reduction effectiveness. IPCA introduces incremental updates into streaming data; its incremental update is specifically a global value-added update, and its formula is: in The new mean after adding new samples is the original mean, μ, which was calculated based on the first n samples. The newly added samples are... Where n is the original number of samples. When a new audio feature sample is added, the mean is updated. The deviation of the new sample from the new mean is adjusted by matrix increment to change the covariance matrix, avoiding a full recalculation. Based on the new covariance matrix, an incremental eigenvalue decomposition algorithm (such as the power iteration method) is used to quickly update the top k principal component vectors, forming a new projection matrix. This allows the principal component matrix to dynamically reflect the latest audio feature distribution. The high-dimensional audio feature vector x, obtained by fusing the time and frequency domains, is projected onto the final principal component matrix W (with dimensions d × k). Projection calculation: y = W^T* (x - μ), where μ is the currently estimated global mean vector, and the resulting y is the k-dimensional lightweight acoustic feature vector.

[0041] Specifically, the dynamic adjustment of the audio principal component matrix includes: a lightweight update based on the statistical information of newly added audio feature samples and on the reuse of historical principal component parameters, and the resulting principal component matrix is ​​used to reflect the real-time audio feature distribution.

[0042] Specifically, the processing of frequency features sensitive to audio quality differences includes: pre-emphasizing the complete segmented audio using the Mel frequency cepstral coefficient algorithm, obtaining high-frequency signal energy using a pre-emphasis filter, and compensating for high-frequency signal attenuation; applying a Hanning window to suppress spectral leakage, and performing a fast Fourier transform on the windowed frame signal to obtain spectral amplitude values; and using a frequency band-differentiated filter bank design based on the spectral amplitude values ​​to achieve a balance between lightweight audio data and high recognizability.

[0043] In this embodiment, the Mel filter bank is a set of overlapping triangular bandpass filters. It has high resolution in the low-frequency range (narrow filter width) and low resolution in the high-frequency range (wide filter width), which perfectly simulates the nonlinear perception of frequency by the human ear (Mel scale). The linear spectrum obtained in the previous step is passed through the Mel filter bank. The output of each filter is the sum of the spectral energy within that frequency band. This step compresses the high-dimensional linear spectrum to the lower-dimensional Mel spectrum, achieving the first lightweighting. DCT is performed on the logarithmically reduced Mel spectrum to obtain the Mel frequency cepstral coefficients. DCT can concentrate the energy of the signal on the first few coefficients. These low-order coefficients contain the envelope information of the sound (related to the shape of the vocal organs and are key to distinguishing the essence of the sound), while the high-order coefficients represent more details and noise. Usually, only the first 12-13 coefficients are taken to complete the second lightweighting and recognition improvement.

[0044] Specifically, the balance between lightweighting and high recognizability of the audio data includes: designing a time-domain window function based on the frame signal characteristics of the complete segmented audio, wherein the length and shape of the window function are dynamically adjusted according to the frequency distribution characteristics of the frame signal; a short window is used for frame signals with a high proportion of high-frequency components to improve time resolution, and a long window is used for frame signals with a high proportion of low-frequency components to ensure frequency resolution; simultaneously, a weighted attenuation coefficient is introduced into the window function design to make the attenuation rate of the signal at the window edge match the energy change gradient of the signal within the frame, further suppressing spectral leakage during the fast Fourier transform process; based on the window function processing, in the filter bank design stage, by screening the frequency bands corresponding to the quality inspection and features, the characteristic frequency band information for defect identification is obtained, resulting in a design that meets the lightweighting requirements for end-side audio and video quality inspection computation complexity, and can obtain the characteristic frequency bands and detailed features of the audio signal corresponding to the quality inspection defects.

[0045] In this embodiment, in the smart home terminal voice command recognition scenario, it is necessary to respond to commands such as "turn on the lights" in the 300Hz-3.4kHz frequency range with a duration of 500ms-1s, while simultaneously meeting the requirements of single command processing time ≤50ms, memory ≤15MB, accuracy ≥99%, and noise resistance. The solution acquires audio at a sampling rate of 16kHz, segments it with a sliding window step size of 200ms and a base length of 500ms, and then cuts it into sub-signals of 15ms / frame. The core command frame and noise frame are distinguished by FFT. For the core frame, high-frequency frames (1.5kHz-3.4kHz, accounting for ≥60%) are improved in temporal resolution using a 12ms short window + 0.7 attenuation coefficient to capture instantaneous features; low-frequency frames (300Hz-1kHz, accounting for ≥60%) are improved in frequency resolution using an 18ms long window + 0.4 attenuation coefficient; and noise frames are reduced in noise using a 20ms ultra-wide window + 0.2 attenuation coefficient to effectively suppress spectral leakage.

[0046] To achieve lightweight design, the solution selects the core frequency bands for commands, simplifying the traditional 40-channel filter bank to 8 core channels, covering key frequency bands for vowels and consonants. This reduces computation by 80%, and the filtering time per frame drops from 4ms to 0.8ms while maintaining a feature retention rate of over 99.2%. Final verification shows that processing a single 500ms command requires only 38ms and 12MB of memory, making it compatible with edge devices. Under 50dB noise, the recognition accuracy of 1000 commands reaches 99.1%, with a false recognition rate of 0.9%. The confusion rate between "turn on the lights" and "turn on the air conditioner" is only 0.3%, perfectly balancing lightweight design and high recognition accuracy to meet the needs of smart home edge devices.

[0047] Specifically, the dual-threshold matching judgment mode includes: the end device retrieves the standard feature vector set of the corresponding scene from the local quality inspection standard feature library according to the scene tag of the audio and video to be inspected, calculates the Euclidean distance between the lightweight feature vector and the scene standard vector, and generates a first-level threshold judgment; based on the threshold judgment, the qualified candidate features enter the second step of feature dimension deviation verification, and if they do not meet the standard, the core reason is recorded as insufficient overall similarity.

[0048] Specifically, the first-level threshold determination includes: the end-side device retrieves the standard feature vector set corresponding to the scene from the local quality inspection standard feature library based on the scene tag of the audio and video to be inspected, performs real-time update and verification of the retrieved standard feature vector set, calculates the feature deviation degree between the newly added qualified samples and the original set, and performs preliminary screening of the audio and video samples.

[0049] In this embodiment, upon power-up, the edge device immediately maps the quality inspection standard feature library from the read-only partition to memory, pre-stores three baseline vectors for each business scenario, clustered from a massive number of clean samples, and configures a circular buffer to continuously store the latest qualified sample features that have passed the secondary verification. When recording ends, and the scene classification model assigns scene labels to the audio and video with sufficient confidence, the device immediately extracts the corresponding vector set based on the labels, decompresses the baseline and newly added vectors to the cache, and initiates a drift check: a single-instruction, multi-data pipeline quickly calculates the centroid of the new sample cluster and the average distance within the cluster, compares the overall drift with the old baseline, and if it exceeds the update threshold, it immediately... The system re-clusters and writes back to the standard library to achieve a small-step, rapid self-refresh of the standard model; otherwise, it uses the latest centroid. Subsequently, the system forms a dynamic first-level threshold by superimposing the average distance within the cluster of the circular buffer with an empirical loose band. The system calculates the Euclidean distance between the high-dimensional feature vector extracted by the lightweight network and the reference center of the segment to be inspected. If the distance is greater than the threshold, it records the overall similarity as insufficient as the core reason and terminates the operation. If it is less than the threshold, it is marked as a first-level candidate and enters the dimensionality deviation verification. No external network is required throughout the process. The threshold expands and contracts in real time with the dispersion of qualified clusters, taking into account environmental changes and quality degradation. This builds an efficient, energy-saving, and adaptive overall similarity screening defense for edge quality inspection.

[0050] Specifically, the feature dimension deviation verification includes: using a weighted grading mechanism, the feature dimension deviation verification labels the deviation rate, standard value, and candidate value of each abnormal dimension, and locates the original audio features corresponding to the abnormal signal based on the abnormal dimension.

[0051] In this embodiment, upon receiving the first-level candidate vector, the edge device immediately initiates dimensional deviation verification. The system first retrieves the "golden vector" from the same scene as a comparison benchmark and pre-assigns higher weights to sensitive dimensions such as the visual low-frequency region and the audio high-frequency region. The verification module calculates the degree of deviation between the candidate value and the standard value dimension by dimension. Once a dimension shows a significant deviation and has a high weight, it is marked as an abnormal dimension. Simultaneously, the deviation rate, the standard value, and the candidate value are recorded to form a clear abnormal triplet.

[0052] In this segment of the conference recording, verification revealed a significant shift in the dimension located in the high-frequency harmonic band, and the system immediately highlighted this dimension in red. Subsequently, using a preset dimension-original feature mapping table, the abnormal dimension was located in reverse to a time-frequency region of the log-Mel spectrum. The device extracted the corresponding waveform segment, and after bandpass filtering, the transient noise of "key collision" could be clearly heard.

[0053] The quality inspection report only sends back a one-sentence description: the abnormal dimension is located in the high-frequency harmonic band, with a significant deviation. The original position corresponds to a brief knocking sound at the end of two seconds. It can accurately locate and annotate the cause without uploading the complete audio, which protects privacy and saves data.

[0054] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A lightweight acoustic feature extraction method for end-side audio and video quality inspection, characterized in that, include: S1: The end-side device separates audio and video stream data using a dual-time anchor point alignment method. It adds audio sampling point anchors and video frame pixel anchors on the basis of audio and video frame timestamps to collect independent audio data. S2: Perform hierarchical preprocessing on the independent audio data. Based on the real-time computing power of the CPU obtained by the end device through the real-time computing power detection submodule, select an adaptive filtering algorithm according to the real-time computing power to obtain the noise-reduced independent audio data. Then, perform sliding window and fixed segment fusion segmentation processing on the noise-reduced independent audio data to obtain segmented audio without fragmented signals. S3: Based on the segmented audio, a time-domain and frequency-domain collaborative extraction algorithm is adopted. The short-time energy algorithm is used to correct the intra-frame energy gradient and obtain multi-dimensional time-domain features. The Mel frequency cepstral coefficient algorithm is used to calculate the spectral centroid and spectral bandwidth to obtain frequency features that are sensitive to audio quality differences. The multi-dimensional time-domain features and frequency features are subjected to incremental principal component analysis to obtain lightweight acoustic feature vectors. S4: Based on the lightweight acoustic feature vector, a dual-threshold matching judgment mode is used to calculate the Euclidean distance between the feature vector and the quality inspection standard feature library to obtain the similarity between the features; based on the feature similarity, the feature dimension deviation is checked to obtain the quality inspection result containing the compliance level and feature anomaly location.

2. The method according to claim 1, characterized in that, The dual-time-series anchor point alignment method for separating audio and video frame pixel anchor points includes: using the inherent timestamps of the audio and video streams as base anchor points, and obtaining initially aligned audio and video streams through linear interpolation; adding sampling point feature anchor points to the audio-specific anchor point module layer based on the audio and video streams, extracting sampling points at fixed intervals, and calibrating the audio stream; and adding pixel feature anchor points to the video stream based on the audio and video streams to obtain a corrected video stream.

3. The method according to claim 1, characterized in that, The sliding window and fixed segment fusion segmentation method includes: dynamically determining the sliding window parameters based on the sampling rate of the noise-reduced independent audio data; performing sliding detection on the noise-reduced audio data and calculating the average energy of the audio signal in each sliding window; using a dynamic threshold calibration mechanism, comparing the average energy of each sliding window with the average energy of the silent segment; setting the sliding window with an average energy greater than or equal to a fixed value as an effective signal interval, and marking the sliding window with an average energy lower than a fixed value as an invalid silent interval.

4. The method according to claim 1, characterized in that, The method for selecting an adaptive filtering algorithm based on real-time computing power is as follows: using the lightweight computing power detection submodule built into the end-side device, the computing power status is classified into levels using the fuzzy comprehensive evaluation method; based on the computing power classification results, an adaptive filtering algorithm is applied to the corresponding computing power device. In the redundant state, an adaptive wavelet threshold filtering algorithm is used to decompose the audio signal into wavelet coefficients of different frequency scales and suppress noise in a layered manner. In the balanced state, the Kalman filter algorithm is used to update the gain parameters in real time according to the noise intensity of the audio signal to suppress mid- and low-frequency noise. Median filtering is applied to the noise-dense regions to suppress high-frequency noise regions; Under load, a lightweight mean and median hybrid filtering algorithm is used to calculate the short-time energy variance of audio frames and identify areas of dense noise.

5. The method according to claim 4, characterized in that, The identification of noise-dense regions includes: dividing the preprocessed audio signal into multiple consecutive audio frames within a fixed time window; using an intra-frame energy gradient correction mechanism to reduce burst signal interference; calculating the corrected short-time energy variance of the audio frames; and comparing the short-time energy variance with a preset noise energy variance threshold to obtain the location result of the noise-dense region.

6. The method according to claim 1, characterized in that, The method for incremental principal component analysis of multi-dimensional time-domain features and frequency features includes: setting a target dimension for lightweight feature vectors based on the quality inspection requirements of batch audio features, and obtaining principal components that retain the core information of the audio features by using a principal component contribution rate threshold; if new audio feature samples are added, dynamically adjusting the audio principal component matrix using an incremental update mechanism, and reflecting the latest audio feature distribution based on the audio principal component matrix; and projecting the multi-dimensional audio feature vectors after the fusion of time and frequency domains onto the final principal component matrix to generate lightweight acoustic feature vectors.

7. The method according to claim 6, characterized in that, The dynamic adjustment of the audio principal component matrix includes: a lightweight update based on the statistical information of newly added audio feature samples and on the reuse of historical principal component parameters, and the resulting principal component matrix is ​​used to reflect the real-time audio feature distribution.

8. The method according to claim 1, characterized in that, The processing of frequency features sensitive to audio quality differences includes: pre-emphasizing the complete segmented audio using the Mel frequency cepstral coefficient algorithm, obtaining high-frequency signal energy using a pre-emphasis filter, and compensating for high-frequency signal attenuation; applying a Hanning window to suppress spectral leakage, and performing a fast Fourier transform on the windowed frame signal to obtain the spectral amplitude value; and using a frequency band-differentiated filter bank design based on the spectral amplitude value to achieve a balance between lightweight audio data and high recognizability.

9. The method according to claim 8, characterized in that, The balance between lightweighting and high recognizability of the audio data includes: designing a time-domain window function based on the frame signal characteristics of the complete segmented audio, wherein the length and shape of the window function are dynamically adjusted according to the frequency distribution characteristics of the frame signal. For frame signals with a high proportion of high-frequency components, a short window is used to improve time resolution, and for frame signals with a high proportion of low-frequency components, a long window is used to ensure frequency resolution. At the same time, a weighted attenuation coefficient is introduced into the window function design to make the attenuation rate of the signal at the window edge adapt to the energy change gradient of the signal within the frame, further suppressing spectral leakage during the fast Fourier transform process. Based on the window function processing, in the filter bank design stage, by screening the frequency bands corresponding to the quality inspection and features, the characteristic frequency band information for defect identification is obtained. This results in a design that meets the lightweighting requirements for end-side audio and video quality inspection computation complexity, and can obtain the characteristic frequency bands and detailed features of the audio signal corresponding to the quality inspection defects.

10. The method according to claim 1, characterized in that, The dual-threshold matching judgment mode includes: the end device retrieves the standard feature vector set of the corresponding scene from the local quality inspection standard feature library according to the scene label of the audio and video to be inspected, calculates the Euclidean distance between the lightweight feature vector and the scene standard vector, and generates a first-level threshold judgment; based on the threshold judgment, the qualified candidate features enter the second step of feature dimension deviation verification, and if they do not meet the standard, the core reason is recorded as insufficient overall similarity.

11. The method according to claim 10, characterized in that, The first-level threshold determination includes: the end-side device retrieves the standard feature vector set corresponding to the scene from the local quality inspection standard feature library according to the scene tag of the audio and video to be inspected, updates and verifies the retrieved standard feature vector set in real time, calculates the feature deviation degree between the newly added qualified samples and the original set, and performs preliminary screening of the audio and video samples.

12. The method according to claim 1, characterized in that, The feature dimension deviation verification includes: using a weighted grading mechanism, the feature dimension deviation verification labels the deviation rate, standard value and candidate value of each abnormal dimension, and locates the original audio features corresponding to the abnormal signal based on the abnormal dimension.

Images