A real-time audio feature extraction method and system based on sliding window

Abstract

This invention provides a real-time audio feature extraction method and system based on a sliding window, belonging to the field of audio feature extraction technology. The method includes: determining the sliding step size for the current feature extraction cycle based on the audio content characteristics of the latest frame of data in the audio buffer; performing a cache index alignment operation to query whether there are reusable features corresponding to overlapping data segments in a preset feature cache; calculating newly added features; generating complete spectral features for the current processing window by reusing reusable features and combining them with the newly added features; generating an audio feature vector; updating the audio buffer and feature cache; and completing the real-time audio feature extraction for the current cycle. This invention achieves adaptive adjustment of the processing frequency by dynamically determining the sliding step size based on audio content characteristics, effectively balancing real-time performance and feature quality; and significantly improves computational efficiency through cache index alignment and feature reuse mechanisms.

Description

Technical Field

[0001] This invention relates to the field of audio feature extraction technology, specifically to a real-time audio feature extraction method and system based on a sliding window. Background Technology

[0002] In audio processing technologies such as digital human-driven systems, real-time speech recognition, and intelligent interaction systems, real-time audio feature extraction is a core technology for achieving high-quality audio-video synchronization and instant voice interaction. Currently, mainstream technologies in this field are generally based on the sliding window mechanism. Specifically, a continuously input audio stream is segmented using a pre-set analysis window of fixed duration. The feature extraction process is triggered whenever the accumulated data in the audio buffer reaches the window size. This process typically includes: preprocessing the raw audio data within the window, extracting frequency domain features, and finally inputting the feature sequence into a pre-trained deep learning model for encoding to generate feature vectors representing the acoustic content. To maintain processing continuity, the window slides with a fixed step size, thus enabling segment-by-segment processing of the audio stream. This type of fixed-parameter sliding window method provides a basic framework for real-time processing, but its inherent limitations are becoming increasingly apparent in practice. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time audio feature extraction method and system based on a sliding window, which dynamically determines the sliding step size by analyzing audio characteristics and reuses historically calculated features using a caching mechanism, thereby reducing processing latency and computational overhead while ensuring feature quality.

[0004] To achieve the above objectives, this invention provides a real-time audio feature extraction method based on a sliding window, comprising: acquiring real-time audio stream data and accumulating it into a preset audio buffer; triggering audio feature extraction when the amount of audio stream data accumulated in the audio buffer equals a preset baseline window size; the audio feature extraction includes: calculating the short-time energy and zero-crossing rate of the latest frame data in the audio buffer to determine the sliding step size of the current feature extraction cycle; determining the overlapping data segment between the current processing window and the previous processing window according to the sliding step size; performing a cache index alignment operation to query whether there are reusable features corresponding to the overlapping data segment in a preset feature cache; extracting new data segments (excluding the overlapping data segment) from the audio buffer and calculating their new features based on the query results; generating complete spectral features of the current processing window by reusing the reusable features and combining them with the new features; inputting the complete spectral features into a pre-trained speech encoder to generate an audio feature vector; and updating the audio buffer and the feature cache according to the sliding step size to complete the real-time audio feature extraction of the current cycle.

[0005] Optionally, triggering audio feature extraction when the amount of audio stream data accumulated in the audio buffer is equal to a preset baseline window size includes: maintaining a preset distance between the write pointer and the read pointer of the audio buffer, the value of which is equal to the baseline window size; updating the position of the write pointer as real-time audio stream data is continuously written to the audio buffer; monitoring the actual distance between the write pointer and the read pointer in real time, and locking the corresponding data block in the audio buffer as the current processing window when the actual distance is detected to be equal to the baseline window size, and triggering audio feature extraction.

[0006] Optionally, after the audio feature extraction is triggered, the writing of new audio stream data to the audio buffer area corresponding to the current processing window is paused until the audio feature extraction of the current feature extraction cycle is completed.

[0007] Optionally, determining the sliding step size of the current feature extraction cycle includes: calculating the short-time energy and zero-crossing rate of the latest frame data; if the short-time energy is less than a first threshold, then the current audio frame is determined to be silent, and the sliding step size is determined to be a first length, which is greater than a reference step size; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is greater than a second threshold, then the current audio frame is determined to be unvoiced, and the sliding step size is determined to be a second length, which is less than the reference step size; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is less than the second threshold, then the current audio frame is determined to be voiced, and the sliding step size is determined to be a reference step size.

[0008] Optionally, the first length is an integer multiple of the reference step size, and the second length is a fractional multiple of the reference step size.

[0009] Optionally, determining the overlapping data segment between the current processing window and the previous processing window includes: calculating the length of the overlapping data segment based on the sliding step size and a preset reference window size; and determining the start and end positions of the overlapping data segment in the audio buffer based on the calculated length.

[0010] Optionally, the step of performing cache index alignment operation and querying whether there is a reusable feature corresponding to the overlapping data segment in the preset feature cache includes: calculating the expected starting index corresponding to the overlapping data segment based on the starting position of the current processing window; comparing the expected starting index with the feature starting index recorded in the feature cache; if the absolute deviation between the two is less than or equal to a preset tolerance threshold, the query result is determined to be that there is a reusable feature; otherwise, the query result is determined to be that there is no reusable feature.

[0011] Optionally, generating the complete spectral features of the current processing window by reusing the reusable features and combining them with the newly added features includes: if the query result indicates the existence of the reusable features, then executing a first path, including: reading the reusable features corresponding to the overlapping data segments from the feature cache; extracting the newly added data segments from the audio buffer and calculating the newly added features corresponding to the newly added data segments; concatenating the reusable features and the newly added features in the time dimension to generate complete spectral features; if the query result indicates the absence of the reusable features, then executing a second path, including: extracting all audio data corresponding to the current processing window from the audio buffer; calculating the complete spectral features of all the audio data as the complete spectral features of the current processing window.

[0012] Optionally, the step of inputting the complete spectral features into a pre-trained speech encoder to generate an audio feature vector includes: inputting the complete spectral features into the pre-trained speech encoder; the speech encoder is an acoustic model encoder based on the Transformer architecture, used to map the input spectral feature sequence into an acoustic feature vector sequence; the speech encoder takes the complete spectral features as input and outputs a fixed-dimensional audio feature vector, the audio feature vector being used to represent the acoustic content within the corresponding time window.

[0013] On the other hand, the present invention provides a real-time audio feature extraction system based on a sliding window, for implementing a real-time audio feature extraction method based on a sliding window. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the real-time audio feature extraction method based on a sliding window.

[0014] The above technical solution dynamically determines the sliding step size by calculating short-time energy and zero-crossing rate, achieving adaptive adjustment of processing frequency and effectively balancing real-time performance and feature quality. Through cache index alignment and feature reuse mechanisms, it significantly reduces the overhead of redundant calculations of overlapping data in adjacent windows, greatly improving computational efficiency. It ensures both low latency and efficient utilization of computing resources.

[0015] Other features and advantages of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the following detailed description to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0017] Figure 1This is a flowchart of a real-time audio feature extraction method based on a sliding window.

[0018] Figure 2 It is a dynamic sliding step decision flowchart. Detailed Implementation

[0019] The following is in conjunction with the appendix Figure 1 - Appendix Figure 2 The specific implementation methods of the embodiments of the present invention will be described in detail below. It should be understood that the specific implementation methods described herein are only for illustrating and explaining the embodiments of the present invention, and are not intended to limit the embodiments of the present invention.

[0020] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0021] In the process of realizing this invention, the inventors of this application discovered that existing real-time audio feature extraction technology based on fixed sliding windows has an inherent defect that it cannot dynamically adjust the calculation strategy according to the audio content due to the use of fixed window size and sliding step size. As a result, when processing continuous audio streams, it always faces the problem of not being able to balance low latency and high computational efficiency.

[0022] Example 1

[0023] Reference Figures 1-2 This is the first embodiment of the present invention, which provides a real-time audio feature extraction method based on a sliding window, including:

[0024] In the embodiments of this application, real-time audio stream data is collected and accumulated into a preset audio buffer. When the amount of audio stream data accumulated in the audio buffer is equal to the preset reference window size, audio feature extraction is triggered.

[0025] In a preferred embodiment of this application, a preset distance is maintained between the write pointer and the read pointer of the audio buffer, and the value of the preset distance is equal to the reference window size; as real-time audio stream data is continuously written to the audio buffer, the position of the write pointer is updated; the actual distance between the write pointer and the read pointer is monitored in real time, and when the actual distance is detected to be equal to the reference window size, the corresponding data block in the audio buffer is locked as the current processing window, and audio feature extraction is triggered.

[0026] Specifically, the audio buffer is first initialized to an empty state, while the initial positions of the write pointer WP and the read pointer RP are aligned (e.g., both pointing to the 0th position of the buffer). A preset distance D between the two pointers is established, and the value of the preset distance D is consistent with the baseline window size N (e.g., D=11040). Subsequently, the real-time audio stream continuously writes audio samples into the audio buffer. Every k audio samples written to the buffer, the position of the write pointer WP is updated synchronously to ensure that the write pointer WP always keeps pace with the data writing progress. The formula for updating the position of the write pointer WP is as follows:

[0027]

[0028] in, This indicates the current position of the write pointer after the update. This indicates the historical position of the write pointer before the update, and k represents the number of audio samples written to the audio buffer this time.

[0029] Furthermore, the read pointer RP remains at its initial position during this stage, waiting for the feature extraction trigger condition to be met. During this process, the actual distance between the write pointer WP and the read pointer RP is calculated and monitored in real time. The actual distance is calculated as: Actual distance = WP - RP (Since audio samples are continuously written, the position of the write pointer WP is always greater than or equal to the position of the read pointer RP, so the actual distance has no negative value). When the actual distance is detected to be equal to the preset distance D (i.e., WP - RP = N), it indicates that enough complete data for one audio feature extraction has accumulated in the audio buffer, at which point the audio feature extraction logic is triggered.

[0030] Furthermore, after audio feature extraction is triggered, writing new audio stream data to the audio buffer area corresponding to the current processing window is paused until the audio feature extraction of the current feature extraction cycle is completed.

[0031] Specifically, after the feature extraction logic is triggered, all data between the read pointer RP and the write pointer WP in the audio buffer is immediately locked (i.e., all data whose position satisfies RP ≤ data position < WP). This locked data block is the current processing window, and its length is equal to the preset spacing D (i.e., equal to the baseline window size N). The purpose of the locking operation is to prevent newly written audio samples from overwriting the data in the current processing window, ensuring the stability of the data during the audio feature extraction process, and thus triggering subsequent audio feature extraction-related processes such as dynamic step size decision, cache index alignment, and feature calculation.

[0032] Furthermore, once the audio feature extraction process for the current cycle is fully executed (including operations such as inputting the complete spectral features into the encoder to generate feature vectors and updating the audio buffer), the read pointer RP is updated according to the sliding step size corresponding to the current feature extraction cycle. The sliding step size position update formula is as follows:

[0033]

[0034] in, This indicates the current position of the read pointer after the update. This indicates the historical position of the pointer before the update. This represents the dynamic sliding step size determined based on short-time energy and zero-crossing rate in the current feature extraction cycle.

[0035] It should be noted that the write pointer WP is continuously updated in position throughout the entire process as audio samples are written. The overall real-time audio stream data writing process is not interrupted. Only after the audio feature extraction is triggered is the writing of new audio stream data to the audio buffer area corresponding to the current processing window paused until the audio feature extraction of the current feature extraction cycle is completed. Then, the cycle of data accumulation, spacing monitoring and triggering extraction begins again.

[0036] Audio feature extraction includes:

[0037] S100: Based on the latest frame of data in the audio buffer, calculate its short-time energy and zero-crossing rate to determine the sliding step size of the current feature extraction cycle.

[0038] In the embodiments of this application, determining the sliding step size of the current feature extraction cycle includes: calculating the short-time energy and zero-crossing rate of the latest frame data; if the short-time energy is less than a first threshold, the current audio frame is determined to be in a silent state, and the sliding step size is determined to be a first length, which is greater than a reference step size; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is greater than a second threshold, the current audio frame is determined to be in a deaf state, and the sliding step size is determined to be a second length, which is less than the reference step size; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is less than the second threshold, the current audio frame is determined to be in a voiced state, and the sliding step size is determined to be a reference step size.

[0039] It should be noted that the baseline step size is preset based on the trade-off between the target's real-time requirements and the accuracy of feature extraction. The first length is an integer multiple of the baseline step size, and the second length is a fractional multiple of the baseline step size.

[0040] In a preferred embodiment of this application, the latest frame of data is a continuous audio sample segment that has just been written into the audio buffer. Its frame length is preset according to the audio sampling rate (for example, when the sampling rate is 16kHz, the frame length is set to 20ms, corresponding to 320 audio samples). In the frame extraction process, Hanning window is used for windowing processing to reduce the impact of spectrum leakage on feature calculation.

[0041] Furthermore, the frame data is extracted, and two core acoustic features are calculated: short-time energy and zero-crossing rate. Short-time energy is calculated by summing the squares of the amplitudes of all audio samples within the frame and taking the mean. This quantitatively characterizes the energy intensity of the audio frame and is the core basis for distinguishing between silent and non-silent states. Zero-crossing rate is calculated by counting the number of times the amplitude sign changes between adjacent audio samples within the frame and normalizing it to a unit time. This characterizes the fluctuation frequency of the audio signal and is a key indicator for distinguishing between unvoiced and voiced states. Then, audio frame state stratification is determined based on a preset first threshold and a second threshold. The first threshold is obtained through statistical calibration of the environmental noise energy in the target scene (e.g., preset to 60dB in a daily office environment), and the second threshold is determined by analyzing the zero-crossing rate distribution of a large number of unvoiced and voiced samples (e.g., preset to 100Hz).

[0042] The determination rule is as follows: if the calculated short-time energy is less than the first threshold, the current audio frame is determined to be in a silent state; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is greater than the second threshold, the current audio frame is determined to be in a clear state; if the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is less than or equal to the second threshold, the current audio frame is determined to be in a voiced state (clearly defining the assignment of edge thresholds to avoid ambiguity in determination).

[0043] Furthermore, the corresponding sliding step size is matched based on the judgment result. The baseline step size needs to be preset by balancing the real-time requirements of the target scenario with the feature extraction accuracy (for example, in real-time voice interaction scenarios, a baseline step size is preset to balance low latency and basic recognition accuracy). =800 samples, corresponding to 50ms. In high-precision speech-to-text scenarios, a preset baseline step size is used. =400 samples, corresponding to 25ms), the sliding step size for the silent state is the first length, which is set to an integer multiple of the base step size (e.g., 2 times, i.e., 1600 samples, corresponding to 100ms), to speed up data processing efficiency during the silent segment and reduce unnecessary computational overhead. The sliding step size for the unresponsive state is the second length, which is set to a fractional multiple of the base step size (e.g., ... The number of samples is increased by 10 times (i.e., 400 samples, corresponding to 25ms) to improve the feature extraction density and accuracy of unvoiced segments with rich speech details. For voiced segments, the baseline step size is directly used to balance processing efficiency and feature integrity.

[0044] The above scheme achieves precise hierarchical determination of audio states by calculating the short-time energy and zero-crossing rate of the latest frame of data. Based on the determination results, it matches differentiated sliding step sizes: a large step size, an integer multiple of the baseline step size, is used for silent segments to significantly reduce unnecessary computational overhead and improve processing throughput; a small step size, a fractional multiple of the baseline step size, is used for unvoiced segments to enhance the feature extraction density and accuracy of speech details; and the baseline step size is retained for voiced segments, balancing processing efficiency and feature integrity. Furthermore, the baseline step size can be flexibly preset according to different scenarios such as real-time interaction and high-precision transcription.

[0045] S200: Determine the overlapping data segment between the current processing window and the previous processing window based on the sliding step size.

[0046] In the embodiments of this application, determining the overlapping data segment between the current processing window and the previous processing window includes: calculating the length of the overlapping data segment based on the sliding step size and the preset reference window size; and determining the start and end positions of the overlapping data segment in the audio buffer based on the calculated length.

[0047] In a preferred embodiment of this application, the core basic parameters are first clearly defined using the number of audio samples as a unified unit. These parameters include: a preset baseline window size N (i.e., the processing window length for a single feature extraction, consistent with the current processing window length, for example, 2400 samples corresponding to 150ms at a 16kHz sampling rate), and the current dynamic sliding step size. (This represents the forward sliding distance of the current processing window relative to the previous processing window, such as 1600 samples in silent mode, 800 samples in voiced mode, and 400 samples in unvoiced mode), and the window index positioning reference (the starting index of the previous processing window is denoted as...). The end index is +N, the starting index of the current processing window. = + The end index is +N).

[0048] Furthermore, the length of the overlapping data segment is calculated using the following formula:

[0049] Length of overlapping data segments = N -

[0050] Where N represents the reference window size, This indicates the current dynamic sliding step size.

[0051] At the same time, clarify the validity constraints, that is, when When <N, the length of the overlapping data segment is >0 (there is a valid overlapping region). When N ≥ N, the length of overlapping data segments is set to 0 (no overlap, and Log recording is triggered when the number of samples exceeds N to ensure robustness. For example, when N = 2400 samples... When there are 800 samples, the length of the overlapping data segment is 2400 - 800 = 1600 samples (corresponding to 100ms).

[0052] Furthermore, using the sample index of the audio buffer as a reference, the boundary of the overlapping data segment is located, with the overlap start position = = + The end position of the overlap = +N (and) +The lengths of overlapping data segments are equivalent, and the derivation logic is the intersection range of the preceding and following windows. + , +N)).

[0053] Finally, to avoid positioning errors caused by buffer overflows, pointer anomalies, etc., a validity verification step is added. This step checks whether the start and end positions of the overlap are within the valid index range of the buffer, and whether the difference between the end and start positions of the overlap matches the calculated length of the overlapped data segment. If they do not match, it is determined to be a positioning anomaly and processed according to the logic of non-overlapping data segments, thus completing the length calculation and position positioning of the overlapped data segment.

[0054] The above scheme, by calculating the range of overlapping data, provides a clear target area for subsequent feature reuse, avoiding invalid data reuse. Through explicit parameter definitions and index positioning logic, the correspondence between overlapping data segments and audio buffer data is unique, providing a precise basis for subsequent cache index alignment and ensuring the feasibility of feature reuse.

[0055] S300: Perform cache index alignment operation to query whether there is a reusable feature corresponding to the overlapping data segment in the preset feature cache area.

[0056] In the embodiments of this application, the expected starting index corresponding to the overlapping data segment is calculated based on the starting position of the current processing window; the expected starting index is compared with the feature starting index recorded in the feature cache; if the absolute deviation between the two is less than or equal to a preset tolerance threshold, the query result is determined to have a reusable feature; otherwise, the query result is determined to have no reusable feature.

[0057] In a preferred embodiment of this application, firstly, the preset structure and initial state of the feature buffer are defined. This feature buffer is used to store feature data corresponding to data segments that may overlap with subsequent windows in the previous processing window, and synchronously records the starting index of the audio buffer sample corresponding to the feature data (feature starting index). The initial state of the feature buffer is set to an invalid flag, specifically, the feature starting index can be assigned a value of (-1) to indicate that there is no valid feature data in the buffer. At the same time, a preset tolerance threshold is set. The tolerance threshold is determined based on the short-term stationarity characteristics of the audio features and is used to tolerate index deviations caused by network jitter, data transmission delay, or small pointer positioning offsets. Its unit is consistent with the reference window size and sliding step size (both are the number of audio samples), for example, it is preset to 320 samples (corresponding to 20ms) at a 16kHz sampling rate.

[0058] Secondly, the expected starting index of the overlapping data segment is determined based on the starting position of the current processing window, and its value is equal to the starting position of the current processing window. This index serves as an ideal reference benchmark for the reusable feature data corresponding to the overlapping data segment in the feature buffer, and is used to match it with the feature index stored in the feature buffer, thereby guiding the determination and invocation of feature reuse.

[0059] Furthermore, the validity of the feature cache is first determined. If the feature starting index is the aforementioned invalid identifier (-1), the feature cache is directly determined to be empty, and the query result is that there is no reusable feature. If the feature starting index is a valid sample index (not -1), the absolute deviation between the expected starting index and the feature starting index is calculated. The absolute deviation = |expected starting index - feature starting index|.

[0060] Finally, the absolute deviation is compared with a preset tolerance threshold. If the absolute deviation is less than or equal to the tolerance threshold, it indicates that the index matching relationship between the feature data stored in the feature cache and the current overlapping data segment meets the preset requirements, and the deviation is within a reasonable and controllable range. The query result is determined to be that a reusable feature exists. If the absolute deviation is greater than the tolerance threshold, it indicates that the index deviation exceeds the range allowed by the short-term stationarity of the audio feature, and the feature data stored in the feature cache does not have a valid matching relationship with the current overlapping data segment. The query result is determined to be that no reusable feature exists.

[0061] S400: Based on the query results, extract the new data segments (excluding overlapping data segments) from the audio buffer in the current processing window and calculate their new features. By reusing reusable features and combining them with the new features, generate the complete spectral features of the current processing window.

[0062] In a preferred embodiment of this application, filter bank characteristics are used as spectral characteristics (Mel frequency cepstral coefficient characteristics can also be used). The spectral characteristics are generated by performing a short-time Fourier transform (STFT) on the audio data to obtain the amplitude spectrum, which is then filtered by the filter bank and normalized to logarithmic energy.

[0063] Simultaneously, based on the determined range of overlapping data segments, the range of newly added data segments in the current processing window is defined as [overlap end position, current processing window end index] (current processing window end index = current processing window start position + baseline window size N), and its length is the current sliding step size. The reusable features stored in the feature buffer are the filter bank feature sequences corresponding to the overlapping data segments in the previous processing window.

[0064] In the embodiments of this application, if the query result indicates the existence of reusable features (filter bank features), the first path is executed, including: reading the reusable features corresponding to the overlapping data segments from the feature buffer; extracting the new data segments from the audio buffer and calculating the new features corresponding to the new data segments [according to the preset filter bank feature calculation process (frame length 20ms, frame shift 10ms, 23 Vimel filter bank)]; and concatenating the reusable features and the new features in the time dimension (the reusable features correspond to the overlapping data segments earlier in the time domain, and the new features correspond to the new data segments later in the time domain) to generate complete spectral features.

[0065] In the embodiments of this application, if the query result is that there is no reusable feature, the second path is executed, including: extracting all audio data corresponding to the current processing window from the audio buffer; and calculating the complete spectral features of all audio data according to the same feature calculation process as the first path, so as to serve as the complete spectral features of the current processing window.

[0066] The above scheme accurately determines the matching degree between cached features and the current overlapping segment by comparing the expected starting index with the feature starting index, avoiding the reuse of irrelevant features that would lead to a decrease in extraction accuracy. The preset tolerance threshold is determined based on the short-term stationarity of audio, which can tolerate common anomalies in real-time scenarios such as network jitter and slight pointer offsets, balancing reuse accuracy and robustness. Furthermore, by first verifying the validity of the feature cache (whether the feature starting index is -1) and then calculating the absolute deviation, the lightweight judgment logic avoids the latency caused by complex matching, ensuring the requirements of real-time processing. The setting of the invalidity flag clearly defines the boundary of the cache without valid data, avoiding invalid judgments of invalid data in subsequent queries and simplifying the overall logic.

[0067] S500: Input the complete spectral features into a pre-trained speech encoder to generate audio feature vectors.

[0068] In the embodiments of this application, the complete spectral features are input to a pre-trained speech encoder; the speech encoder is an acoustic model encoder based on the Transformer architecture, used to map the input spectral feature sequence into an acoustic feature vector sequence; the speech encoder takes the complete spectral features as input and outputs a fixed-dimensional audio feature vector, which is used to represent the acoustic content within the corresponding time window.

[0069] In a preferred embodiment of this application, the dimensional format of the complete spectral features is [T, D] (where T is the number of time frames, determined by the reference window size N and the frame shift parameter for feature calculation, and D is the feature dimension corresponding to the 23-Vimel filter bank). Before input, the feature sequence needs to be reshaped into a three-dimensional tensor of [1, T, D] according to the input tensor specification of the speech encoder (1 is the batch dimension, adapted to single-window inference scenarios).

[0070] Preferably, the core structure of the acoustic model encoder includes 6 encoder blocks, each of which consists of an 8-head self-attention mechanism (attention dimension 64-dimensional), layer normalization, a feedforward neural network with a hidden layer dimension of 2048-dimensional, and residual connections. Furthermore, this encoder has been trained on large-scale speech datasets such as LibriSpeech, using speech recognition as the pre-training task, and possesses the ability to deeply encode and semantically represent acoustic features.

[0071] Furthermore, mean-variance normalization (CMVN) preprocessing is performed on the complete spectral features. The preprocessed complete spectral features are then input into the speech encoder. The speech encoder captures the temporal dependencies of the feature sequence through a multi-layer self-attention mechanism. After the features undergo nonlinear transformation through a feedforward neural network, the output is an intermediate feature sequence with dimensions [1, T, 512]. Subsequently, global average pooling is performed on the temporal dimension of this intermediate feature sequence, aggregating the sequence features of [1, T, 512] into a fixed-dimensional audio feature vector of [1, 1, 512].

[0072] The above scheme uses a pre-trained speech encoder based on the Transformer architecture, which has powerful acoustic feature semantic representation capabilities and generates feature vectors with higher recognizability.

[0073] S600: Update the audio buffer and feature buffer according to the sliding step size to complete the real-time audio feature extraction of the current cycle.

[0074] In a preferred embodiment of this application, the method for updating the audio buffer is to remove the previously processed audio data from the audio buffer. One audio sample (or update the read pointer position to its original position in the circular buffer + ...) (At this point), only unprocessed audio data is retained, which both releases buffer storage resources and ensures that feature extraction can be performed on the latest unprocessed data in subsequent cycles.

[0075] In a preferred embodiment of this application, the update of the feature cache area needs to be combined with the differential processing of query results.

[0076] If the query result indicates the existence of reusable features, the tail feature frame corresponding to the overlapping data segment between the current processing window and the next processing window from the currently generated complete spectral features is stored in the feature cache. Simultaneously, the feature start index recorded in the feature cache is updated to the start position of the next processing window, providing a valid benchmark for the cache index alignment operation in the next cycle.

[0077] If the query result indicates that no reusable feature exists, the feature starting index recorded in the feature cache will be reset to an invalid flag (such as -1), and the stored feature data will be cleared to avoid interfering with feature queries in subsequent cycles.

[0078] Through the above operations, both the audio buffer and the feature buffer are updated to a ready state, which can seamlessly take over the data accumulation and feature extraction process of the next cycle.

[0079] The present invention also provides a real-time audio feature extraction system based on a sliding window, for implementing a real-time audio feature extraction method based on a sliding window. The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the real-time audio feature extraction method based on a sliding window.

[0080] This invention provides a storage medium storing a program that, when executed by a processor, implements a real-time audio feature extraction method based on a sliding window.

[0081] This invention provides a processor for running a program, wherein the program executes a real-time audio feature extraction method based on a sliding window.

[0082] This invention provides a device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements a real-time audio feature extraction method based on a sliding window. The device described herein can be a server, PC, tablet, mobile phone, etc.

[0083] This application also provides a computer program product that, when executed on a data processing device, is suitable for performing a sliding window-based real-time audio feature extraction method.

[0084] Those skilled in the art will understand that embodiments of this application can provide methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0088] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0089] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0090] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0091] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0092] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A real-time audio feature extraction method based on sliding window, characterized in that, include: Real-time audio stream data is collected and accumulated into a preset audio buffer. When the amount of audio stream data accumulated in the audio buffer is equal to the preset reference window size, audio feature extraction is triggered. The audio feature extraction includes: Based on the latest frame of data in the audio buffer, its short-time energy and zero-crossing rate are calculated to determine the sliding step size of the current feature extraction cycle. The process of determining the sliding step size for the current feature extraction cycle includes: Calculate the short-time energy and zero-crossing rate of the latest frame of data; If the short-time energy is less than the first threshold, the current audio frame is determined to be in a silent state, and the sliding step size is determined to be the first length, which is greater than the reference step size. If the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is greater than the second threshold, then the current audio frame is determined to be in a clear state, and the sliding step size is determined to be the second length, which is less than the reference step size; If the short-time energy is greater than or equal to the first threshold and the zero-crossing rate is less than the second threshold, then the current audio frame is determined to be in a voiced state, and the sliding step size is determined to be the reference step size. Based on the sliding step size, determine the overlapping data segment between the current processing window and the previous processing window; Perform a cache index alignment operation to query whether there is a reusable feature corresponding to the overlapping data segment in the preset feature cache area; Based on the query results, new data segments other than the overlapping data segments are extracted from the audio buffer and their new features are calculated. By reusing the reusable features and combining them with the new features, the complete spectral features of the current processing window are generated. The complete spectral features are input into a pre-trained speech encoder to generate an audio feature vector; Based on the sliding step size, the audio buffer and the feature buffer are updated to complete the real-time audio feature extraction for the current period.

2. The real-time audio feature extraction method based on sliding window according to claim 1, characterized in that, When the amount of audio stream data accumulated in the audio buffer equals the preset baseline window size, audio feature extraction is triggered, including: A preset distance is maintained between the write pointer and the read pointer of the audio buffer, the value of which is equal to the reference window size; As real-time audio stream data is continuously written to the audio buffer, the position of the write pointer is updated; The actual distance between the write pointer and the read pointer is monitored in real time. When the actual distance is found to be equal to the reference window size, the corresponding data block in the audio buffer is locked as the current processing window, and audio feature extraction is triggered.

3. The real-time audio feature extraction method based on sliding window according to claim 2, characterized in that, After the audio feature extraction is triggered, the writing of new audio stream data to the audio buffer area corresponding to the current processing window is paused until the audio feature extraction of the current feature extraction cycle is completed.

4. The real-time audio feature extraction method based on a sliding window according to claim 1, characterized in that, The first length is an integer multiple of the reference step size, and the second length is a fractional multiple of the reference step size.

5. The real-time audio feature extraction method based on a sliding window according to claim 1, characterized in that, The step of determining the overlapping data segment between the current processing window and the previous processing window includes: The length of the overlapping data segment is calculated based on the sliding step size and the preset reference window size. The start and end positions of the overlapping data segments in the audio buffer are determined based on the calculated length.

6. The real-time audio feature extraction method based on a sliding window according to claim 1, characterized in that, The step of performing cache index alignment operation, which involves querying a preset feature cache area to see if there is a reusable feature corresponding to the overlapping data segment, includes: Based on the starting position of the current processing window, calculate the expected starting index corresponding to the overlapping data segment; The expected starting index is compared with the feature starting index recorded in the feature cache. If the absolute deviation between the two is less than or equal to the preset tolerance threshold, the query result is determined to have reusable features; otherwise, the query result is determined to have no reusable features.

7. The real-time audio feature extraction method based on a sliding window according to claim 6, characterized in that, The process of generating complete spectral features for the current processing window by reusing the reusable features and combining them with the newly added features includes: If the query result indicates the existence of the reusable feature, then the first path is executed, including: Read the reusable features corresponding to the overlapping data segments from the feature buffer; Extract the newly added data segment from the audio buffer and calculate the new features corresponding to the newly added data segment; The reusable features and the newly added features are concatenated along the time dimension to generate complete spectral features; If the query result indicates that the reusable feature does not exist, then the second path is executed, including: Extract all audio data corresponding to the current processing window from the audio buffer; Calculate the complete spectral characteristics of all the audio data to serve as the complete spectral characteristics of the current processing window.

8. The real-time audio feature extraction method based on a sliding window according to claim 1, characterized in that, The step of inputting the complete spectral features into a pre-trained speech encoder to generate an audio feature vector includes: The complete spectral features are input into the pre-trained speech encoder; The speech encoder is an acoustic model encoder based on the Transformer architecture, used to map the input spectral feature sequence into an acoustic feature vector sequence; The speech encoder takes the complete spectral features as input and outputs a fixed-dimensional audio feature vector, which is used to characterize the acoustic content within the corresponding time window.

9. A real-time audio feature extraction system based on a sliding window, characterized in that, The system includes a control module, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the real-time audio feature extraction method based on a sliding window according to any one of claims 1-8.

Images