An audio feature extraction method and processing terminal based on perception-semantic dual domains

Abstract

This invention discloses an audio feature extraction method based on a perceptual-semantic dual-domain approach, comprising the following steps: Step 1: Obtaining an audio signal A and its critical frequency band feature B; Step 2: Calculating the simultaneous masking effect of the audio signal A, characterized by a joint masking threshold. Spectral components below the joint masking value will not be perceived by the human ear. After calculating the joint masking threshold, the perceptual importance of the audio signal A is determined by comparing the joint masking threshold with the actual energy of the audio signal A; Step 3: Inputting the critical frequency band feature B into an audio semantic encoder, and projecting the audio features of the audio signal A, which incorporates type labels and emotional attributes, into a 256-dimensional compact semantic space through a semantic embedding generation network to generate a semantic embedding vector. This invention effectively obtains both the perceptual importance and semantic embedding vector features of audio.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to an audio feature extraction method and processing terminal based on a perceptual-semantic dual-domain approach. Background Technology

[0002] Audio compression involves a series of processes, including audio transformation, feature extraction, quantization, encoding, and reconstruction. Feature extraction, in particular, involves audio analysis. Current audio compression technologies often lack processing of the semantic content of the audio, resulting in a lack of semantic feature extraction based on the semantic domain. Furthermore, current technologies often fail to fully utilize perceptual importance information for adaptive bit allocation optimization, meaning they often lack perceptual feature extraction based on the perceptual domain. Because current technologies typically do not consider both the perceptual and semantic domains in audio feature extraction, they often fail to obtain perceptual importance and semantic embedding vectors, which affects the quality of quantization and encoding in subsequent audio compression processes, ultimately impacting the final audio compression performance. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the purpose of this invention is to provide an audio feature extraction method and processing terminal based on a perceptual-semantic dual-domain approach, which can solve the problems described in the background technology.

[0004] The technical solution to achieve the objective of this invention is: an audio feature extraction method based on a perceptual-semantic dual domain, comprising the following steps: Step 1: Obtain audio signal A and critical band features B of audio signal A. The dimensions of audio signal A are F×T×C, where F represents the number of frequency bands, T represents the number of time frames, and C represents the number of channels in each frequency band. Critical band features B include at least two Bark critical bands. Step 2: Calculate the simultaneous masking effect of audio signal A. The simultaneous masking effect is characterized by a joint masking threshold; spectral components below the joint masking threshold will not be perceived by the human ear. After calculating the joint masking threshold, the perceptual importance of the audio signal A is determined by comparing the joint masking threshold with the actual energy of the audio signal A. The perceptual importance characterizes the degree of contribution of each video position in the audio signal A to the final auditory perception. Step 3: Input the critical band feature B into the audio semantic encoder. The audio semantic encoder is used to perform semantic learning on the critical band feature B to model the long-range dependencies of the audio sequences in the critical band feature B. Input the critical band feature B processed by the audio semantic encoder into the hierarchical feature aggregation module. The hierarchical feature aggregation module is used to extract audio features of critical band features B at different depths in the audio semantic encoder. Based on the audio features and long-range dependencies, the semantic classification head of the critical band feature B is obtained. The semantic classification head is used to predict the type label and sentiment attribute of the audio frame of audio signal A. Then, the audio features of audio signal A, which are fused with type label and sentiment attribute, are projected into a 256-dimensional compact semantic space through the semantic embedding generation network to generate semantic embedding vectors.

[0005] Furthermore, the formula for calculating the joint masking threshold is as follows:

[0006] In the formula, This represents the joint masking threshold at the b-th critical frequency band in critical frequency band feature B and the m-th audio frame of audio signal A. This represents the simultaneous masking threshold, which is calculated by superimposing the masking effects of all maskers at the b-th critical frequency band at the current moment. This represents the forward temporal masking threshold, calculated from the masking effect of the signal from previous time steps on the current time step. This represents the backward temporal masking threshold, calculated from the masking effect of signals from subsequent time points on the current time. This represents the adaptive fusion index, with a value range of [0,1]. This represents the actual energy value at the b-th critical frequency band and the m-th audio frame of audio signal A. This is a reference energy value.

[0007] Furthermore, the perceptual importance of the b-th critical frequency band in critical frequency band feature B and the m-th audio frame of audio signal A is denoted as... It represents the degree of contribution of the audio frame at that time-frequency position to the final auditory perception. Its value range is [0,1] after normalization. The larger the value of the perception importance, the more important the audio frame at that time-frequency position is, and the higher the requirement for coding accuracy.

[0008] Furthermore, the audio semantic encoder adopts an audio semantic encoder based on the Transformer network architecture.

[0009] A processing terminal, comprising: Memory, used to store program instructions; A processor is configured to run the program instructions to perform steps of an audio feature extraction method based on a perceptual-semantic dual-domain approach.

[0010] The beneficial effects of this invention are: This invention can effectively combine the perceptual domain and semantic domain for audio feature extraction, and can obtain two types of features: perceptual importance of audio and semantic embedding vector, thereby ensuring the quality of quantization and encoding in the subsequent audio compression process, and thus ensuring the final audio compression effect. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a preferred embodiment of the method of the present invention; Figure 2 This is a schematic diagram of the processing terminal. Detailed Implementation

[0012] The present invention will be further described below with reference to the accompanying drawings and specific embodiments: like Figure 1 As shown, an audio feature extraction method based on a perceptual-semantic dual-domain approach includes the following steps: Step 1: Obtain audio signal A and its critical band feature B. The dimensions of audio signal A are F×T×C, where F represents the number of frequency bands (typically 64), T represents the number of time frames, and C represents the number of channels in each frequency band. Critical band feature B includes at least two Bark critical bands.

[0013] It is understandable that obtaining audio signal A and critical frequency band characteristics B can be achieved using existing technologies or through the following steps: Step S1: Obtain discrete time-series audio signals .

[0014] audio signal It can be a pulse code modulated audio signal. The bit depth can be 16 bits, 24 bits, or higher to ensure high-quality audio signals. That is, audio signal A.

[0015] Step S2: For the audio signal Preprocessing and framing are performed sequentially to obtain the framed audio signal. Preprocessing is used to process the audio signal. Perform filtering.

[0016] For example, preprocessing includes filtering with a pre-emphasis filter to enhance the audio signal. The energy of the high-frequency components, thereby balancing the audio signal. The spectral characteristics improve the numerical stability of subsequent transformation processing.

[0017] The pre-emphasis filter is a first-order high-pass filter, and its transfer function is: , This represents the pre-emphasis coefficient, which is typically 0.97.

[0018] For example, framing includes: the pre-emphasized audio signal (i.e., the pre-processed audio signal) is divided into overlapping short frames. The frame length of each short frame can be set to 2048 sampling points (approximately 42.7 milliseconds), and the frame shift is 512 sampling points (approximately 10.7 milliseconds). There is 75% overlap between two adjacent short frames to ensure the temporal continuity of the framed audio signal. Each short frame is multiplied by a sine window function to obtain the final framed audio signal. The purpose of multiplying by the sine window function is to reduce the spectral leakage effect.

[0019] Step S3: Determine whether each audio frame in the segmented audio signal is a transient frame or a steady-state frame. Perform wavelet packet decomposition on the audio frames that are transient frames to obtain the first type of audio frames. Perform improved discrete cosine transform on the audio frames that are steady-state frames to obtain the second type of audio frames.

[0020] For example, determining whether an audio frame is a transient frame or a steady-state frame is achieved by calculating the transient attitude value. If the transient attitude value is greater than or equal to a preset threshold, it is determined to be a transient frame; otherwise, it is determined to be a steady-state frame.

[0021] For example, the instantaneous attitude value is calculated using the following formula:

[0022] In the formula, This represents the transient magnitude value of the m-th audio frame. The larger the value, the stronger the transient characteristics of the audio frame. This represents the signal value of the nth sample point in the m-th audio frame, where N is the frame length of the m-th audio frame. This represents the signal value of the (n-1)th sample point in the m-th audio frame. When n=0, This indicates that the signal value of the last sample point of the previous frame (i.e., the (m-1)th frame) is taken. This represents the sum of the squares of the differences between adjacent sample points formed by the audio frames of the m-th frame and the (m-1)-th frame (i.e., the sum of the squares). The total variation energy represents the total variation of an audio signal. Transient signals, due to their drastic changes, will produce a large total variation. This represents the total energy of the m-th audio frame, used to normalize the transient energy so that it is independent of the signal amplitude. It is a positive constant between 0 and 1 (excluding the endpoint 0), and usually takes a small value (its value is generally 10). -8(), used to prevent the denominator from being zero. This represents the zero-crossing rate of the m-th audio frame, which is defined as the number of times the signal crosses a zero point within that frame. The long-term average value representing the zero-crossing rate is calculated by taking the moving average of the zero-crossing rates over the most recent few frames. By introducing the relative change in the zero-crossing rate as a modulation factor, the instantaneous attitude magnitude is increased when the zero-crossing rate is significantly higher than the average value. The multiplication operation combines the total variation ratio with the zero-crossing rate modulation factor, and the final instantaneous attitude magnitude is obtained by integrating the two characteristics.

[0023] The wavelet packet decomposition employs a four-level decomposition structure based on the Daubechies-8 wavelet basis to provide fine temporal resolution, thereby capturing transient details. The improved discrete cosine transform uses a 50% overlapping sinusoidal window to achieve fine resolution in the frequency domain, accurately characterizing the steady-state harmonic structure.

[0024] Step S4: Fuse the first type of audio frames and the second type of audio frames to obtain the fused audio signal. Divide the fused audio signal non-uniformly according to the Bark critical band scale to obtain the divided audio signal. This divided audio signal includes the multi-scale time-frequency representation A and the critical band feature B. The dimension of the divided audio signal (i.e., the dimension of the multi-scale time-frequency representation A) is... F represents the number of frequency bands (typically 64), T represents the number of time frames, and C represents the number of channels in each frequency band. The critical band feature B of the divided audio signal has a dimension of 25×T, corresponding to the energy distribution of 25 Bark critical bands. Non-uniform division allows for finer segmentation of the low-frequency region (i.e., lower bandwidth, such as approximately 100Hz) and coarser segmentation of the high-frequency region (i.e., larger bandwidth, such as reaching several kiloHz), thereby ensuring that the resulting audio signal matches the frequency selectivity characteristics of the human auditory system.

[0025] Step 2: Calculate the simultaneous masking effect of audio signal A. The simultaneous masking effect is characterized by a joint masking threshold. Spectral components below the joint masking value will not be perceived by the human ear. The formula for calculating the joint masking threshold is as follows:

[0026] In the formula, This represents the joint masking threshold at the b-th critical frequency band in critical frequency band feature B and the m-th audio frame of audio signal A. This represents the simultaneous masking threshold, which is calculated by superimposing (i.e., summing) the masking effects produced by all maskers at the b-th critical frequency band at the current time. This represents the forward temporal masking threshold, calculated from the masking effect of the strong signal from the previous time step (especially the previous time step) on the current time step. It reflects the temporal integral characteristics of human auditory perception. This represents the backward time masking threshold, calculated from the masking effect of a strong signal from a time later on the current time. This effect is particularly pronounced during the attack phase of transient signals. `max()` indicates taking the maximum value; here, it takes the maximum of the three masking thresholds in parentheses because, as long as any masking mechanism is in effect, this spectral component cannot be perceived. This represents the adaptive fusion index, with a value range of [0,1]. It is dynamically adjusted based on the local characteristics of the audio signal. For audio frames that are transient frames, A larger value is chosen to emphasize the effect of the masking threshold. For audio frames that are in a steady state, Choose a smaller value to give more consideration to the energy of the audio signal itself. This represents the actual energy value at the m-th audio frame of the b-th critical frequency band and audio signal A. The reference energy value is set to the energy level corresponding to the hearing threshold. Taking the absolute energy of audio signal A into account ensures that high-energy components receive a higher protection priority.

[0027] The entire formula described above combines the masking effect and signal energy by using maximum value calculation and exponential weighting.

[0028] After calculating the joint masking threshold, the perceptual importance of audio signal A is determined by comparing the joint masking threshold with the actual energy of audio signal A. Perceptual importance characterizes the degree of contribution of each video position in audio signal A to the final auditory perception.

[0029] Among them, the perceptual importance of the b-th critical frequency band in the critical frequency band feature B and the m-th audio frame of the audio signal A is denoted as . It represents the degree of contribution of the audio frame at that time-frequency position to the final auditory perception. Its value range is [0,1] after normalization. The larger the value of the perception importance, the more important the audio frame at that time-frequency position is, and the higher the requirement for coding accuracy.

[0030] Step 3: Input the critical band feature B into the audio semantic encoder. The audio semantic encoder performs semantic learning on the critical band feature B to model the long-range dependencies of the audio sequences in the critical band feature B. Then, input the critical band feature B processed by the audio semantic encoder into the hierarchical feature aggregation module. The hierarchical feature aggregation module extracts audio features from critical band features B at different depths in the audio semantic encoder. Based on the audio features and long-range dependencies, a semantic classification head for the critical band feature B is obtained. The semantic classification head is then used to predict the type label and sentiment attribute of the audio frames of audio signal A. Finally, a semantic embedding generation network projects the audio features of audio signal A, which incorporates type labels and sentiment attributes, into a 256-dimensional compact semantic space to generate semantic embedding vectors.

[0031] An audio semantic encoder based on a Transformer network architecture is employed to process audio with multi-scale time-frequency representations. This encoder consists of 12 self-attention layers, each including a multi-head self-attention sublayer and a feedforward neural network sublayer, capable of modeling long-range dependencies in audio sequences. The input to the encoder is a flattened audio feature sequence based on its time-frequency representation, which, after incorporating learnable positional encodings, is fed into the Transformer layers. The encoder is pre-trained on a large-scale audio dataset, learning rich audio semantic knowledge.

[0032] like Figure 2 As shown, the present invention also provides a processing terminal 100, which includes: Memory 101 is used to store program instructions; Processor 102 is configured to run the program instructions to perform the steps of the audio feature extraction method based on the perceptual-semantic dual domain.

[0033] The embodiments disclosed in this specification are merely illustrative of one aspect of the invention, and the scope of protection of the invention is not limited to these embodiments. Any other functionally equivalent embodiments fall within the scope of protection of the invention. Those skilled in the art can make various other corresponding changes and modifications based on the technical solutions and concepts described above, and all such changes and modifications should fall within the scope of protection of the claims of this invention.

Claims

1. An audio feature extraction method based on perception-semantic dual domain, characterized in that, Includes the following steps: Step 1: Obtain audio signal A and critical band features B of audio signal A. The dimensions of audio signal A are F×T×C, where F represents the number of frequency bands, T represents the number of time frames, and C represents the number of channels in each frequency band. Critical band features B include at least two Bark critical bands. Step 2: Calculate the simultaneous masking effect of audio signal A. The simultaneous masking effect is characterized by a joint masking threshold; spectral components below the joint masking threshold will not be perceived by the human ear. After calculating the joint masking threshold, the perceptual importance of the audio signal A is determined by comparing the joint masking threshold with the actual energy of the audio signal A. The perceptual importance characterizes the degree of contribution of each video position in the audio signal A to the final auditory perception. Step 3: Input the critical band feature B into the audio semantic encoder. The audio semantic encoder is used to perform semantic learning on the critical band feature B to model the long-range dependencies of the audio sequences in the critical band feature B. Input the critical band feature B processed by the audio semantic encoder into the hierarchical feature aggregation module. The hierarchical feature aggregation module is used to extract audio features of critical band features B at different depths in the audio semantic encoder. Based on the audio features and long-range dependencies, the semantic classification head of the critical band feature B is obtained. The semantic classification head is used to predict the type label and sentiment attribute of the audio frame of audio signal A. Then, the audio features of audio signal A, which are fused with type label and sentiment attribute, are projected into a 256-dimensional compact semantic space through the semantic embedding generation network to generate semantic embedding vectors. 2.The audio feature extraction method based on perception-semantic dual domain according to claim 1, characterized in that, The formula for calculating the joint masking threshold is as follows: In the formula, This represents the joint masking threshold at the b-th critical frequency band in critical frequency band feature B and the m-th audio frame of audio signal A. This represents the simultaneous masking threshold, which is calculated by superimposing the masking effects of all maskers at the b-th critical frequency band at the current moment. This represents the forward temporal masking threshold, calculated from the masking effect of the signal from previous time steps on the current time step. This represents the backward temporal masking threshold, calculated from the masking effect of signals from subsequent time points on the current time. This represents the adaptive fusion index, with a value range of [0,1]. This represents the actual energy value at the b-th critical frequency band and the m-th audio frame of audio signal A. This is a reference energy value.

3. The audio feature extraction method based on perceptual-semantic dual domains according to claim 2, characterized in that, The perceptual importance of the b-th critical frequency band in critical frequency band feature B and the m-th audio frame of audio signal A is denoted as . It represents the degree of contribution of the audio frame at that time-frequency position to the final auditory perception. Its value range is [0,1] after normalization. The larger the value of the perception importance, the more important the audio frame at that time-frequency position is, and the higher the requirement for coding accuracy.

4. The audio feature extraction method based on perceptual-semantic dual domains according to claim 1, characterized in that, The audio semantic encoder is an audio semantic encoder based on the Transformer network architecture.

5. A processing terminal, characterized in that, It includes: Memory, used to store program instructions; A processor for running the program instructions to perform the steps of the audio feature extraction method based on the perceptual-semantic dual domain as described in any one of claims 1-4.

Images