An intelligent sound signal perception method and system based on time-frequency feature fusion

Abstract

The application discloses an intelligent sound signal perception method and system based on time-frequency feature fusion, relates to the technical field of intelligent sound signal processing, and comprises the following steps: acquiring an original audio signal; performing short-time Fourier transform on the original audio signal to obtain a complex spectrum feature; inputting the complex spectrum feature into a complex neural network for processing to extract a frequency domain deep feature; processing the original audio signal by adopting a one-dimensional convolutional neural network to extract a time domain deep feature; splicing and fusing the frequency domain deep feature and the time domain deep feature in a channel dimension to obtain a time-frequency fusion feature; and inputting the time-frequency fusion feature into a full connection layer classifier to obtain a final audio scene recognition result. Through the design of the double-branch network architecture for parallel processing of the time domain and the frequency domain, the time domain waveform information and the amplitude and phase information of the frequency domain are effectively fused, and in an end-to-end training scene, the accuracy of audio recognition is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent sound signal processing technology, and more specifically to an intelligent sound signal perception method and system based on time-frequency feature fusion. Background Technology

[0002] Acoustic Scene Recognition (ASR) is a crucial task in environmental sound understanding. In recent years, multimodal or self-supervised models pre-trained on large-scale datasets have achieved significant results in audio classification tasks. For example, the method proposed by Siddharth et al., based on multimodal data and pre-training, achieved near-perfect classification accuracy on the standard ESC-50 dataset. However, the success of these models heavily relies on massive amounts of training data and enormous computational resources. Their high training and deployment costs limit their research and application in resource-constrained scenarios, such as edge computing devices or mobile terminals. Therefore, exploring how to design more efficient and powerful models in a setting where no pre-training is required and end-to-end training is performed solely using the target task dataset remains of significant academic and practical value.

[0003] Traditional audio modeling methods typically focus on single-dimensional features. One type of method is based on frequency domain features, which can effectively capture the spectral structure of sound, but often ignores phase information, which is crucial for distinguishing certain subtle acoustic scenes. Another type of method directly models the original time-domain waveform, which theoretically can retain the most complete information. However, learning multi-scale acoustic features from the original waveform places higher demands on network structure design and optimization, and may be difficult to directly learn the intuitive harmonic structure in the frequency domain.

[0004] It is evident that existing technologies mainly suffer from the following problems: existing high-performance models heavily rely on large-scale pre-training, which is costly, and the number of model parameters is often large, making it difficult to apply on resource-constrained devices; single-domain (time domain or frequency domain) modeling methods cannot fully utilize the complementary information contained in audio signals, especially the phase information in the frequency domain, which is often ignored, limiting further improvement in model performance.

[0005] Therefore, how to provide an intelligent sound signal perception method and system based on time-frequency feature fusion, which can efficiently fuse time-domain and frequency-domain information without pre-training and improve the accuracy of audio recognition, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides an intelligent sound signal perception method and system based on time-frequency feature fusion. By designing a dual-branch network architecture that processes time and frequency in parallel, it effectively integrates time-domain waveform information with frequency-domain amplitude and phase information, significantly improving the accuracy of audio recognition in end-to-end training scenarios.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: an intelligent sound signal perception method based on time-frequency feature fusion, comprising: acquiring the original audio signal;

[0008] Frequency domain feature extraction and time domain feature extraction are performed on the original audio signal respectively to obtain frequency domain depth features and time domain depth features;

[0009] The process of extracting frequency domain features from the original audio signal includes:

[0010] Perform a short-time Fourier transform on the original audio signal to obtain complex spectral features;

[0011] The complex spectral features are input into a complex neural network for processing to extract frequency domain depth features;

[0012] Temporal feature extraction of the original audio signal includes:

[0013] The original audio signal is processed using a one-dimensional convolutional neural network to extract temporal depth features;

[0014] The frequency domain depth features and the time domain depth features are concatenated and fused along the channel dimension to obtain the time-frequency fusion features;

[0015] The time-frequency fusion features are input into a fully connected layer classifier to obtain the final audio scene recognition result.

[0016] Preferably, the complex neural network is composed of multiple complex residual modules stacked together, and the complex residual module includes a complex convolutional layer, a complex batch normalization layer, a complex activation function, and a complex attention module connected in sequence.

[0017] Preferred options also include:

[0018] After acquiring the original audio signal, the original audio signal is resampled to obtain the sampling features.

[0019] Preferably, performing a short-time Fourier transform on the original audio signal to obtain complex spectral features includes:

[0020] A short-time Fourier transform is performed on the original audio signal using a preset window function, number of Fourier transform points, and frame shift length to obtain a complex tensor containing real and imaginary parts;

[0021] Using the pre-calculated global mean and standard deviation, the real and imaginary parts of the complex tensor are globally normalized to obtain the complex spectral features.

[0022] Preferably, the complex feature map is input into a complex convolutional layer for processing, as shown below:

[0023] ;

[0024] in, and Represents complex convolution kernel The real and imaginary parts; and Representing the first The characteristics of the real and imaginary parts output by each complex residual module. and They represent and Input the real and imaginary features of the output of the complex convolutional layer;

[0025] The complex batch normalization layer and Normalization is performed to obtain normalized features. and ;

[0026] The complex activation function calculates the modulus of the normalized feature. and scaling factor The calculated scaling factor is then applied to both the real and imaginary parts of the input features, resulting in:

[0027] ;

[0028] ;

[0029] in, and express and Real and imaginary part features after complex convolutional layers, complex normalization layers, and complex activation functions;

[0030] The complex attention module is based on and The modulus is used to obtain the complex channel weights:

[0031] ;

[0032] ;

[0033] in, This is the first The input feature maps of each complex residual module are dynamically generated, with complex weights for each channel. and It consists of real and imaginary parts; and express and Real and imaginary features after complex convolutional layers, complex normalization layers, complex activation layers, and complex attention layers;

[0034] Output features Add each term to the residual term and perform pooling to obtain the first term. Complex feature map after processing by each module .

[0035] Preferably, the complex feature map output by the last complex residual module is subjected to global average pooling to obtain the real and imaginary part vectors. and ;

[0036] By processing the spatial dimensions, we obtain the flattened real and imaginary vectors. and ;

[0037] The flattened real and imaginary vectors are concatenated along the channel dimension, and the concatenated feature vector is input into a fully connected layer to obtain the final frequency domain depth feature vector.

[0038] Preferably, the one-dimensional convolutional neural network includes a one-dimensional convolutional layer, an anti-aliasing pooling layer, a one-dimensional residual module, and a temporal aggregation module connected in sequence.

[0039] Preferably, the one-dimensional convolutional layer is used to perform preliminary feature extraction on the input waveform;

[0040] The anti-overlapping pooling layer is used for downsampling;

[0041] The one-dimensional residual module is used to extract deep temporal features;

[0042] The temporal aggregation module is used to aggregate sequence features and generate temporal deep features.

[0043] Preferably, an intelligent sound signal perception system based on time-frequency feature fusion includes:

[0044] The signal acquisition module is used to acquire the raw audio signal;

[0045] The feature extraction module is used to extract frequency domain features and time domain features from the original audio signal to obtain frequency domain depth features and time domain depth features.

[0046] The feature fusion module is used to concatenate and fuse the frequency domain depth features and the time domain depth features in the channel dimension to obtain time-frequency fusion features;

[0047] The scene recognition module is used to input the time-frequency fusion features into the fully connected layer classifier to obtain the final audio scene recognition result.

[0048] As can be seen from the above technical solution, compared with the prior art, the present invention discloses an intelligent sound signal perception method and system based on time-frequency feature fusion, which has the following beneficial effects:

[0049] (1) By using the architecture of parallel processing of time and frequency domains, the complementarity between the original waveform information and the complex spectrum (including amplitude and phase) information is fully utilized, which enhances the model’s comprehensive understanding of audio signals.

[0050] (2) The frequency domain branch adopts a fully complex neural network design, including complex convolution, complex normalization, complex activation and complex attention mechanisms, which can directly process and model the phase information in the complex spectrum, and improve the recognition of the fine structure of audio.

[0051] (3) This invention focuses on end-to-end scenarios, which is different from the path that relies on large-scale pre-trained models. It gets rid of the dependence on large-scale pre-trained data, significantly reduces the resource and data costs of model training, and makes it more adaptable in resource-constrained research and application scenarios. At the same time, it effectively improves classification accuracy in end-to-end scenarios, and has both higher efficiency and practicality. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0053] Figure 1 This is a schematic diagram of a method for intelligent sound signal perception based on time-frequency feature fusion provided by the present invention.

[0054] Figure 2 The diagram shows the overall architecture of the time-frequency fusion neural network (FusionNet) provided in this embodiment of the invention.

[0055] Figure 3 This is a schematic diagram of the internal structure of the complex residual module (ComplexResBlock) in the frequency domain branch provided in an embodiment of the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] This invention discloses an intelligent sound signal perception method based on time-frequency feature fusion, such as... Figure 1 As shown, it includes:

[0058] S100: Acquire the raw audio signal;

[0059] S200. Frequency domain feature extraction and time domain feature extraction are performed on the original audio signal respectively to obtain frequency domain depth features and time domain depth features;

[0060] The process of extracting frequency domain features from the original audio signal includes:

[0061] Perform a short-time Fourier transform on the original audio signal to obtain complex spectral features;

[0062] The complex spectral features are input into a complex neural network for processing to extract frequency domain depth features;

[0063] Temporal feature extraction of the original audio signal includes:

[0064] The original audio signal is processed using a one-dimensional convolutional neural network to extract temporal depth features;

[0065] S300. The frequency domain depth feature and the time domain depth feature are spliced ​​and fused in the channel dimension to obtain the time-frequency fusion feature;

[0066] S400. Input the time-frequency fusion features into the fully connected layer classifier to obtain the final audio scene recognition result.

[0067] In this embodiment of the invention, a time-domain feature extraction branch and a frequency-domain feature extraction branch are established in parallel for the input raw audio waveform signal. In the frequency-domain branch, the raw audio signal is subjected to a short-time Fourier transform (STFT) to obtain complex spectral features, and the complex spectral features are input into a complex neural network for processing to extract frequency-domain depth features. In the time-domain branch, a one-dimensional convolutional neural network is used to process the raw audio signal to extract time-domain depth features.

[0068] This invention employs a parallel dual-branch neural network architecture to perform deep modeling of the original waveform and complex spectrum of audio, and effectively fuses the features of both. The aim is to improve the accuracy and robustness of audio scene recognition without relying on large-scale pre-training.

[0069] Specifically, the original audio waveform signal is acquired and resampled to a specific frequency to obtain the sampling features. .

[0070] Specifically, in the frequency domain network, a short-time Fourier transform is performed on the original audio signal to obtain complex spectral features, including:

[0071] A short-time Fourier transform is performed on the original audio signal using a preset window function, number of Fourier transform points, and frame shift length to obtain a complex tensor containing real and imaginary parts;

[0072] Using the pre-calculated global mean and standard deviation, the real and imaginary parts of the complex tensor are globally normalized to obtain the complex spectral features.

[0073] Specifically, the complex spectrum is obtained by performing a short-time Fourier transform on the sampled features. The formula is as follows:

[0074] ;

[0075] in, Indicates sampling characteristics; It is a complex matrix, represented as:

[0076] ;

[0077] The imaginary unit, and These are the real and imaginary part matrices, respectively;

[0078] Global normalization is performed on the real and imaginary matrices respectively to obtain the complex spectral characteristics.

[0079] Global normalization is performed on the real and imaginary parts separately to stabilize model training:

[0080] ;

[0081] in, , This represents the global mean and standard deviation of the real part of the STFT, which are pre-computed over the entire training dataset. , This represents the global mean and standard deviation of the imaginary part of the STFT, pre-computed across the entire training dataset. The input to the frequency domain network is a complex feature map. :

[0082] ;

[0083] First, we pass through the main part of the frequency domain network, namely... k A continuous complex residual module.

[0084] Specifically, the complex neural network is composed of multiple complex residual modules (ComplexResBlock), each of which includes a complex convolutional layer (ComplexConv2d), a complex batch normalization layer (ComplexBatchNorm2d), a complex activation function (ComplexModReLU), and a complex attention module (ComplexSEBlock) connected in sequence.

[0085] Each complex residual module receives the output of the previous complex residual module and generates a new, more abstract complex feature map. This embodiment uses the first... k This will be illustrated using a complex residual module as an example.

[0086] No. Complex feature maps input from each complex residual module Its internal calculation process consists of four steps. Residual connection: Input It goes through a shortcut connection. If the number of input and output channels is different, the shortcut will go through a... The complex convolutional layer (ComplexConv2d) is used to match the dimensions to obtain the residual terms:

[0087] ;

[0088] Main Path: Input The process proceeds sequentially through the following sub-modules:

[0089] a. Complex Convolutional Layer (ComplexConv2d): The complex feature map is input into the complex convolutional layer for processing, represented as:

[0090] ;

[0091] in, and These are all standard real-valued two-dimensional convolution kernels. Conceptually, they represent complex convolution kernels, respectively. The real and imaginary parts; and Representing the first The characteristics of the real and imaginary parts output by each complex residual module. and They represent and Input the real and imaginary features of the output of the complex convolutional layer.

[0092] b. Complex batch normalization layer, wherein the complex batch normalization layer... and Normalization is performed to obtain normalized features. and ;

[0093] c. Complex Modulus Activation Function (ComplexModReLU), which calculates the modulus of the normalized features. and scaling factor The activation function uses a scaling factor acting on the modulus of the complex number. To implement a nonlinear transformation, first calculate the modulus of the input feature map. :

[0094] ;

[0095] Then calculate the scaling factor. :

[0096] ;

[0097] Where b is the learnable activation bias, It is a very small positive number, ensuring the stability of numerical calculations;

[0098] Finally, the calculated scaling factor is applied to both the real and imaginary parts of the input feature, resulting in:

[0099] ;

[0100] ;

[0101] in, and express and Real and imaginary features after complex convolutional layers, complex normalization layers, and complex activation functions.

[0102] d. Complex attention module, the complex attention module according to and The modulus is used to obtain the complex channel weights:

[0103] ;

[0104] ;

[0105] in, This is the first The input feature maps of each complex residual module are dynamically generated, with complex weights for each channel. and It consists of real and imaginary parts; and Indicating the residual main path, and Real and imaginary features after complex convolutional layers, complex normalization layers, complex activation layers, and complex attention layers.

[0106] These features extend the attention mechanism beyond amplitude adjustment in the real domain to the complex domain, enabling adaptive modulation of both feature intensity and phase relationship. This is one of the key design features of the frequency domain branch in this invention, allowing it to effectively process and utilize phase information.

[0107] Output features Add each term to the residual term and perform pooling to obtain the first term. Complex feature map after processing by each module .

[0108] Specifically, the main path output features are added to the residual terms:

[0109] ;

[0110] .

[0111] Specifically, the complex feature map output by the last complex residual module is subjected to global average pooling to obtain the real and imaginary part vectors. and ;

[0112] By processing the spatial dimensions, we obtain the flattened real and imaginary vectors. and ;

[0113] The flattened real and imaginary vectors are concatenated along the channel dimension, and the concatenated feature vector is input into a fully connected layer to obtain the final frequency domain depth feature vector.

[0114] Specifically, pooling: The output of some residual modules is connected to a max pooling layer (MaxPool2d), which is applied independently to each module. and To reduce the spatial dimension of the feature map, the first... k Complex feature map after processing by each module The complex feature map output by the last complex residual module (ComplexResBlock) is obtained after global average pooling. and Flatten each part, remove the 1x1 spatial dimension, and obtain and The flattened real and imaginary vectors are concatenated along the channel dimension, and the concatenated feature vector is input into a fully connected layer to obtain the final frequency domain depth feature vector. .

[0115] Specifically, the one-dimensional convolutional neural network includes a one-dimensional convolutional layer, an anti-aliasing pooling layer (AADownsample), multiple one-dimensional residual modules (ResBlock1dTF), and a temporal aggregation module (TAggregate) based on a Transformer encoder, connected in sequence.

[0116] Specifically, the one-dimensional convolutional layer is used to perform preliminary feature extraction on the input waveform;

[0117] The anti-overlapping pooling layer is used for downsampling;

[0118] The one-dimensional residual module is used to extract deep temporal features;

[0119] The temporal aggregation module is used to aggregate sequence features and generate temporal deep features.

[0120] Specifically, in the time-domain processing branch, Input to time domain network The time-domain features are denoted as , where the function This represents a deep one-dimensional convolutional neural network, which internally contains, in sequence, a one-dimensional convolutional layer for feature extraction, an anti-aliasing pooling layer for downsampling, a one-dimensional residual module for deepening the network, and a temporal aggregation module for aggregating global temporal information.

[0121] Specifically, the classifier is a fully connected layer (Linear), whose input dimension is the sum of the dimensions of the frequency domain depth features and the time domain depth features, and whose output dimension is the preset number of audio categories.

[0122] Specifically, the frequency domain depth features and time domain depth features are concatenated and fused along the channel dimension to obtain a time-frequency fusion feature vector. The fused features are input into a fully connected layer classifier to obtain audio scene recognition results. :

[0123] ;

[0124] and It is the weight matrix and bias vector that the classifier can learn.

[0125] In one specific embodiment of the present invention, an intelligent sound signal perception system based on time-frequency feature fusion includes:

[0126] The signal acquisition module is used to acquire the raw audio signal;

[0127] The feature extraction module is used to extract frequency domain features and time domain features from the original audio signal to obtain frequency domain depth features and time domain depth features.

[0128] The feature fusion module is used to concatenate and fuse the frequency domain depth features and the time domain depth features in the channel dimension to obtain time-frequency fusion features;

[0129] The scene recognition module is used to input the time-frequency fusion features into the fully connected layer classifier to obtain the final audio scene recognition result.

[0130] In one specific embodiment of the present invention, an intelligent sound signal perception method based on time-frequency feature fusion is provided, such as... Figure 2 As shown, this includes: parallel branch establishment:

[0131] For an input raw audio waveform signal (typically a one-dimensional tensor), it is first logically fed into two parallel processing branches: a time-domain processing branch (TimeNet) and a frequency-domain processing branch (FrequencyNet). These two branches will independently extract features from the signal.

[0132] In the frequency domain processing branch, the goal is to extract deep frequency domain features containing amplitude and phase information: First, a Short-Time Fourier Transform (STFT) is performed on the original audio waveform. For example, the number of Fourier transform points n_fft = 512, the frame shift length hop_length = 256, and a Hanning window hann_window can be used. The result of the STFT is a complex tensor with dimensions [batch size, frequency points, time frames]. This complex tensor is then separated into two real tensors: a real tensor and an imaginary tensor.

[0133] Next, to stabilize training, an STFTNormalizer module is used to globally normalize the real and imaginary parts of the STFT, that is, by subtracting the mean pre-calculated over the entire training set and then dividing by the standard deviation.

[0134] The normalized real and imaginary tensors are stacked into a four-dimensional tensor of [batch size, 2, frequency point, time frame], which serves as the input to the subsequent complex neural network.

[0135] The complex neural network FrequencyNet is mainly composed of a series of stacked complex residual modules (such as...). Figure 3 (As shown). Each ComplexResBlock contains:

[0136] ComplexConv2d: A complex two-dimensional convolutional layer. It simulates complex multiplication (a+bi). (c+di)=(ac-bd)+i(ad+bc) processes the real and imaginary parts of the input through two parallel real convolution kernels, and then combines them to obtain the real and imaginary parts of the output.

[0137] ComplexBatchNorm2d: A complex batch normalization layer. It concatenates the real and imaginary parts of the input along the channel dimension, feeds them into a standard BatchNorm2d layer for normalization, and then separates them back into real and imaginary parts.

[0138] ComplexModReLU: A complex activation function. It calculates a scaling factor based on the magnitude of the input complex number and a learnable bias, and then multiplies the real and imaginary parts by this factor to achieve nonlinear activation.

[0139] ComplexSEBlock: Complex Channel Attention Module. It learns the amplitude and phase weights of the channel dimension by performing a Squeeze-and-Excitation operation on the modulus of the input features, thereby generating a complex weight that adaptively recalibrates different channels.

[0140] After being stacked by multiple ComplexResBlocks, the feature map goes through a global adaptive average pooling layer, then flattens and passes through a linear projection layer, finally outputting a fixed-dimensional frequency domain depth feature vector.

[0141] The time-domain processing branch TimeNet (SoundNetRaw in this embodiment) directly processes the one-dimensional raw audio waveform.

[0142] The network first extracts preliminary features from the input waveform through a convolutional layer.

[0143] The signal is then passed through alternating stacks of multiple downsampling modules (Down) and one-dimensional residual modules (ResBlock1dTF). The downsampling module (Down) includes an AADownsample layer for anti-aliasing, preserving useful information while reducing temporal resolution. The ResBlock1dTF layer is used to deepen the network and extract more complex temporal features.

[0144] Deep within the network, the feature sequences are fed into a TAggregate module. This module is essentially a standard Transformer encoder that aggregates information from the entire time series using a self-attention mechanism. By introducing a learnable classification token (cls_token) and concatenating it with the sequence features, a global temporal deep feature vector representing the entire audio segment is finally extracted from the output corresponding to this token.

[0145] Feature fusion and classification:

[0146] After extracting the frequency domain deep features (feat_freq) and time domain deep features (feat_time) from the two branches respectively, these two feature vectors are concatenated in the channel dimension (dim=1) to form a longer and more informative fused feature vector fused_feat.

[0147] Finally, this fused feature vector is fed into a final fully connected classifier fc. This classifier maps the fused features to a preset number of audio categories and outputs a score (logits) for each category, thus completing the audio scene recognition task.

[0148] During the training phase, the entire FusionNet model is optimized end-to-end. Loss functions such as CrossEntropyLoss or LabelSmoothing CrossEntropyLoss are used to calculate the gap between the predicted score and the true label, and an optimizer such as AdamW is used to update all learnable parameters in the network via backpropagation.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0150] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for intelligent sound signal perception based on time-frequency feature fusion, characterized in that, include: Acquire the raw audio signal; The time-domain feature extraction branch and the frequency-domain feature extraction branch are established in parallel; Frequency domain feature extraction and time domain feature extraction are performed on the original audio signal respectively to obtain frequency domain depth features and time domain depth features; The process of extracting frequency domain features from the original audio signal includes: Perform a short-time Fourier transform on the original audio signal to obtain complex spectral features; The complex spectral features are input into a complex neural network for processing to extract frequency domain depth features; Temporal feature extraction of the original audio signal includes: The original audio signal is processed using a one-dimensional convolutional neural network to extract temporal depth features; The frequency domain depth features and the time domain depth features are concatenated and fused along the channel dimension to obtain the time-frequency fusion features; The time-frequency fusion features are input into a fully connected layer classifier to obtain the final audio scene recognition result; The complex neural network is composed of multiple complex residual modules stacked together. Each complex residual module includes a complex convolutional layer, a complex batch normalization layer, a complex activation function, and a complex attention module connected in sequence. The complex spectral features are input into a complex convolutional layer for processing, as follows: ； in, and Represents complex convolution kernel The real and imaginary parts; and Representing the first The characteristics of the real and imaginary parts output by each complex residual module. and They represent and Input the real and imaginary features of the output of the complex convolutional layer; The complex batch normalization layer and Normalization is performed to obtain normalized features. and ; The complex activation function calculates the modulus of the normalized feature. and scaling factor The calculated scaling factor is then applied to both the real and imaginary parts of the input features, resulting in: ； ； in, and express and Real and imaginary part features after complex convolutional layers, complex normalization layers, and complex activation functions; The complex attention module is based on and The modulus is used to obtain the complex channel weights: ； ； in, This is the first The input feature maps of each complex residual module are dynamically generated, with complex weights for each channel. and It consists of real and imaginary parts; and express and Real and imaginary features after complex convolutional layers, complex normalization layers, complex activation layers, and complex attention layers; Output features Add each term to the residual term and perform pooling to obtain the first term. Complex feature map after processing by each module ; The complex feature map output by the last complex residual module is then subjected to global average pooling to obtain the real and imaginary vectors. and ; By processing the spatial dimensions, we obtain the flattened real and imaginary vectors. and ; The flattened real and imaginary vectors are concatenated along the channel dimension, and the concatenated feature vector is input into a fully connected layer to obtain the final frequency domain depth feature vector.

2. The intelligent sound signal perception method based on time-frequency feature fusion according to claim 1, characterized in that, Also includes: After acquiring the original audio signal, the original audio signal is resampled to obtain the sampling features.

3. The intelligent sound signal perception method based on time-frequency feature fusion according to claim 1, characterized in that, Perform a short-time Fourier transform on the original audio signal to obtain complex spectral features, including: A short-time Fourier transform is performed on the original audio signal using a preset window function, number of Fourier transform points, and frame shift length to obtain a complex tensor containing real and imaginary parts; Using the pre-calculated global mean and standard deviation, the real and imaginary parts of the complex tensor are globally normalized to obtain the complex spectral features.

4. The intelligent sound signal perception method based on time-frequency feature fusion according to claim 1, characterized in that, The one-dimensional convolutional neural network includes a one-dimensional convolutional layer, an anti-aliasing pooling layer, a one-dimensional residual module, and a temporal aggregation module connected in sequence.

5. The intelligent sound signal perception method based on time-frequency feature fusion according to claim 4, characterized in that, The one-dimensional convolutional layer is used to perform preliminary feature extraction on the input waveform; The anti-overlapping pooling layer is used for downsampling; The one-dimensional residual module is used to extract deep temporal features; The temporal aggregation module is used to aggregate sequence features and generate temporal deep features.

6. An intelligent sound signal perception system based on time-frequency feature fusion, employing the intelligent sound signal perception method based on time-frequency feature fusion as described in any one of claims 1-5, characterized in that, include: The signal acquisition module is used to acquire the raw audio signal; The feature extraction module is used to extract frequency domain features and time domain features from the original audio signal to obtain frequency domain depth features and time domain depth features. The feature fusion module is used to concatenate and fuse the frequency domain depth features and the time domain depth features in the channel dimension to obtain time-frequency fusion features; The scene recognition module is used to input the time-frequency fusion features into the fully connected layer classifier to obtain the final audio scene recognition result.

Images