Intelligent recognition method for flying animal sound based on feature fusion

Abstract

This invention relates to the field of sound recognition technology for flying animals, and discloses a method for intelligent sound recognition of flying animals based on feature fusion. The method includes acquiring the sound signal of the target flying animal and performing time-frequency transformation to obtain a time-frequency feature matrix. Local feature vectors are extracted from the matrix, their gradient fields are calculated, local extrema are located, and their directional connectivity is encoded to form a local topological structure. High-energy extrema are selected as key local features. The Mel-frequency cepstral coefficient sequence of the signal is extracted, the statistical characteristics of each Mel-frequency band are calculated, and the coefficients corresponding to high-variance frequency bands are selected as key acoustic features. After fusing the two types of key features, the result is input into a pre-trained convolutional neural network to complete category recognition. This method enhances the feature discrimination ability by focusing on the essential structure and significant change information of sound, thereby improving recognition accuracy and robustness.

Description

Technical Field

[0001] This invention relates to the field of flying animal sound recognition technology, specifically to a method for intelligent recognition of flying animal sounds based on feature fusion. Background Technology

[0002] Existing technologies for recognizing the sounds of flying animals typically rely directly on global features from Mel-frequency cepstral coefficients or time-spectral graphs for classification. This approach treats sound signals as ordinary data sequences or images, failing to effectively separate crucial information highly relevant to the identification task. Key local patterns representing specific physiological or behavioral states in sound signals are often submerged in global features, making the model insensitive to subtle but important acoustic changes and resulting in insufficient recognition stability in complex acoustic environments or when the target's state is variable.

[0003] In attempts to improve the performance of feature fusion methods, a common practice is to directly concatenate or superimpose multiple extracted acoustic features. This approach lacks a mechanism for evaluating the inherent information content and redundancy of the features. Unfiltered, full-dimensional fusion can result in feature vectors containing a large amount of irrelevant or even interfering information, increasing model complexity and potentially obscuring core discriminative features. This prevents the expected fusion effect from being fully realized, limiting the improvement in model generalization ability.

[0004] The purpose of this invention is to overcome the aforementioned shortcomings and propose a recognition method that can automatically focus on and efficiently fuse essential information of sound signals. This method requires the extraction of topological features reflecting the intrinsic structure of sound from time-frequency representations and adaptive screening of significantly changing key components from traditional acoustic features, ultimately achieving more accurate and robust identification of flying animal categories. Summary of the Invention

[0005] The purpose of this invention is to provide a method for intelligent recognition of the sounds of flying animals based on feature fusion, so as to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, this invention provides a method for intelligent recognition of the sounds of flying animals based on feature fusion, the method comprising:

[0007] The sound signals generated by various target flying animals under different physiological and behavioral states were collected, and the sound signals were transformed by time and frequency to obtain the time and frequency feature matrix.

[0008] Multiple local feature vectors are extracted from the time-frequency feature matrix, the gradient field of the local feature vectors in the feature space is calculated, the local extreme points of the gradient field and the corresponding vector directions are determined, the position of each local extreme point and its directional connection relationship with adjacent local extreme points are encoded to form a local topology, and local extreme points with energy intensity exceeding a preset energy threshold are selected from the local topology as key local features.

[0009] Mel frequency cepstral coefficients are extracted from the sound signal to form a Mel frequency cepstral coefficient sequence. The statistical characteristics of the Mel frequency cepstral coefficient sequence in different Mel frequency bands are calculated to form a Mel frequency feature vector. The coefficients corresponding to the frequency bands whose variance exceeds a preset variance threshold are selected from the Mel frequency feature vector as key acoustic features.

[0010] The key local features and key acoustic features are fused to form a fused feature vector;

[0011] The fused feature vector is input into a pre-trained convolutional neural network to obtain the target flying animal category.

[0012] Preferably, the step of collecting sound signals generated by various target flying animals under different physiological and behavioral states, performing time-frequency transformation on the sound signals, and obtaining a time-frequency feature matrix includes:

[0013] Calculate the short-time spectrum matrix of the sound signal, extract the time-frequency spectrum from the short-time spectrum matrix, and perform block processing on the time-frequency spectrum to obtain multiple spectrum blocks;

[0014] Time-frequency domain features are extracted from each of the spectrum blocks to form a time-frequency feature matrix that includes the number of spectrum blocks and the number of feature dimensions.

[0015] Preferably, the step of calculating the short-time spectrum matrix of the sound signal and extracting the time-spectrum from the short-time spectrum matrix includes:

[0016] The short-time Fourier transform is used to calculate the spectrum of the preprocessed audio signal, resulting in short-time spectrum matrices in the frequency, time, and amplitude dimensions.

[0017] Amplitude information within a preset frequency band is extracted from the short-time spectrum matrix, and a grayscale image is plotted with time point number as the horizontal axis, frequency point as the vertical axis, and amplitude magnitude as the intensity, thus forming a time-spectrum diagram.

[0018] Preferably, the step of dividing the time-spectrum graph into multiple spectrum blocks includes:

[0019] The time-frequency spectrum is uniformly divided into rectangular grids of the same size along the time axis and frequency axis, and each rectangular grid is a spectrum block;

[0020] Each spectrum block is assigned a unique frequency block index and a time block index, resulting in a set containing multiple spectrum blocks.

[0021] Preferably, the step of extracting time-frequency domain features for each of the spectrum blocks to form a time-frequency feature matrix including the number of spectrum blocks and the number of feature dimensions includes:

[0022] For each spectral block, calculate its spectral energy, extract the centroid frequency of the spectral block, calculate the spectral flatness of the spectral block, extract the maximum amplitude of the spectral block and its corresponding frequency, calculate the zero-crossing rate of the spectral block, and construct a time-frequency domain feature vector by combining the spectral energy, centroid frequency, spectral flatness, frequency corresponding to the maximum amplitude, and zero-crossing rate.

[0023] Arrange the time-frequency domain feature vectors corresponding to all spectrum blocks in the order of spectrum block index to form a two-dimensional matrix containing the number of spectrum blocks and the number of feature dimensions as the time-frequency feature matrix.

[0024] Preferably, the step of extracting multiple local feature vectors from the time-frequency feature matrix and calculating the gradient field of the local feature vectors in the feature space includes:

[0025] In the time-frequency feature matrix, each row of data corresponding to each feature dimension is used as a local feature vector;

[0026] For each local eigenvector, calculate its first-order difference along the spectral block index direction, and arrange the first-order difference results of all local eigenvectors to form the gradient field matrix.

[0027] Preferably, determining the local extrema of the gradient field and the corresponding vector directions, and encoding the position of each local extrema and its directional connection with adjacent local extrema to form a local topology, includes:

[0028] In the gradient field matrix, points whose difference value is greater than the difference value of adjacent points are detected along both the feature dimension and the spectral block index, and are marked as local extreme points;

[0029] Calculate the gradient direction at each local extremum point, the gradient direction being determined by the direction from that point toward neighboring points with larger gradient values;

[0030] Using local extrema as nodes and gradient directions as directed edges, nodes with adjacent relationships are connected to form a topological graph representing the change in the flow direction of the feature space, which serves as the local topological structure.

[0031] Preferably, the step of extracting Mel-frequency cepstral coefficients from the sound signal to form a Mel-frequency cepstral coefficient sequence includes:

[0032] The spectrum of the preprocessed audio signal is obtained by performing a fast Fourier transform.

[0033] The spectrum is passed through a set of Mel-scale triangular filters, and the logarithmic energy of the output of each Mel filter is calculated.

[0034] Perform a discrete cosine transform on the logarithmic energy sequence and take the first preset number of coefficients to obtain the Mel frequency cepstral coefficient sequence.

[0035] Preferably, the process of fusing the key local features and key acoustic features to form a fused feature vector includes:

[0036] The selected key local feature vectors are standardized, and the selected key acoustic feature coefficients are also standardized.

[0037] The standardized key local feature vectors and key acoustic feature coefficients are concatenated end-to-end along the feature dimension.

[0038] Principal component analysis is performed on the concatenated feature vectors to reduce their dimensionality, and a fused feature vector with fixed dimensions is output.

[0039] Preferably, the step of inputting the fused feature vector into a pre-trained convolutional neural network to obtain the target flying animal category includes:

[0040] Local topological consistency constraints are introduced during the training process of the convolutional neural network, and the convolutional neural network is used to extract deep abstract features.

[0041] Deep abstract features are input into a fully connected network, which contains multiple hidden layers, and the fully connected network outputs the probability distribution for each target flying animal category.

[0042] Based on the probability distribution, the sound signal to be identified is classified into the target flying animal category with the highest probability.

[0043] The introduction of local topological consistency constraints during the training process of the convolutional neural network includes:

[0044] A regularization constraint term is added to the loss function of the convolutional neural network. This regularization constraint term calculates the difference norm between the local gradient structure of the feature map extracted from the deep layers of the network and the local topology structure corresponding to the key local features. The network weights are adjusted through backpropagation so that the features extracted by the network maintain a topological flow relationship similar to the original time-frequency features.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] By calculating the gradient field of local eigenvectors in the time-frequency feature matrix and determining their local extrema and directional connectivity to encode a local topological structure, extreme points with energy exceeding a threshold are selected from this structure as key local features. This allows the feature extraction process to capture the structured regions and spatial relationships of sound signals that change significantly or remain stable in the time-frequency domain, surpassing the simple utilization of amplitude information by traditional methods. The generated topological features directly describe the core contours and morphology of sound events, enhancing the model's ability to identify and structurally understand the key acoustic patterns of the target sound, and improving the feature representation capability.

[0047] By calculating the statistical characteristics of the Mel frequency cepstral coefficient sequence across different Mel frequency bands and selecting coefficients corresponding to bands with significant variations as key acoustic features based on a variance threshold, this feature is then fused with the aforementioned key local features. This technique achieves adaptive band selection based on the signal's own statistical properties before the feature input network, filtering out stable or low-information band components. This early fusion strategy based on explicit rules constructs a more complementary feature representation, reducing data redundancy and noise interference while strengthening the contribution of acoustic cues most relevant to the classification task. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating the working principle of the intelligent recognition method for the sounds of flying animals based on feature fusion described in this invention.

[0049] Figure 2 A flowchart illustrating the formation of a local topology;

[0050] Figure 3 A flowchart for extracting the Mel frequency cepstral coefficient sequence;

[0051] Figure 4 A heatmap showing the distribution of local extrema points in the gradient field of the time-frequency characteristic matrix;

[0052] Figure 5 A comparison chart showing the performance of sound recognition for flying animals at different stages of feature fusion. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Please see Figure 1This invention provides a method for intelligent recognition of the sounds of flying animals based on feature fusion. The method includes: collecting sound signals generated by various target flying animals under different physiological and behavioral states; performing time-frequency transformation on the sound signals to obtain a time-frequency feature matrix; extracting multiple local feature vectors from the time-frequency feature matrix; calculating the gradient field of the local feature vectors in the feature space; determining the local extrema of the gradient field and the corresponding vector directions; encoding the position of each local extrema and its directional connection relationship with adjacent local extrema to form a local topology; and selecting local extrema with energy intensities exceeding a preset energy threshold from the local topology as key local features; extracting Mel-frequency cepstral coefficients from the sound signals to form a Mel-frequency cepstral coefficient sequence; calculating the statistical characteristics of the Mel-frequency cepstral coefficient sequence in different Mel-frequency bands to form a Mel-frequency feature vector; and selecting coefficients corresponding to frequency bands with variances exceeding a preset variance threshold from the Mel-frequency feature vector as key acoustic features; fusing the key local features and key acoustic features to form a fused feature vector; and inputting the fused feature vector into a pre-trained convolutional neural network to obtain the target flying animal category.

[0055] Example 1: In the steps of collecting sound signals generated by various target flying animals under different physiological and behavioral states, performing time-frequency transformation on the sound signals, and obtaining a time-frequency feature matrix, the short-time spectrum matrix of the sound signal is calculated, a time-frequency spectrum map is extracted from the short-time spectrum matrix, and the time-frequency spectrum map is divided into blocks to obtain multiple spectrum blocks. Time-frequency domain features are extracted from each spectrum block to form a time-frequency feature matrix containing the number of spectrum blocks and the number of feature dimensions. The process of calculating the short-time spectrum matrix of the sound signal and extracting the time-frequency spectrum map from the short-time spectrum matrix includes using a short-time Fourier transform on the preprocessed sound signal to calculate its spectrum, obtaining a short-time spectrum matrix with frequency, time, and amplitude dimensions. Amplitude information within a preset frequency band is extracted from the short-time spectrum matrix, and a grayscale image is plotted with the time point number as the horizontal axis, the frequency point as the vertical axis, and the amplitude magnitude as the intensity, forming a time-frequency spectrum map.

[0056] In specific implementation, the short-time spectrum matrix of the sound signal is calculated, and a time-frequency spectrum is extracted from the short-time spectrum matrix. The time-frequency spectrum is then divided into multiple spectrum blocks, and time-frequency domain features are extracted from each spectrum block to form a time-frequency feature matrix containing the number of spectrum blocks and the number of feature dimensions. The specific process of calculating the short-time spectrum matrix of the sound signal and extracting the time-frequency spectrum is as follows: the preprocessed sound signal is subjected to a short-time Fourier transform to calculate its spectrum, resulting in a short-time spectrum matrix with frequency, time, and amplitude dimensions. In specific implementation, a Hamming window is used as the window function for the short-time Fourier transform, with a window length of 256 sampling points and an overlap of 128 sampling points. In some embodiments, amplitude information within a preset frequency band is extracted from the short-time spectrum matrix, and a grayscale image is plotted with the time point number as the horizontal axis, the frequency point as the vertical axis, and the amplitude magnitude as the intensity, forming a time-frequency spectrum. The preset frequency band is set to 0 Hz to 8000 Hz based on the main energy distribution of the target flying animal's sound. It is understandable that the time-frequency graph is stored in matrix form, with rows corresponding to frequency points, columns corresponding to time points, and element values ​​being the amplitude at the corresponding time-frequency point.

[0057] In specific implementations, the time-frequency spectrum is uniformly divided into rectangular grids of equal size along the time and frequency axes, with each rectangular grid representing a spectrum block. In some embodiments, the height of the rectangular grid on the frequency axis is set to 32 frequency points, and the width on the time axis is set to 32 time points. A unique frequency block index and time block index are assigned to each spectrum block, resulting in a set containing multiple spectrum blocks. The frequency block index represents the position number of the block on the frequency axis, and the time block index represents the position number of the block on the time axis. It can be understood that the process of extracting time-frequency domain features for each spectrum block to form a time-frequency feature matrix containing the number of spectrum blocks and the number of feature dimensions includes calculating the spectral energy of each spectrum block, extracting the centroid frequency of the spectrum block, calculating the spectral flatness of the spectrum block, extracting the maximum amplitude and its corresponding frequency of the spectrum block, calculating the zero-crossing rate of the spectrum block, and constructing a time-frequency domain feature vector from the spectral energy, centroid frequency, spectral flatness, frequency corresponding to the maximum amplitude, and zero-crossing rate. In practice, the spectral energy of a spectrum block is obtained by calculating the sum of squares of all amplitudes within that block, and the centroid frequency of a spectrum block is obtained by calculating the weighted average of the frequencies within the block with respect to amplitude. All time-frequency domain feature vectors corresponding to all spectrum blocks are arranged in spectral block index order, forming a two-dimensional matrix containing the number of spectrum blocks and the number of feature dimensions, which serves as the time-frequency feature matrix. Optionally, the spectral block index order can be arranged in ascending order first along the frequency direction and then along the time direction. The number of rows in the time-frequency feature matrix equals the total number of spectrum blocks, and the number of columns in the time-frequency feature matrix equals the number of dimensions of the time-frequency domain feature vectors.

[0058] Example 2: The time-frequency spectrum is uniformly divided into rectangular grids of equal size along the time and frequency axes, with each rectangular grid representing a spectrum block. A unique frequency block index and time block index are assigned to each spectrum block, resulting in a set containing multiple spectrum blocks. The process of extracting time-frequency domain features for each spectrum block to form a time-frequency feature matrix containing the number of spectrum blocks and the number of feature dimensions includes calculating the spectral energy of each spectrum block, extracting the centroid frequency of the spectrum block, calculating the spectral flatness of the spectrum block, extracting the maximum amplitude and its corresponding frequency of the spectrum block, and calculating the zero-crossing rate of the spectrum block. The spectral energy, centroid frequency, spectral flatness, maximum amplitude frequency, and zero-crossing rate are combined to form a time-frequency domain feature vector. All time-frequency domain feature vectors corresponding to all spectrum blocks are arranged in spectral block index order to form a two-dimensional matrix containing the number of spectrum blocks and the number of feature dimensions, serving as the time-frequency feature matrix.

[0059] In specific implementation, the process of dividing the time-spectrum graph into multiple spectrum blocks includes uniformly dividing the time-spectrum graph into rectangular grids of the same size along the time and frequency axes. Each rectangular grid is a spectrum block, and each spectrum block is assigned a unique frequency block index and a time block index, resulting in a set containing multiple spectrum blocks. In some embodiments, the total size of the time-spectrum graph is 256 frequency points multiplied by 512 time points. When dividing the time-spectrum graph into rectangular grids, the height of each rectangular grid is set to 32 frequency points, and the width is set to 64 time points, thereby generating 8 frequency blocks along the frequency axis and 8 time blocks along the time axis, for a total of 64 spectrum blocks. It can be understood that assigning a unique frequency block index and a time block index to each spectrum block, with the frequency block index ranging from 1 to 8 and the time block index also ranging from 1 to 8, allows for the unique identification and location of each spectrum block within the original time-spectrum graph. This step generates a set containing multiple spectrum blocks, providing structured input for subsequent feature extraction.

[0060] In specific implementation, the spectral energy of each spectral block is calculated, the centroid frequency of the spectral block is extracted, the spectral flatness of the spectral block is calculated, the maximum amplitude and its corresponding frequency of the spectral block are extracted, and the zero-crossing rate of the spectral block is calculated. The spectral energy, centroid frequency, spectral flatness, frequency corresponding to the maximum amplitude, and zero-crossing rate are then combined to form a time-frequency domain feature vector. In some embodiments, the spectral energy of a spectral block is calculated by summing the squares of all amplitude data within that spectral block. The centroid frequency of the spectral block reflects the concentration of energy within the spectral block, and its calculation formula is as follows:

[0061] ;

[0062] in: Indicates the centroid frequency. This represents the frequency value of the i-th frequency point within the spectrum block. This represents the amplitude at the corresponding frequency point. This represents the total number of frequency points within the spectrum block. The spectral flatness of a spectrum block describes the flatness of the spectrum shape; its value is calculated as the ratio of the geometric mean to the arithmetic mean of the spectrum block. The maximum amplitude of the spectrum block and its corresponding frequency are determined by traversing all amplitude data within the block to find the maximum value and its corresponding frequency. The zero-crossing rate of the spectrum block is calculated by counting the number of times the signal amplitude crosses zero points within the block.

[0063] In practice, all time-frequency domain feature vectors corresponding to the spectral blocks are arranged in spectral block index order to form a two-dimensional matrix containing the number of spectral blocks and the number of feature dimensions, which serves as the time-frequency feature matrix. The spectral block index order follows the rule of first arranging them by frequency block index from smallest to largest, and then by time block index from smallest to largest within the same frequency block index. Optionally, each time-frequency domain feature vector contains five feature values: spectral energy, centroid frequency, spectral flatness, frequency corresponding to the maximum amplitude, and zero-crossing rate. For a set with 64 spectral blocks, the generated time-frequency feature matrix will have 64 rows and 5 columns. Each row of the time-frequency feature matrix corresponds to a spectral block, and each column corresponds to a time-frequency domain feature type.

[0064] Example 3: See Figure 2 The process of extracting multiple local feature vectors from the time-frequency feature matrix and calculating the gradient field of these local feature vectors in the feature space includes: taking one row of data corresponding to each feature dimension as a local feature vector in the time-frequency feature matrix; calculating the first-order difference of each local feature vector along the spectral block index direction and arranging the first-order difference results of all local feature vectors to form a gradient field matrix; determining the local extrema and corresponding vector directions of the gradient field, and encoding the position of each local extrema and its directional connection relationship with adjacent local extrema to form a local topology; detecting points in the gradient field matrix whose difference value is greater than the difference value of adjacent points along both the feature dimension and the spectral block index direction, and marking them as local extrema points; calculating the gradient direction at each local extrema point, which is determined by the direction from that point to the adjacent point with a larger gradient value; and connecting adjacent local extrema point nodes with the gradient direction as directed edges to form a topology graph representing the change in the flow direction of the feature space as the local topology.

[0065] In practical implementation, multiple local eigenvectors are extracted from the time-frequency feature matrix. The gradient field of these local eigenvectors in the feature space is calculated, and the local extrema and corresponding vector directions of the gradient field are determined. The position of each local extremum and its directional connection with adjacent local extrema are encoded to form a local topology. Specifically, the process of extracting multiple local eigenvectors from the time-frequency feature matrix and calculating their gradient field in the feature space includes taking one row of data corresponding to each feature dimension as a local eigenvector. For a time-frequency feature matrix containing 5 feature dimensions and 64 spectral blocks, 5 local eigenvectors will be generated. Each local eigenvector is a sequence of 64 elements, where each element corresponds to the value of a specific spectral block in that feature dimension. It can be understood that for each local feature vector, its first difference is calculated along the direction of the spectrum block index, and the first difference results of all local feature vectors are arranged to form a gradient field matrix. For the example above, calculating the first difference of each local feature vector with a length of 64 will result in a difference sequence with a length of 63. Arranging the difference sequences corresponding to the 5 local feature vectors in rows will finally form a gradient field matrix of 5 rows and 63 columns. Each element in the gradient field matrix represents the rate of change of the feature value between adjacent spectrum blocks in a specific feature dimension.

[0066] In some embodiments, determining the local extrema of the gradient field and their corresponding vector directions, and encoding the position of each local extremum and its directional connection with adjacent local extrema to form a local topology, includes detecting points in the gradient field matrix whose difference value is greater than the difference value of their neighboring points along both the feature dimension and the spectral block index, and marking them as local extrema. Specifically, for each internal element in the gradient field matrix, the values ​​of its four adjacent elements (upper, lower, left, and right) are compared. If the value of the element is strictly greater than the values ​​of its four adjacent elements, then the point is marked as a local extremum. It can be understood that the gradient direction at each local extremum is calculated, and the gradient direction is determined by the direction from that point towards the neighboring point with the larger gradient value. Since the local extremum itself is already the maximum value in its neighborhood, the gradient direction here is defined as an indication of its changing trend in the original time-frequency feature space, as shown in the formula:

[0067] ;

[0068] in: Indicates the location at the feature dimension index and spectrum block index The gradient direction vector at the local extremum point. The components are determined by comparing the sign of the numerical differences between the extreme point and the elements above and below it in the gradient field matrix. The components are determined by comparing the sign of the numerical differences between the extreme point and its left and right elements in the gradient field matrix. In some embodiments, local extreme points are used as nodes, and gradient directions are used as directed edges. Connecting adjacent local extreme point nodes forms a topological graph representing the change in the flow direction of the feature space as a local topological structure. The adjacency relationship is defined as the sum of the absolute values ​​of the difference between the row index and the column index of two local extreme points in the gradient field matrix being equal to 1.

[0069] Optionally, the local topology is stored using a graph data structure. Each node in the graph records the feature dimension index and spectral block index coordinates of the corresponding local extremum point, and each directed edge records the gradient direction vector from the source node to the target node. Optionally, the process of selecting key local features from the local topology requires backtracking to the original time-frequency feature matrix, extracting the original feature value corresponding to the local extremum point as its energy intensity based on the location coordinates of the local extremum point, and comparing it with a preset global energy threshold. The feature vectors associated with local extremum points whose energy intensity exceeds the preset energy threshold are retained as key local features.

[0070] Example 4: See Figure 3 The process of extracting Mel-frequency cepstral coefficients from an audio signal to construct a Mel-frequency cepstral coefficient sequence includes performing a Fast Fourier Transform (FFT) on the preprocessed audio signal to obtain its spectrum. The spectrum is then passed through a set of Mel-scale triangular filter banks, and the logarithmic energy of each Mel filter output is calculated. A Discrete Cosine Transform (DCT) is performed on the logarithmic energy sequence, and the first predetermined number of coefficients are taken to obtain the Mel-frequency cepstral coefficient sequence. The process of fusing the key local features and key acoustic features to form a fused feature vector includes standardizing the selected key local feature vectors and standardizing the selected key acoustic feature coefficients. The standardized key local feature vectors and key acoustic feature coefficients are concatenated end-to-end along the feature dimension. Principal component analysis is performed on the concatenated feature vector to reduce its dimensionality, outputting a fused feature vector with a fixed dimension.

[0071] In specific implementation, Mel-frequency cepstral coefficients are extracted from the sound signal to form a Mel-frequency cepstral coefficient sequence. Key local features and key acoustic features are then fused to form a fused feature vector. The process of extracting Mel-frequency cepstral coefficients from the sound signal to form the Mel-frequency cepstral coefficient sequence includes performing a Fast Fourier Transform (FFT) on the preprocessed sound signal to obtain its spectrum. The FFT points are set to 512. In some embodiments, the spectrum is passed through a set of Mel-scale triangular filter banks, and the logarithmic energy of each Mel filter output is calculated. The number of Mel-scale triangular filter banks is set to 26, covering a frequency range from 0 Hz to half the sampling frequency. The logarithmic energy of each Mel filter output is obtained by taking the base-10 logarithm of the sum of the squares of the filtered spectrum amplitudes. It can be understood that performing a Discrete Cosine Transform (DCT) on the logarithmic energy sequence and taking the first preset number of coefficients yields the Mel-frequency cepstral coefficient sequence. The preset number is typically set to 13. The formula for the Discrete Cosine Transform is expressed as:

[0072] ;

[0073] in: Represents the cepstral coefficients of the k-th order Mel frequency. This represents the total number of Mel filters. This represents the output energy of the m-th Mel filter. This represents the logarithmic value of the corresponding energy, and the value of k ranges from 0 to a preset quantity minus 1.

[0074] In practice, the statistical characteristics of the Mel frequency cepstral coefficient sequence across different Mel frequency bands are calculated to form a Mel frequency feature vector. Coefficients corresponding to frequency bands with variances exceeding a preset variance threshold are selected from the Mel frequency feature vector as key acoustic features. The Mel frequency bands divide the Mel frequency cepstral coefficient sequence into several continuous segments, each corresponding to a Mel frequency band. In some embodiments, a Mel frequency cepstral coefficient sequence of length 13 is divided into four Mel frequency bands, with the first three bands each containing three coefficients and the last band containing four coefficients. Four statistical characteristics—mean, variance, maximum, and minimum—are calculated for each Mel frequency band, resulting in a Mel frequency feature vector containing 16 elements. Table 1 shows a fragment of a calculated Mel frequency feature vector; the complete vector should be constructed by sequentially concatenating the statistical characteristics of all Mel frequency bands.

[0075] Table 1: Statistical characteristics of the Mel frequency cepstral coefficient sequence

[0076]

[0077] It is understandable that coefficients corresponding to frequency bands with variances exceeding a preset variance threshold are selected from the Mel frequency eigenvectors as key acoustic features. The preset variance threshold is determined by analyzing the distribution of variance features within all Mel frequency eigenvectors in the training dataset. Optionally, the selection process first identifies Mel frequency bands with variance statistical feature values ​​exceeding the preset variance threshold from the Mel frequency eigenvectors, and then extracts the original Mel frequency cepstral coefficients corresponding to these Mel frequency bands. For example, if the variances of the 2nd and 4th Mel frequency bands exceed the threshold, then the 3 coefficients corresponding to the 2nd Mel frequency band and the 4th Mel frequency band are extracted from the original 13-dimensional Mel frequency cepstral coefficient sequence, and these 7 coefficients are used as key acoustic features.

[0078] In practice, the process of fusing key local features and key acoustic features to form a fused feature vector includes standardizing the selected key local feature vectors and standardizing the selected key acoustic feature coefficients. The standardization process uses the Z-score method, which involves subtracting the mean and then dividing by the standard deviation. The mean and standard deviation parameters are calculated from the training dataset. Essentially, the standardized key local feature vectors and key acoustic feature coefficients are concatenated end-to-end along the feature dimension. Assuming the key local feature vectors are 5-dimensional and the key acoustic feature coefficients are 7-dimensional, the concatenation results in a 12-dimensional temporary feature vector. Optionally, principal component analysis (PCA) is performed on the concatenated feature vector to reduce its dimensionality, outputting a fused feature vector with a fixed dimension. The target dimension for PCA dimensionality reduction is set to 8 dimensions. The projection matrix required for PCA is obtained by learning from all the concatenated temporary feature vectors in the training dataset. The final output fused feature vector is an 8-dimensional vector, used as input to the subsequent convolutional neural network.

[0079] See Figure 4 In a feature fusion-based method for recognizing the sound of flying animals, the spatial distribution of the gradient field of the time-frequency feature matrix and the localization results of local extrema are shown. The horizontal axis represents the spectral block index, the vertical axis represents the feature dimension, and the color gradient represents the gradient value of the gradient field (from blue to red, corresponding to 0.2 to 0.8). "☆" marks the local extrema of the gradient field. Specifically, the time-frequency feature matrix is ​​formed by arranging the time-frequency domain features (spectral energy, centroid frequency, etc.) of each spectral block according to their indices. The gradient field matrix is ​​constructed by calculating the first-order difference of the local feature vectors along the spectral block index direction. Subsequently, local maxima (i.e., "☆" in the figure) are detected along the feature dimension and the spectral block index direction, and these extrema are used as nodes, with the gradient direction as directed edges to form a local topology. The extrema are concentrated in areas with high gradient values ​​(red and yellow blocks), reflecting the core nodes of the topology in the time-frequency feature space. Subsequently, extrema with sufficient energy intensity in these areas will be selected as key local features to provide a foundation for feature fusion.

[0080] Example 5: The process of inputting the fused feature vector into a pre-trained convolutional neural network to obtain the target flying animal category includes introducing local topological consistency constraints during the training of the convolutional neural network and extracting deep abstract features using the convolutional neural network. The deep abstract features are input into a fully connected network containing multiple hidden layers, and the fully connected network outputs a probability distribution for each target flying animal category. Based on the probability distribution, the sound signal to be identified is classified into the target flying animal category with the highest probability. Introducing local topological consistency constraints during the training of the convolutional neural network includes adding a regularization constraint term to the loss function of the convolutional neural network. This regularization constraint term calculates the difference norm between the local gradient structure of the feature map extracted from the deep layers of the network and the local topological structure corresponding to the key local features. The network weights are adjusted through backpropagation so that the features extracted by the network maintain a topological flow relationship similar to the original time-frequency features.

[0081] In specific implementations, the process of inputting the fused feature vector into a pre-trained convolutional neural network (CNN) to obtain the target flying animal category includes introducing local topological consistency constraints during the training of the CNN and utilizing the CNN to extract deep abstract features. In some embodiments, the fused feature vector is reshaped into a two-dimensional feature map to adapt to the input format of the CNN; for example, an 8-dimensional fused feature vector is reshaped into a 1x8 single-channel feature map. The CNN contains two convolutional layers and one pooling layer. The first convolutional layer uses 16 1x3 kernels, the second convolutional layer uses 32 1x3 kernels, and the pooling layer uses max pooling.

[0082] It is understandable that introducing local topological consistency constraints during the training process of convolutional neural networks involves adding a regularization constraint term to the loss function. This regularization constraint term calculates the difference norm between the local gradient structure of the feature map extracted from deep layers of the network and the local topological structure corresponding to key local features. The network weights are then adjusted through backpropagation to ensure that the features extracted by the network maintain a similar topological flow relationship to the original time-frequency features. The specific implementation of local topological consistency constraints is illustrated by the formula... Incorporate into the total loss function:

[0083] ;

[0084] in: This represents the regularization constraint term. The weighting coefficients represent the strength of the control constraints. This represents the matrix representation of the local topological structure corresponding to the key local features extracted and encoded from the original time-frequency feature matrix. This represents the feature map output from the last convolutional layer of the convolutional neural network. The local gradient structure matrix calculated in the middle, This represents the square of the Frobenius norm of the matrix. It can be understood that the total loss function is a weighted sum of the cross-entropy loss term and the regularization constraint term, which simultaneously optimizes the network's accuracy in class prediction and its preservation of deep feature topology through the backpropagation algorithm.

[0085] In some embodiments, deep abstract features are input into a fully connected network containing multiple hidden layers. The fully connected network outputs a probability distribution for each target flying animal category. The fully connected network contains two hidden layers: the first hidden layer has 64 neurons, and the second hidden layer has 32 neurons. Each hidden layer is followed by a ReLU activation function for non-linear transformation. It can be understood that, based on the probability distribution, the sound signal to be identified is classified into the target flying animal category with the highest probability. The output layer of the fully connected network uses a softmax activation function to convert the final value into the probability corresponding to each target flying animal category. The probability distribution is a vector, and the length of the vector is equal to the total number of flying animal categories to be identified. Optionally, during the model training phase, the overall model containing the convolutional neural network and the fully connected network is trained end-to-end using a training dataset with labeled categories, and the Adam optimizer is used for optimization. Optionally, during the model application phase, the fused feature vector obtained from the aforementioned steps of the flying animal sound signal to be identified is input into the trained model. The category corresponding to the maximum value in the probability distribution output by the model is the identified target flying animal category.

[0086] See Figure 5 In a feature fusion-based intelligent recognition method for the sounds of flying animals, the variation pattern of recognition performance under different feature stages is demonstrated. Specifically, the figure compares the performance of four stages—"local features only," "acoustic features only," "feature fusion stage" (horizontal axis), and accuracy, precision, and recall (vertical axis)—using the "feature fusion stage" as the horizontal axis and accuracy, precision, and recall as the vertical axis. In the local features only stage, accuracy, precision, and recall were 0.82, 0.81, and 0.83, respectively, indicating relatively low performance. In the acoustic features only stage, the three metrics improved to 0.85, 0.84, and 0.86, demonstrating the performance gain from acoustic features. In the feature fusion stage (without dimensionality reduction), accuracy, precision, and recall further improved to 0.93, 0.92, and 0.94, validating the effectiveness of fusing local and acoustic features. In the feature fusion stage (after dimensionality reduction), the three metrics reached 0.95, 0.94, and 0.95, indicating that principal component analysis dimensionality reduction maintained high recognition performance while compressing feature dimensions, achieving efficient feature utilization.

[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent recognition of the sounds of flying animals based on feature fusion, characterized in that, include: The sound signals generated by various target flying animals under different physiological and behavioral states were collected, and the sound signals were transformed by time and frequency to obtain the time and frequency feature matrix. Multiple local feature vectors are extracted from the time-frequency feature matrix, the gradient field of the local feature vectors in the feature space is calculated, the local extreme points of the gradient field and the corresponding vector directions are determined, the position of each local extreme point and its directional connection relationship with adjacent local extreme points are encoded to form a local topology, and local extreme points with energy intensity exceeding a preset energy threshold are selected from the local topology as key local features. Mel frequency cepstral coefficients are extracted from the sound signal to form a Mel frequency cepstral coefficient sequence. The statistical characteristics of the Mel frequency cepstral coefficient sequence in different Mel frequency bands are calculated to form a Mel frequency feature vector. The coefficients corresponding to the frequency bands whose variance exceeds a preset variance threshold are selected from the Mel frequency feature vector as key acoustic features. The key local features and key acoustic features are fused to form a fused feature vector; The fused feature vector is input into a pre-trained convolutional neural network to obtain the target flying animal category; The step of extracting multiple local feature vectors from the time-frequency feature matrix and calculating the gradient field of the local feature vectors in the feature space includes: In the time-frequency feature matrix, each row of data corresponding to each feature dimension is used as a local feature vector; For each local eigenvector, calculate its first-order difference along the spectral block index direction, and arrange the first-order difference results of all local eigenvectors to form the gradient field matrix.

2. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 1, characterized in that, The process involves collecting sound signals from various target flying animals under different physiological and behavioral states, performing time-frequency transformation on the sound signals, and obtaining a time-frequency feature matrix, including: Calculate the short-time spectrum matrix of the sound signal, extract the time-frequency spectrum from the short-time spectrum matrix, and perform block processing on the time-frequency spectrum to obtain multiple spectrum blocks; Time-frequency domain features are extracted from each of the spectrum blocks to form a time-frequency feature matrix that includes the number of spectrum blocks and the number of feature dimensions.

3. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 2, characterized in that, The calculation of the short-time spectrum matrix of the sound signal, and the extraction of the time-spectrum graph from the short-time spectrum matrix, includes: The short-time Fourier transform is used to calculate the spectrum of the preprocessed audio signal, resulting in short-time spectrum matrices in the frequency, time, and amplitude dimensions. Amplitude information within a preset frequency band is extracted from the short-time spectrum matrix, and a grayscale image is plotted with time point number as the horizontal axis, frequency point as the vertical axis, and amplitude magnitude as the intensity, thus forming a time-spectrum diagram.

4. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 3, characterized in that, The process of dividing the time-spectrum graph into blocks to obtain multiple spectrum blocks includes: The time-frequency spectrum is uniformly divided into rectangular grids of the same size along the time axis and frequency axis, and each rectangular grid is a spectrum block; Each spectrum block is assigned a unique frequency block index and a time block index, resulting in a set containing multiple spectrum blocks.

5. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 2, characterized in that, The step of extracting time-frequency domain features for each of the spectrum blocks to form a time-frequency feature matrix including the number of spectrum blocks and the number of feature dimensions includes: For each spectral block, calculate its spectral energy, extract the centroid frequency of the spectral block, calculate the spectral flatness of the spectral block, extract the maximum amplitude of the spectral block and its corresponding frequency, calculate the zero-crossing rate of the spectral block, and construct a time-frequency domain feature vector by combining the spectral energy, centroid frequency, spectral flatness, frequency corresponding to the maximum amplitude, and zero-crossing rate. Arrange the time-frequency domain feature vectors corresponding to all spectrum blocks in the order of spectrum block index to form a two-dimensional matrix containing the number of spectrum blocks and the number of feature dimensions as the time-frequency feature matrix.

6. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 5, characterized in that, The step of determining the local extrema of the gradient field and the corresponding vector directions, and encoding the position of each local extrema and its directional connection with adjacent local extrema to form a local topology includes: In the gradient field matrix, points whose difference value is greater than the difference value of adjacent points are detected along both the feature dimension and the spectral block index, and are marked as local extreme points; Calculate the gradient direction at each local extremum point, the gradient direction being determined by the direction from that point toward neighboring points with larger gradient values; Using local extrema as nodes and gradient directions as directed edges, nodes with adjacent relationships are connected to form a topological graph representing the change in the flow direction of the feature space, which serves as the local topological structure.

7. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 1, characterized in that, The step of extracting Mel-frequency cepstral coefficients from the sound signal to form a Mel-frequency cepstral coefficient sequence includes: The spectrum of the preprocessed audio signal is obtained by performing a fast Fourier transform. The spectrum is passed through a set of Mel-scale triangular filters, and the logarithmic energy of the output of each Mel filter is calculated. Perform a discrete cosine transform on the sequence of logarithmic energy, and take the first preset number of coefficients to obtain the Mel frequency cepstral coefficient sequence.

8. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 7, characterized in that, The fusion of the key local features and key acoustic features to form a fused feature vector includes: The selected key local feature vectors are standardized, and the selected key acoustic feature coefficients are also standardized. The standardized key local feature vectors and key acoustic feature coefficients are concatenated end-to-end along the feature dimension. Principal component analysis is performed on the concatenated feature vectors to reduce their dimensionality, and a fused feature vector with fixed dimensions is output.

9. The intelligent recognition method for the sounds of flying animals based on feature fusion according to claim 1, characterized in that, The process of inputting the fused feature vector into a pre-trained convolutional neural network to obtain the target flying animal category includes: Local topological consistency constraints are introduced during the training process of the convolutional neural network, and the convolutional neural network is used to extract deep abstract features. Deep abstract features are input into a fully connected network, which contains multiple hidden layers, and the fully connected network outputs the probability distribution for each target flying animal category. Based on the probability distribution, the sound signal to be identified is classified into the target flying animal category with the highest probability. The introduction of local topological consistency constraints during the training process of the convolutional neural network includes: A regularization constraint term is added to the loss function of the convolutional neural network. This regularization constraint term calculates the difference norm between the local gradient structure of the feature map extracted from the deep layers of the network and the local topology structure corresponding to the key local features. The network weights are adjusted through backpropagation so that the features extracted by the network maintain a topological flow relationship similar to the original time-frequency features.

Images