Bird identification and ecological monitoring method and system based on multimodal data fusion

Abstract

The present invention discloses a bird identification and ecological monitoring method and system based on multimodal data fusion. The method comprises: performing multimodal data fusion processing based on image data and voiceprint data to perform type identification and obtain a type identification result; the multimodal data fusion processing includes: preprocessing the collected data to obtain image data and voiceprint data; performing cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features; and performing hierarchical adaptive dynamic multimodal fusion on the extracted image features and voiceprint features and performing type identification to obtain a type identification result. The present invention discloses a bird identification and ecological monitoring method based on multimodal data fusion, which can provide high-precision, real-time assessment of bird population dynamics and ecological environmental health.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a bird identification and ecological monitoring method and system based on multimodal data fusion. Background Art

[0002] Birds, as important indicators of ecosystem health, are commonly used in ecological monitoring and assessment. Existing bird recognition methods primarily rely on traditional image processing and machine learning techniques, or deep learning recognition methods based on a single modality (such as images or voiceprints). For example, patent CN201610123456.7 discloses a bird image recognition method based on a convolutional neural network. This method uses a convolutional neural network (CNN) to extract and classify bird images. While this method achieves high recognition accuracy in scenarios with rich image data, its drawbacks include sensitivity to image quality, shooting angle, and background complexity. Furthermore, a single image cannot fully reflect bird behavior and ecological changes. Furthermore, single-image recognition suffers from significant shortcomings at night or in low-light environments, where degraded image quality further impacts recognition accuracy and effectiveness.

[0003] The main method for current biodiversity monitoring and ecological environment assessment is generally image recognition: using cameras to collect bird images; using preprocessing, feature extraction (usually using CNN), classification and recognition steps to automatically analyze bird images.

[0004] In summary, existing technologies have the following deficiencies in bird automatic identification and ecological monitoring:

[0005] 1. Single modality data is easily affected by external interference in complex environments, resulting in insufficient recognition accuracy and robustness.

[0006] 2. Single image recognition suffers from insufficient image information at night or in low-light conditions, further reducing the recognition effect.

[0007] 3. Data collection and processing lack the integration of multi-dimensional and cross-modal information, making it difficult to form a complete ecological and environmental assessment system.

[0008] 4. Existing monitoring methods have limited support for data display and statistical analysis, and are unable to intuitively display ecological monitoring data and trend changes, limiting the effectiveness of decision-making support. Summary of the Invention

[0009] The present invention overcomes the shortcomings of the existing technology and provides a bird identification and ecological monitoring method based on multimodal data fusion, which can perform high-precision and real-time assessment of bird population dynamics and ecological environmental health status.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a bird identification and ecological monitoring method based on multimodal data fusion, comprising:

[0011] Acquire image data and voiceprint data;

[0012] Perform multimodal data fusion processing based on image data and voiceprint data, perform type recognition, and obtain type recognition results;

[0013] The multimodal data fusion process includes:

[0014] Preprocessing the collected data to obtain image data and voiceprint data;

[0015] Perform cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features;

[0016] The extracted image features and voiceprint features are hierarchically adaptively fused with dynamic multimodal methods and type recognition is performed to obtain type recognition results.

[0017] In a preferred embodiment of the present invention, acquiring image data comprises the following steps:

[0018] The collected bird images are obtained and standardized, and features of the standardized bird images are extracted through a residual network. The last layer full connection of the residual network is removed to obtain image data.

[0019] In a preferred embodiment of the present invention, obtaining voiceprint data includes the following steps:

[0020] The collected voiceprint signal is obtained, and the voiceprint signal is standardized, and the artificial spectrum feature is extracted from the standardized voiceprint signal to obtain the voiceprint data.

[0021] In a preferred embodiment of the present invention, cross-modal feature extraction includes the following steps: Image feature extraction includes: using a backbone network to extract multi-level visual features from image data, and the image feature extraction algorithm includes: defining the first The layer output is: ;in, The image is preprocessed. The superscripts B, C, H, and W are batch, channel, height, and width, respectively. is the field of real numbers, is the residual network;

[0022] Voiceprint feature extraction: Based on a self-supervised learning mechanism, temporal acoustic features are extracted from voiceprint data. The image spatial dimensions are aligned through time domain interpolation. The voiceprint feature extraction algorithm includes:

[0023] ; ;in, is the preprocessed voiceprint Mel spectrum, is the time domain interpolation operation, is the extracted voiceprint feature, T is the time dimension, H×W is the space dimension; Reshape is the tensor reshaping operation, is the pseudo-spatial audio representation of the final output.

[0024] In a preferred embodiment of the present invention, hierarchical adaptive dynamic multimodal fusion comprises the following steps: performing cross-modal spatial interaction and decomposition factorized bilinear fusion and dynamic gated fusion on image features and voiceprint features;

[0025] Cross-modal spatial interaction includes the following steps: local feature alignment of image and voiceprint features through an attention mechanism; dynamically establishing the spatial association between beak morphology and song by calculating the cross-modal attention weights of image and voiceprint features;

[0026] Decomposition-based factorized bilinear fusion includes the following steps: projecting the image and voiceprint features into the factor space after global average pooling, and using bilinear interaction to capture the nonlinear correlation between image and audio;

[0027] Dynamic gating fusion involves the following steps: Dynamically adjust feature weights through channel-level gating vectors.

[0028] In a preferred embodiment of the present invention, local feature alignment of image features and voiceprint features is performed through an attention mechanism, including the following algorithm: ;

[0029] ;

[0030] ;

[0031] ;

[0032] Among them, γ is the fusion weight of the learning scaling parameter, is the number of channels after dimensionality reduction, Q is the query vector, K is the key vector, is a value vector, is the cross-modal attention weight, is a 1×1 convolutional layer, is a 3×3 convolutional layer, is the normalization algorithm, is the scaling factor, is the multimodal feature fused through the cross attention mechanism, It is a spatial feature.

[0033] In a preferred embodiment of the present invention, the algorithm for modeling global high-order interactions in factorized bilinear fusion includes: ; ; ; , where k is the decomposition factor, GAP is the global average pooling operation, 、 are the global feature vectors obtained by global average pooling of spatial features and pseudo-spatial audio representation, Are projection matrices, I, V are the features after projection, To globally fuse bilinear features; It is Hadamard.

[0034] In a preferred embodiment of the present invention, the feature weights are dynamically adjusted by channel-level gating vectors, including an algorithm for adaptively integrating two types of features: ; ;

[0035] in, is the gate vector, ; is the learnable weight matrix, is the Sigmoid function, Represents a splicing operation, is the final fusion feature; It is Hadamard.

[0036] In a preferred embodiment of the present invention, a bird identification and ecological monitoring system based on multimodal data fusion is used to implement a bird identification and ecological monitoring method based on multimodal data fusion, comprising:

[0037] Data recognition module, used to obtain image data and voiceprint data;

[0038] The data module is used to perform type recognition based on the image data and voiceprint data and the pre-trained multimodal data fusion model to obtain the type recognition result;

[0039] The multimodal data fusion model includes: an interconnected data preprocessing module, a cross-modal feature extraction module, and a hierarchical adaptive dynamic multimodal fusion module;

[0040] The data preprocessing module is used to preprocess the collected data to obtain image data and voiceprint data;

[0041] The cross-modal feature extraction module is used to perform cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features.

[0042] The hierarchical adaptive dynamic multimodal fusion module is used to perform hierarchical adaptive dynamic multimodal fusion on the extracted image features and voiceprint features and perform type recognition to obtain type recognition results.

[0043] In a preferred embodiment of the present invention, the hierarchical adaptive dynamic multimodal fusion module includes: an interconnected cross-modal spatial interaction module, a decomposition factorized bilinear fusion module, and a dynamic gated fusion module;

[0044] The cross-modal spatial interaction module is used to align local features of image and voiceprint features through an attention mechanism. By calculating the cross-modal attention weights of image and voiceprint features, the spatial association between beak morphology and song is dynamically established.

[0045] The decomposition-factored bilinear fusion module is used to project the features of image and voiceprint features into the factor space after global average pooling. The bilinear interaction captures the nonlinear correlation between image and audio.

[0046] Dynamic gated fusion module, which is used to dynamically adjust feature weights through channel-level gating vectors.

[0047] The present invention solves the defects existing in the technical background, and the beneficial technical effects of the present invention are:

[0048] A bird identification and ecological monitoring method and driving method based on multimodal data fusion integrates image and voiceprint data, utilizes deep learning and cross-modal fusion technology to achieve high-precision identification of bird species, and combines a dedicated system statistics and data visualization front end with a high-quality image and voiceprint database to provide a comprehensive and intuitive ecological environment monitoring and assessment solution, thereby effectively making up for the shortcomings of existing technologies.

[0049] 1. Multimodal data fusion: Integrate bird images and voiceprint data, and use deep learning and cross-modal fusion technology to improve recognition accuracy and robustness in complex shooting conditions and noisy environments.

[0050] 2. Comprehensive ecological monitoring and assessment: Through detailed analysis of bird populations and distribution, quantitative assessment of ecosystem health can be achieved.

[0051] 3. Data statistics and visualization: Design a dedicated system statistics and data visualization front-end to display monitoring data in the form of intuitive charts, maps and time series, so that decision makers and scientific researchers can understand and analyze ecological dynamics in a timely manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and examples.

[0053] Figure 1The workflow of the bird identification and ecological monitoring method based on multimodal data fusion in the preferred embodiment of the present invention is as follows: Figure 1 ;

[0054] Figure 2 The workflow of the bird identification and ecological monitoring method based on multimodal data fusion in the preferred embodiment of the present invention is as follows: Figure 2 ;

[0055] Figure 3 The original bird data collected in the bird identification and ecological monitoring method based on multimodal data fusion in the preferred embodiment of the present invention Figure 1 ;

[0056] Figure 4 for Figure 3 Grad-CAM maps corresponding to the corresponding birds;

[0057] Figure 5 for Figure 3 The corresponding audio time domain graphs for the birds;

[0058] Figure 6 for Figure 3 The corresponding bird species correspond to the converted mel-spectrogram;

[0059] Figure 7 The original bird data collected in the bird identification and ecological monitoring method based on multimodal data fusion in the preferred embodiment of the present invention Figure 2 ;

[0060] Figure 8 for Figure 7 Grad-CAM maps corresponding to the corresponding birds;

[0061] Figure 9 for Figure 7 The corresponding audio time domain graphs for the corresponding birds;

[0062] Figure 10 for Figure 7 The corresponding bird species correspond to the converted mel-spectrogram;

[0063] Figure 11 The original bird data collected in the bird identification and ecological monitoring method based on multimodal data fusion in the preferred embodiment of the present invention Figure 3 ;

[0064] Figure 12 for Figure 11 Grad-CAM maps corresponding to the birds;

[0065] Figure 13 for Figure 11 The corresponding audio time domain graphs for the corresponding birds;

[0066] Figure 14 for Figure 11 The corresponding bird species correspond to the converted mel-spectrogram. DETAILED DESCRIPTION

[0067] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. These drawings are simplified schematic diagrams that only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0068] It should be noted that if the embodiments of the present invention involve directional indications (such as up, down, bottom, top, etc.), the directional indications are only used to explain the relative positional relationship, movement status, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly. The terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. Unless otherwise clearly specified and defined, the terms "set", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a communication between the internal parts of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0069] Example 1, as Figure 1-Figure 3 As shown, a bird identification and ecological monitoring method based on multimodal data fusion includes:

[0070] Acquire image data and voiceprint data;

[0071] Multimodal data fusion processing is performed based on the image data and voiceprint data to perform type recognition and obtain a type recognition result; the multimodal data fusion processing includes: preprocessing the collected data to obtain image data and voiceprint data; cross-modal feature extraction of the acquired image data and voiceprint data to obtain image features and voiceprint features; hierarchical adaptive dynamic multimodal fusion of the extracted image features and voiceprint features and type recognition to obtain a type recognition result.

[0072] Furthermore, image data acquisition includes the following steps: acquiring bird images, normalizing them, extracting features from the normalized bird images using a residual network, and removing the fully connected residual network's final layer to obtain image data. Specifically, normalizing the bird images includes cropping the input images to a uniform resolution of 224×224, applying random rotations (±15° range) and horizontal flips (with a probability of 50%) during training to enhance data diversity. The input data is normalized by normalizing pixel values ​​to the ImageNet mean ([0.485, 0.456, 0.406]) and standard deviation ([0.229, 0.224, 0.225]). Feature extraction from the normalized bird images includes extracting features using a ResNet-50 model pre-trained on ImageNet.

[0073] Furthermore, acquiring voiceprint data includes the following steps: acquiring a collected voiceprint signal, performing standardization processing on the voiceprint signal, and performing manual spectral feature extraction on the standardized voiceprint signal to obtain voiceprint data. Specifically, standardization of the voiceprint signal includes: converting the original audio into a mono WAV file, resampling it to a 16kHz sampling rate, truncating the duration / padding silence to 5 seconds, and linearly normalizing the waveform amplitude to the range [-1, 1]. Manual spectral feature extraction of the standardized voiceprint signal includes: directly inputting the preprocessed waveform data into a model, and performing manual spectral feature extraction based on the existing wav2vec model (unsupervised speech pre-training model).

[0074] Furthermore, cross-modal feature extraction includes the following steps:

[0075] Image feature extraction includes: using backbone network to extract multi-level visual features from image data, and image feature extraction algorithm includes: defining the first The layer output is: ;in, The image is preprocessed. The superscripts B, C, H, and W are batch, channel, height, and width, respectively. is the field of real numbers, is the residual network.

[0076] Voiceprint feature extraction: Based on the wav2vec model, temporal acoustic features are extracted from voiceprint data. The image spatial dimensions are aligned through time domain interpolation. The voiceprint feature extraction algorithm includes: ; ;in, is the preprocessed voiceprint Mel spectrum, is the time domain interpolation operation, is the extracted voiceprint feature, T is the time dimension, H×W is the space dimension; Reshape is the tensor reshaping operation, is the pseudo-spatial audio representation of the final output.

[0077] Furthermore, the hierarchical adaptive dynamic multimodal fusion includes the following steps: cross-modal spatial interaction and decomposition factorized bilinear fusion and dynamic gated fusion of image features and voiceprint features.

[0078] Specifically, cross-modal spatial interaction includes the following steps: local feature alignment of image features and voiceprint features through the attention mechanism; dynamically establishing the spatial association between beak morphology and song by calculating the cross-modal attention weights of image features and voiceprint features; focusing on the matching area of ​​image features and voiceprint features in the visible part, while retaining the original image features, injecting voiceprint-guided local enhancement information of the voiceprint features. Among them, local feature alignment of image features and voiceprint features through the attention mechanism includes the following algorithms: ;

[0079] ;

[0080] ;

[0081] ;

[0082] Among them, γ is the fusion weight of the learning scaling parameter, is the number of channels after dimensionality reduction, Q is the query vector, K is the key vector, is a value vector, is the cross-modal attention weight, is a 1×1 convolutional layer, is a 3×3 convolutional layer, is the normalization algorithm, is the scaling factor, is the multimodal feature fused through the cross attention mechanism, It is a spatial feature.

[0083] For example, in occlusion scenarios (e.g., leaves obscuring 50% of a bird's body), the attention mechanism can guide the model to focus on the visible parts that match the voiceprint features, improving recognition accuracy. The residual connection design retains the original image features while injecting local enhancement information guided by the voiceprint, and the learnable parameter γ effectively suppresses cross-modal noise interference.

[0084] Specifically, factorized bilinear fusion includes the following steps: projecting the image and voiceprint features after global average pooling into the factor subspace, and using bilinear interaction to capture the nonlinear correlation between image and audio. In factorized bilinear fusion, global high-order interactions are modeled, and the algorithm includes the following: ; ; ; , where k is the decomposition factor, GAP is the global average pooling operation, 、 are the global feature vectors obtained by global average pooling of spatial features and pseudo-spatial audio representation, Are projection matrices, I, V are the features after projection, To globally fuse bilinear features; is the Hadamard product. Compared with the standard bilinear pooling, the decomposition strategy reduces the number of parameters from down to , the training speed is increased by 2.1 times under the ResNet-50 backbone, and the recognition accuracy of long-distance shooting samples (image resolution <100px) is improved.

[0085] Specifically, dynamic gating fusion includes the following steps: Dynamically adjust feature weights through channel-level gating vectors. Dynamically adjust feature weights through channel-level gating vectors, including algorithms for adaptively integrating two types of features: ; ;in, is the gate vector, ; is the learnable weight matrix, is the Sigmoid function, Represents a splicing operation, is the final fusion feature; It is Hadamard.

[0086] Dynamic gating fusion via channel-level gating vectors Dynamically adjust feature weights, such as the egret crest class that is sensitive to texture, The mean value of the relevant channels reaches 0.87, focusing on the local details of CMSIM; while for the nightingale singing scene that relies on behavioral analysis, The mean value is reduced to 0.35, enhancing the global temporal correlation of DFBFM. This mechanism improves the overall robustness of the system in complex environments such as heavy rain (image occlusion rate > 40%), and the Sigmoid function design ensures gradient stability during the fusion process.

[0087] Working principle:

[0088] The present invention provides a bird identification and ecological monitoring method and driving method based on multimodal data fusion. By integrating image and voiceprint data, deep learning and cross-modal fusion technology are used to achieve high-precision identification of bird species. In combination with a dedicated system statistics and data visualization front end and a high-quality image and voiceprint database, a comprehensive and intuitive ecological environment monitoring and assessment solution is provided, thereby effectively making up for the shortcomings of the existing technology.

[0089] This invention utilizes a fusion of image and voiceprint data to effectively address the blind spots of single-modality recognition in specific environments, such as at night or under obstruction. In scenarios with poor lighting and complex backgrounds, image enhancement and a cross-modal attention mechanism dynamically select the optimal recognition channel, achieving greater robustness and adaptability. The complementary nature of image and voiceprint sensing allows the monitoring system to operate continuously, day and night, and in varying weather conditions, significantly improving the integrity and timeliness of ecological monitoring.

[0090] In Example 2, based on Example 1, an image and voiceprint database is established using the bird recognition and ecological monitoring method based on multimodal data fusion of Example 1. The image and voiceprint database contains bird images and corresponding bird song voiceprint data.

[0091] The image and voiceprint database contains a dedicated database of key protected bird data, providing solid data support for the training of deep learning models in existing technologies and ecological monitoring and evaluation.

[0092] Data management and sharing of image and voiceprint databases include: Image and voiceprint databases use storage engines (such as MySQL, Mongodb) combined with cloud storage technology to ensure data security, stability and scalability.

[0093] The image and voiceprint database establishes a standardized data upload interface and automatic backup mechanism, which can support multi-party data access and data sharing and cross-institutional collaboration in subsequent ecological protection research.

[0094] Embodiment 3, a bird identification and ecological monitoring system based on multimodal data fusion, is used to implement the bird identification and ecological monitoring method based on multimodal data fusion in embodiment 1, including:

[0095] Data acquisition module, used to obtain image data and voiceprint data;

[0096] The data recognition module is used to perform type recognition based on image data and voiceprint data and a pre-trained multimodal data fusion model to obtain a type recognition result.

[0097] Specifically, the multimodal data fusion model includes: an interconnected data preprocessing module, a cross-modal feature extraction module, and a hierarchical adaptive dynamic multimodal fusion module;

[0098] Specifically, the data preprocessing module is used to preprocess the collected data to obtain image data and voiceprint data;

[0099] Specifically, the cross-modal feature extraction module is used to perform cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features.

[0100] Specifically, the hierarchical adaptive dynamic multimodal fusion module is used to perform hierarchical adaptive dynamic multimodal fusion on the extracted image features and voiceprint features and perform type recognition to obtain type recognition results. Furthermore, the hierarchical adaptive dynamic multimodal fusion module includes: an interconnected cross-modal spatial interaction module, a decomposition-factored bilinear fusion module, and a dynamic gated fusion module; the cross-modal spatial interaction module is used to perform local feature alignment on image features and voiceprint features through an attention mechanism; by calculating the cross-modal attention weights of image features and voiceprint features, the spatial association between bird beak morphology and song is dynamically established; the decomposition-factored bilinear fusion module is used to project the features of image features and voiceprint features after global average pooling into the factor space, and the bilinear interaction captures the nonlinear association between image and audio; the dynamic gated fusion module is used to dynamically adjust the feature weights through channel-level gating vectors.

[0101] Example 4: Based on Example 3, a bird identification and ecological monitoring system based on multimodal data fusion also establishes a system statistics and data visualization front end, including an interactively connected front-end system and a back-end data system.

[0102] Among them, the front-end system includes: setting up a web platform dedicated to statistics and data visualization, built based on the Flask framework, and using Query or Echarts visualization tools to display deep learning recognition results and ecological monitoring data in the form of intuitive charts, maps, time series, etc.

[0103] The front-end system and the back-end data system use the RESTful API interface to achieve regular data updates, ensuring that monitoring data and statistical information reflect bird dynamics and ecological environment changes in real time and accurately. The visualization platform provides researchers and decision makers with intuitive data display and trend analysis. The front-end system can count the changing trends of bird populations in various regions, support a variety of visualization methods such as data heat maps, and provide real-time assistance for scientific research analysis and ecological decision-making. It can be used to quickly identify and warn of the appearance of rare or migratory birds, effectively improving resource allocation and patrol efficiency. It reduces labor costs and time consumption, replaces manual visual inspection and manual identification, and improves the level of monitoring automation. To promote the open sharing of scientific research resources, the system supports standardized data management and interface sharing mechanisms.

[0104] Working principle:

[0105] The present invention provides a bird identification and ecological monitoring method based on multimodal data fusion. Through the integration of multimodal fusion identification, high-quality database support and data visualization platform, the accuracy, adaptability and practicality of the bird identification system are significantly improved, which can produce significant technical and social benefits in ecological monitoring, resource protection, scientific research and other aspects, and provide strong technical support for biodiversity protection. The use of system statistics and data visualization front-end makes it easy to display monitoring data in the form of intuitive charts, maps and time series, which is convenient for decision makers and scientific researchers to understand and analyze ecological dynamics in a timely manner. Figure 3-Figure 14 This is a diagram of the processing process of bird identification and ecological monitoring based on multimodal data fusion for three groups of birds. Figure 3 、 Figure 7 、 Figure 11 These are three sets of original pictures of birds. Figure 4 、 Figure 8 、 Figure 12 Respectively Figure 3 、 Figure 7 、 Figure 11 Grad-CAM graph for the corresponding image; indicates the image region that the model pays most attention to when making the current classification decision. Figure 5 、 Figure 9 、 Figure 13 is with Figure 3 、 Figure 7 、 Figure 11 The audio time domain diagram corresponding to the birds in the picture, Figure 6 、 Figure 10 、 Figure 14 is with Figure 3 、 Figure 7 、 Figure 11 The converted mel-spectrograms of the birds.

[0106] The above specific implementation methods are specific support for the scheme ideas proposed in the present invention, and cannot be used to limit the scope of protection of the present invention. Any equivalent changes or equivalent modifications made on the basis of this technical scheme in accordance with the technical ideas proposed in the present invention still fall within the scope of protection of the technical scheme of the present invention.

Claims

1. A bird identification and ecological monitoring method based on multimodal data fusion, characterized in that: include: Acquire image data and voiceprint data; Perform multimodal data fusion processing based on image data and voiceprint data, perform type recognition, and obtain type recognition results; The multimodal data fusion process includes: Preprocessing the collected data to obtain image data and voiceprint data; Perform cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features; The extracted image features and voiceprint features are fused hierarchically and adaptively with dynamic multimodality to perform type recognition and obtain type recognition results; Cross-modal feature extraction includes the following steps: Image feature extraction includes: using backbone network to extract multi-level visual features from image data, and image feature extraction algorithm includes: defining the first The layer output is: ;in, The image is preprocessed. The superscripts B, C, H, and W are batch, channel, height, and width, respectively. is the field of real numbers, is the residual network; Voiceprint feature extraction: Based on a self-supervised learning mechanism, temporal acoustic features are extracted from voiceprint data. The image spatial dimensions are aligned through time domain interpolation. The voiceprint feature extraction algorithm includes: ; ;in, is the preprocessed voiceprint Mel spectrum, is the time domain interpolation operation, is the extracted voiceprint feature, T is the time dimension, H×W is the space dimension; Reshape is the tensor reshaping operation, is the pseudo-spatial audio representation of the final output.

2. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 1 is characterized by: The acquisition of image data includes the following steps: The collected bird images are obtained and standardized, and features of the standardized bird images are extracted through a residual network. The last layer full connection of the residual network is removed to obtain image data.

3. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 2 is characterized by: The acquisition of voiceprint data includes the following steps: The collected voiceprint signal is obtained, and the voiceprint signal is standardized, and the artificial spectrum feature is extracted from the standardized voiceprint signal to obtain the voiceprint data.

4. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 3 is characterized by: Hierarchical adaptive dynamic multimodal fusion includes the following steps: cross-modal spatial interaction and factorized bilinear fusion of image features and voiceprint features, as well as dynamic gated fusion; Cross-modal spatial interaction includes the following steps: local feature alignment of image and voiceprint features through an attention mechanism; dynamically establishing the spatial association between beak morphology and song by calculating the cross-modal attention weights of image and voiceprint features; Decomposition-based factorized bilinear fusion includes the following steps: projecting the image and voiceprint features into the factor space after global average pooling, and using bilinear interaction to capture the nonlinear correlation between image and audio; Dynamic gating fusion involves the following steps: Dynamically adjust feature weights through channel-level gating vectors.

5. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 4 is characterized by: The image features and voiceprint features are locally aligned through the attention mechanism. The following algorithms are included: ; ； ； ； Among them, γ is the fusion weight of the learning scaling parameter, is the number of channels after dimensionality reduction, Q is the query vector, K is the key vector, is a value vector, is the cross-modal attention weight, is a 1×1 convolutional layer, is a 3×3 convolutional layer, is the normalization algorithm, is the scaling factor, is the multimodal feature fused through the cross attention mechanism, It is a spatial feature.

6. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 4, characterized in that: In factorized bilinear fusion, global high-order interactions are modeled using algorithms such as: ; ; ; , where k is the decomposition factor, GAP is the global average pooling operation, 、 are the global feature vectors obtained by global average pooling of spatial features and pseudo-spatial audio representation, Are projection matrices, I, V are the features after projection, To globally fuse bilinear features; It is Hadamard.

7. The bird identification and ecological monitoring method based on multimodal data fusion according to claim 4 is characterized by: Dynamically adjust feature weights through channel-level gating vectors, including algorithms that adaptively integrate two types of features: ; ; in, is the gate vector, ; is the learnable weight matrix, is the Sigmoid function, Represents a splicing operation, is the final fusion feature, It is Hadamard.

8. A bird identification and ecological monitoring system based on multimodal data fusion, characterized by: A method for bird identification and ecological monitoring based on multimodal data fusion for implementing any one of claims 1 to 6, comprising: Data acquisition module, used to obtain image data and voiceprint data; The data recognition module is used to perform type recognition based on the image data and voiceprint data and a pre-trained multimodal data fusion model to obtain a type recognition result; The multimodal data fusion model includes: an interconnected data preprocessing module, a cross-modal feature extraction module, and a hierarchical adaptive dynamic multimodal fusion module; The data preprocessing module is used to preprocess the collected data to obtain image data and voiceprint data; A cross-modal feature extraction module is used to perform cross-modal feature extraction on the acquired image data and voiceprint data to obtain image features and voiceprint features; The hierarchical adaptive dynamic multimodal fusion module is used to perform hierarchical adaptive dynamic multimodal fusion on the extracted image features and voiceprint features and perform type recognition to obtain type recognition results.

9. The bird identification and ecological monitoring system based on multimodal data fusion according to claim 8, characterized in that: The hierarchical adaptive dynamic multimodal fusion module includes: an interconnected cross-modal spatial interaction module, a decomposition factorized bilinear fusion module, and a dynamic gated fusion module; The cross-modal spatial interaction module is used to align local features of image and voiceprint features through an attention mechanism. By calculating the cross-modal attention weights of image and voiceprint features, the spatial association between beak morphology and song is dynamically established. The decomposition-factored bilinear fusion module is used to project the features of image and voiceprint features into the factor space after global average pooling. The bilinear interaction captures the nonlinear correlation between image and audio. Dynamic gated fusion module, which is used to dynamically adjust feature weights through channel-level gating vectors.

Images