Intelligent bird repelling method and system based on multi-modal perception

Abstract

The invention relates to the technical field of intelligent monitoring and automatic control detection, and discloses an intelligent bird repelling method and system based on multi-modal sensing. According to the method, sound sensor data and depth camera image data are fused to comprehensively monitor bird activities. Through a deep learning algorithm and a multi-modal fusion technology, a high-precision bird detection model is established, and the behavior pattern and the real-time position of the bird can be accurately identified. According to different behavior modes of birds, various bird repelling means such as sound interference, light beam bird repelling and mechanical bird frightening are designed, and accurate intervention on bird activities is achieved. Compared with a traditional bird repelling method, the method has higher efficiency and a better effect, loss and risks caused by bird activities can be remarkably reduced, meanwhile, the method has the advantages of being environmentally friendly, easy to deploy and maintain and the like, and wide application prospects and market value are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent monitoring and automatic control, and in particular to an intelligent bird-repelling method and system based on multimodal perception. Background Art

[0002] In scenarios such as agriculture, airports, and power facilities, bird activities often bring a series of problems, such as crop damage, flight safety hazards, and short circuits in power facilities. Traditional bird-repelling methods, such as setting up dummies, using chemicals, or using sound to repel birds, often have problems such as low efficiency, short-term effects, or pollution to the environment. With the advancement of science and technology, people have begun to try to use intelligent means to repel birds. However, most of the existing intelligent bird-repelling methods only rely on sensor information of a single mode, such as relying solely on sound or image data, resulting in incomplete and inaccurate monitoring of bird activities. Therefore, how to achieve accurate monitoring of bird activities and effective repelling has become an urgent problem to be solved. Summary of the invention

[0003] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide an intelligent bird-repelling method and system based on multimodal perception, so as to solve the problems of low efficiency and poor effect of bird-repelling methods in the prior art, and difficulty in achieving accurate monitoring and effective repelling of bird activities.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions: In a first aspect, the present invention provides an intelligent bird-repelling method based on multimodal perception, the method comprising: Step S100: using a sound sensor and a depth camera to obtain real-time bird information; Step S200: performing data preprocessing on real-time bird information; Step S300: Establishing a bird detection model based on historical bird information and a deep learning algorithm based on multimodal fusion; Step S400: input the pre-processed real-time bird information into the bird detection model, and output the behavior pattern and real-time position of the bird; Step S500: Intervene by driving away birds according to the behavior patterns and real-time positions of the birds.

[0005] Preferably, in a possible implementation manner of the first aspect, the bird information includes sound sensor data and bird depth image data.

[0006] Preferably, in a possible implementation manner of the first aspect, the data preprocessing includes performing spectrum analysis and feature extraction on the sound sensor data and performing image enhancement and correction processing on the bird depth image data.

[0007] Preferably, in a possible implementation manner of the first aspect, the historical bird information includes historical sound sensor data and historical bird depth image data.

[0008] Preferably, in a possible implementation of the first aspect, the deep learning algorithm based on multimodal fusion includes: A data embedding layer is used to embed the historical sound sensor data and historical bird depth image data into a high-dimensional space to form their respective modal feature vectors; The graph construction layer constructs multimodal graphs for each monitoring area based on the embedded modal feature vectors. These graphs include homogeneous nodes representing features within the same modality, heterogeneous nodes representing features between different modalities, and edges representing sequential relationships within the modality and interdependencies between modalities. The graph attention mechanism layer applies a dual attention mechanism on the constructed multimodal sequence graph to independently aggregate heterogeneous and homogeneous node information and adaptively focus on important nodes and relationships; Asynchronous update layer, which asynchronously updates the representation of each node through update gates, with each modality prioritizing its inherent sequentiality or its interaction with another modality; The output layer applies classification and regression algorithms to output the behavior patterns and real-time positions of birds based on the node representations output by the asynchronous update layer.

[0009] Preferably, in a possible implementation manner of the first aspect, the behavior pattern includes a flying posture, a perching state, and a foraging state.

[0010] Preferably, in a possible implementation of the first aspect, the step S400 specifically includes: inputting the preprocessed sound sensor data and bird depth image data into a bird detection model, the model first converts the input data into a modal feature vector through a data embedding layer, then constructs a multimodal graph in a graph construction layer, then applies a dual attention mechanism to aggregate node information in a graph attention mechanism layer, updates node representations through an asynchronous update layer, and finally applies classification and regression algorithms in an output layer to output the behavior patterns and real-time positions of birds.

[0011] Preferably, in a possible implementation of the first aspect, the bird-repelling means comprises: When birds are in a roosting state, a sound interference device is used to play sounds of a specific frequency to drive the birds away; When the birds are in a flying posture, a beam bird repellent device is used to emit a strong light beam to disrupt the flight path of the birds; When birds are foraging, mechanical bird-scare devices are used to generate sudden sounds or physical vibrations to disturb the birds.

[0012] In a second aspect, the present invention provides an intelligent bird-repelling system based on multimodal perception, the system comprising: A data acquisition module for acquiring real-time bird information using sound sensors and depth cameras; Data preprocessing module, which preprocesses the collected sound and image data, including spectrum analysis, feature extraction, image enhancement and correction, etc.; Model training module, which builds a bird detection model based on historical bird information and a deep learning algorithm with multimodal fusion; The real-time detection module inputs the pre-processed real-time bird information into the trained model and outputs the bird’s behavior pattern and real-time location; The bird repellent control module selects appropriate bird repellent means for intervention based on the detected behavior patterns and location information.

[0013] The beneficial effects of the present invention are as follows: by fusing sound sensor data and depth camera image data, comprehensive monitoring of bird activities is achieved. Using deep learning algorithms and multimodal fusion technology, a high-precision bird detection model is established, which can accurately identify the behavior patterns and real-time positions of birds. At the same time, according to the different behavior patterns of birds, the present invention also designs a variety of bird-repelling methods to achieve precise intervention in bird activities. Compared with traditional bird-repelling methods, the present invention has higher efficiency and better effects, and can significantly reduce the losses and risks caused by bird activities. In addition, the present invention also has the advantages of being environmentally friendly, easy to deploy and maintain, and has broad application prospects and market value. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flow chart of an intelligent bird-repelling method based on multimodal perception is provided for this application.

[0016] Figure 2 A structural diagram of an intelligent bird-repelling system based on multimodal perception is provided for this application.

[0017] Explanation of the accompanying drawings: 1-data acquisition module, 2-data preprocessing module, 3-model training module, 4-real-time detection module, 5-bird repelling control module. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Embodiment 1: Figure 1 As shown, the present invention provides an intelligent bird-repelling method based on multimodal perception, comprising: Step S100: using a sound sensor and a depth camera to obtain real-time bird information.

[0020] Specifically, the bird information includes sound sensor data and bird depth image data.

[0021] In this embodiment, in order to accurately capture the sound characteristics of birds, the sound sensor uses the SPM0404HD5H model, which has the characteristics of high sensitivity and strong anti-interference ability, and can accurately capture the subtle differences in bird calls. At the same time, in order to obtain the three-dimensional morphological information of birds, the depth camera uses the CamBoard pico flexx model, which has high resolution and depth perception capabilities, and can generate depth image data of birds in real time, providing an important basis for subsequent bird detection and identification.

[0022] The collected bird information is transmitted to the data center via a wireless network. The SPM0404HD5H sound sensor first captures the bird's calls, and these sound signals are converted into digital data. The CamBoard pico flexx depth camera captures the bird's three-dimensional morphological information and generates bird depth image data. These data are packaged at the acquisition end, including the audio feature parameters of the sound sensor data and the image frame information of the depth image data.

[0023] These data packets are then transmitted through the 5G communication module. The wireless network equipment encodes these data packets into a wireless transmission format and sends them to the wireless access point where the data center is located. After receiving these data packets, the wireless access point decodes them and restores them to the original sound sensor data and depth image data.

[0024] Step S200: performing data preprocessing on real-time bird information.

[0025] Specifically, data preprocessing includes spectrum analysis and feature extraction of sound sensor data and image enhancement and correction processing of bird depth image data.

[0026] In this embodiment, spectrum analysis and feature extraction are performed on the sound sensor data. First, the audio file is loaded. Specify the path audio_path of the audio file and the sampling rate sr (set to 22050Hz in this embodiment). The audio file is loaded by the librosa.load function to obtain the audio signal y and the actual sampling rate sr. Then, the short-time Fourier transform (STFT) of the audio signal is calculated. The absolute value function np.abs of the numpy library is used to calculate the absolute value of the complex short-time Fourier transform result returned by the librosa.stft function to obtain the amplitude spectrum D. Then, the librosa.amplitude_to_db function is used to convert the amplitude spectrum into a power spectrum DB expressed in decibels (dB), where the reference value is the maximum value np.max of the amplitude spectrum. Then, the Mel-frequency cepstral coefficient (MFCC) features are extracted. The librosa.feature.mfcc function is used to extract MFCC features from the audio signal y, specifying the sampling rate sr and the number of MFCC features to be extracted n_mfcc (set to 13 in this embodiment). The function returns the MFCC feature matrix mfccs. Finally, the audio signal y and the MFCC feature matrix mfccs are returned as the output of the method.

[0027] Image enhancement and correction are performed on bird depth image data. First, the OpenCV library (cv2) is used to load the depth image. Specify the path image_path of the depth image, and read the image in an unchanged way (cv2.IMREAD_UNCHANGED) through the cv2.imread function to obtain the image data image. Then, the contrast of the image is enhanced using the contrast-limited adaptive histogram equalization (CLAHE) technique. Create a CLAHE object through the cv2.createCLAHE function, and set the contrast limit (clipLimit) to 2.0 and the grid size (tileGridSize) to 8x8. Use the apply method to apply CLAHE to the normalized image to obtain the contrast-enhanced image enhanced_image. According to the image size (height h and width w) and the camera's intrinsic parameters and distortion coefficients, the cv2.getOptimalNewCameraMatrix function is used to calculate the optimal new camera matrix new_camera_mtx and the region of interest roi. Then, the cv2.initUndistortRectifyMap function is used to calculate the mapping relationship mapx and mapy for distortion correction and image correction. Finally, the enhanced image is remapped using the cv2.remap function according to the mapping relationship to obtain the distorted image undistorted_image. According to the region of interest roi (including the starting coordinates x, y, width w, and height h), the rectified image is cropped to remove the invalid area.

[0028] Step S300: According to historical bird information, a bird detection model is established based on a deep learning algorithm of multimodal fusion.

[0029] Specifically, historical bird information includes historical sound sensor data and historical bird deep image data. In the deep learning algorithm based on multimodal fusion, the multimodal data are sound and image. The algorithm specifically includes: data embedding layer, which is used to embed historical sound sensor data and historical bird deep image data into high-dimensional space to form their own modal feature vectors; graph construction layer, based on the embedded modal feature vectors, multimodal graphs are constructed for each monitoring area. These graphs include homogeneous nodes representing features within the same modality, heterogeneous nodes representing features between different modalities, and edges representing sequential relationships within the modality and interdependencies between modalities; graph attention mechanism layer, on the constructed multimodal sequence graph, a dual attention mechanism is applied to independently aggregate heterogeneous and homogeneous node information, and adaptively focus on important nodes and relationships; asynchronous update layer, which asynchronously updates the representation of each node through an update gate, and each modality gives priority to its inherent sequentiality or its interaction with another modality; output layer, based on the node representation output by the asynchronous update layer, classification and regression algorithms are applied to output the behavior pattern and real-time position of birds. Behavioral patterns include flight posture, perching status, and foraging status.

[0030] In this embodiment, the data set is first annotated, and the data set is made using historical bird information. For each piece of information in the data set, the corresponding bird location and behavior pattern are annotated. The annotated data set is divided into a training set and a test set in an 8:2 ratio.

[0031] For the deep learning algorithm based on multimodal fusion, data embedding is first performed to embed the historical sound sensor data and historical bird deep image data into a high-dimensional space to form their respective modal feature vectors. First, the DataEmbeddingLayer class is defined, which inherits from the "nn.Module" class in the PyTorch framework. It has three main parts: the initialization function, the forward propagation function, and two embedding layers. In the initialization function, three parameters are received: mfcc_dim (the dimension of MFCC features), image_feature_dim (the dimension of image features), and embedding_dim (the dimension of features after embedding). Then two embedding layers are defined: mfcc_embedding and image_embedding. For mfcc_embedding, a linear layer is first used to reduce the dimension of the MFCC feature from mfcc_dim to 128, then a ReLU activation function is passed, and finally a linear layer is passed to reduce the dimension from 128 to embedding_dim. For image_embedding, its structure is similar to mfcc_embedding, except that the input dimension of the first linear layer is image_feature_dim, the output dimension is 256, and the output dimension of the second linear layer is embedding_dim. Next, the forward propagation function receives two inputs: mfcc_features (MFCC feature matrix) and image_features (image feature matrix). These two inputs are then passed through the previously defined mfcc_embedding and image_embedding embedding layers to obtain the embedded MFCC features and image features.

[0032] Secondly, the graph is constructed. According to the embedded modal feature vector, a multimodal graph is constructed for each monitoring area. These graphs include homogeneous nodes representing features within the same modality, heterogeneous nodes representing features between different modalities, and edges representing sequential relationships within the modality and interdependencies between modalities. First, the GraphConstructionLayer class is defined, which inherits from torch.nn.Module. According to the input MFCC (Mel Frequency Cepstral Coefficient) embedding features and image embedding features, a graph structure containing homogeneous nodes and heterogeneous nodes is constructed. In the initialization method of the GraphConstructionLayer class, three parameters are received: embedding_dim represents the dimension of the embedded features, num_nodes_per_modality represents the number of nodes per modality, and num_modalities represents the number of modalities. The feature matrix of homogeneous nodes, homogeneous_node_features, is initialized as a trainable parameter with a shape of (num_nodes_per_modality * num_modalities, embedding_dim). The feature matrix heterogeneous_node_features of the heterogeneous nodes is also initialized as a trainable parameter with a shape of (num_modalities, embedding_dim). The list of edges edge_list is initialized to an empty list for subsequent storage of edges in the graph. During the initialization of the edges, self-loops are first added to each homogeneous node, that is, edges from the node to itself. Then connecting edges are added between different modalities, connecting the first num_nodes_per_modality nodes of the first modality to the first num_nodes_per_modality nodes of the second modality, and so on. Finally, the list of edges is converted to torch.tensor type. The forward propagation method forward of the GraphConstructionLayer class receives the MFCC embedding features mfcc_embeddings and the image embedding features image_embeddings as input. First, these two embedding features are concatenated to form the features of the homogeneous nodes. Then the corresponding parts in the homogeneous node feature matrix are updated to include the input features. Next, a directed graph G is constructed using the NetworkX library. First, add nodes and features to homogeneous nodes. The node features come from the homogeneous node feature matrix. Then add nodes and features to heterogeneous nodes (i.e. modes). The node features come from the heterogeneous node feature matrix. Finally, add edges to the graph according to the list of edges. Return the constructed graph object G and the updated homogeneous node feature matrix homogeneous_node_features.

[0033] Then, on the constructed multimodal sequence graph, a dual attention mechanism is applied to independently aggregate heterogeneous and homogeneous node information, and adaptively focus on important nodes and relationships. First, define the GraphAttentionLayer class, which inherits from nn.Module. In the initialization method of the GraphAttentionLayer class, multiple parameters are defined, including the number of input channels (in_channels), the number of output channels (out_channels), the number of attention heads (num_heads), whether to connect the output (concat, default is True), the dropout rate (dropout, default is 0.6), the alpha parameter (default is 0.2), and the negative_slope parameter (default is 0.2). In the initialization method, create a nn.ModuleList object named self.gat_convs. This object contains two GATConv objects, which are used for the attention mechanism of homogeneous nodes and heterogeneous nodes respectively. The parameters of each GATConv object are set according to the parameters passed in. If the concat parameter is False, the number of output channels self.out_channels is equal to the passed in out_channels; otherwise, self.out_channels is equal to out_channels multiplied by num_heads. In the forward propagation method of the GraphAttentionLayer class, the features of homogeneous nodes and heterogeneous nodes are first calculated based on the input feature x and the edge index edge_index. Homogeneous node features are obtained by intercepting the first num_nodes_per_modality * num_modalities elements of x, and heterogeneous node features are obtained by intercepting the remaining part of x. Then the attention mechanism is applied to homogeneous nodes and heterogeneous nodes separately. For homogeneous nodes, each GATConv object is applied to the homogeneous node features and the results are concatenated along the feature dimension. For heterogeneous nodes, the heterogeneous node features are expanded to the shape of (num_modalities, -1, -1), and the attention mechanism is applied to the heterogeneous nodes of each modality. The results are then average pooled along the modality dimension to obtain the final heterogeneous node output. Finally, the homogeneous node outputs and heterogeneous node outputs are concatenated along the node dimension to obtain the final output combined_out.

[0034] Then, an asynchronous update is performed. The representation of each node is asynchronously updated through the update gate. Each modality gives priority to its inherent order or its interaction with another modality. First, the asynchronous update layer receives four parameters when initialized: input feature dimension (in_features), output feature dimension (out_features), number of modalities (num_modalities), and number of nodes per modality (num_nodes_per_modality). Next, three linear transformation parameters are defined inside the asynchronous update layer: update gate parameter (update_gate), which maps the input feature to a single scalar value to control the update degree of node features. It is implemented using the nn.Linear class. The input feature dimension is in_features and the output feature dimension is 1. Node update parameter (node_update), which calculates new node features based on the input features. It is also implemented using the nn.Linear class. The input feature dimension is in_features and the output feature dimension is out_features. Inter-modality interaction parameter, which processes the interaction information between modalities. In the forward method, the asynchronous update layer receives as input the node feature matrix (node_features), the edge index matrix (edge_index), and the optional edge weight matrix (edge_weights). Then the updated node features are calculated according to the following steps: the update gate is calculated, the node features are linearly transformed using the update gate parameters, and the result is mapped to a value between 0 and 1 through the sigmoid function, which represents the degree of update of each node feature, and then the result is compressed from a two-dimensional matrix (one value per node) to a one-dimensional vector; the updated node feature matrix is ​​initialized, and an all-zero matrix with the same number of nodes as the input node feature matrix but with feature dimension out_features is created to store the updated node features; each mode is traversed, and for each mode, the updated value of the corresponding node feature is calculated, including linear transformation of the node feature using the node update parameters, and weighted summation of the new node feature with the original node feature according to the value of the update gate; the update gate is applied, and the updated node features are assigned to the node positions of the corresponding mode in the initialization matrix; finally, the asynchronous update layer returns the updated node feature matrix as output.

[0035] Finally, according to the node representation output by the asynchronous update layer, classification and regression algorithms are applied to output the behavior pattern and real-time position of the bird. The behavior pattern includes flight posture, perching state, and foraging state. First, define the OutputLayer class, which inherits from torch.nn.Module. In the initialization method of the OutputLayer class, five parameters are received: num_nodes_per_modality (the number of nodes per modality), num_modalities (the number of modalities), embedding_dim (the dimension of the embedded feature), num_classes_behavior (the number of categories of the behavior pattern), and num_outputs_position (the output dimension of the position information). In the initialization method, first calculate the total number of nodes total_num_nodes, which is equal to the number of nodes per modality multiplied by the number of modalities. Then define two fully connected layers: behavior_classifier and position_regressor. Behavior_classifier is used to classify behavior patterns. Its input feature dimension is embedding_dim and its output feature dimension is num_classes_behavior. The output of this layer will be processed by the Softmax activation function to obtain the probability distribution of the behavior pattern. position_regressor is used for the regression of position information. Its input feature dimension is also embedding_dim, but the output feature dimension is num_outputs_position. In the forward propagation method of the OutputLayer class, the node_features input parameter is received, which represents the node feature matrix output by the asynchronous update layer, and its shape is (total_num_nodes, embedding_dim). In the forward propagation process, the node_features are first classified into behavior patterns through the behavior_classifier layer to obtain the prediction results of the behavior patterns behavior_predictions. Then the Softmax activation function is applied to the behavior_predictions to obtain the probability distribution of the behavior patterns. Then the node_features are regressed through the position_regressor layer to obtain the prediction results of the position information position_predictions. Finally, the prediction results of the behavior patterns behavior_predictions and the prediction results of the position information position_predictions are returned.

[0036] Step S400: input the pre-processed real-time bird information into the bird detection model, and output the behavior pattern and real-time position of the bird.

[0037] Specifically, the preprocessed sound sensor data and bird depth image data are input into the bird detection model. The model first converts the input data into a modal feature vector through the data embedding layer, then constructs a multimodal graph in the graph construction layer, and then applies the dual attention mechanism to aggregate node information in the graph attention mechanism layer, updates the node representation through the asynchronous update layer, and finally applies the classification and regression algorithm in the output layer to output the behavior patterns and real-time positions of birds.

[0038] In this embodiment, the pre-processed sound sensor data and bird depth image data are input into the bird detection model, and the data is first processed by the data embedding layer. The data embedding layer uses the trained embedding layer (such as the MFCC embedding layer and the image embedding layer) to embed the MFCC features in the sound sensor data and the image features in the depth image data into the high-dimensional space to form their respective modal feature vectors.

[0039] These modal feature vectors are then fed into the graph construction layer. In the graph construction layer, a multimodal graph containing homogeneous nodes and heterogeneous nodes is constructed for each monitoring area based on the embedded modal feature vectors. Homogeneous nodes represent features within the same modality, while heterogeneous nodes represent features between different modalities. At the same time, according to the order relationship within the modality and the mutual dependence relationship between modalities, corresponding edges are added to the graph to form a complete multimodal graph structure.

[0040] Next, in the graph attention mechanism layer, a dual attention mechanism is applied to independently aggregate heterogeneous and homogeneous node information. It adaptively focuses on important nodes and relationships, thereby effectively extracting key features on the constructed multimodal sequence graph. The asynchronous update layer then asynchronously updates the node representations output by the dual attention mechanism. The representation of each node is updated according to its inherent sequentiality or interaction with another modality to ensure that the model can capture the latest dynamic information of the bird.

[0041] Finally, in the output layer, classification and regression algorithms are applied to process the updated node representations to output the behavior patterns and real-time locations of birds. The behavior patterns include flight posture, perching status, and foraging status, which are predicted by the classification algorithm. The real-time location is predicted by the regression algorithm to obtain the specific location information of the birds in the monitoring area.

[0042] Step S500: Intervene by driving away birds according to the behavior patterns and real-time positions of the birds.

[0043] Specifically, the bird-repelling methods include: when birds are in a perching state, using a sound interference device to play sounds of a specific frequency to drive away the birds; when birds are in a flying posture, using a beam bird-repelling device to emit a strong light beam to interfere with the birds' flight path; when birds are in a foraging state, using a mechanical bird-scare device to generate sudden sounds or physical vibrations to disturb the birds.

[0044] In this embodiment, when it is detected that the bird is in a perching state, that is, the bird stays at a certain position without moving or performing slight activities, the sound interference device is used as the main bird repelling means. The sound interference device plays sounds of specific frequencies, which have a repelling effect on the birds. By controlling the playing position, volume and frequency of the sound, the perched birds are forced to leave their resting place.

[0045] If a bird is detected in flight, that is, it is flying in the air or moving quickly, the light beam bird repellent device will be used to intervene. The light beam bird repellent device can emit strong light beams that can interfere with the bird's visual system, making it difficult for it to accurately determine the flight path. By adjusting the emission angle, intensity and color of the light beam, the flying bird is forced to change its flight direction.

[0046] When birds are identified as being in a foraging state, that is, birds looking for food on the ground or in the low air, mechanical bird scaring devices are used to drive them away. Mechanical bird scaring devices scare foraging birds by generating sudden sounds or physical vibrations. These sounds and vibrations can attract the attention of birds, forcing them to interrupt their foraging behavior and flee the scene.

[0047] When implementing these bird-repelling measures, the system will comprehensively consider the real-time location and behavior patterns of birds, as well as the characteristics and constraints of the surrounding environment to ensure the effectiveness and safety of the intervention measures. At the same time, the system will also record detailed information on each intervention, including intervention time, method selection, intervention effect, etc., for subsequent data analysis and optimization adjustments.

[0048] Embodiment 2: Figure 2 As shown, the present invention provides an intelligent bird-repelling system based on multimodal perception, comprising: Data collection module 1, used to obtain real-time bird information using sound sensors and depth cameras; Data preprocessing module 2, preprocesses the collected sound and image data, including spectrum analysis, feature extraction, image enhancement and correction, etc.; Model training module 3, based on historical bird information, establishes a bird detection model based on a deep learning algorithm of multimodal fusion; The real-time detection module 4 inputs the pre-processed real-time bird information into the trained model and outputs the behavior pattern and real-time location of the bird; The bird-repelling control module 5 selects appropriate bird-repelling means to intervene according to the detected behavior pattern and location information.

[0049] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An intelligent bird-repelling method based on multimodal perception, characterized in that: The method comprises: Step S100: using a sound sensor and a depth camera to obtain real-time bird information; Step S200: performing data preprocessing on real-time bird information; Step S300: Establishing a bird detection model based on historical bird information and a deep learning algorithm based on multimodal fusion; Step S400: input the pre-processed real-time bird information into the bird detection model, and output the behavior pattern and real-time position of the bird; Step S500: Intervene by driving away birds according to the behavior patterns and real-time positions of the birds.

2. The intelligent bird-repelling method according to claim 1, characterized in that: The bird information includes sound sensor data and bird depth image data.

3. The intelligent bird-repelling method according to claim 1, characterized in that: The data preprocessing includes spectrum analysis and feature extraction of the sound sensor data and image enhancement and correction processing of the bird depth image data.

4. The intelligent bird-repelling method according to claim 1, characterized in that: The historical bird information includes historical sound sensor data and historical bird depth image data.

5. The intelligent bird-repelling method according to claim 1, characterized in that: The deep learning algorithm based on multimodal fusion includes: A data embedding layer is used to embed the historical sound sensor data and historical bird depth image data into a high-dimensional space to form their respective modal feature vectors; The graph construction layer constructs multimodal graphs for each monitoring area based on the embedded modal feature vectors. These graphs include homogeneous nodes representing features within the same modality, heterogeneous nodes representing features between different modalities, and edges representing sequential relationships within the modality and interdependencies between modalities. The graph attention mechanism layer applies a dual attention mechanism on the constructed multimodal sequence graph to independently aggregate heterogeneous and homogeneous node information and adaptively focus on important nodes and relationships; Asynchronous update layer, which asynchronously updates the representation of each node through update gates, with each modality prioritizing its inherent sequentiality or its interaction with another modality; The output layer applies classification and regression algorithms to output the behavior patterns and real-time positions of birds based on the node representations output by the asynchronous update layer.

6. The intelligent bird-repelling method according to claim 5, characterized in that: The behavior patterns include flying posture, perching state and foraging state.

7. The intelligent bird-repelling method according to claim 5, characterized in that: The step S400 specifically includes: inputting the preprocessed sound sensor data and bird depth image data into the bird detection model, the model first converts the input data into a modal feature vector through a data embedding layer, then constructs a multimodal graph in a graph construction layer, then applies a dual attention mechanism to aggregate node information in a graph attention mechanism layer, updates node representations through an asynchronous update layer, and finally applies a classification and regression algorithm in an output layer to output the behavior pattern and real-time position of the bird.

8. The intelligent bird-repelling method according to claim 1, characterized in that: The bird-repelling means include: When birds are in a roosting state, a sound interference device is used to play sounds of a specific frequency to drive the birds away; When the birds are in a flying posture, a beam bird repellent device is used to emit a strong light beam to disrupt the flight path of the birds; When birds are foraging, mechanical bird-scare devices are used to generate sudden sounds or physical vibrations to disturb the birds.

9. The intelligent bird-repelling system based on multimodal perception is characterized by: The system comprises: A data acquisition module for acquiring real-time bird information using sound sensors and depth cameras; Data preprocessing module, which preprocesses the collected sound and image data, including spectrum analysis, feature extraction, image enhancement and correction, etc.; Model training module, which builds a bird detection model based on historical bird information and a deep learning algorithm with multimodal fusion; The real-time detection module inputs the pre-processed real-time bird information into the trained model and outputs the bird’s behavior pattern and real-time location; The bird repellent control module selects appropriate bird repellent means for intervention based on the detected behavior patterns and location information.

Images