Airport surrounding bird risk analysis method based on image processing

Abstract

The invention relates to the field of intelligent risk analysis, in particular to an airport surrounding bird risk analysis method based on image processing, which creatively performs conjoint analysis and prediction on the flight path of a bird flock, constructs a dynamic graph containing three dimensions of space, association and time, fits the airspace-bird flock behavior association of reality, and improves the bird flock risk analysis accuracy. Information fusion from local bird individuals to a time sequence and then to a global bird flock is realized through mixed attention, meanwhile, a multi-granularity decoding strategy is designed, and bottom layer trajectory generation is guided and constrained by predicting a high-level intention, so that trajectory prediction is accurate and conforms to a bird flock joint flight law, and finally accurate prediction of risks is realized. According to the method, a graph segmentation method is adopted to segment the birds in the sky graph, accurate segmentation masks and texts describing the spatial positions of the birds are output, bird image-text pairs containing position labels are generated, an accurate position basis is provided for flight path detection and prediction of the birds, and then the bird risk analysis capability is improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent risk analysis, specifically a method for risk analysis of birds around airports based on image processing. Background Technology

[0002] With global air passenger and cargo volumes increasing year by year, flight takeoffs and landings are becoming more frequent, significantly increasing the probability of aircraft encountering birds in airspace and around airports. Bird strikes can cause serious malfunctions such as engine damage or failure, cockpit windshield cracks, and nose radome damage, and have historically caused numerous fatal aviation accidents. Therefore, bird strike risk analysis at airports is of great importance; however, existing risk analysis methods cannot accurately predict risks based on bird movements. Summary of the Invention

[0003] To address the above issues and overcome the shortcomings of existing technologies, this invention provides an image processing-based method for risk analysis of birds around airports. Addressing the problem that existing risk analysis methods cannot accurately predict risks based on bird movement, this invention creatively employs a joint analysis and prediction method based on dynamic hierarchical graphs and multi-granularity feature prediction to analyze and predict bird flock flight trajectories. It innovatively constructs a dynamic graph containing spatial, relational, and temporal dimensions through an explicit hierarchical graph structure, which more closely reflects the real-world airspace-bird flock behavior association than traditional global attention methods, and can more accurately capture local and structured interactions. Through a hybrid attention encoding mechanism, graph attention, temporal convolution, and global self-attention are organically combined in a single encoding layer, achieving information fusion from individual birds to temporal sequences and then to the global bird flock, resulting in a more expressive model. Simultaneously, a multi-granularity decoding strategy is designed, introducing… The dual-path design of intent decoding and trajectory decoding guides and constrains the generation of low-level trajectories by predicting high-level intents, making trajectory prediction not only accurate but also consistent with the joint flight patterns of bird flocks, and more logical. Building upon this, the invention employs a Unet segmentation method based on a multimodal large language model to segment birds in sky images. By understanding natural language questions through the multimodal large language model, it can simultaneously output accurate segmentation masks and text describing the spatial location of birds. This allows the model to not only identify birds but also indicate their locations, achieving a combination of perception and cognition. Furthermore, a cross-attention mechanism dynamically fuses visual and linguistic features, replacing traditional splicing or addition operations, improving semantic alignment capabilities and generating bird image-text pairs with location annotations. This provides a precise location basis for bird flight trajectory detection and prediction, thereby enhancing the ability for bird risk analysis.

[0004] This invention provides a method for risk analysis of birds around airports based on image processing, which specifically includes the following steps:

[0005] Step S1: Image acquisition, real-time acquisition of sky images around the airport using a camera;

[0006] Step S2: Bird identification. A convolutional neural network is used to identify and detect birds in the sky image to obtain the location detection results of all birds.

[0007] Step S3: Bird segmentation. A Unet segmentation method based on a multimodal large language model is used to segment the birds in the sky image, obtaining a bird flock segmentation map. This includes the following steps:

[0008] Step S31: Visual feature extraction. The sky map is visually encoded using an image encoder based on a lightweight visual Transformer architecture to obtain a feature map.

[0009] Step S32: Joint semantic embedding generation, using a multimodal large language model to perform multimodal semantic understanding and joint semantic embedding generation on the text descriptions of the sky map and location detection results;

[0010] Step S33: Dynamic fusion, using a cross-attention mechanism to dynamically fuse visual features with semantic embeddings to generate conditional segmentation features: ;

[0011] In the formula, and This represents the projection layer, which maps features to the same dimension. and Representing visual features and semantic embedding, respectively. Represents the scaling factor. Represents the softmax function. Represents self-attention processing, Represents segmentation features;

[0012] Step S34: Mask prediction and segmentation. The segmentation features are progressively upsampled using a transposed convolutional layer to obtain a binary segmentation mask. The birds in the sky image are segmented according to the segmentation mask to obtain a bird flock segmentation map.

[0013] Step S4: Intelligent prediction. A method based on dynamic hierarchical maps and multi-granularity feature prediction is used to predict the flight path of each bird in the bird flock segmentation map, resulting in a predicted flight path. This includes the following steps:

[0014] Step S41: Dynamic hierarchical graph construction. A dynamic hierarchical graph is constructed based on the positions of all birds in the flock and their states at different time points. The dynamic hierarchical graph consists of nodes, spatial neighborhood edges, flight association edges, and temporal continuity edges.

[0015] Step S411: Node definition, defining each bird as a node at each time step t. During the real-time data collection period T, there are a total of N×T nodes, where N is the total number of birds;

[0016] Step S412: Define spatial neighborhood edges. Based on the spatial distance between birds, a bird is considered a node when the distance between them is less than a preset threshold 'a'. With nodes Establish undirected edges in the neighborhood between them;

[0017] Step S413: Define flight-related edges. Based on the relative positions and flight paths of the birds in space, two birds are considered nodes when their flight paths are the same and the distance between their relative positions is less than a preset threshold b. With nodes Establish directed edges connecting them;

[0018] Step S414: Define a time-coherent edge, which is a node of consecutive time steps of the same bird. and Establish a coherent directed edge;

[0019] Step S415: Define a dynamic layered graph by connecting all nodes and all edges to form a dynamic layered graph;

[0020] Step S42: Node feature initialization, setting the nodes of each bird at each time step t. The flight speed, flight direction, and relative three-dimensional coordinates are defined as node features;

[0021] Step S43: Node embedding, mapping node features to a high-dimensional space through a linear layer to obtain feature representations. , where t is the time step node, and l is the feature representation of the node in the l-th linear layer;

[0022] Step S44: Hierarchical map encoding. Perform multi-type dynamic encoding on the dynamic hierarchical map to obtain the flight behavior code for each bird. This includes the following steps:

[0023] Step S441: Intra-layer graph attention network processing. For each node, graph attention processing is performed on the set of its spatial neighborhood edges and flight-related edges. That is, graph attention coefficients are calculated through feature representations, and the feature representations are aggregated through attention coefficients and their corresponding feature representations to obtain intra-layer features that understand the influence of local spatial environment and flight between birds.

[0024] Step S442: Inter-layer temporal convolutional network processing. For each node, inter-layer temporal convolutional network processing is performed on its temporal coherence edges. That is, for each bird, the feature representations of all its time steps are stacked into a feature tensor. One-dimensional causal convolution is applied to the feature tensor to obtain inter-layer features that understand the flight pattern of each bird within a time period.

[0025] Step S443: Global attention processing. For each bird, its intra-layer features and inter-layer features are concatenated to obtain a summary representation. Global self-attention processing is performed on the summary representations of all birds to obtain global features that understand the global scene characteristics of the bird flock.

[0026] Step S444: Feature fusion, weighted fusion of intra-layer features, inter-layer features and global features for each bird to obtain flight behavior code;

[0027] Step S45: Intent decoding, encoding the flight behavior of each bird at the last time step. Intent recognition is performed based on a multilayer perceptron (MLP) to obtain intent features;

[0028] Step S46: Trajectory decoding, encoding the flight behavior of each bird at the last time step. The trajectory features are obtained by concatenating the trajectory features with their corresponding intent features and then decoding the trajectory using a multilayer perceptron.

[0029] Step S47: Trajectory synthesis and prediction output. The intention features and trajectory features are weighted and fused and predicted to obtain a weighted combination of the micro trajectory and the macro flock flight path derived from the intention, i.e. the predicted flight path of each bird.

[0030] Step S5: Risk analysis, setting up hierarchical safety zones for the airport space, with each level corresponding to a different risk level. When the predicted flight route reaches the corresponding safety zone, the corresponding risk level is output.

[0031] The beneficial results achieved by the present invention using the above solution are as follows:

[0032] (1) In view of the problem that existing risk analysis cannot accurately predict the risk based on the movement of birds, this invention creatively adopts a dynamic hierarchical graph and multi-granularity feature prediction method to jointly analyze and predict the flight trajectory of bird flocks. Through the explicit hierarchical graph structure, a dynamic graph containing three dimensions of space, association and time is innovatively constructed. It is more in line with the real airspace-bird flock behavior association than the traditional global attention and can more accurately capture local and structured interactions. Through the hybrid attention coding mechanism, graph attention, temporal convolution and global self-attention are organically combined in one coding layer, realizing the information fusion from local individual birds to temporal sequence and then to the global bird flock. The model has stronger expressive power. At the same time, a multi-granularity decoding strategy is designed, and a dual-path design of intent decoding and trajectory decoding is introduced. By predicting the high-level intent, the low-level trajectory generation is guided and constrained, so that the trajectory prediction is not only accurate, but also conforms to the joint flight law of bird flocks and has stronger logic.

[0033] (2) This invention uses a Unet segmentation method based on a multimodal large language model to segment birds in sky images. By understanding natural language questions through the multimodal large language model, it can simultaneously output accurate segmentation masks and text describing the spatial location of birds. This allows the model to not only circle birds but also indicate their locations, achieving a combination of perception and cognition. At the same time, it uses a cross-attention mechanism to dynamically fuse visual and linguistic features, replacing traditional splicing or addition operations, improving semantic alignment capabilities, and generating bird image-text pairs with location annotations. This provides an accurate location basis for bird flight trajectory detection and prediction, thereby increasing the ability of bird risk analysis. Attached Figure Description

[0034] Figure 1 A flowchart of an image processing-based method for analyzing bird risk around airports provided by this invention;

[0035] Figure 2 This is a flowchart illustrating a method for predicting features based on dynamic hierarchical graphs and multi-granularity features.

[0036] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0037] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0038] Example 1, see Figure 1 The present invention provides a method for risk analysis of birds around airports based on image processing, characterized by the following steps:

[0039] Step S1: Image acquisition, real-time acquisition of sky images around the airport using a camera;

[0040] Step S2: Bird identification. A convolutional neural network is used to identify and detect birds in the sky image to obtain the location detection results of all birds.

[0041] Step S3: Bird segmentation. A Unet segmentation method based on a multimodal large language model is used to segment the birds in the sky map to obtain a bird flock segmentation map.

[0042] Step S4: Intelligent prediction. A prediction method based on dynamic hierarchical maps and multi-granularity features is used to predict the flight path of each bird in the bird flock segmentation map, and the predicted flight path is obtained.

[0043] Step S5: Risk analysis, setting up hierarchical safety zones for the airport space, with each level corresponding to a different risk level. When the predicted flight route reaches the corresponding safety zone, the corresponding risk level is output.

[0044] Example 2: This example is based on the above example. Step S3: Bird segmentation. A Unet segmentation method based on a multimodal large language model is used to segment the birds in the sky image to obtain a bird flock segmentation map. Specifically, it includes the following steps:

[0045] Step S31: Visual feature extraction. The sky map is visually encoded using an image encoder based on a lightweight visual Transformer architecture to obtain a feature map.

[0046] Step S32: Joint semantic embedding generation, using a multimodal large language model to perform multimodal semantic understanding and joint semantic embedding generation on the text descriptions of the sky map and location detection results;

[0047] Step S33: Dynamic fusion, using a cross-attention mechanism to dynamically fuse visual features with semantic embeddings to generate conditional segmentation features: ;

[0048] In the formula, and This represents the projection layer, which maps features to the same dimension. and Representing visual features and semantic embedding, respectively. Represents the scaling factor. Represents the softmax function. Represents self-attention processing, Represents segmentation features;

[0049] Step S34: Mask prediction and segmentation. The segmentation features are progressively upsampled using a transposed convolutional layer to obtain a binary segmentation mask. The birds in the sky image are segmented according to the segmentation mask to obtain a bird flock segmentation map.

[0050] Example 3, this example is based on the above example, step S4 specifically includes the following steps:

[0051] Step S41: Dynamic hierarchical graph construction. A dynamic hierarchical graph is constructed based on the positions of all birds in the flock and their states at different time points. The dynamic hierarchical graph consists of nodes, spatial neighborhood edges, flight association edges, and temporal continuity edges.

[0052] Step S42: Node feature initialization, setting the nodes of each bird at each time step t. The flight speed, flight direction, and relative three-dimensional coordinates are defined as node features;

[0053] Step S43: Node embedding, mapping node features to a high-dimensional space through a linear layer to obtain feature representations. , where t is the time step node, and l is the feature representation of the node in the l-th linear layer;

[0054] Step S44: Layered graph encoding, perform multi-type dynamic encoding on the dynamic layered graph to obtain the flight behavior code for each bird;

[0055] Step S45: Intent decoding, encoding the flight behavior of each bird at the last time step. Intent recognition is performed based on a multilayer perceptron (MLP) to obtain intent features;

[0056] Step S46: Trajectory decoding, encoding the flight behavior of each bird at the last time step. The trajectory features are obtained by concatenating the trajectory features with their corresponding intent features and then decoding the trajectory using a multilayer perceptron.

[0057] Step S47: Trajectory synthesis and prediction output. The intention features and trajectory features are weighted and fused to produce a prediction output, which is a weighted combination of the micro trajectory and the macro flock flight path derived from the intention, i.e., the predicted flight path of each bird.

[0058] Example 4: This example is based on the above examples. Step S41 specifically includes the following steps:

[0059] Step S411: Node definition, defining each bird as a node at each time step t. During the real-time data collection period T, there are a total of N×T nodes, where N is the total number of birds;

[0060] Step S412: Define spatial neighborhood edges. Based on the spatial distance between birds, a bird is considered a node when the distance between them is less than a preset threshold 'a'. With nodes Establish undirected edges in the neighborhood between them;

[0061] Step S413: Define flight-related edges. Based on the relative positions and flight paths of the birds in space, two birds are considered nodes when their flight paths are the same and the distance between their relative positions is less than a preset threshold b. With nodes Establish directed edges connecting them;

[0062] Step S414: Define a time-coherent edge, which is a node of consecutive time steps of the same bird. and Establish a coherent directed edge;

[0063] Step S415: Define a dynamic layered graph by connecting all nodes and all edges to form a dynamic layered graph.

[0064] Example 5: This example, based on the above examples, applies the above scheme to risk warning of multiple bird flocks crossing flights:

[0065] Scenario Description: At dusk, two flocks of birds were observed flying towards the runway takeoff and landing area from different directions at an international airport. Flock A was a migratory flock flying at a higher altitude; Flock B was a native foraging flock flying at a lower altitude, but their flight paths intersected.

[0066] Implementation steps:

[0067] Image acquisition (S1)

[0068] High-definition cameras deployed on the east and west sides of the airport continuously capture images of the sky at a frame rate of 30fps and a resolution of 1920×1080.

[0069] Bird identification (S2)

[0070] The YOLOv7 model was used to detect images in real time, identifying two groups of approximately 50 birds in total, and outputting the bounding box coordinates of each bird.

[0071] Bird body segmentation (S3)

[0072] The Unet segmentation network, based on a multimodal large language model, is used as input. The input image and text prompt are: "Segment all birds in the image and output their location descriptions".

[0073] By fusing visual and semantic features through a cross-attention mechanism, a pixel-level mask for each bird is generated, and a text description such as "the bird is located in the upper left area of ​​the image, coordinates (120, 80)" is output.

[0074] Intelligent Prediction (S4)

[0075] Dynamic hierarchical graph construction:

[0076] Node: Each bird is a node in each frame;

[0077] Spatial neighborhood edge: Set a threshold a=50 pixels to connect neighboring birds;

[0078] Flight-related edges: Set a threshold of b=30 pixels to connect birds flying in the same direction;

[0079] Temporally coherent edges: connect consecutive frame nodes of the same bird;

[0080] Node features include velocity, orientation angle, and three-dimensional coordinates (x, y, height).

[0081] Layered graph coding:

[0082] Graph attention captures local interactions (such as avoidance behavior within a group).

[0083] Temporal convolution extracts the flight time sequence pattern of each bird;

[0084] Global self-attention identification of the cross-flight trends of two groups of birds;

[0085] Intent decoding: The MLP outputs the intent of each bird (such as "migration", "foraging", "avoidance");

[0086] Trajectory decoding: Integrating intent and behavior encoding to predict flight path for the next 30 seconds;

[0087] Risk Analysis (S5)

[0088] Set up three levels of security zones:

[0089] Red Zone: Within 100 meters of the runway, high risk;

[0090] Yellow Zone: Within 500 meters of the runway, medium risk;

[0091] Green zone: 500 meters away, low risk;

[0092] The system predicts that flock A will enter the yellow zone in 15 seconds and flock B will enter the red zone in 20 seconds, triggering a medium-to-high risk warning.

[0093] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0094] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0095] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. An image processing based bird risk analysis method for an airport perimeter, characterized in that: Specifically comprising the following steps: Step S1: image acquisition, acquiring the sky map of the airport surroundings in real time through a camera; Step S2: bird identification, using a convolutional neural network to identify and detect birds in the sky map to obtain the position detection results of all birds; Step S3: bird segmentation, using a Unet segmentation method based on a multi-modal large language model to segment the birds in the sky map to obtain a bird group segmentation map; Step S4: intelligent prediction, using a dynamic hierarchical graph and multi-granularity feature prediction method to predict the flight route of each bird in the bird group segmentation map to obtain a flight prediction route; Step S5: risk analysis, setting hierarchical safety zones for the airport space, each hierarchical zone corresponding to a different risk level, and when the flight prediction route reaches the corresponding safety zone, outputting the corresponding risk level.

2. The method of claim 1, wherein the method is characterized by: Step S4, specifically comprising the following steps: Step S41: dynamic hierarchical graph construction, constructing a dynamic hierarchical graph according to the positions of all birds in the bird group and their states at different time nodes, the dynamic hierarchical graph being composed of nodes, spatial neighborhood edges, flight correlation edges, and time coherence edges; Step S42: node feature initialization, defining the flight speed, flight direction and relative three-dimensional coordinates of each bird at each time step t as the node feature of each bird at each time step t as the node feature of each bird Step S43: Node embedding, mapping node features to a high-dimensional space through a linear layer to obtain feature representation where t is the time step node, and l is the feature representation of the node at the l-th layer linear layer. Step S44: hierarchical graph encoding, performing multi-type dynamic encoding on the dynamic hierarchical graph to obtain flight behavior encoding of each bird; Step S45: intention decoding, encoding the flight behavior of each bird at the last time step An intent recognition based on a multi-layer perception MLP is performed to obtain an intent feature. Step S46: trajectory decoding, encoding the flight behavior of each bird at the last time step and its corresponding intention feature are spliced, and trajectory features are obtained through multi-layer perception trajectory decoding. Step S47: trajectory synthesis and prediction output, weighting and fusing the intention features and trajectory features and outputting the prediction to obtain a weighted combination of the microscopic trajectory and the macroscopic bird group flight route derived from the intention, i.e., the flight prediction route of each bird.

3. The method of claim 2, wherein the method further comprises: determining a bird risk level for each of the plurality of areas based on the bird risk score and the bird risk level of the corresponding area. Step S41, specifically comprising the following steps: Step S411: Node definition, defining each bird as a node at each time step t In the time period T of real-time collection, there are N×T nodes in total, where N is the total number of birds; Step S412: Spatial neighborhood edge definition, based on the spatial distance between the positions of the birds, when the distance between the birds is less than a preset threshold a, a node neighborhood undirected edge is established between the nodes ; Step S413: flight correlation edge definition, based on the relative position of the birds in space and the flight route, when the flight routes of two birds are the same, and the distance of the relative position is less than a preset threshold b, an associated directed edge is established between the nodes and nodes ​ Step S414: Time consecutive edge definition, for the same bird's continuous time step node With Establish consecutive directed edge; Step S415: dynamic hierarchical graph definition, constructing a dynamic hierarchical graph from all nodes and edges.

4. The method of claim 3, wherein the method further comprises: Step S44, specifically comprising the following steps: Step S441: intra-layer graph attention network processing, for each node, performing graph attention processing in the set of its spatial neighborhood edges and flight correlation edges, i.e., calculating graph attention coefficients through feature representation, and performing aggregation operations on the feature representation through the attention coefficients and their corresponding feature representations to obtain intra-layer features that understand the local spatial environment and the flight influence between birds; Step S442: inter-layer time convolution network processing, for each node, performing inter-layer time convolution network processing in its time coherence edges, i.e., stacking the feature representations of all time steps of each bird into a feature tensor, applying one-dimensional causal convolution to the feature tensor to obtain inter-layer features that understand the flight regularity of each bird in a time period; Step S443: global attention processing, for each bird, concatenating its intra-layer features and inter-layer features to obtain an abstract representation, and performing global self-attention processing on the abstract representations of all birds to obtain global features that understand the global scene characteristics of the bird group; Step S444: feature fusion, weighting and fusing the intra-layer features, inter-layer features, and global features of each bird to obtain flight behavior encoding.

Images