Airport clearance zone scene twin modeling and dynamic target safety prediction method

Abstract

The invention discloses an airport clearance area scene twin modeling and dynamic target safety prediction method, belongs to the technical field of digital twin and computer modeling, and solves the problem of high difficulty in automatic modeling of a large airport scene. The method comprises the following steps: acquiring point cloud data of a static target in an airport clearance area and preprocessing the point cloud data; performing curved surface reconstruction by adopting a point cloud projection reconstruction method to obtain rapid three-dimensional modeling of the static target, including the three-dimensional coordinate of the static target and the semantic relationship of the static target; performing dynamic target recognition on the obtained video of the airport clearance area in real time by using an improved YOLOv8 network to obtain classified dynamic targets and a semantic relationship between the dynamic targets; performing real-time positioning on the identified dynamic target based on the improved BP network to obtain a three-dimensional coordinate of the dynamic target; and fusing the three-dimensional coordinates of the static target and the dynamic target in an un-ity engine to realize twin modeling of the clearance area, and dynamically updating in real time. The method is used for airport clearance area scene twin modeling.

Description

Technical Field

[0001] An airport clearance area scenario twin modeling and dynamic target safety prediction method is used for airport clearance area scenario twin modeling and belongs to the technical fields of digital twin and computer modeling. Background Art

[0002] The airport clearance area is the core area to ensure the safe takeoff, landing and flight of aircraft. Its scope usually expands several kilometers to dozens of kilometers outward centered on the airport runway, forming a three-dimensional airspace protection scope. With the rapid growth of global air transportation volume and the acceleration of urbanization, the protection of the clearance area faces multiple challenges: the expansion of urban high-rise buildings, the construction of wind power facilities, the popularization of low-altitude aircraft such as drones, and the dynamic changes of natural terrain and landforms all pose severe tests to the traditional clearance area management methods. According to statistics, there are more than a thousand aviation safety hazard incidents caused by obstacles in the clearance area globally every year, directly threatening aviation safety and operation efficiency.

[0003] There have been incidents where drones, birds, etc. entered the clearance area at the airport, affecting the alternate landing and takeoff of multiple flights and having a serious impact on aviation safety. Therefore, to strengthen the construction of the airport clearance area, it is necessary to enhance the intelligent construction and management level of the airport clearance area, promote the transformation of aviation safety supervision to digital and intelligent, which has far-reaching significance for improving the national airspace resource management level.

[0004] Digital twin technology is a technical system that constructs a "mirror model" of physical entities (such as equipment, systems, environments, etc.) in a virtual space through digital means and realizes the linkage between the virtual and the real through real-time data interaction. Its core lies in using technologies such as sensors, the Internet of Things (IoT), artificial intelligence (AI), big data, and simulation modeling to dynamically map the state, behavior, and environment of the physical world into the virtual space, forming a "digital copy" that can be updated in real time and analyzed and predicted. In addition, digital twin technology can also be applied to the airport clearance area. By constructing real-time mapping and interaction between the physical space and the digital space, it significantly improves the accuracy, dynamics, and intelligent level of clearance area management. Such as precise obstacle identification, real-time data fusion, risk prediction and active prevention and control, and regional planning. Therefore, constructing a twin model of the airport clearance area plays a key role in the informatization and intelligent management of the airport. However, the airport clearance area scenario not only involves static basic scenarios such as the terminal building, control tower, runway, and surrounding buildings, but also includes dynamic objects such as airplanes, birds, drones, and vehicles. The dynamics and uncertainties of various objects in the large airport scenario make the model association relationship complex, resulting in great difficulty in automatic scene modeling.

[0005] In summary, the following technical problems exist in the prior art:

[0006] The dynamic and uncertain nature of various objects within a large airport scenario makes the model association relationships complex, leading to difficulties in automatic scene modeling. These difficulties include low model accuracy, long processing times, high costs, and the inability to update in real-time dynamically. For example, in point cloud reconstruction, the speed is slow and the accuracy is low due to the influence of point cloud quality. Detection errors occur due to blurred target boundaries and similar target types. There are also problems with low positioning accuracy for complex small dynamic targets and the inability to achieve fast and accurate updates. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for twin modeling and dynamic target safety prediction in the airport clearance area, which solves the problem that the dynamic and uncertain nature of various objects within a large airport scenario makes the model association relationships complex, leading to difficulties in automatic scene modeling.

[0008] To achieve the above objective, the technical solution adopted by the present invention is as follows:

[0009] A method for twin modeling of the airport clearance area scene includes the following steps:

[0010] Step 1: Collect point cloud data of static targets in the airport clearance area and perform preprocessing.

[0011] Step 2: Use the point cloud projection reconstruction method to perform surface reconstruction on the preprocessed point cloud data of static targets, obtaining a rapid three-dimensional modeling of static targets, including the three-dimensional coordinates of static targets and the semantic relationships of static targets.

[0012] Step 3: Use the improved YOLOv8 network to perform real-time dynamic target recognition on the acquired video of the airport clearance area, obtaining classified dynamic targets and the semantic relationships of dynamic targets.

[0013] Step 4: Based on the improved BP network, perform real-time positioning on the recognized dynamic targets to obtain the three-dimensional coordinates of dynamic targets.

[0014] Step 5: Based on the three-dimensional coordinates of static targets and dynamic targets, fuse and implement twin modeling of the clearance area in the unity engine and update in real-time dynamically.

[0015] Furthermore, the specific steps of Step 1 are as follows:

[0016] Step 1.1: Use the LIO-SAM algorithm to collect point cloud data of static targets in the airport clearance area, obtaining the original static target data, where the static targets include the airport terminal building, runway, tower, and buildings around the airport terminal building.

[0017] Step 1.2: Preprocess the point cloud data of the static target, including removing discrete points from the point cloud data of the static target using open-source PCL software, that is, first performing voxel filtering for downsampling on the point cloud data of the static target, and then performing smoothing processing.

[0018] Furthermore, the specific steps of Step 2 are as follows:

[0019] Step 2.1: Use the point cloud projection reconstruction method to reconstruct the framework of the point cloud data of the static target obtained by preprocessing, that is, project the point cloud data of the static target obtained by preprocessing onto a plane to obtain the two-dimensional coordinates of each point cloud data, and arrange the point cloud data on the projected plane in a clockwise order. After arrangement, use the python loop code method to calculate multiple areas enclosed by the point cloud data in the plane, and select the polygon with the largest area and the most point cloud data inside. Among them, the python loop code method includes the polygon area calculation method and the grid method;

[0020] Step 2.2: Based on the selected polygon, use a point cloud data as the starting point, select the point cloud data within the distance threshold range, and screen the two point cloud data with the smallest angle formed by the starting point cloud data and the remaining point cloud data. Form a triangular mesh with the starting point cloud data and the two point cloud data obtained through screening, and then process the remaining point cloud data in the polygon with the most point cloud data until all point cloud data is processed. Finally, obtain multiple triangular meshes;

[0021] Step 2.3: Import multiple triangular meshes into the virtual environment to form the static target of the airport clearance area, that is, obtain the rapid three-dimensional modeling of the static target, including the three-dimensional coordinates of the static target and the semantic relationship of the static target.

[0022] Furthermore, in the improved YOLOv8 network in Step 3, dilated convolutional layers DConv are respectively added behind the third Conv layer and the fourth Conv layer of the backbone network of the YOLOv8 network, and a CAM attention mechanism module that assigns weights to the output of the backbone network according to importance and inputs it to the first layer of the neck network is added. Among them, the dilated convolutional layer DConv includes three identical dilated convolutions processed in sequence;

[0023] The dilated convolutional layer DConv behind the third Conv layer performs dilated convolution processing on the output of the third Conv layer and then inputs it to the next layer and the second connection layer Concat in the neck network;

[0024] The dilated convolutional layer DConv behind the fourth Conv layer performs dilated convolution processing on the output of the fourth Conv layer and then inputs it to the next layer and the first connection layer Concat in the neck network.

[0025] Furthermore, the improved BP network in step 4 is an improvement on the ASPP (Atrous Spatial Pyramid Pooling) in the original BP network, that is, the dilated convolution of 3*3, the dilated convolution of 7*7, the dilated convolution of 9*9, the deformable convolution of 3*3, and the asymmetric convolution of 3*1 are respectively used to replace the first, second, third, fourth, and fifth dilated convolutions of the original ASPP.

[0026] Furthermore, the specific steps of step 5 are as follows:

[0027] Step 5.1: Obtain the semantic relationships between dynamic targets and the semantic relationships between static targets in the airport clearance area;

[0028] Step 5.2: Based on the semantic relationships, establish a multi-level semantic constraint mechanism. The multi-level semantic constraint mechanism includes geometric information, attribute information, and topological relationships, and is set according to the actual physical security and national management of the airport clearance area;

[0029] Step 5.3: Determine the relevant parameters for the twin modeling of the clearance area through the method of "classification result + knowledge model", and map and instantiate the dynamic and static targets in the airport clearance area in the Unity engine based on the multi-level semantic constraint mechanism to establish a twin model of the airport clearance area scene. Among them, the relevant parameters include geometric parameters, physical parameters, environmental parameters, dynamic monitoring parameters, and classification parameters. The geometric parameters include the height, width, and three-dimensional boundary coordinates of the clearance area, the height, width, and three-dimensional coordinates of the dynamic and static targets. The physical parameters include the speed of the dynamic targets. The environmental parameters include wind speed and wind direction. The dynamic detection parameters include real-time displacement and real-time speed. The classification parameters include the classification of different static and dynamic targets;

[0030] Step 5.4: To achieve real-time dynamic update of dynamic targets, a video input of more than 1000fps is used to improve the optical flow prediction network to obtain real-time dynamic targets and update the dynamic targets in the twin model of the airport clearance area scene in real time.

[0031] Furthermore, the improved optical flow prediction network in step 5.4 means that the four convolutional long short-term memory networks in the spatio-temporal information module of the optical flow prediction network are respectively replaced by a 3*3 convolutional long short-term memory network, an asymmetric convolutional long short-term memory network of 3*1 and 1*3, an asymmetric convolutional long short-term memory network of 5*1 and 1*5, and a dilated convolutional long short-term memory network with a dilation rate of 3;

[0032] And a self-attention module is added between the spatio-temporal information module and the upsampling layer.

[0033] Furthermore, it is characterized in that:

[0034] Based on the initially obtained or updated airport clearance area scene twin model, combined with safety distance constraints, distance-based detection, vertical movement constraints, dynamic target endurance radius, and / or the influence of wind speed and direction on the speed of dynamic targets, predict whether the dynamic targets in the airport clearance area scene twin model are safe. Among them, distance-based detection is used to determine whether a dynamic target will collide with a static target during movement, distance-based detection is used to determine whether multiple dynamic targets are safe during movement, vertical movement constraints are used to determine whether a dynamic target is safe in the vertical direction during takeoff and landing, the dynamic target endurance radius constraint is to judge whether the fuel of a dynamic target that requires fuel supports the subsequent movement, and the influence of wind speed and direction on the speed of dynamic targets is to judge whether the influence on the speed and direction of a dynamic target during movement is safe.

[0035] Furthermore, the safety distance constraint represents the safety distance constraint parameters determined according to the type and movement state of the dynamic target. Among them, the types of dynamic targets include aircraft, vehicles, drones, and birds, and the movement states include stationary, taxiing, and takeoff. The calculation formula for the safety distance constraint parameters is:

[0036]

[0037] In the formula: v x represents the speed of dynamic target x at a certain moment, a max represents the maximum braking acceleration of dynamic target x, h x represents the size of dynamic target x, p x represents the safety distance coefficient of dynamic target x, and δ, ε, ∈ respectively represent the given speed influence coefficient, size influence coefficient, and safety distance influence coefficient;

[0038] Distance-based detection is to calculate whether the distance d ij (t) between dynamic target i and dynamic target j at time t is less than the safety threshold If it is less, it is considered that there is a path conflict between dynamic targets; otherwise, there is no conflict. The judgment formula is:

[0039]

[0040] In the formula, (x i , y i , z i ) represents the position of dynamic target i, and (x j , y j , z j ) represents the position of dynamic target j;

[0041] The vertical motion constraint restricts the vertical height of other dynamic targets x′ according to the current aircraft motion mode. Here, the current aircraft motion modes include landing, takeoff, and waiting. The calculation formula for the vertical height is as follows:

[0042]

[0043] In the formula, RReLU represents the randomized ReLU function, z x′ (y) represents the actual height of the dynamic target x′, represents the lowest allowable height of the dynamic target x′, represents the highest allowable height of the dynamic target x′;

[0044] The calculation formula for the maximum endurance radius of the dynamic target m that requires fuel is as follows:

[0045]

[0046] In the formula, represents the remaining fuel of the dynamic target m at the maximum endurance radius, represents the fuel consumption of the dynamic target m per unit of motion length, v m represents the real-time speed of the dynamic target i;

[0047] The speed influence of the dynamic target based on wind speed and wind direction means that the wind speed and wind direction will affect the speed and direction of the dynamic target during motion. The motion formula based on wind speed and wind direction is as follows:

[0048] v f = v i + βθ(α·v w )

[0049] In the formula, v f represents the speed of the dynamic target i after being affected by the wind speed and wind direction, v i represents the initial speed of the dynamic target i, v w represents the real-time wind speed, β represents the proportional coefficient of the wind direction in the motion direction, θ represents the wind direction, and α represents the proportional coefficient of the wind speed in the motion direction.

[0050] Compared with the prior art, the advantages of the present invention are as follows:

[0051] The present invention proposes an integrated method for constructing a digital twin airport clearance area with static fast modeling, dynamic real-time perception, and fusion. It uses point cloud data and the SLAM algorithm to perform three-dimensional reconstruction of the static scene in the clearance area. Secondly, for the dynamic targets in the area, an improved YOLOv8 network and dynamic positioning technology are used to achieve target discrimination and positioning. Then, the dynamic and static targets are fused to realize the simulation and real-time update of the clearance area, specifically as follows:

[0052] 1. In the process of establishing a point cloud dataset of static targets in the airport clearance area and performing surface reconstruction of point cloud data projection in the present invention, first, considering the complexity of the airport environment, the LIO-SAM algorithm is selected to collect point cloud data of the area, which is used as the basic scene data. This algorithm effectively improves the accuracy of the point cloud data. Secondly, preprocessing operations are performed on the point cloud data. The preprocessing can further improve the accuracy of the point cloud data. The PCL open-source software is used for the operation. Statistical discrete points of the point cloud data are removed. The quality of the point cloud data is improved through preprocessing. The point cloud projection reconstruction method is used to reconstruct the surface, forming a more complete triangular mesh. The formed triangular mesh is re-projected onto the surface using the topological relationship between points. This method effectively reduces the influence brought by the open point cloud data and reduces the generation of redundant faces. It not only improves the data collection speed and quality of the airport clearance area, but also effectively improves the quality of the point cloud data, improves the effect and accuracy of surface reconstruction, and fully maps the appearance and size of the static scene, laying a foundation for the subsequent work;

[0053] 2. The present invention performs real-time dynamic target detection on static and dynamic scenes (i.e., videos) based on the improved YOLOV8 network. Two dilated convolutional layers DConv are added to the backbone network of the original YOLOv8 network. Each dilated convolutional layer DConv includes three identical dilated convolutions processed in sequence, which are used to process feature maps of different scales, expand the receptive field, and improve the extraction accuracy. For the complex situation and diverse objects in the airport clearance area, the speed and accuracy of identifying static and dynamic targets are improved. The application of the CAM attention mechanism enhances the sensitivity of the network to different objects in complex environments, enabling it to more accurately identify the static and dynamic targets of the airport, thereby improving the accuracy and timeliness of airport area targets. The improved YOLOV8 network effectively integrates dilated convolution and the attention mechanism, enabling the improved YOLOV8 network to have strong adaptability in the face of complex situations, solving the problems of low accuracy and slow speed of dynamic target recognition in previous models in complex scenes. By improving and introducing different hyperparameters under the YOLOv8 architecture, the robustness of the network model is enhanced, and it remains effective even in complex scenes. Moreover, the accuracy and efficiency of dynamic target monitoring have been significantly improved, realizing efficient lightweight recognition. It can accurately identify dynamic targets and provide more accurate and effective technical support for the construction of an intelligent and information-based airport clearance area, that is, the improved YOLOv8 network solves the detection errors caused by fuzzy target boundaries and similar target types, improving the detection accuracy;

[0054] III. The present invention calculates spatial coordinates through an improved BP network based on the binocular vision positioning principle in combination with the projection matrix and camera coordinates, realizes the positioning of dynamic targets, significantly improves the real-time performance and accuracy of the control of the airport clearance area, the system can effectively identify and locate dynamic targets in the airport clearance area, ensuring the accurate perception of the geometric information, spatial range and type of dynamic targets, that is, through the strong global context of the improved BP network, reducing the calculation amount, enhancing the direction sensitivity and effectively targeting complex small objects, improving the positioning accuracy in complex situations;

[0055] IV. The present invention uses multi-layer semantic constraint conditions based on the obtained semantic relationships between dynamic targets and static targets in the airport clearance area to perform entity object mapping and instantiation of the airport clearance area, establish a twin model of the clearance area scene, and the technology of dynamic and static fusion enables the construction of the static scene to provide a real and reliable physical scene for the simulation of the dynamic scene, greatly improving the authenticity of the model;

[0056] The simulation of the airport clearance area based on Unity realizes the efficient and accurate simulation of the airport clearance area by using advanced physical models and rapid modeling techniques. Compared with other traditional methods, this method realizes the advantages of improving the accuracy of simulation and real-time dynamic update, which is beneficial to the intelligent and information-based monitoring and management of the airport clearance area;

[0057] V. The present invention is beneficial for modeling to more realistically map the movement trajectory, direction and speed of dynamic targets, and at the same time facilitates the movement prediction of dynamic targets in the airport clearance area, and judges whether the next movement is safe in combination with constraint conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and therefore should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0059] Figure 1 It is a schematic flow chart of point cloud reconstruction of the present invention;

[0060] Figure 2 It is a schematic structural diagram of the improved YOLOv8 network of the present invention. In the figure, inptu represents input, conv represents convolution operation, c2f represents compression of the feature map and feature aggregation, Dconv represents dilated convolution, concat represents feature map splicing, backbone represents the backbone network, Head represents the head network, neck represents the neck network, CAM represents the attention mechanism, and detect represents target detection inference;

[0061] Figure 3 This is a schematic diagram of the structure of the dilated convolution group added to the YOLOv8 network. The three figures from left to right show that the dilated convolution expands the receptive field;

[0062] Figure 4 This is a schematic diagram of the structure of the improved BP network in the present invention. In the figure, rate represents the dilation factor;

[0063] Figure 5 This is a schematic diagram of the structure of the improved optical flow prediction network in the present invention. Specific implementation manners

[0064] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0065] This article mainly focuses on the twin modeling and dynamic real-time update of the airport clearance area with the fusion of dynamic and static data. The targets in the airport clearance area are divided into static and dynamic targets, and a method of separate processing and then data fusion is adopted. The types of dynamic and static targets are determined through target recognition. For static targets, point cloud data collection, preprocessing and surface reconstruction are carried out. For dynamic targets, positioning technology is used to perform real-time positioning on dynamic targets, and the data of dynamic and static targets are fused to realize the simulation and dynamic real-time update of the airport clearance area.

[0066] A method for twin modeling of the airport clearance area scene includes the following steps:

[0067] The scene content of the airport clearance area is complex, involving not only static basic scenes such as the terminal building, tower, runway, and surrounding buildings, but also dynamic objects such as airplanes, birds, drones, and vehicles. There are differences in the modeling methods for static and dynamic objects. The number of static and dynamic objects in the airport is huge and the update speed is fast. There are complex connection relationships and uncertainties between the same and different objects. The current main modeling methods, namely manual and parametric modeling methods, are for static objects. However, the airport clearance area has significant dynamic characteristics, with high modeling difficulty and slow speed, making it difficult to meet the modeling requirements of the airport clearance area. To solve the complex and fast-updating airport clearance area, traditional methods have problems such as low simulation accuracy, long time consumption, high cost, and inability to update dynamically in real time. By using point cloud data and surface reconstruction, rapid modeling of the static scene in the clearance area is achieved. For static and dynamic scenes, target extraction and dynamic target positioning are adopted to achieve rapid scene modeling, and on this basis, dynamic update of the airport clearance area is realized. This achievement can directly serve the intelligent and information management of the airport, strengthen the development of methods and theories for smart airports, and has universal application value and strong practical significance.

[0068] Step 1: Collect the point cloud data of static targets in the airport clearance area and perform preprocessing;

[0069] Specific steps:

[0070] Step 1.1: Use the LIO-SAM algorithm to collect the point cloud data of static targets in the airport clearance area to obtain the original static target data. Among them, the static targets include the airport terminal building, runway, tower, and buildings around the airport terminal building;

[0071] Step 1.2: Perform preprocessing on the point cloud data of static targets, including using the PCL open-source software to remove discrete points from the point cloud data of static targets, that is, first perform voxel filtering for downsampling on the point cloud data of static targets, and then perform smoothing processing. This can solve the problem of incorrect point cloud data caused by discrete points and noise in traditional point cloud data reconstruction.

[0072] Step 2: Adopt the point cloud projection reconstruction method to perform surface reconstruction on the preprocessed point cloud data of static targets to obtain a rapid three-dimensional model of static targets, including the three-dimensional coordinates of static targets and the semantic relationships of static targets;

[0073] Specific steps:

[0074] Step 2.1: Use the point cloud projection reconstruction method to reconstruct the framework of the point cloud data of the static target obtained by preprocessing. That is, project the point cloud data of the static target obtained by preprocessing onto a plane to obtain the two-dimensional coordinates of each point cloud data, and arrange the point cloud data on the projected plane in a clockwise order. After arrangement, use the Python loop code method to calculate the areas of multiple polygons enclosed by the point cloud data in the plane, and select the polygon with the largest area and the most point cloud data inside. Among them, the Python loop code method includes the polygon area calculation method and the grid method; select the polygon with the largest area and the most point cloud data inside. The number of point clouds and the area included can ensure that no correct point cloud data is ignored and ensure the accuracy of the point cloud model, forming a framework to determine the basis for the subsequent point cloud reconstruction.

[0075] Step 2.2: Based on the selected polygon, take a point cloud data as the starting point, select the point cloud data within the distance threshold range, and screen the two point cloud data with the smallest angle formed by the starting point cloud data and the remaining point cloud data. Form a triangular mesh with the starting point cloud data and the two point cloud data obtained through screening. Then, process the remaining point cloud data in the polygon with the most point cloud data until all point cloud data is processed. Finally, obtain multiple triangular meshes. Select the two point clouds with the smallest angle formed with the starting point cloud data to avoid the appearance of long and narrow triangles in the plane that affect the reconstruction quality and make the reconstruction effect smoother.

[0076] Step 2.3: Import multiple triangular meshes into a virtual environment to form the static target of the airport clearance area, that is, obtain the rapid three-dimensional modeling of the static target, including the three-dimensional coordinates of the static target and the semantic relationship of the static target.

[0077] Step 3: Use the improved YOLOv8 network to perform real-time dynamic target recognition on the acquired video of the airport clearance area to obtain the classified dynamic targets and the semantic relationships of the dynamic targets.

[0078] The improved YOLOv8 network adds dilated convolutional layers DConv after the third Conv layer and the fourth Conv layer of the backbone network of the YOLOv8 network respectively, and adds a CAM attention mechanism module that assigns weights to the output of the backbone network according to importance and inputs it to the first layer of the neck network. Among them, the dilated convolutional layer DConv includes three identical dilated convolutions processed in sequence; the three identical dilated convolutions solve the problems of losing semantic information and ignoring local consistency in traditional methods, which have a greater impact on extracting target details and lead to inaccurate extraction when there are similar targets and the distance between targets is close. The added dilated convolution effectively expands the receptive field. Traditional dilated convolutions have a grid effect that affects extraction accuracy, while the improved dilated convolution avoids the grid effect. The DConv in the structure diagram is the improved dilated convolution with dilation rates of 1, 3, and 9 respectively, after the third Conv and fourth Conv operations and before the C2F feature aggregation. The CAM attention mechanism assigns weights according to importance.

[0079] The dilated convolutional layer DConv behind the third Conv layer performs dilated convolution processing on the output of the third Conv layer and then inputs it to the next layer and the second connection layer Concat in the neck network;

[0080] The dilated convolutional layer DConv behind the fourth Conv layer performs dilated convolution processing on the output of the fourth Conv layer and then inputs it to the next layer and the first connection layer Concat in the neck network.

[0081] Step 4: Based on the improved BP network, perform real-time positioning on the recognized dynamic targets to obtain the three-dimensional coordinates of the dynamic targets;

[0082] The improved BP network improves the ASPP atrous spatial pyramid in the original BP network, that is, replaces the first, second, third, fourth, and fifth dilated convolutions of the original ASPP atrous spatial pyramid with 3*3 dilated convolution, 7*7 dilated convolution, 9*9 dilated convolution, 3*3 deformable convolution, and 3*1 asymmetric convolution respectively. The traditional ASPP model has insufficient processing of global context information and poor performance for complex small objects. Therefore, the improved ASPP atrous spatial pyramid is added between the last convolution layer and the fully connected layer of the convolutional feature extraction layer, reducing the computational amount, enhancing direction sensitivity, and effectively targeting complex small objects. The improved BP network for binocular vision positioning can improve the positioning accuracy in complex situations.

[0083] Step 5: Based on the three-dimensional coordinates of the static targets and dynamic targets, fuse and implement the twin modeling of the clearance area in the unity engine and update it in real time dynamically.

[0084] The specific steps are as follows:

[0085] Step 5.1: Obtain the semantic relationships between dynamic objects and the semantic relationships between static objects in the airport clearance area;

[0086] Step 5.2: Based on the semantic relationships, establish a multi-level semantic constraint mechanism. The multi-level semantic constraint mechanism includes geometric information, attribute information, and topological relationships, and is set according to the actual physical safety and national management of the airport clearance area;

[0087] Step 5.3: Determine the relevant parameters for the twin modeling of the clearance area through the method of "classification result + knowledge model", and map and instantiate the dynamic and static objects in the airport clearance area in the Unity engine based on the multi-level semantic constraint mechanism to establish a twin model of the airport clearance area scene. Among them, the relevant parameters include geometric parameters, physical parameters, environmental parameters, dynamic monitoring parameters, and classification parameters. The geometric parameters include the height, width, and three-dimensional boundary coordinates of the clearance area, the height, width, and three-dimensional coordinates of the dynamic and static objects. The physical parameters include the speed of the dynamic object. The environmental parameters include wind speed and wind direction. The dynamic detection parameters include real-time displacement and real-time speed. The classification parameters include the classification of different static and dynamic objects;

[0088] Step 5.4: To achieve real-time dynamic update of dynamic objects, a video with a frame rate above 1000fps is used to input the improved optical flow prediction network to obtain real-time dynamic objects and update the dynamic objects in the twin model of the airport clearance area scene in real time.

[0089] The improved optical flow prediction network refers to replacing the four convolutional long short-term memory networks in the spatio-temporal information module of the optical flow prediction network with 3*3 convolutional long short-term memory networks, 3*1 and 1*3 asymmetric convolutional long short-term memory networks, 5*1 and 1*5 asymmetric convolutional long short-term memory networks, and dilated convolutional long short-term memory networks with a dilation rate of 3; and adding a self-attention module between the spatio-temporal information module and the upsampling layer.

[0090] A dynamic target safety prediction method for airport clearance area scenarios, which predicts whether the dynamic targets in the twin model of the airport clearance area scenario are safe based on the initially obtained or updated twin model of the airport clearance area scenario in combination with safety distance constraints, distance-based detection, vertical movement constraints, dynamic target endurance radius, and / or the speed influence of the dynamic target based on wind speed and wind direction. Among them, distance-based detection is used to determine whether a dynamic target will collide with a static target during movement, distance-based detection is used to determine whether multiple dynamic targets are safe during movement, vertical movement constraints are used to determine whether a dynamic target is safe in the vertical direction during takeoff and landing, the dynamic target endurance radius constraint is to judge whether the fuel of a dynamic target that requires fuel supports the next movement, and the speed influence of the dynamic target based on wind speed and wind direction is to judge, and the speed influence of the dynamic target based on wind speed and wind direction is used to determine whether the influence on the speed and direction of the dynamic target during movement is safe.

[0091] The safety distance constraint refers to the safety distance constraint parameters determined according to the type and movement state of the dynamic target. Among them, the types of dynamic targets include aircraft, vehicles, drones, and birds, and the movement states include stationary, taxiing, and takeoff. The calculation formula for the safety distance constraint parameters is:

[0092]

[0093] In the formula: v x represents the speed of dynamic target x at a certain moment, a max represents the maximum braking acceleration of dynamic target x, h x represents the size of dynamic target x, p x represents the safety distance coefficient of dynamic target x, and δ, ε, ∈ respectively represent the given speed influence coefficient, size influence coefficient, and safety distance influence coefficient;

[0094] Distance-based detection is to calculate whether the distance d ij (t) between dynamic target i and dynamic target j at time t is less than the safety threshold If it is less, it is considered that there is a path conflict between the dynamic targets, otherwise, there is no. The judgment formula is:

[0095]

[0096] In the formula, (x i , y i , z i ) represents the position of dynamic target i, and (x j , y j , z j ) represents the position of dynamic target j;

[0097] The vertical motion constraint restricts the vertical height of other dynamic targets x′ according to the current aircraft motion mode, where the current aircraft motion mode includes landing, takeoff, and waiting. The calculation formula for the vertical height is as follows:

[0098]

[0099] In the formula, RReLU represents the randomized ReLU function, z x′ (y) represents the actual height of the dynamic target x′, represents the lowest allowable height of the dynamic target x′, represents the highest allowable height of the dynamic target x′;

[0100] The calculation formula for the maximum endurance radius of the dynamic target m that requires fuel is as follows:

[0101]

[0102] In the formula, represents the remaining fuel of the dynamic target m at the maximum endurance radius, represents the fuel consumption of the dynamic target m per unit of motion length, v m represents the real-time speed of the dynamic target i;

[0103] The speed influence of the dynamic target based on wind speed and wind direction means that the wind speed and wind direction will affect the speed and direction of the dynamic target during motion. The motion formula based on wind speed and wind direction is as follows:

[0104] v f = v i + βθ(α·v w )

[0105] In the formula, v f represents the speed of the dynamic target i after being affected by wind speed and wind direction, v i represents the initial speed of the dynamic target i, v w represents the real-time wind speed, β represents the proportional coefficient of the wind direction in the motion direction, θ represents the wind direction, and α represents the proportional coefficient of the wind speed in the motion direction.

Claims

1. A method for twin modeling of airport clearance area scenarios, characterized in that, It includes the following steps: Step 1: Collect the point cloud data of static objects in the airport clearance area and perform preprocessing; Step 2: Use the point cloud projection reconstruction method to reconstruct the surface of the point cloud data of the static objects after preprocessing, and obtain a rapid 3D model of the static objects, including the 3D coordinates of the static objects and the semantic relationships of the static objects; Step 3: Use the improved YOLOv8 network to perform real-time dynamic object recognition on the video of the airport clearance area obtained, and obtain the classified dynamic objects and the semantic relationships of the dynamic objects; Step 4: Based on the improved BP network, perform real-time positioning on the recognized dynamic objects to obtain the 3D coordinates of the dynamic objects; Step 5: Based on the 3D coordinates of the static objects and the dynamic objects, fuse and implement the twin modeling of the clearance area in the unity engine, and update it in real time dynamically.

2. The twin modeling method for airport clearance area scenes according to claim 1, wherein, The specific steps of Step 1: Step 1.1: Use the LIO-SAM algorithm to collect the point cloud data of static objects in the airport clearance area to obtain the original static object data, where the static objects include the airport terminal building, runway, tower, and buildings around the airport terminal building; Step 1.2: Perform preprocessing on the point cloud data of the static objects, including using the PCL open-source software to remove discrete points from the point cloud data of the static objects, that is, first perform voxel filtering for downsampling on the point cloud data of the static objects, and then perform smoothing processing.

3. A method for twin modeling of an airport clearance area scene according to claim 1 or 2, characterized in that, The specific steps of Step 2: Step 2.1: Use the point cloud projection reconstruction method to perform framework reconstruction on the point cloud data of the static objects obtained after preprocessing, that is, project the point cloud data of the static objects obtained after preprocessing onto a plane to obtain the two-dimensional coordinates of each point cloud data, and arrange the point cloud data on the projected plane in a clockwise order. After arranging, use the python loop code method to calculate the multiple areas enclosed by the point cloud data in the plane, and select the polygon with the largest area and the most point cloud data inside. Among them, the python loop code method includes the polygon area calculation method and the grid method; Step 2.2: Based on the selected polygon, use a point cloud data as the starting point, select the point cloud data within the distance threshold range, and screen the two point cloud data with the smallest angle formed by the starting point cloud data and the remaining point cloud data. Form a triangular mesh with the starting point cloud data and the two point cloud data obtained after screening, and then process the remaining point cloud data in the polygon with the most point cloud data until all the point cloud data is processed. Finally, obtain multiple triangular meshes; Step 2.3: Import the multiple triangular meshes into the virtual environment to form the static objects in the airport clearance area, that is, obtain a rapid 3D model of the static objects, including the 3D coordinates of the static objects and the semantic relationships of the static objects.

4. A method for twin modeling of an airport clearance area scene according to claim 3, characterized in that, In step 3, the improved YOLOv8 network adds dilated convolutional layers DConv after the third Conv layer and the fourth Conv layer of the backbone network of the YOLOv8 network, and adds a CAM attention mechanism module that assigns weights to the output of the backbone network according to importance and inputs it to the first layer of the neck network. Among them, the dilated convolutional layer DConv includes three identical dilated convolutions processed sequentially; The dilated convolutional layer DConv after the third Conv layer performs dilated convolution processing on the output of the third Conv layer and then inputs it to the next layer and the second connection layer Concat in the neck network; The dilated convolutional layer DConv after the fourth Conv layer performs dilated convolution processing on the output of the fourth Conv layer and then inputs it to the next layer and the first connection layer Concat in the neck network.

5. A method for twin modeling of an airport clearance area scene according to claim 4, characterized in that: In step 4, the improved BP network improves the ASPP atrous spatial pyramid in the original BP network, that is, replaces the first, second, third, fourth, and fifth dilated convolutions of the original ASPP atrous spatial pyramid with 3*3 dilated convolution, 7*7 dilated convolution, 9*9 dilated convolution, 3*3 deformable convolution, and 3*1 asymmetric convolution respectively.

6. The twin modeling method for airport clearance area scenarios according to claim 5, characterized in that The specific steps of step 5 are as follows: Step 5.1, obtain the semantic relationships between dynamic objects and the semantic relationships between static objects in the airport clearance area; Step 5.2, based on the semantic relationships, establish a multi-level semantic constraint mechanism. The multi-level semantic constraint mechanism includes geometric information, attribute information, and topological relationships, and the multi-level semantic constraint mechanism is set according to the actual physical safety and national management of the airport clearance area; Step 5.3, determine the relevant parameters for the twin modeling of the clearance area through the method of "classification result + knowledge model", and map and instantiate the dynamic objects and static objects in the airport clearance area in the Unity engine based on the multi-level semantic constraint mechanism to establish a twin model of the airport clearance area scene. Among them, the relevant parameters include geometric parameters, physical parameters, environmental parameters, dynamic monitoring parameters, and classification parameters. The geometric parameters include the height, width, and three-dimensional boundary coordinates of the clearance area, the height, width, and three-dimensional coordinates of the dynamic and static objects. The physical parameters include the speed of the dynamic object. The environmental parameters include wind speed and wind direction. The dynamic detection parameters include real-time displacement and real-time speed. The classification parameters include the classification of different static objects and dynamic objects; Step 5.4, to achieve real-time dynamic update of dynamic objects, a video input with more than 1000fps is used to improve the optical flow prediction network to obtain real-time dynamic objects and update the dynamic objects in the twin model of the airport clearance area scene in real time.

7. A method for twin modeling of an airport clearance area scene according to claim 6, characterized in that, The improved optical flow prediction network in step 5.4 refers to replacing the four convolutional long short-term memory networks in the spatio-temporal information module of the optical flow prediction network with 3*3 convolutional long short-term memory network, 3*1, 1*3 asymmetric convolutional long short-term memory network, 5*1, 1*5 asymmetric convolutional long short-term memory network, and dilated convolutional long short-term memory network with a dilation rate of 3; A self-attention module is added between the spatio-temporal information module and the upsampling layer.

8. A method for dynamically predicting the safety of moving targets in the airport clearance area according to any one of claims 1-7, characterized in that: Based on the initially obtained or updated twin model of the airport clearance area scene, combined with safety distance constraints, distance-based detection, vertical movement constraints, the endurance radius of moving targets, and / or the speed influence of moving targets based on wind speed and wind direction, it is predicted whether the moving targets in the twin model of the airport clearance area scene are safe. Among them, distance-based detection is used to determine whether a moving target will collide with a static target during movement, distance-based detection is used to determine whether multiple moving targets are safe during movement, vertical movement constraints are used to determine whether a moving target is safe in the vertical direction during takeoff and landing, the endurance radius constraint of a moving target is to judge whether the fuel of a moving target that requires fuel supports the subsequent movement, and the speed influence of a moving target based on wind speed and wind direction is to judge, and the speed influence of a moving target based on wind speed and wind direction is used to determine whether the influence on the speed and direction of a moving target during movement is safe.

9. A method for dynamically predicting the safety of moving targets in the airport obstacle-free zone scene according to claim 8, characterized in that: The safety distance constraint represents the safety distance constraint parameters determined according to the type and movement state of the moving target. Among them, the types of moving targets include airplanes, vehicles, drones, and birds, and the movement states include stationary, taxiing, and takeoff. The calculation formula for the safety distance constraint parameters is: where: v x represents the velocity of the dynamic target x at a certain moment, a max represents the maximum braking acceleration of the dynamic target x, h x represents the size of the dynamic target x, p x represents the safety distance coefficient of the dynamic target x, and δ, ε, ∈ respectively represent the given velocity influence coefficient, size influence coefficient, and safety distance influence coefficient; Distance-based detection is performed by calculating the distance d between the dynamic object i and the dynamic object j at time t ij (t) is less than the safety threshold If it is less, it is considered that there is a path conflict between the dynamic objects; otherwise, there is no conflict. The judgment formula is as follows: where (x i , y i , z i ) represents the position of dynamic target i, and (x j , y j , z j ) represents the position of dynamic target j; The vertical movement constraint is to constrain the vertical height of other moving targets x′ according to the current movement mode of the aircraft. Among them, the current movement modes of the aircraft include landing, takeoff, and waiting. The calculation formula for the vertical height is: wherein, RReLU represents the randomized ReLU function, and z x′ (y) represents the actual height of the dynamic target x′, represents the lowest height allowed for the dynamic target x′, represents the highest height allowed for the dynamic target x′; The calculation formula for the maximum endurance radius of a moving target m that requires fuel is: In the formula, represents the remaining fuel of the dynamic target m at the maximum endurance radius, represents the fuel consumed by the dynamic target m per unit of movement length, v m represents the real-time speed of the dynamic target i; The speed influence of a moving target based on wind speed and wind direction means that wind speed and wind direction will affect the speed and direction of a moving target during movement. The movement formula based on wind speed and wind direction is: v f = v i + βθ(α · v w ) where, v f represents the speed of the dynamic target i after being affected by the wind speed and direction, v i represents the initial speed of the dynamic target i, v w represents the real-time wind speed, β represents the proportionality coefficient of the wind direction in the moving direction, θ represents the wind direction, and α represents the proportionality coefficient of the wind speed in the moving direction.

Images