Method and system for visual monitoring of airport concealed works

Abstract

The application discloses an airport concealed engineering visual monitoring method and system, relates to the technical field of intelligent airports, and comprises the following steps: collecting image data, depth data and environmental parameters of airport concealed engineering in real time through a plurality of source sensors, performing dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters to obtain concealed engineering feature data, extracting structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, performing dynamic threshold segmentation on the abnormal features, positioning abnormal areas, constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal areas in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal. Through the above method, the technical problems of low abnormal state monitoring accuracy and low abnormal positioning efficiency caused by manual inspection or single sensor fixed-point monitoring in the airport concealed engineering are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of smart airport, in particular to an airport concealed engineering visual monitoring method and system. BACKGROUND

[0002] Airport concealed engineering (such as underground pipe network, foundation structure, internal support, etc.) is a key component to ensure the safe and stable operation of the airport, but its structure state is difficult to directly observe. Traditional monitoring methods mainly rely on manual inspection or single sensor fixed-point monitoring, which has limited coverage, poor real-time performance, and is prone to missed detection. In the prior art, although attempts have been made to combine image or point cloud data for modeling analysis, it is still difficult to achieve high-precision anomaly recognition and intuitive visual alarm due to factors such as noise interference, data heterogeneity, and dynamic environmental changes, resulting in difficulty in timely discovering and locating safety hazards.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide an airport concealed engineering visual monitoring method and system, which aims to solve the technical problems of low accuracy of abnormal state monitoring and low efficiency of abnormal positioning in the prior art of airport concealed engineering manual inspection or single sensor fixed-point monitoring.

[0005] To achieve the above purpose, the present application provides an airport concealed engineering visual monitoring method, the method comprising:

[0006] Real-time acquisition of image data, depth data and environmental parameters of the airport concealed engineering by multi-source sensors, dynamic noise filtering and data alignment processing of the acquired image data, depth data and environmental parameters to obtain concealed engineering feature data;

[0007] Extracting structure features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, the concealed engineering perception model being obtained based on optimization of an initial concealed engineering perception model, the initial concealed engineering perception model being used for abnormal perception of the concealed engineering feature data;

[0008] Dynamic threshold segmentation of the abnormal features to locate abnormal areas;

[0009] Constructing a concealed engineering real scene model based on the structure features, superimposing the abnormal areas in highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal.

[0010] In an embodiment, the step of performing dynamic noise filtering and data alignment on the collected image data, depth data and environmental parameters to obtain concealed engineering feature data comprises:

[0011] processing the image data based on a hybrid denoising algorithm of guided filtering and adaptive median filtering to obtain denoised image data;

[0012] performing dynamic noise suppression on the depth data through a temporal Kalman filter, and removing drift points in combination with a point cloud statistical outlier removal algorithm to obtain noise-suppressed depth data;

[0013] performing pixel-level spatial alignment of the denoised image data and the noise-suppressed depth data through feature point matching and epipolar constraint methods, and establishing a unified coordinate system;

[0014] constructing a data credibility weight based on the environmental parameters, and performing weighted fusion of the denoised image data and the noise-suppressed depth data based on the data credibility weight to generate concealed engineering feature data with spatio-temporal consistency in the unified coordinate system.

[0015] In an embodiment, the concealed engineering perception model is composed of an improved multi-scale feature pyramid network and a bidirectional long short-term memory network fusion, and the step of extracting structural features and abnormal features in the concealed engineering feature data based on the concealed engineering perception model comprises:

[0016] inputting the concealed engineering feature data into an improved multi-scale feature pyramid network to extract local detail features and global context features;

[0017] temporally modeling the local detail features and the global context features based on a bidirectional long short-term memory network to determine dynamic evolution features of a concealed engineering state;

[0018] determining a saliency weight of each region based on the dynamic evolution features of the concealed engineering state, and updating a response strength of an abnormal region based on the saliency weight;

[0019] performing redundancy compression on the local detail features and the global context features according to the response strength to obtain structural features and abnormal features.

[0020] In an embodiment, before the step of extracting structural features and abnormal features in the concealed engineering feature data based on the concealed engineering perception model, the method further comprises:

[0021] obtaining an airport concealed engineering historical data set containing normal and multiple typical abnormal states, and finely labeling abnormal regions in the historical data set to construct a training sample set;

[0022] taking a pre-trained convolutional neural network as a backbone network of an initial concealment engineering perception model, and initializing a bidirectional long short-term memory network;

[0023] training the initial concealment engineering perception model based on a focal loss function and the training sample set to obtain concealment engineering perception model features;

[0024] migrating the concealment engineering perception model features to a multi-scale feature pyramid network to obtain an improved multi-scale feature pyramid network;

[0025] obtaining the concealment engineering perception model according to the improved multi-scale feature pyramid network and the initialized bidirectional long short-term memory network.

[0026] In an embodiment, the step of dynamically threshold segmenting the abnormal features and locating abnormal regions includes:

[0027] determining an abnormal feature map based on the abnormal features, determining a mean value and a standard deviation in a neighborhood of each pixel point in the abnormal feature map, and obtaining an adaptive segmentation threshold value according to the mean value and the standard deviation;

[0028] performing binaryzation processing on the abnormal feature map based on the adaptive segmentation threshold value to obtain a binaryzation feature map, and eliminating scattered point noise of the binaryzation feature map and smoothing an abnormal region boundary based on morphological opening operation and closing operation to obtain the abnormal region.

[0029] In an embodiment, the step of eliminating scattered point noise of the binaryzation feature map and smoothing an abnormal region boundary based on morphological opening operation and closing operation to obtain the abnormal region includes:

[0030] eliminating scattered point noise of the binaryzation feature map and smoothing an abnormal region boundary based on morphological opening operation and closing operation to obtain an initial abnormal region;

[0031] determining a positioning result of an abnormal region of a previous time sequence frame, performing position prediction and state updating on the initial abnormal region of the current frame to obtain a corrected abnormal region;

[0032] determining a feature intensity mean value and an area of the corrected abnormal region, and determining the corrected abnormal region as an abnormal region when the feature intensity mean value and the area are both greater than a preset threshold.

[0033] In an embodiment, the step of constructing a concealment engineering real scene model based on the structure features, superimposing the abnormal region in a highlighted form into the concealment engineering real scene model, and outputting a visual alarm signal includes:

[0034] reconstructing the structural features to generate a three-dimensional network model of the concealed engineering, and performing texture mapping based on the image data to obtain a real scene model of the concealed engineering;

[0035] determining spatial coordinates of the abnormal area, mapping the abnormal area to the real scene model of the concealed engineering based on the spatial coordinates, determining an alarm color according to the abnormal type of the abnormal area, and performing highlight rendering on the alarm color;

[0036] generating a visual alarm signal based on the spatial coordinates and the abnormal type.

[0037] In an embodiment, the step of reconstructing the structural features to generate a three-dimensional network model of the concealed engineering, and performing texture mapping based on the image data to obtain a real scene model of the concealed engineering includes:

[0038] processing the point cloud data in the structural features based on a Poisson surface reconstruction algorithm to generate a smooth and closed three-dimensional grid model of the concealed engineering;

[0039] calculating an accurate mapping relationship between the image data and the vertices of the three-dimensional grid model based on a unified coordinate system;

[0040] generating a seamless texture map according to the image data fusion, and mapping the texture map to the three-dimensional grid model according to the accurate mapping relationship to output a real scene model of the concealed engineering.

[0041] In an embodiment, after the step of constructing a real scene model of the concealed engineering based on the structural features, superimposing the abnormal area in a highlighted form into the real scene model of the concealed engineering, and outputting a visual alarm signal, the method further includes:

[0042] determining a severity level of the abnormal type, and dividing the visual alarm signal into three levels of pre-alarm, alarm, and emergency alarm based on the severity level;

[0043] dynamically displaying an alarm information panel in a color corresponding to the level on a user interface of the real scene model of the concealed engineering, and driving a peripheral audible and light alarm to emit an audible and light alarm of a corresponding level;

[0044] generating an alarm log containing the spatial coordinates of the abnormal area, the abnormal type, the alarm level, and a time stamp, and pushing the alarm log and the visual alarm signal to a remote monitoring center in real time through a network interface.

[0045] In addition, to achieve the above-mentioned purpose, the present application also provides an airport concealed engineering visual monitoring system, which comprises:

[0046] The data processing module is configured to collect image data, depth data and environmental parameters of the airport concealed engineering in real time through the multi-source sensor, perform dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters, and obtain concealed engineering feature data.

[0047] The anomaly perception module is configured to extract structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, wherein the concealed engineering perception model is obtained by optimizing an initial concealed engineering perception model, and the initial concealed engineering perception model is used for anomaly perception of the concealed engineering feature data.

[0048] The anomaly positioning module is configured to perform dynamic threshold segmentation on the abnormal features and locate abnormal regions.

[0049] The visual monitoring module is configured to construct a concealed engineering real scene model based on the structural features, superimpose the abnormal regions in a highlighted form into the concealed engineering real scene model, and output a visual alarm signal.

[0050] In addition, to achieve the above object, the application further provides an airport concealed engineering visual monitoring device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the above airport concealed engineering visual monitoring method.

[0051] In addition, to achieve the above object, the application further provides a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the above airport concealed engineering visual monitoring method.

[0052] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above airport concealed engineering visual monitoring method.

[0053] The application provides an airport concealed engineering visual monitoring method, which comprises the following steps: collecting image data, depth data and environmental parameters of the airport concealed engineering in real time through a multi-source sensor, performing dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters to obtain concealed engineering feature data, extracting structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, performing dynamic threshold segmentation on the abnormal features, positioning an abnormal area, constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal area in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal. Through the above method, the technical problems of low abnormal state monitoring accuracy and low abnormal positioning efficiency of manual inspection or single sensor fixed-point monitoring in the airport concealed engineering are solved. BRIEF DESCRIPTION OF DRAWINGS

[0054] The accompanying drawings incorporated in and forming a part of the specification illustrate embodiments consistent with the present application and serve to explain the principles of the application, together with the description.

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0056] Figure 1 The flowchart of the airport concealed engineering visual monitoring method of the first embodiment of the present application;

[0057] Figure 2 The flowchart of determining the abnormality of the first embodiment of the airport concealed engineering visual monitoring method of the present application;

[0058] Figure 3 The visual diagram of the first embodiment of the airport concealed engineering visual monitoring method of the present application;

[0059] Figure 4 The module structure diagram of the airport concealed engineering visual monitoring system of the embodiment of the present application;

[0060] Figure 5 The device structure diagram of the hardware running environment involved in the airport concealed engineering visual monitoring method of the embodiment of the present application.

[0061] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0062] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application, and are not used to limit the present application.

[0063] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0064] The main solution of the embodiment of the present application is: real-time collection of image data, depth data and environmental parameters of the airport concealed engineering by a multi-source sensor, dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters, to obtain concealed engineering feature data;

[0065] Extracting structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, the concealed engineering perception model being obtained based on optimization of an initial concealed engineering perception model, the initial concealed engineering perception model being used for abnormal perception of the concealed engineering feature data;

[0066] Dynamic threshold segmentation on the abnormal features to locate abnormal areas;

[0067] Constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal areas in highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal.

[0068] At present, airport concealed engineering (such as underground pipe network, foundation structure, internal support, etc.) is a key component to ensure the safe and stable operation of the airport, but its structural state is difficult to directly observe. The traditional monitoring means mainly rely on manual inspection or single sensor fixed-point monitoring, which has the problems of limited coverage, poor real-time performance, easy to miss detection, etc. In the prior art, although attempts have been made to combine image or point cloud data for modeling analysis, but affected by noise interference, data heterogeneity, dynamic environmental changes and other factors, it is still difficult to achieve high-precision abnormal identification and intuitive visual alarm, resulting in that safety hazards are difficult to be found and located in time.

[0069] The present application provides a solution, which real-time collects image data, depth data and environmental parameters of the airport concealed engineering by a multi-source sensor, performs dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters, to obtain concealed engineering feature data, extracts structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, performs dynamic threshold segmentation on the abnormal features to locate abnormal areas, constructs a concealed engineering real scene model based on the structural features, superimposes the abnormal areas in highlighted form into the concealed engineering real scene model, and outputs a visual alarm signal. Through the above-mentioned manner, the technical problems of low accuracy of abnormal state monitoring and low efficiency of abnormal positioning by manual inspection or single sensor fixed-point monitoring in the airport concealed engineering are solved.

[0070] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an airport covert engineering visualization monitoring device, etc., and the embodiment does not make specific limitations. The following takes the airport covert engineering visualization monitoring device as an example to describe the embodiment and the following embodiments.

[0071] All actions of obtaining signals, information or data in this application are carried out in accordance with the corresponding data protection regulations and policies of the country where the system is located, and with the authorization given by the corresponding system owner.

[0072] The embodiment of the application provides an airport covert engineering visualization monitoring method, which refers to Figure 1 , Figure 1 The flowchart of the first embodiment of the airport covert engineering visualization monitoring method of the application is shown in the figure.

[0073] In the embodiment, the airport covert engineering visualization monitoring method comprises steps S10-S40:

[0074] Step S10, real-time collection of image data, depth data and environmental parameters of the airport covert engineering by a multi-source sensor, dynamic noise filtering and data alignment processing of the collected image data, depth data and environmental parameters, and obtaining of covert engineering feature data.

[0075] It should be noted that the multi-source sensor refers to a combination of multiple different types of sensors with complementary functions. It usually includes image sensors such as high-definition cameras and infrared thermal imagers, which are used to obtain visual information such as color, texture, cracks and leakage of the surface of the covert engineering. Depth sensors such as LiDAR, three-dimensional scanners and depth cameras are used to obtain three-dimensional geometric information such as coordinates, shapes, contours and settlement deformation of the surface of the covert engineering. Environmental parameter sensors such as temperature and humidity sensors, strain gauges and vibration sensors are used to collect environmental data (temperature, humidity) and structural response data (stress, vibration frequency).

[0076] It can be understood that, due to the complexity of the airport environment, the original sensor data contains a large amount of noise, and direct use will seriously affect the accuracy of subsequent analysis. At the same time, image, depth and other data sources come from different sensors, and they have differences in spatial position and timestamp. If not accurately aligned, effective cross-modal feature correlation and analysis cannot be achieved. Therefore, first, a hybrid strategy of guided filtering and adaptive median filtering is adopted to effectively smooth the noise while protecting the image edge; through the time sequence Kalman filter and the point cloud denoising algorithm, the depth data is dynamically smoothed and the abnormal points are removed, to ensure the accuracy of the three-dimensional structure. Subsequently, using computer vision techniques such as feature point matching, the visual texture and three-dimensional geometry are pixel-level aligned, and a unified coordinate system is established. Finally, the reliability weight based on the environmental parameters is introduced for data fusion, and a set of pure, aligned and reliable hidden engineering feature data is finally generated, providing high-quality input for the subsequent abnormal perception model, and fundamentally guaranteeing the accuracy and reliability of the entire monitoring system.

[0077] In a feasible implementation manner, the step of performing dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters to obtain hidden engineering feature data comprises:

[0078] The image data is processed based on a hybrid denoising algorithm of guided filtering and adaptive median filtering to obtain denoised image data;

[0079] The depth data is dynamically noise-suppressed through a time sequence Kalman filter, and drift points are removed in combination with a point cloud statistical outlier removal algorithm to obtain noise-suppressed depth data;

[0080] The denoised image data and the noise-suppressed depth data are pixel-level spatially aligned through feature point matching and epipolar constraint methods, and a unified coordinate system is established;

[0081] A data reliability weight is constructed based on the environmental parameters, and the denoised image data and the noise-suppressed depth data are weighted and fused based on the data reliability weight to generate hidden engineering feature data with spatio-temporal consistency in the unified coordinate system.

[0082] It can be understood that, when denoising the image, the idea of can be adopted, wherein is the original image. The adaptive median filter first processes impulse noise, and the window size is adaptive to the local noise level; the guided filter takes the original image as the guide, uses a linear model to smooth the pixels in the window , wherein is a parameter solved by minimizing the cost function . is the input pixel, is a regularization parameter, so that the edges are preserved while denoising, is the denoised image.

[0083] Temporal and spatial filters are combined. The Kalman filter smoothes in the temporal domain, whose state vector can be position and velocity, and the state transition equation is used to recursively estimate the state vector is the state transition matrix, is the process noise) and the observation equation is used to recursively estimate the state vector is the observation matrix, is the observation noise, to obtain more stable depth values. Then, statistical outlier removal is applied in three-dimensional space: for a point in the point cloud P, the average distance to its k-nearest neighbors is calculated, the average distance of the entire point cloud is calculated, and the standard deviation is calculated. All points satisfying (n is usually taken as 1.0-3.0) are removed to obtain pure depth data.

[0084] Cross-modal data alignment is achieved through computer vision geometric principles. SIFT, ORB, and other algorithms are used to extract image feature points, and matching algorithms under epipolar constraints (such as descriptor distance-based and epipolar verification) are used to find accurate correspondence between the image and the depth map, and then through solving the fundamental matrix or homography matrix, pixel-level spatial registration is completed, and a unified world coordinate system or camera coordinate system is established.

[0085] Finally, the fusion is not a simple splicing, but a weighted sum based on credibility. According to the environmental parameter vector E, the credibility weight of each data source is calculated through a predefined or learned function. In the unified coordinate system, the generated feature data can be represented as a weighted fusion of each modal feature. This dynamic weighting ensures that in harsh environments, the system can automatically rely on more reliable data sources, thereby generating high spatiotemporal consistency and stable quality of hidden engineering feature data, laying a solid foundation for subsequent anomaly detection.

[0086] Step S20, extracting structural features and abnormal features in the hidden engineering feature data based on a hidden engineering perception model, wherein the hidden engineering perception model is obtained by optimizing an initial hidden engineering perception model, and the initial hidden engineering perception model is used for anomaly perception of the hidden engineering feature data.

[0087] It should be noted that the concealed engineering perception model is the core artificial intelligence analysis module of the embodiment, which is a deep learning network specially designed and optimized for understanding and analyzing the state of concealed engineering, and can simultaneously analyze the structural features and abnormal features from the input data. The initial concealed engineering perception model refers to a convolutional neural network that has been pre-trained on a large-scale general-purpose dataset and has basic feature extraction capabilities, which serves as the basis for building a specialized perception model. Structural features refer to the inherent properties of concealed engineering in a normal and healthy state, such as the flatness of the pipeline surface, the contour shape of the structure, and the high-dimensional abstract representation of the texture of the material, which is the basis for building a visual reality model. Abnormal features refer to features that are significantly different from the learned normal structure pattern, which quantify the likelihood and extent of potential defects such as cracks, deformations, and corrosion, and are the basis for abnormal positioning and alarm.

[0088] It can be understood that the perception model usually adopts an encoder-decoder architecture. The encoder part is based on a pre-trained backbone network and is improved, such as adding a multi-scale feature pyramid FPN, which is responsible for efficiently extracting rich spatial features from local details to global context from the preprocessed fused data; At the same time, in order to capture the dynamic behavior of the structure state over time, such as crack propagation and settlement trend, the model will integrate a bidirectional long short-term memory network to model the temporal characteristics of these spatial features. Subsequently, the decoder part will calculate a saliency map based on the learned dynamic pattern to weight the feature responses of different regions, highlighting abnormal regions and suppressing irrelevant backgrounds. Finally, the network separates the processed features into two parts through specific design: one is the structural features used for accurate reconstruction of the appearance of the engineering, and the other is the abnormal features used to identify potential problems. This process essentially combines prior knowledge with domain-specific data, enabling the model to decouple normal structures and abnormal patterns from complex multi-source data.

[0089] In a feasible implementation, the concealed engineering perception model is composed of an improved multi-scale feature pyramid network and a bidirectional long short-term memory network, and the step of extracting structural features and abnormal features from the concealed engineering feature data based on the concealed engineering perception model includes:

[0090] inputting the concealed engineering feature data into the improved multi-scale feature pyramid network to extract local detail features and global context features;

[0091] temporally modeling the local detail features and the global context features based on the bidirectional long short-term memory network to determine the dynamic evolution features of the concealed engineering state;

[0092] determining the saliency weights of each region based on the dynamic evolution features of the concealed engineering state, and updating the response strength of the abnormal region based on the saliency weights.

[0093] According to the response intensity, the local detail features and the global context features are redundantly compressed to obtain structure features and anomaly features.

[0094] In a specific implementation, with reference to Figure 2 , Figure 2 a flowchart for determining anomalies. The preprocessed hidden engineering feature data with spatiotemporal consistency is input into an improved multi-scale feature pyramid network. The network takes a pre-trained backbone network as an encoder to extract feature maps at different levels. Let the input data be , T is the time length, H, W, and C are the height, width, and channel number. Then the feature map output by the encoder at the i-th layer is . .

[0095] The traditional feature pyramid network fuses multi-scale features through a top-down path and lateral connection. The improvement of the embodiment is that an adaptive spatial attention mechanism is introduced to reweight the features at each layer before fusion, so that they pay more attention to the key structural regions. For the feature map , its attention weight is calculated as follows:

[0096]

[0097] where is a Sigmoid function, is a 1x1 convolution used for dimension reduction and channel information integration, and are global pooling operations. The weighted feature , is an element-wise multiplication. Finally, the fused multi-scale feature contains rich information from local detail features to global context features.

[0098] The obtained multi-scale feature sequence is flattened along the spatial dimension and converted into sequence data , which is then input into a bidirectional long short-term memory network for temporal modeling. Bi-LSTM contains two LSTMs in forward and backward directions, which process the sequence in forward and reverse order respectively. For each time step t, the output of Bi-LSTM is the concatenation of the forward hidden state and the backward hidden state :

[0099]

[0100] The final output H of Bi-LSTM is the dynamic evolution characteristics of the hidden engineering state that captures the historical and future context information. The dynamic evolution characteristics H are passed through a lightweight convolution layer and a Sigmoid function to generate a saliency weight map that identifies the degree of temporal evolution anomaly of each spatial position. Finally, the model performs redundant compression and feature separation on the enhanced features through two different decoder branches. One decoder (usually composed of transpose convolution or up-sampling layers) is used to accurately reconstruct the normal appearance of the input data, and the loss function is usually L1 or L2 reconstruction loss. The output of this branch is the structural feature representing the engineering health state. Another decoder is used to highlight the abnormal area by complementing the structural feature. The output of this branch is the abnormal feature.

[0101] In a feasible implementation, before the step of extracting the structural feature and the abnormal feature in the hidden engineering feature data based on the hidden engineering perception model, the method further comprises:

[0102] obtaining an airport hidden engineering historical data set containing normal and multiple typical abnormal states, and finely labeling the abnormal areas in the historical data set to construct a training sample set;

[0103] using a pre-trained convolutional neural network as the backbone network of the initial hidden engineering perception model, and initializing a bidirectional long short-term memory network;

[0104] training the initial hidden engineering perception model based on a focal loss function and the training sample set to obtain a hidden engineering perception model feature;

[0105] migrating the hidden engineering perception model feature to a multi-scale feature pyramid network to obtain an improved multi-scale feature pyramid network;

[0106] obtaining a hidden engineering perception model according to the improved multi-scale feature pyramid network and the initialized bidirectional long short-term memory network.

[0107] In a specific implementation, this is the cornerstone of the success of the model. First, a large amount of historical data of airport concealed engineering needs to be obtained, which should cover as many scenarios as possible (such as terminal, runway, and apron), different pipeline materials (metal, PVC), and various typical abnormal states (such as cracks, corrosion, leakage, deformation, and loose bolts, etc.). The key step is to finely label the abnormal area. This not only needs a bounding box, but also a pixel-level fine segmentation mask to accurately define the shape and range of the anomaly. For example, a crack needs to be accurately outlined with its winding contour, rather than simply boxed in a rectangle. The labeled data is divided into training set, validation set and test set, which together constitute the training sample set. In order to improve the training efficiency and model performance, a transfer learning strategy is adopted. A convolutional neural network pre-trained on a large general image dataset is selected as the encoder of the initial concealed engineering perception model. The parameters of the bidirectional long short-term memory network are usually randomly initialized, because the time sequence pattern is quite different between general data and concealed engineering data. Next, the feature extraction layer of the pre-trained backbone network is "grafted" into the multi-scale feature pyramid network. Specifically, the bottom-up path of FPN is directly composed of the corresponding stages of the pre-trained backbone network. In this way, FPN has the ability to output high-quality feature maps from different levels at the beginning of training, which is the "improvement" of the original. It inherits powerful pre-trained feature representation and lays a high starting point for subsequent multi-scale feature fusion.

[0108] Since the concealed engineering anomaly usually appears as a small and discontinuous pixel area in the image, there is an extreme class imbalance problem. The standard cross-entropy loss function will be "overwhelmed" by the dominant normal pixels, causing the model to tend to predict all pixels as normal. To this end, the focal loss function is used in this embodiment to solve this problem. The focal loss is introduced on the basis of the standard cross-entropy loss , to reduce the weight of easily classified samples and let the model focus on difficult-to-classify samples. For a binary classification problem (abnormal / normal), the focal loss is defined as:

[0109]

[0110] where, is the probability that the model predicts the sample to belong to its true class, is a modulation factor (usually ≥ 1) to control the rate of weight reduction of easily classified samples. The greater the value, the smaller the contribution of easily classified samples, is a factor for balancing class weights.

[0111] In training, the total loss of the model is usually the weighted sum of the focal loss and the reconstruction loss. Through this loss function design, the model is forced to learn two goals simultaneously: 1) accurately reconstruct the entire scene; 2) sensitively identify abnormal regions with subtle deviations from the normal pattern.

[0112] After the above training is completed, an encoder with strong feature extraction capability is obtained. Finally, the trained improved multi-scale feature pyramid network is integrated with a bidirectional long short-term memory network trained and converged on time series data, and two decoder branches, to form a mature covert engineering perception model that can be used for actual deployment.

[0113] Step S30, dynamically thresholding the abnormal features to locate the abnormal regions.

[0114] It should be noted that the abnormal features refer to high-dimensional data output by the covert engineering perception model, which can quantitatively represent the possibility of each pixel or region being abnormal, usually embodied as an abnormal heat map or saliency map, where a higher value represents a higher possibility of abnormality. Locating the abnormal regions refers to extracting the pixels exceeding the threshold in the abnormal feature map through segmentation techniques to form connected regions with clear geometric boundaries, thereby accurately marking the location, shape, and range of abnormality in the original engineering scene.

[0115] It should be understood that first, a statistical analysis is performed on the abnormal feature map, or the local context information of the image is combined to adaptively determine an optimal segmentation threshold; then, the threshold is applied to binarize the abnormal feature map to generate a binary mask, where white pixels (value 1) represent regions determined to be abnormal; finally, a series of post-processing operations are usually performed, such as using morphological opening operation to remove small noise points, or using region growing, contour finding, and other algorithms to merge adjacent abnormal pixels into complete abnormal regions, thereby achieving accurate abnormal positioning.

[0116] In one possible implementation, the step of dynamically thresholding the abnormal features to locate the abnormal regions includes:

[0117] determining an abnormal feature map based on the abnormal features, determining the mean and standard deviation in the neighborhood of each pixel point in the abnormal feature map, and obtaining an adaptive segmentation threshold based on the mean and the standard deviation;

[0118] performing binarization processing on the abnormal feature map based on the adaptive segmentation threshold to obtain a binary feature map, and eliminating scattered point noise and smoothing the abnormal region boundary of the binary feature map based on morphological opening and closing operations to obtain an abnormal region.

[0119] In a specific implementation, using a global threshold directly on the whole abnormal feature map usually does not work well, because the abnormal response intensity output by the model can vary across the whole image. Therefore, an adaptive threshold based on the local neighborhood of each pixel is calculated here. After obtaining the abnormal feature map where each pixel value represents the probability or response intensity of the position belonging to an abnormality. For each pixel point in the feature map a local window of size is defined with it as the center. Within this local window, the mean and standard deviation of all pixel values are calculated. The mean reflects the average response level of the background in this local region. The standard deviation reflects the degree of dispersion, i.e. variability, of the response in this local region. Then the adaptive threshold is determined, which is linearly combined from the mean and the standard deviation, and the calculation formula is usually:

[0120]

[0121] where is an adjustable sensitivity coefficient, the larger the value, the higher the threshold, and the more conservative the segmented abnormal region, the smaller the value, the lower the threshold, and the more sensitive the segmentation result.

[0122] After obtaining the adaptive threshold of each pixel point, binarization segmentation and optimization are performed, and the abnormal feature value of each pixel point is compared with its corresponding adaptive threshold to generate a binarized feature map. The preliminary binarized map often contains isolated noise points and the boundaries of the abnormal region can be rough and uneven. Therefore, an erosion operation is performed first, followed by a dilation operation. Erosion can effectively eliminate small and isolated noise points (scattered point noise), while subsequent dilation can restore pixels that belong to the true abnormal region but have been mistakenly deleted, without restoring isolated noise that has been eroded. Then a dilation operation is performed, followed by an erosion operation. Closing operation can fill small holes inside the abnormal region and smooth its boundaries, making the outline more complete and coherent. In practical applications, open and close operations are usually performed in sequence according to the morphological characteristics of noise and abnormal regions, or only one of them is performed. The final purified binarized map, i.e. the located and connected abnormal region, can be directly superimposed on the original engineering image or model to complete precise abnormal visualization positioning.

[0123] In a feasible implementation, the step of eliminating scattered point noise in the binarized feature map and smoothing the boundaries of the abnormal region based on morphological open and close operations to obtain the abnormal region comprises:

[0124] Eliminate the scatter noise of the binarized feature map and smooth the abnormal region boundary based on morphological opening and closing operations to obtain an initial abnormal region;

[0125] Determine the positioning result of the abnormal region of the previous time sequence frame, perform position prediction and state update on the initial abnormal region of the current frame to obtain a modified abnormal region;

[0126] Determine the feature intensity mean and area of the modified abnormal region, and when the feature intensity mean and the area are both greater than a preset threshold, determine the modified abnormal region as an abnormal region.

[0127] In a specific implementation, a simple morphological processing only considers the spatial features of a single frame of image, while an anomaly of a hidden engineering (such as a slowly developing crack or a leakage) has a time sequence continuity. Therefore, state prediction and update can be performed, and the system can maintain a tracking list to record the position, size, motion speed and other state information of each identified abnormal region in a previous time sequence frame (such as a t-1 moment). For each known abnormal region in the t-1 frame, a simple linear motion model is used to predict the possible position and boundary box of the abnormal region in the current t frame. A commonly used algorithm is Kalman filtering, which can optimally combine historical motion information and current observation uncertainty for prediction. The initial abnormal region obtained by the morphological processing of the current frame is associated and matched with the predicted abnormal region. The initial region that is successfully matched is considered as the current observation value of the corresponding abnormal region. Subsequently, the observation value is used to update the state of the abnormal region to obtain a more smooth and accurate modified abnormal region. For the initial region that cannot be matched with any existing track, it is temporarily determined as a “newly appeared candidate region”.

[0128] The region corrected in time and space also needs to be examined for “reasonableness” in intensity and area to prevent misjudgment of small and insignificant fluctuations as anomalies that need to be concerned. The average value of the corresponding values of all pixels in the modified abnormal region on the original abnormal feature map is calculated. A preset threshold is set. This threshold ensures that the finally confirmed anomaly must have a high enough confidence or significant degree, and filters out those regions with weak model response and ambiguous boundaries. The pixel area of the modified abnormal region is calculated. A preset threshold is set. This threshold is based on prior knowledge and excludes point noise or small defects with an area that is too small and does not have practical significance in engineering, for example, the difference of a few pixels.

[0129] Step S40, based on the structural features, a hidden engineering real scene model is constructed, the abnormal region is superimposed on the hidden engineering real scene model in a highlighted form, and a visual alarm signal is output.

[0130] It is necessary to explain that the structural feature refers to the information collected by a sensing device (such as an RGB-D camera, a laser radar) for describing the three-dimensional spatial structure of the concealed project, such as point cloud data, mesh model, depth map, etc. It is the basis for constructing a digital model with real geometric size and spatial relationship. The real scene model of the concealed project is a three-dimensional model for digital reconstruction of the concealed project (such as tunnel segment, underground pipe network, internal structure of building) in the real world. The model reflects the geometric shape, texture and spatial position of the project, and provides an intuitive context for anomaly positioning.

[0131] It can be understood that first, the internal and external parameters (calibration parameters) of the camera and the pose information are used to establish the accurate mapping relationship (i.e. inverse projection) between the two-dimensional image pixel points and the surface points of the three-dimensional real scene model. Then, the binary abnormal area contour is calculated on the corresponding area of the three-dimensional model through the mapping relationship. Then, in the rendering engine, a specific highlight material (such as light emission, semi-transparent coloring) is applied to the three-dimensional area, so that it is highlighted in the overall model. Finally, the system automatically generates and pops up an interactive three-dimensional alarm view, and the user can view the abnormal details from any angle, which may be accompanied by sound and light prompts and a detailed list of abnormal properties, so as to complete the whole process from data sensing to intelligent warning.

[0132] In a feasible implementation, the step of constructing a real scene model of the concealed project based on the structural feature, superimposing the abnormal area in a highlight form into the real scene model of the concealed project, and outputting a visual alarm signal comprises:

[0133] Surface reconstruction is performed on the structural feature to generate a three-dimensional network model of the concealed project, and texture mapping is performed based on the image data to obtain a real scene model of the concealed project;

[0134] The spatial coordinates of the abnormal area are determined, the abnormal area is mapped to the real scene model of the concealed project based on the spatial coordinates, the warning color is determined according to the abnormal type of the abnormal area, and the warning color is highlighted rendered;

[0135] A visual alarm signal is generated based on the spatial coordinates and the abnormal type.

[0136] In a specific implementation, the system utilizes pre-acquired structural features to generate a three-dimensional mesh model of the concealed engineering through a surface reconstruction algorithm (converts these discrete point clouds into a continuous, complete three-dimensional mesh model. This mesh model is composed of countless triangular facets, accurately describing the geometric morphology of the concealed engineering (such as the inner wall of the tunnel, the surface of the pipeline). The only geometric model lacks realism. To this end, the system uses the acquired image data (high-definition RGB images) as the texture source, determines the shooting angle and position of each photo through camera calibration parameters (internal and external parameters), calculates a set of two-dimensional coordinates (UV coordinates) for each vertex of the three-dimensional mesh, which is equivalent to "unfolding" the three-dimensional model onto a two-dimensional plane, projects multiple images from different angles onto the UV map of the model according to the registration information, and fuses and stitches them to eliminate seams and lighting differences, and finally generates a seamless high-definition texture map. After the mapping is completed, a realistic concealed engineering model is obtained.

[0137] Using the internal and external parameters of the camera, the mapping relationship from the image pixel point to the three-dimensional space ray can be established. The intersection point of this ray and the above-mentioned three-dimensional mesh model is calculated. This intersection point is the spatial coordinate of the pixel point on the surface of the three-dimensional model. Repeat this process for all pixels within the abnormal area contour to obtain the distribution range of the abnormal area in three-dimensional space. Thus, the pixel position of the two-dimensional abnormal area is converted to the three-dimensional world coordinate. After obtaining the three-dimensional space coordinate, the system will mark the three-dimensional model facet corresponding to the coordinate as an abnormal facet. The system internally presets a color mapping table, which applies a highlight material to these marked abnormal facets in the graphics rendering engine. Usually, semi-transparent coloring or glowing effect is used, for example, a 50% transparent red layer is superimposed on the original texture, so that the abnormal area can be highlighted and identified, and the original surface texture details are not completely blocked, facilitating engineers to observe the specific morphology of the abnormal area. Refer to Figure 3 , Figure 3 For visualized schematic diagram. The system automatically generates structured alarm information based on the mapped spatial coordinates and abnormal types. This is not just a simple pop-up window, but an integrated visualization interface that is presented to the user through a monitor screen, computer terminal, or mobile device App. At the same time, the system usually accompanies with sound and light alarm or message push to ensure that relevant personnel can timely learn about the alarm information.

[0138] In a feasible implementation, the step of surface reconstruction of the structural features, generating a three-dimensional mesh model of the concealed engineering, and texture mapping based on the image data to obtain a concealed engineering model includes:

[0139] Processing the point cloud data in the structural features based on the Poisson surface reconstruction algorithm to generate a smooth and closed three-dimensional mesh model of the concealed engineering;

[0140] calculating an accurate mapping relationship between the image data and vertices of the three-dimensional mesh model based on the unified coordinate system;

[0141] generating a seamless texture map according to the image data fusion, and mapping the texture map to the three-dimensional mesh model according to the accurate mapping relationship, and outputting a concealed engineering real scene model.

[0142] In a specific implementation, first, the point cloud data and its normal vector information are converted into a continuous vector field. The direction of the vector field points to the inside and outside of the model. Then, the problem of the vector field is converted into a problem of solving a Poisson equation. In simple terms, it is to find a three-dimensional function whose gradient best matches the vector field, and the isosurface of the function is the final generated smooth surface. Input the dense point cloud with normal vector information. The normal vector can be directly provided by the scanning device or calculated by analyzing the local neighborhood of the point cloud. Then an octree data structure is used to efficiently discretize the three-dimensional space. The depth of the octree is a key parameter that determines the resolution of the reconstructed model. The greater the depth, the more detailed the model, but the greater the amount of calculation. Solve the Poisson equation in the octree represented space to get the indicator function. Finally, use the marching cubes algorithm to extract the triangular mesh surface from the indicator function.

[0143] In the data acquisition stage, through precise calibration, the data of all sensors is converted to the same world coordinate system. This means that each three-dimensional point cloud and an image pixel point know their exact position in the global space. For each vertex of the three-dimensional mesh model, use the camera parameters to project it from the world coordinate system to the image plane that may capture it, to get one or more two-dimensional pixel coordinates, to determine the accurate mapping relationship between the image data and the vertices of the three-dimensional mesh model. Then generate a seamless texture map according to the image data fusion, and map the texture map to the three-dimensional mesh model according to the accurate mapping relationship, and output a concealed engineering real scene model.

[0144] In a feasible implementation, after the step of constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal area in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal, the method further comprises:

[0145] determining the severity level of the abnormal type, and dividing the visual alarm signal into three levels of pre-warning, warning, and emergency warning based on the severity level;

[0146] dynamically displaying an alarm information panel in a color corresponding to the level on the user interface of the concealed engineering real scene model, and driving a peripheral audible and light alarm to issue an audible and light warning of the corresponding level;

[0147] Generate an alarm log containing the spatial coordinates of the abnormal area, the type of anomaly, the alarm level and the timestamp, and push the alarm log and the visual alarm signal to the remote monitoring center in real time through the network interface.

[0148] In a specific implementation, first, the severity level of the abnormal type is automatically determined according to a preset rule base, for example, fine cracks are classified as a warning, significant cracks are classified as an alarm, and through cracks that may cause structural instability are classified as an emergency alarm. Subsequently, the visual interface dynamically responds to the alarm level: a color corresponding alarm information panel is rendered beside the three-dimensional real scene model, and a physical sound and light alarm is driven through a GPIO interface or an industrial bus protocol to provide intuitive warnings with different frequencies of flashing and tones. At the same time, the system encapsulates a structured alarm log (in JSON or XML format) that accurately records the spatial coordinates of the abnormal area, the type, the level and the timestamp accurate to milliseconds, and finally pushes the log and the real-time rendered three-dimensional alarm picture stream to the remote monitoring center through network protocols such as MQTT or WebSocket, ensuring that monitoring personnel can immediately obtain panoramic visual information and structured data to achieve rapid positioning and decision-making.

[0149] The embodiment provides an airport concealed engineering visual monitoring method, image data, depth data and environment parameters of an airport concealed engineering are collected in real time through a plurality of source sensors, dynamic noise filtering and data alignment processing are carried out on the collected image data, depth data and environment parameters, concealed engineering feature data is obtained, structure features and abnormal features in the concealed engineering feature data are extracted based on a concealed engineering perception model, dynamic threshold segmentation is carried out on the abnormal features, an abnormal area is located, a concealed engineering real scene model is constructed based on the structure features, the abnormal area is superimposed on the concealed engineering real scene model in a highlighted form, and a visual alarm signal is output. Through the above manner, the technical problems of low accuracy of abnormal state monitoring and low efficiency of abnormal positioning in manual inspection or single sensor fixed-point monitoring of the airport concealed engineering are solved.

[0150] It should be noted that the above examples are only used for understanding the present application and do not constitute a limitation on the airport concealed engineering visual monitoring method of the present application, and more forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0151] The present application also provides an airport concealed engineering visual monitoring system, please refer to Figure 4 The airport concealed engineering visual monitoring system comprises:

[0152] The data processing module 10 is configured to collect image data, depth data and environment parameters of an airport concealed engineering in real time through a plurality of source sensors, perform dynamic noise filtering and data alignment processing on the collected image data, depth data and environment parameters, and obtain concealed engineering feature data.

[0153] an anomaly perception module 20 configured to extract structural features and anomaly features from the concealment engineering feature data based on a concealment engineering perception model, wherein the concealment engineering perception model is optimized based on an initial concealment engineering perception model, and the initial concealment engineering perception model is used for anomaly perception of the concealment engineering feature data;

[0154] an anomaly positioning module 30 configured to perform dynamic threshold segmentation on the anomaly features and locate an anomaly region;

[0155] a visual monitoring module 40 configured to construct a concealment engineering real scene model based on the structural features, superimpose the anomaly region in a highlighted form into the concealment engineering real scene model, and output a visual alarm signal.

[0156] In an embodiment, the data processing module 10 is further configured to process the image data based on a hybrid denoising algorithm of guided filtering and adaptive median filtering to obtain denoised image data.

[0157] The depth data is subjected to dynamic noise suppression by a time series Kalman filter, and drift points are removed by a point cloud statistical outlier removal algorithm to obtain noise-suppressed depth data.

[0158] The denoised image data and the noise-suppressed depth data are subjected to pixel-level spatial alignment by a feature point matching and epipolar constraint method, and a unified coordinate system is established.

[0159] A data credibility weight is constructed based on the environmental parameters, and the denoised image data and the noise-suppressed depth data are weighted and fused based on the data credibility weight to generate concealment engineering feature data with spatio-temporal consistency in the unified coordinate system.

[0160] In an embodiment, the anomaly perception module 20 is further configured to input the concealment engineering feature data into an improved multi-scale feature pyramid network to extract local detail features and global context features.

[0161] The local detail features and the global context features are subjected to time series modeling based on a bidirectional long short-term memory network to determine dynamic evolution features of a concealment engineering state.

[0162] Significance weights of regions are determined based on the dynamic evolution features of the concealment engineering state, and response intensities of anomaly regions are updated based on the significance weights.

[0163] The local detail features and the global context features are subjected to redundancy compression according to the response intensities to obtain structural features and anomaly features.

[0164] In an implementable embodiment, the anomaly perception module 20 is further configured to obtain a historical data set of airport concealed engineering containing normal and multiple typical abnormal states, and finely label abnormal regions in the historical data set to construct a training sample set;

[0165] The pre-trained convolutional neural network is used as a backbone network of an initial concealed engineering perception model, and a bidirectional long short-term memory network is initialized;

[0166] The initial concealed engineering perception model is trained based on a focal loss function and the training sample set to obtain concealed engineering perception model features;

[0167] The concealed engineering perception model features are migrated to a multi-scale feature pyramid network to obtain an improved multi-scale feature pyramid network;

[0168] The concealed engineering perception model is obtained according to the improved multi-scale feature pyramid network and the initialized bidirectional long short-term memory network.

[0169] In an implementable embodiment, the anomaly positioning module 30 is further configured to determine an abnormal feature map based on the abnormal features, determine a mean value and a standard deviation in a neighborhood of each pixel point in the abnormal feature map, and obtain an adaptive segmentation threshold value according to the mean value and the standard deviation;

[0170] The abnormal feature map is binarized based on the adaptive segmentation threshold value to obtain a binarized feature map, and the binarized feature map is subjected to morphological opening and closing operations to eliminate scattered point noise and smooth an abnormal region boundary, so as to obtain an abnormal region.

[0171] In an implementable embodiment, the anomaly positioning module 30 is further configured to eliminate scattered point noise and smooth an abnormal region boundary of the binarized feature map based on morphological opening and closing operations to obtain an initial abnormal region;

[0172] The positioning result of the abnormal region of a previous time sequence frame is determined, and the initial abnormal region of the current frame is subjected to position prediction and state updating to obtain a corrected abnormal region;

[0173] The mean value of feature intensity and the area of the corrected abnormal region are determined, and when the mean value of feature intensity and the area are both greater than a preset threshold, the corrected abnormal region is determined as an abnormal region.

[0174] In an implementable embodiment, the visual monitoring module 40 is further configured to perform surface reconstruction on the structural features to generate a three-dimensional network model of the concealed engineering, and perform texture mapping based on the image data to obtain a real scene model of the concealed engineering;

[0175] determine a spatial coordinate of the abnormal area, map the abnormal area to the real scene model of the concealed project based on the spatial coordinate, determine an alarm color according to the abnormal type of the abnormal area, and highlight render the alarm color;

[0176] generate a visual alarm signal based on the spatial coordinate and the abnormal type.

[0177] In an implementable embodiment, the visual monitoring module 40 is further configured to process the point cloud data in the structural features based on a Poisson surface reconstruction algorithm to generate a smooth and closed three-dimensional grid model of the concealed project.

[0178] based on a unified coordinate system, calculate an accurate mapping relationship between the image data and the vertices of the three-dimensional grid model;

[0179] generate a seamless texture map according to the image data fusion, map the texture map to the three-dimensional grid model according to the accurate mapping relationship, and output a real scene model of the concealed project.

[0180] In an implementable embodiment, the visual monitoring module 40 is further configured to determine a severity level of the abnormal type, and divide the visual alarm signal into three levels of pre-alarm, alarm, and emergency alarm based on the severity level.

[0181] on a user interface of the real scene model of the concealed project, dynamically display an alarm information panel in a color corresponding to the level, and drive a peripheral audible and visual alarm to issue an audible and visual alarm of the corresponding level;

[0182] generate an alarm log containing the spatial coordinate of the abnormal area, the abnormal type, the alarm level, and a time stamp, and push the alarm log and the visual alarm signal to a remote monitoring center in real time through a network interface.

[0183] The airport concealed project visual monitoring system provided in the present application adopts the airport concealed project visual monitoring method in the above embodiments, and can solve the technical problems of low accuracy of abnormal state monitoring and low efficiency of abnormal positioning caused by manual inspection or single sensor fixed-point monitoring in the airport concealed project. Compared with the prior art, the airport concealed project visual monitoring system provided in the present application has the same beneficial effects as the airport concealed project visual monitoring method provided in the above embodiments, and other technical features in the airport concealed project visual monitoring system are the same as the features disclosed in the above embodiments, which will not be repeated here.

[0184] This application provides a visual monitoring device for concealed airport works. The visual monitoring device for concealed airport works includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the visual monitoring method for concealed airport works in the above embodiment 1.

[0185] The following is for reference. Figure 5 This document illustrates a structural schematic diagram of an airport concealed works visualization monitoring device suitable for implementing embodiments of this application. The airport concealed works visualization monitoring device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), vehicle terminals (e.g., vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The airport concealed engineering visualization monitoring equipment shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0186] like Figure 5As shown, the airport concealed engineering visualization monitoring device can include a processing system 1001 (e.g., a central processor, a graphics processor, etc.) that can perform various appropriate actions and processes according to programs stored in a ROM (Read Only Memory) 1002 or programs loaded from a storage system 1003 into a RAM (Random Access Memory) 1004. Various programs and data required for operation of the airport concealed engineering visualization monitoring device are also stored in the RAM 1004. The processing system 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input system 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output system 1008 including, for example, an LCD (Liquid Crystal Display), a speaker, a vibrator, etc.; the storage system 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication system 1009. The communication system 1009 can allow the airport concealed engineering visualization monitoring device to communicate with other devices wirelessly or by wire to exchange data. Although the airport concealed engineering visualization monitoring device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0187] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through a communication system, or installed from the storage system 1003, or installed from the ROM 1002. When the computer program is executed by the processing system 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are performed.

[0188] The airport concealed engineering visualization monitoring device provided by the present application adopts the airport concealed engineering visualization monitoring method in the above-mentioned embodiments, and can solve the technical problem of airport concealed engineering visualization monitoring. Compared with the prior art, the airport concealed engineering visualization monitoring device provided by the present application has the same beneficial effects as the airport concealed engineering visualization monitoring method provided by the above-mentioned embodiments, and other technical features in the airport concealed engineering visualization monitoring device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0189] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0190] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0191] The present application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e. computer programs) for performing the airport concealed engineering visualization monitoring method in the above embodiments.

[0192] The computer readable storage medium provided by the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to: an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory or flash memory), an optical fiber, a CD-ROM (CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to: electric wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0193] The above computer readable storage medium can be contained in the airport concealed engineering visualization monitoring device; or can exist separately without being assembled into the airport concealed engineering visualization monitoring device.

[0194] The computer readable storage medium carries one or more programs, when the one or more programs are executed by the airport concealed engineering visual monitoring device, the airport concealed engineering visual monitoring device: through the multi-source sensor, the image data, the depth data and the environment parameter of the airport concealed engineering are collected in real time, the collected image data, the depth data and the environment parameter are carried out dynamic noise filtering and data alignment processing, and the concealed engineering feature data is obtained;

[0195] The structural features and abnormal features in the concealed engineering feature data are extracted based on a concealed engineering perception model, the concealed engineering perception model is obtained based on an initial concealed engineering perception model, and the initial concealed engineering perception model is used for abnormal perception of the concealed engineering feature data;

[0196] The abnormal features are dynamically threshold segmented, and the abnormal area is located;

[0197] The abnormal area is superimposed on the concealed engineering real scene model in a highlighted form based on the structural features, and a visual alarm signal is output.

[0198] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0199] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or combinations of hardware and software.

[0200] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the unit itself.

[0201] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer programs) for executing the above-mentioned airport concealed engineering visualization monitoring method, and can solve the technical problem of airport concealed engineering visualization monitoring. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the airport concealed engineering visualization monitoring method provided by the above-mentioned embodiments, which will not be repeated here.

[0202] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned airport concealed engineering visualization monitoring method.

[0203] The computer program product provided by the present application can solve the technical problem of airport concealed engineering visualization monitoring. Compared with the prior art, the computer program product provided by the present application has the same beneficial effects as the airport concealed engineering visualization monitoring method provided by the above-mentioned embodiments, which will not be repeated here.

[0204] The above is only some embodiments of the present application, and does not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. An airport concealed engineering visual monitoring method, characterized in that, The airport concealed engineering visual monitoring method comprises: Real-time collection of image data, depth data and environmental parameters of the airport concealed engineering by a multi-source sensor, dynamic noise filtering and data alignment processing of the collected image data, depth data and environmental parameters to obtain concealed engineering feature data; Extracting structural features and abnormal features in the concealed engineering feature data based on a concealed engineering perception model, wherein the concealed engineering perception model is obtained based on optimization of an initial concealed engineering perception model, and the initial concealed engineering perception model is used for abnormal perception of the concealed engineering feature data; Dynamic threshold segmentation of the abnormal features to locate abnormal regions; Constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal regions in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal; Wherein, the concealed engineering perception model is composed of an improved multi-scale feature pyramid network and a bidirectional long short-term memory network, and the step of extracting the structural features and abnormal features in the concealed engineering feature data based on the concealed engineering perception model comprises: Inputting the concealed engineering feature data into the improved multi-scale feature pyramid network to extract local detail features and global context features; Time series modeling of the local detail features and the global context features based on the bidirectional long short-term memory network to determine the dynamic evolution features of the concealed engineering state; Determining the saliency weight of each region based on the dynamic evolution features of the concealed engineering state, and updating the response strength of the abnormal region based on the saliency weight; According to the response strength, the local detail features and global context features are redundantly compressed to obtain structural features and abnormal features.

2. The method of claim 1, wherein, The step of performing dynamic noise filtering and data alignment processing on the collected image data, depth data and environmental parameters to obtain concealed engineering feature data comprises: Processing the image data based on a hybrid denoising algorithm of guided filtering and adaptive median filtering to obtain denoised image data; Dynamic noise suppression of the depth data by a time series Kalman filter, and removal of drift points by a point cloud statistical outlier removal algorithm to obtain noise-suppressed depth data; Through feature point matching and polar constraint method, the denoised image data and the noise-suppressed depth data are pixel-level spatially aligned, and a unified coordinate system is established; Based on the environmental parameters, a data reliability weight is constructed, and the denoised image data and the noise-suppressed depth data are weighted and fused based on the data reliability weight to generate concealed engineering feature data with spatio-temporal consistency in the unified coordinate system.

3. The method of claim 1, wherein, Before the step of extracting the structural features and abnormal features in the concealed engineering feature data based on the concealed engineering perception model, the following steps are further included: Obtaining an airport concealed engineering historical data set containing normal and multiple typical abnormal states, and finely labeling the abnormal regions in the historical data set to construct a training sample set; Using a pre-trained convolutional neural network as the backbone network of the initial concealed engineering perception model, and initializing the bidirectional long short-term memory network; Train the initial concealed engineering perception model based on a focal loss function and the training sample set to obtain a concealed engineering perception model feature; Migrate the concealed engineering perception model feature to a multi-scale feature pyramid network to obtain an improved multi-scale feature pyramid network; Obtain the concealed engineering perception model according to the improved multi-scale feature pyramid network and an initialized bidirectional long short-term memory network.

4. The method of claim 1, wherein, The step of performing dynamic threshold segmentation on the abnormal feature to locate an abnormal region comprises: Determine an abnormal feature map based on the abnormal feature, determine the mean value and the standard deviation in the neighborhood of each pixel point in the abnormal feature map, and obtain an adaptive segmentation threshold value according to the mean value and the standard deviation; Perform binaryzation processing on the abnormal feature map based on the adaptive segmentation threshold value to obtain a binaryzation feature map, eliminate the scattered point noise of the binaryzation feature map and smooth the abnormal region boundary based on morphological opening operation and closing operation to obtain an abnormal region.

5. The method of claim 4, wherein, The step of eliminating the scattered point noise of the binaryzation feature map and smoothing the abnormal region boundary based on morphological opening operation and closing operation to obtain an abnormal region comprises: Eliminate the scattered point noise of the binaryzation feature map and smooth the abnormal region boundary based on morphological opening operation and closing operation to obtain an initial abnormal region; Determine the positioning result of the abnormal region of the previous time sequence frame, perform position prediction and state updating on the initial abnormal region of the current frame to obtain a corrected abnormal region; Determine the mean value of the feature intensity and the area of the corrected abnormal region, and determine the corrected abnormal region as the abnormal region when the mean value of the feature intensity and the area are both greater than a preset threshold.

6. The method of claim 1, wherein, The step of constructing a concealed engineering real scene model based on the structure feature, superimposing the abnormal region in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal comprises: Perform surface reconstruction on the structure feature to generate a three-dimensional network model of the concealed engineering, and perform texture mapping based on the image data to obtain a concealed engineering real scene model; Determine the spatial coordinates of the abnormal region, map the abnormal region to the concealed engineering real scene model based on the spatial coordinates, determine the warning color according to the abnormal type of the abnormal region, and perform highlighted rendering on the warning color; Generate a visual alarm signal based on the spatial coordinates and the abnormal type.

7. The method of claim 6, wherein, The step of performing surface reconstruction on the structure feature to generate a three-dimensional network model of the concealed engineering, and performing texture mapping based on the image data to obtain a concealed engineering real scene model comprises: Process the point cloud data in the structure feature based on a Poisson surface reconstruction algorithm to generate a smooth and closed three-dimensional mesh model of the concealed engineering; Calculate the accurate mapping relationship between the image data and the vertices of the three-dimensional mesh model based on a unified coordinate system; Generate seamless texture mapping according to the image data fusion, and map the texture mapping to the three-dimensional mesh model according to the accurate mapping relationship to output a concealed engineering real scene model.

8. The method of claim 6, wherein, The step of constructing a concealed engineering real scene model based on the structural features, superimposing the abnormal area in a highlighted form into the concealed engineering real scene model, and outputting a visual alarm signal further comprises: determining a severity level of the abnormal type, dividing the visual alarm signal into three levels of early warning, alarm, and emergency alarm based on the severity level; dynamically displaying an alarm information panel in a color corresponding to the level on a user interface of the concealed engineering real scene model, and driving a peripheral audible and light alarm to issue an audible and light warning of the corresponding level; generating an alarm log containing the spatial coordinates of the abnormal area, the abnormal type, the alarm level, and a time stamp, and pushing the alarm log and the visual alarm signal to a remote monitoring center in real time through a network interface.

9. An airport concealed engineering visual monitoring system, characterized in that, The airport concealed engineering visual monitoring system comprises: a data processing module configured to collect image data, depth data, and environmental parameters of the airport concealed engineering in real time through multiple source sensors, perform dynamic noise filtering and data alignment processing on the collected image data, depth data, and environmental parameters, and obtain concealed engineering feature data; an abnormality perception module configured to extract structural features and abnormal features from the concealed engineering feature data based on a concealed engineering perception model, wherein the concealed engineering perception model is obtained by optimizing an initial concealed engineering perception model, and the initial concealed engineering perception model is configured to perform abnormality perception on the concealed engineering feature data; an abnormality positioning module configured to perform dynamic threshold segmentation on the abnormal features and locate abnormal areas; a visual monitoring module configured to construct a concealed engineering real scene model based on the structural features, superimpose the abnormal area in a highlighted form into the concealed engineering real scene model, and output a visual alarm signal; wherein the concealed engineering perception model is composed of an improved multi-scale feature pyramid network and a bidirectional long short-term memory network, and the step of extracting the structural features and abnormal features from the concealed engineering feature data based on the concealed engineering perception model comprises: inputting the concealed engineering feature data into the improved multi-scale feature pyramid network to extract local detail features and global context features; performing time series modeling on the local detail features and the global context features based on the bidirectional long short-term memory network to determine dynamic evolution features of a concealed engineering state; determining a saliency weight of each area based on the dynamic evolution features of the concealed engineering state, and updating a response intensity of the abnormal area based on the saliency weight; performing redundancy compression on the local detail features and the global context features according to the response intensity to obtain the structural features and abnormal features.

Images