Intelligent foreign matter recognition and dynamic monitoring method based on remote sensing data fusion

Abstract

This invention provides a method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion, belonging to the field of image recognition. It solves the problem of low efficiency in foreign object identification. Specifically, the method involves: acquiring and extracting fusion features from remote sensing images of various types of road foreign objects; performing similarity learning on the fusion features to establish a high-dimensional index structure; acquiring real-time remote sensing images of the road to determine if foreign objects exist; marking foreign objects if they exist, and leaving them untouched if they do not; acquiring historical natural images of the target area, designing a temporal feature analysis module, a spatial feature analysis module, and a spatiotemporal joint prediction module to output a predicted natural environment image of the target area for future times; acquiring actual remote sensing images, and marking abnormal areas in the actual remote sensing images based on the predicted natural environment image. This invention improves the efficiency of foreign object identification by acquiring and extracting features from remote sensing images of roads and the natural environment in the target area.

Description

Technical Field

[0001] This invention relates to a method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion, and pertains to the field of image processing. Background Technology

[0002] Existing methods for identifying foreign objects using remote sensing data have the following shortcomings:

[0003] Complexity of feature extraction and fusion: Some existing foreign object identification methods only use a single or simple feature extraction method, which cannot make full use of the rich information of multi-source remote sensing data. For example, using only color features for foreign object identification, when faced with foreign objects with complex textures or different shapes, it is difficult to accurately distinguish foreign objects from the background due to the lack of fusion of key information such as texture and shape, resulting in low identification accuracy.

[0004] Inaccuracy of recognition results: Some existing foreign object recognition methods use fixed weights or simple distance calculation methods, which cannot be dynamically adjusted according to the specific characteristics of the image; for foreign object images with complex geometric structures, if the weight of shape features is not increased, shape information may be ignored due to the dominant role of other features, resulting in misjudgment or omission of foreign objects.

[0005] Insufficient automation: Traditional foreign object monitoring methods usually require a lot of manual intervention, such as manually selecting features and manually comparing images. This not only increases labor costs, but also easily introduces subjective errors due to the speed limitations of manual operation, which also affects the efficiency and timeliness of monitoring. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion, aiming to solve the problem of low efficiency in foreign object identification.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: A method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion includes:

[0008] Remote sensing images of various types of road debris are acquired, and the texture, shape, and color features corresponding to each type of road debris are extracted and fused to obtain fused features. Similarity learning is performed on the fused features to establish a high-dimensional index structure. Real-time remote sensing images of the road are acquired to determine whether there are any debris on the road. If there are, the debris is marked; if there are no, no action is taken.

[0009] Historical natural images of the target area are acquired, and a temporal feature analysis module, a spatial feature analysis module, and a spatiotemporal joint prediction module are designed to output a natural environment prediction image of the target area at future times. Actual remote sensing images are acquired, and the natural environment prediction image is time-aligned with the actual remote sensing image. Anomaly areas are marked in the actual remote sensing image based on the natural environment prediction image.

[0010] Continuously monitor foreign objects on roads and abnormal areas in the natural environment of the target area.

[0011] Furthermore, the specific steps for extracting texture features are as follows:

[0012] Construct and iterate through all elements of the gray-level co-occurrence matrix corresponding to the remote sensing image of road debris, and calculate the statistics of the gray-level co-occurrence matrix corresponding to each road debris;

[0013] The statistics of the gray-level co-occurrence matrix corresponding to each road object are combined into a row vector;

[0014] Summarize and merge all row vectors for each type of road obstacle to obtain the original texture matrix for each type of road obstacle; the original texture matrix has the following format (m （te） ×n （te） ), where m （te） n represents the total number of row vectors for this type of road debris. （te） The number of types representing all statistics for road debris;

[0015] The original texture matrices of each type of road debris are collected, standardized, and the covariance matrix is ​​calculated.

[0016] Eigenvalues ​​and eigenvectors of each type of road debris are obtained by performing eigenvalue decomposition on the covariance matrix.

[0017] Calculate the cumulative variance contribution rate of the feature vector corresponding to each type of road debris, and calculate the product of the cumulative variance contribution rate of different feature vectors and the corresponding feature value of the feature vector, which is used as the feature selection coefficient of different feature vectors for each type of road debris.

[0018] The feature vectors corresponding to the top 30% of feature selection coefficients for each type of road obstacle are used as the main feature vectors for that type of road obstacle; the main feature vectors are in the form of (n （te） A column vector of (×1);

[0019] Combine all the main feature vectors of road debris into (n) （te） ×k （te） The projection matrix of the road object is multiplied by the original texture matrix of each type of road object to obtain the texture features of each type of road object.

[0020] Furthermore, the specific steps for extracting shape and color features are as follows:

[0021] Extracting shape features:

[0022] The remote sensing image of road debris was converted from a color image to a grayscale image, and the road debris image and background in the grayscale image were segmented to obtain a binary image; in this image, the road debris was set to white and the background was set to black.

[0023] Using a lightweight segmentation network, the semantic mask of white pixels in the binary image is set to 1, and the semantic mask of black pixels is set to 0, to obtain the mask image corresponding to each type of road foreign object.

[0024] Mark the outline of the foreign object region in the binary image of each type of road foreign object, and then calculate the geometric moments of the foreign object region outline; convert the geometric moments corresponding to each type of road foreign object into Hu moments that are invariant to translation, rotation and scaling, and use them as the shape features of each type of road foreign object.

[0025] Extract color features:

[0026] The images of road debris were converted into three color spaces: RGB, HSV, and Lab. Gaussian pyramid decomposition was then performed on the images in each color space to obtain a multi-scale image set for each type of road debris.

[0027] Based on the mask image corresponding to each type of road foreign object, calculate the local color histogram of the multi-scale image set corresponding to each type of road foreign object in RGB, HSV and Lab color spaces;

[0028] By using a channel attention mechanism, the features of three color spaces are weighted and fused to obtain the color features of each type of road object.

[0029] Furthermore, the specific steps of feature fusion are as follows:

[0030] The texture, shape, and color features corresponding to each type of road debris are fused using a gating attention mechanism. The specific fusion process is as follows:

[0031] The three modal features of each type of road foreign object are standardized, and the standardized three modal features are projected through a linear layer.

[0032] The three modal features after projection are L2 normalized; the ReLU function is used to activate the three normalized modal features, and then they are concatenated in dimension 1 to form a context vector.

[0033] The three modal features of the first type of road foreign object are fused through a gating network:

[0034] Input the context vector of the first type of road foreign object, and output 3 logits values;

[0035] The Softmax function is used to transform the three logits values ​​into a single probability distribution alpha containing three probability values.

[0036] Multiply each element in alpha by the projection features corresponding to the texture, shape, and color features respectively, and then add them together to obtain the trimodal mixed feature;

[0037] The three-modal hybrid features are then L2 normalized to obtain the fused features of the first type of road foreign objects.

[0038] By fusing the three modal features of other road debris, the fused features of each type of road debris are obtained.

[0039] Furthermore, the specific steps of similarity learning are as follows:

[0040] Add labels to all remote sensing images of road debris, marking the type of road debris to which each remote sensing label belongs;

[0041] The remote sensing images of all road debris are combined into sample pairs, and the loss function of the Siamese network is designed as follows:

[0042] Let the sample pair be (x （i） x （j） ), where i and j represent the labels of the remote sensing images;

[0043] Calculate remote sensing image x （i） and x （j） Embedding distance d （ij） ;

[0044] If labels i and j are the same, then parameter y （ij） The value of is 1; if they are different, then the parameter y （ij） The value is 0;

[0045] Let the loss function be L(x) （i） x （j） ):

[0046] ;

[0047] Where, n （mar） Indicates the interval constant;

[0048] Input all sample pairs into the Siamese network and adjust n. （mar） Find the value of that minimizes the loss function and construct the feature space;

[0049] Clustering algorithms are used to cluster the fusion features corresponding to all foreign object images, and the fusion features corresponding to each type of foreign object image are grouped into one category to generate a multi-level index tree.

[0050] Based on the distribution of various road debris fusion features in the feature space in the multi-level index tree, a high-dimensional index structure is used to store the fusion features corresponding to various road debris.

[0051] The fusion features of real-time remote sensing images on the road are acquired and extracted, the fusion features of real-time remote sensing images are mapped into the feature space, and it is determined whether the fusion features of real-time remote sensing images match the existing fusion features in the high-dimensional index structure.

[0052] If a match is found, the foreign object is marked and reported to the traffic management department.

[0053] If there is no match, then nothing is done.

[0054] Furthermore, the specific steps of the time series feature analysis module are as follows:

[0055] Historical natural images are decomposed at multiple scales and feature-encoded to obtain natural features; external environmental variables are fused as additional features.

[0056] The natural features and additional features are standardized and normalized to obtain the global feature vector of the historical natural image.

[0057] The global feature vector is fed into the input of the LSTM / GRU network. The input of the LSTM / GRU network receives the global feature vector in chronological order. The time dependency is dynamically adjusted through the forget gate, input gate and output gate to learn the long-term trend of natural environment change.

[0058] During the training of the LSTM / GRU network, the TimeSformer module is introduced to calculate the correlation weights of each parameter of the global feature vector at different times.

[0059] By introducing ARIMA or SARIMA models to learn the natural changes in the periodic or seasonal patterns of the natural environment, and merging them with the long-term trend of natural environment changes, the temporal change trend of the natural environment in the target area can be obtained.

[0060] Furthermore, the specific steps of the spatial feature analysis module are as follows:

[0061] The spatial feature analysis module consists of a CNN model and a GNN model;

[0062] CNN models extract local spatial structure:

[0063] The CNN model learns the spatial features of historical natural images through multi-layer convolution and pooling operations. It uses shallow convolution to capture the texture and edge information of spatial features and uses deep convolution to abstract them into high-dimensional spatial semantic features, dividing historical natural images into environmental units.

[0064] GNN models model spatial adjacency relationships:

[0065] The Generative Neural Network (GNN) model treats each environmental unit as a graph node, defining edge weights based on spatial proximity or geographical similarity to form a spatial adjacency graph between regions. The GNN model propagates features between adjacent nodes through an information transfer mechanism, enabling cross-regional environmental change impact modeling. The specific process is as follows:

[0066] Each graph node first integrates its own initial features, and then sends a message to its neighboring graph nodes. The message content integrates its own features and edge weight information. After receiving the message, the neighboring nodes combine it with their original features to perform aggregation and update, thereby obtaining the spatial change trend of the natural environment of the target area.

[0067] Furthermore, the specific steps of the spatiotemporal joint prediction module are as follows:

[0068] The ConvLSTM model learns the dependence between the temporal and spatial changes of the natural environment in the target area through a convolutional gating mechanism, and then adjusts the weight distribution in the temporal and spatial dimensions through a dual self-attention mechanism to output a predicted image of the natural environment in the target area at future times.

[0069] The system acquires real-time remote sensing images of the natural environment of the target area, aligns the predicted natural environment images with the actual remote sensing images in time, marks areas in the actual remote sensing images that conflict with the predicted natural environment images as anomalous areas, and provides feedback.

[0070] Compared with the prior art, the beneficial effects of the present invention are:

[0071] Efficient Feature Extraction and Fusion: This invention employs Feature Normalization and Attention Fusion (GAF) technology to standardize multi-channel features such as color, texture, and shape using Z-score. After eliminating dimensional differences, the importance of each feature channel is dynamically learned and fused through a gated attention mechanism. This fusion method can adaptively weight feature importance, enabling the model to focus more on texture features in images with high texture complexity and on shape features in images with clear structure. This fully utilizes the complementary information from multi-source remote sensing data to extract more comprehensive and discriminative features, effectively improving the accuracy of foreign object identification.

[0072] Fast similarity retrieval: This invention utilizes a contrastive learning network for deep metric learning to construct a distinguishable feature embedding space, and combines it with high-dimensional indexing structures such as FAISS or HNSW to achieve approximate nearest neighbor retrieval. Through continuous training, the network learns feature representations that "small semantic distance equals similarity." With the help of an efficient indexing structure, it can quickly find a set of images similar to the target image in a database containing massive amounts of remote sensing images, meeting the real-time requirements of dynamic monitoring and promptly grasping the appearance and changes of foreign objects.

[0073] Accurate identification results: This invention utilizes an adaptive similarity reweighting strategy to dynamically adjust weights based on the complexity of the sample's texture, shape, and color, further improving the accuracy of similarity measurement and making foreign object identification results more reliable. At the same time, this invention achieves a high degree of automation throughout the entire process from feature extraction and fusion to similarity retrieval and foreign object identification, reducing the need for manual intervention and minimizing false positives and false negatives. Attached Figure Description

[0074] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0075] Figure 1 This is a schematic diagram of the method of the present invention;

[0076] Figure 2 This is a schematic diagram of the road foreign object identification method of the present invention;

[0077] Figure 3 This is a schematic diagram illustrating the identification of foreign objects in the natural environment according to the present invention. Detailed Implementation

[0078] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] Please see Figure 1 and Figure 2 Methods for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion include:

[0080] Step S1: Obtain remote sensing images of various types (of previously encountered) road debris, extract and fuse the texture, shape, and color features corresponding to each type of road debris to obtain fused features; perform similarity learning on the fused features to establish a high-dimensional index structure; obtain real-time remote sensing images of the road to determine whether there are debris on the road; if there are, mark the debris (and report it to the traffic management department); if there are no, do not process it.

[0081] It should be noted that, in this invention, "target area" refers to the area where foreign objects are identified using this invention (a method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion);

[0082] Acquire remote sensing images of various types of (previously encountered) road debris and extract the fusion features of the road debris;

[0083] The remote sensing images of road foreign objects are preprocessed (radiometric calibration, atmospheric correction, geometric correction, etc.) to eliminate sensor errors and atmospheric effects, and then the texture features, shape features and color features of the remote sensing images corresponding to various types of road foreign objects are extracted (after preprocessing).

[0084] Extracting texture features:

[0085] Construct the gray-level co-occurrence matrix (GLCM) of remote sensing images of road debris;

[0086] The elements in the gray-level co-occurrence matrix represent the frequency or probability of a pair of gray-level pixels appearing in a remote sensing image of road debris at a specific distance and direction.

[0087] Iterate through all elements of the gray-level co-occurrence matrix (GLCM), calculate the mean and standard deviation of all pixel gray levels corresponding to each row and column direction in the GLCM, and calculate the statistics of the GLCM corresponding to each road object (statistics include but are not limited to: contrast, correlation, energy, entropy, etc.).

[0088] The statistics of the gray-level co-occurrence matrix corresponding to each road object are combined into a row vector;

[0089] Summarize and merge all row vectors for each type of road obstacle to obtain the original texture matrix for each type of road obstacle; the original texture matrix has the following format (m （te） ×n （te） ), where m （te） n represents the total number of row vectors for this type of road debris. （te） The number of types representing all statistics for road debris;

[0090] Summarize all the original texture matrices for each type of road debris (to eliminate the difference in dimensions between different statistics), and perform Z-score standardization to calculate the covariance matrix of the original texture matrix corresponding to each type of road debris.

[0091] Eigenvalues ​​and eigenvectors of each type of road debris are obtained by performing eigenvalue decomposition on the covariance matrix.

[0092] Calculate the cumulative variance contribution rate of the feature vector corresponding to each type of road debris, and calculate the product of the cumulative variance contribution rate of different feature vectors and the corresponding feature value of the feature vector, which is used as the feature selection coefficient of different feature vectors for each type of road debris.

[0093] (In descending order of feature selection coefficient) the feature vectors corresponding to the top 30% of feature selection coefficients for each type of road debris are taken as the main feature vectors for that type of road debris; (It should be noted that the "30%" here is the default selection ratio of the main feature vectors, and users or relevant technicians can adjust the selection ratio of the main feature vectors according to actual needs).

[0094] The main feature vector pattern is (n （te） A column vector of (×1);

[0095] Combine all the main feature vectors of road debris into (n) （te） ×k （te） The projection matrix of the road debris is multiplied by the original texture matrix of each type of road debris to obtain the texture features of each type of road debris.

[0096] Extracting shape features:

[0097] Convert the road debris remote sensing image from a color image to a grayscale image (skip this step if the road debris remote sensing image is already a grayscale image). Use the OTSU adaptive thresholding algorithm to segment the road debris image and the background in the grayscale image ("background" refers to the "non-road debris image" in the grayscale image). Then convert the segmented grayscale image into a binary image, where the road debris is set to white and the background is set to black.

[0098] Using a lightweight segmentation network (such as DeepLabv3-MobileNet or Segment Anything model), the semantic mask of white pixels in the binary image is set to 1, and the semantic mask of black pixels is set to 0, to obtain the mask image corresponding to each type of road foreign object.

[0099] The cv2.findContours() function is used to mark the contour of the foreign object region in the binary image of each type of road foreign object (i.e., the white pixel region in the binary image of each type of road foreign object), and then the geometric moments of the foreign object region contour are calculated using the moments function.

[0100] The geometric moments corresponding to each type of road debris are converted into Hu moments that are invariant to translation, rotation, and scaling, and used as the shape features of each type of road debris.

[0101] Extract color features:

[0102] The CLAHE algorithm (contrast-limited adaptive histogram equalization) is used to enhance the image contrast of remote sensing images of each type of road debris, thus completing the image preprocessing.

[0103] The preprocessed images (corresponding to each type of road debris) are converted into three color spaces: RGB, HSV, and Lab. Gaussian pyramid decomposition is then performed on the images in each color space to obtain a multi-scale image set for each type of road debris.

[0104] Based on the mask image corresponding to each type of road foreign object, calculate the local color histogram of the multi-scale image set corresponding to each type of road foreign object in RGB, HSV and Lab color spaces;

[0105] The channel attention mechanism is used to weightedly fuse features from three color spaces (RGB, HSV, Lab, etc.). The specific implementation process is as follows:

[0106] Obtain the local color histograms for the corresponding color channels of RGB, HSV, and Lab, and calculate the mean of the local color histogram for each color channel to obtain mean(RGB), mean(HSV), and mean(Lab).

[0107] Introduce a (learnable) weight parameter W for each color channel. （RGB） W （HSV） and W （Lab） ;

[0108] For each color channel, calculate exp(W) separately. （RGB） ×mean(RGB)), exp(W （HSV） ×mean(HSV)) and exp(W （Lab） ×mean(Lab)), which is the (unnormalized) attention score for each color channel; where exp() denotes the exponential function;

[0109] The attention score (unnormalized) for each color channel is normalized using the softmax function to obtain the attention weight α for each color channel. （RGB） α （HSV） and α （Lab） ;

[0110] Based on attention weight α （RGB） α （HSV） and α （Lab） We perform weighted fusion of the local color histograms corresponding to different color channels (i.e., RGB, HSV, and Lab color channels) for each type of road debris to obtain the (fused) color features of each type of road debris.

[0111] Feature fusion:

[0112] The texture, shape, and color features corresponding to each type of road obstacle are fused using a gated attention fusion (GAF) mechanism. The specific fusion process is as follows:

[0113] Z-score normalization is performed on the three modal features (i.e., texture features, shape features, and color features) of each type of road debris (to eliminate the dimensional differences between different feature sources), and then the normalized three modal features are projected through a linear layer (to map the three modal features of each type of road debris to the same dimension).

[0114] The three modal features after projection are L2 normalized; the ReLU function is used to activate the three normalized modal features, and then they are concatenated in dimension 1 to form a context vector.

[0115] The three modal features of the first type of road foreign object are fused using a gating network (the gating network consists of two fully connected layers, with ReLU as the activation function in between):

[0116] Input the context vector of the first type of road foreign object, and output 3 logits values ​​(i.e., 3 logits values, which correspond to the attention weights of the three modalities respectively).

[0117] The three logits values ​​are transformed into a single probability distribution alpha containing three probability values ​​using the Softmax function; (this distribution describes the dynamic weighting of the three modal features in the similarity measure).

[0118] Multiply each element in alpha by the projection features corresponding to the texture, shape, and color features respectively, and then sum them to obtain the trimodal mixed feature; (here, "projection feature" refers to the feature "after linear layer, L2 normalization, and ReLU activation").

[0119] The three-modal hybrid features are then L2 normalized to obtain the fused features of the first type of road foreign objects.

[0120] (Repeat the same steps as obtaining the fusion features of the first type of road foreign object) Fuse the three modal features of other road foreign objects to obtain the fusion features of each type of road foreign object;

[0121] Similarity learning:

[0122] Add labels to all remote sensing images of road debris to indicate the type of road debris to which each remote sensing label belongs;

[0123] Combine all remote sensing images of road debris into sample pairs (the remote sensing images in a sample pair can be two remote sensing images of the same type of road debris or two remote sensing images of different types of road debris).

[0124] Design the loss function for Siamese networks (such as Siamese Networks):

[0125] Let the sample pair be (x （i） x （j） ), where i and j represent the labels of the remote sensing images;

[0126] Calculate remote sensing image x （i） and x （j） Embedding distance d （ij） ;

[0127] Determine if labels i and j are the same; if they are the same, then parameter y... （ij） The value of is 1; if they are different, then the parameter y （ij） The value is 0;

[0128] Let the loss function be L(x) （i） x （j） ):

[0129] ;

[0130] Where, n （mar） This represents the interval constant (i.e., the hyperparameters preset for the input Siamese network).

[0131] All sample pairs are input into a Siamese network (such as a Siamese Network), and n is adjusted through continuous training. （mar） The value of minimizes the loss function (of the Siamese network), allowing the network to learn the embedding space features of "small semantic distance equals similarity," thus completing the construction of the feature space;

[0132] After completing the feature space construction, clustering algorithms (such as K-means or Product Quantization) are used to cluster the fusion features corresponding to all foreign object images, and the fusion features corresponding to each class of foreign object images are grouped into one class to generate a multi-level index tree.

[0133] Based on the distribution of various road debris fusion features in the feature space in the multi-level index tree, a high-dimensional index structure (such as FAISS or HNSW) is used to store the fusion features corresponding to various road debris.

[0134] Acquire real-time remote sensing images of the road (repeat the same steps as extracting fusion features from remote sensing images of road debris), extract fusion features from the real-time remote sensing images, map the fusion features of the real-time remote sensing images into the feature space, and compare them with the fusion features stored in the high-dimensional index structure to determine whether the fusion features of the real-time remote sensing images match a certain existing fusion feature in the "high-dimensional index structure".

[0135] If a match is found, it indicates the presence of a foreign object on the road. The object will be marked and reported to the traffic management department.

[0136] If there is no match, it means there are no foreign objects on the road, and no action is taken.

[0137] Step S2: Acquire historical natural images of the target area, design a temporal feature analysis module, a spatial feature analysis module, and a spatiotemporal joint prediction module, and output a natural environment prediction image of the target area at future times; acquire actual remote sensing images, and time-align the natural environment prediction image with the actual remote sensing image, and mark abnormal areas in the actual remote sensing image based on the natural environment prediction image.

[0138] Please see Figure 3 Remote sensing images of the natural environment in the target area in recent years are collected as historical natural images. Geometric registration and radiometric correction are performed on the historical natural images (so that the pixel positions of the same geographical area at different times are consistent). Then, image enhancement and noise filtering algorithms (such as histogram equalization and adaptive filtering) are used to improve the image clarity and texture discernibility.

[0139] Time series feature analysis module:

[0140] In the feature extraction stage, historical natural images are decomposed and encoded at multiple scales to obtain natural features. Natural features include three categories: texture features (automatically learned through gray-level co-occurrence matrix or convolutional kernel); spectral features (reflecting differences in reflectance of vegetation, soil, and water); and geometric features (such as boundaries, corners, and shape contours).

[0141] External environmental variables (such as temperature, precipitation, NDVI, and surface temperature) are integrated as additional features (to enhance the model's ability to perceive complex natural evolution patterns).

[0142] The natural features and additional features are standardized and normalized to obtain the global feature vector of the historical natural image.

[0143] The global feature vector is fed into the input of the LSTM / GRU network. The LSTM / GRU network receives the global feature vector in chronological order and dynamically adjusts the temporal dependence through forget gates, input gates, and output gates to learn the long-term trend of natural environment change (i.e., LSTM uses forget gates, input gates, and output gates to filter and memorize the global feature vectors corresponding to different historical natural images, automatically determining which historical information to retain (such as the cumulative effect of continuous drought) and ignoring short-term fluctuations (such as one-off climate anomalies); GRU, as a simplified structure, achieves a similar function to the "LSTM model" through update gates and reset gates; after the model completes multi-temporal sequence training, it can abstract the patterns of evolution over time from past environmental images, such as the annual decline in vegetation cover and seasonal fluctuations in water area, so as to infer future trends based on the current state in the prediction stage and realize the long-term evolution prediction of natural environment change).

[0144] During the training of the LSTM / GRU network, a TimeSformer module based on the Transformer structure is introduced. This module adopts a self-attention mechanism to calculate the correlation weights of each parameter of the global feature vector at different times along the entire time dimension.

[0145] By introducing ARIMA or SARIMA models to learn the natural changes in the periodic or seasonal patterns of the natural environment in the target area, and merging them with the long-term trend of natural environment changes, the temporal change trend of the natural environment in the target area can be obtained.

[0146] Spatial Feature Analysis Module:

[0147] This module consists of two parts: a convolutional neural network (CNN) and a graph neural network (GNN).

[0148] CNN models extract local spatial structure:

[0149] CNN models learn the spatial features (such as topographic relief, river direction, building density, and vegetation coverage) of historical natural images (target area) through multi-layer convolution and pooling operations. They use shallow convolution to capture the texture and edge information of spatial features and use deep convolution to abstract them into high-dimensional spatial semantic features, dividing historical natural images into environmental units (such as typical environmental units such as "mountainous vegetation area", "urban construction area", and "water body area").

[0150] GNN models model spatial adjacency relationships:

[0151] (Based on CNN) The GNN model treats each environmental unit as a graph node, defining edge weights based on spatial proximity or geographical similarity to form a spatial adjacency graph between regions. The GNN model propagates features between adjacent nodes through a message passing mechanism, enabling cross-regional environmental change impact modeling. The specific process is as follows:

[0152] Each graph node first integrates its own initial features (such as vegetation coverage, river area, biological activity area, and other environmental information). Each graph node sends a message to its neighboring graph nodes. The message content integrates its own features and edge weight information. After receiving the message, the neighboring nodes combine it with their own original features to perform aggregation and update, thereby obtaining the spatial change trend of the natural environment of the target area.

[0153] Spatiotemporal Joint Prediction Module:

[0154] The ConvLSTM model introduces convolution operations into the gated units of the LSTM model, preserving local structural information of spatial features in the time dimension.

[0155] The ConvLSTM model (which takes a sequence of images from past moments as input) learns the dependence between the temporal and spatial trends of the natural environment in the target area through a convolutional gating mechanism. Then, it adjusts the weight distribution in the temporal and spatial dimensions through a dual self-attention mechanism to output a predicted image of the natural environment in the target area at future moments.

[0156] The system acquires real-time remote sensing images of the natural environment of the target area and aligns the predicted natural environment images with the actual remote sensing images in time. It then marks areas in the actual remote sensing images that conflict with the predicted natural environment images (the "conflicting area" refers to an environmental unit area in the actual remote sensing image that does not match the predicted natural environment image) as anomalous areas and provides feedback.

[0157] Step S3: Continuously monitor foreign objects on roads and abnormal areas in the natural environment of the target area.

[0158] The above formulas are all dimensionless calculations. The formulas are derived from software simulations using a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation. For example, there are weighting coefficients and proportional coefficients. The values ​​set are to quantify each parameter to obtain a specific value, which is convenient for subsequent comparison. The values ​​of the weighting coefficients and proportional coefficients are only required to not affect the proportional relationship between the parameters and the quantified values.

[0159] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion, characterized in that, include: Remote sensing images of various types of road debris are acquired, and the texture, shape, and color features corresponding to each type of road debris are extracted and fused to obtain fused features. The texture, shape, and color features corresponding to each type of road debris are then fused through a gating attention mechanism. Similarity learning is performed on the fused features to establish a high-dimensional index structure; real-time remote sensing images of the road are acquired to determine whether there are foreign objects on the road; if they exist, they are marked; if they do not exist, no action is taken. Historical natural images of the target area are acquired, and a temporal feature analysis module, a spatial feature analysis module, and a spatiotemporal joint prediction module are designed to output a predicted natural environment image of the target area at future times. Acquire actual remote sensing images and time-align the predicted natural environment images with the actual remote sensing images. Mark abnormal areas in the actual remote sensing images based on the predicted natural environment images. The specific steps of the time series feature analysis module are as follows: Historical natural images are decomposed at multiple scales and feature-encoded to obtain natural features; external environmental variables are fused as additional features. The natural features and additional features are standardized and normalized to obtain the global feature vector of the historical natural image. The global feature vector is fed into the input of the LSTM / GRU network. The input of the LSTM / GRU network receives the global feature vector in chronological order. The time dependency is dynamically adjusted through the forget gate, input gate and output gate to learn the long-term trend of natural environment change. During the training of the LSTM / GRU network, the TimeSformer module is introduced to calculate the correlation weights of each parameter of the global feature vector at different times. By introducing ARIMA or SARIMA models to learn the natural changes in the periodic or seasonal patterns of the natural environment, and merging them with the long-term trend of natural environment changes, the temporal change trend of the natural environment in the target area can be obtained. The specific steps of the spatial feature analysis module are as follows: The spatial feature analysis module consists of a CNN model and a GNN model; CNN models extract local spatial structure: By using multi-layer convolution and pooling operations, the spatial features of historical natural images are learned. Shallow convolution is used to capture the texture and edge information of spatial features, and deep convolution is used to abstract them into high-dimensional spatial semantic features, dividing historical natural images into environmental units. GNN models model spatial adjacency relationships: Each environmental unit is treated as a graph node, and edge weights are defined based on spatial proximity or geographical similarity to form a spatial adjacency graph between regions. Features are propagated between adjacent nodes through an information transfer mechanism to achieve cross-regional environmental change impact modeling. The specific process is as follows: Each graph node first integrates its own initial features, and then sends a message to its neighboring graph nodes. The message content integrates its own features and edge weight information. After receiving the message, the neighboring nodes combine it with their own original features to perform aggregation and update, thereby obtaining the spatial change trend of the natural environment of the target area. The specific steps of the spatiotemporal joint prediction module are as follows: The system learns the dependence between the temporal and spatial changes of the natural environment in the target area through a convolutional gating mechanism, and then adjusts the weight distribution in the temporal and spatial dimensions through a dual self-attention mechanism to output a predicted image of the natural environment in the target area at future times. The system acquires real-time remote sensing images of the natural environment of the target area, aligns the predicted natural environment images with the actual remote sensing images in time, marks areas in the actual remote sensing images that conflict with the predicted natural environment images as anomalous areas, and provides feedback. Continuously monitor foreign objects on roads and abnormal areas in the natural environment of the target area.

2. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps for extracting texture features are as follows: Construct and iterate through all elements of the gray-level co-occurrence matrix corresponding to the remote sensing image of road debris, and calculate the statistics of the gray-level co-occurrence matrix corresponding to each road debris; The statistics of the gray-level co-occurrence matrix corresponding to each road object are combined into a row vector; Summarize and merge all row vectors of each type of road debris to obtain the original texture matrix of each type of road debris; The original texture matrices of each type of road debris are collected, standardized, and the covariance matrix is ​​calculated. Eigenvalues ​​and eigenvectors of each type of road debris are obtained by performing eigenvalue decomposition on the covariance matrix. Calculate the cumulative variance contribution rate of the feature vector corresponding to each type of road debris, and calculate the product of the cumulative variance contribution rate of different feature vectors and the corresponding feature value of the feature vector, which is used as the feature selection coefficient of different feature vectors for each type of road debris. The feature vectors of the top 30% of the feature selection coefficients for each type of road debris are used as the main feature vectors of that type of road debris. The main feature vectors of all road debris are combined into a projection matrix. The original texture matrix of each type of road debris is multiplied by its corresponding projection matrix to obtain the texture features of each type of road debris.

3. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps for extracting shape and color features are as follows: Extracting shape features: The remote sensing image of road debris was converted from a color image to a grayscale image, and the road debris image and background were segmented in the grayscale image to obtain a binary image. Using a lightweight segmentation network, the semantic mask of white pixels in the binary image is set to 1, and the semantic mask of black pixels is set to 0, to obtain the mask image corresponding to each type of road foreign object. Mark the outline of the foreign object region in the binary image of each type of road foreign object, and then calculate the geometric moments of the foreign object region outline; convert the geometric moments corresponding to each type of road foreign object into Hu moments that are invariant to translation, rotation and scaling, and use them as the shape features of each type of road foreign object. Extract color features: The images of road debris were converted into three color spaces: RGB, HSV, and Lab. Gaussian pyramid decomposition was then performed on the images in each color space to obtain a multi-scale image set for each type of road debris. Based on the mask image corresponding to each type of road foreign object, calculate the local color histogram of the multi-scale image set corresponding to each type of road foreign object in RGB, HSV and Lab color spaces; By using a channel attention mechanism, the features of three color spaces are weighted and fused to obtain the color features of each type of road object.

4. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of feature fusion are as follows: The three modal features of each type of road foreign object are standardized, and the standardized three modal features are projected through a linear layer. The three modal features after projection are L2 normalized; the three normalized modal features are activated and then concatenated in dimension 1 to form a context vector. The three modal features of the first type of road foreign object are fused: the context vector of the first type of road foreign object is input, and the output is 3 logits values; Transform the three logits values ​​into a single probability distribution alpha containing three probability values; Multiply each element in alpha by the projection features corresponding to the texture, shape, and color features respectively, and then add them together to obtain the trimodal mixed feature; The three-modal hybrid features are then L2 normalized to obtain the fused features of the first type of road foreign objects. By fusing the three modal features of other road debris, the fused features of each type of road debris are obtained.

5. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of similarity learning are as follows: Add labels to all remote sensing images of road debris, marking the type of road debris to which each remote sensing label belongs; The remote sensing images of all road debris are combined into sample pairs, and the loss function of the Siamese network is designed as follows: Let the sample pair be (x （i） x （j） ), where i and j represent the labels of the remote sensing images; Calculate remote sensing image x （i） and x （j） Embedding distance d （ij） ; If labels i and j are the same, then parameter y （ij） The value of is 1; if they are different, then the parameter y （ij） The value is 0; Let the loss function be L(x) （i） x （j） ): ； Where, n （mar） Indicates the interval constant; Input all sample pairs into the Siamese network and adjust n. （mar） Find the value of that minimizes the loss function and construct the feature space; Clustering is performed on the fusion features corresponding to all foreign object images, and the fusion features corresponding to each type of foreign object image are grouped into one category to generate a multi-level index tree. Based on the distribution of various road debris fusion features in the feature space in the multi-level index tree, a high-dimensional index structure is used to store the fusion features corresponding to various road debris. The fusion features of real-time remote sensing images on the road are acquired and extracted, the fusion features of real-time remote sensing images are mapped into the feature space, and it is determined whether the fusion features of real-time remote sensing images match the existing fusion features in the high-dimensional index structure. If a match is found, mark the foreign object. If there is no match, then nothing is done.

Images