Foreign matter intelligent identification and dynamic monitoring method based on remote sensing data fusion

Abstract

The invention provides a foreign matter intelligent identification and dynamic monitoring method based on remote sensing data fusion, and belongs to the field of image identification. The problem of low foreign matter recognition efficiency is solved; the method specifically comprises the following steps: acquiring and extracting fusion features of remote sensing pictures of various road foreign matters; similarity learning is carried out on the fusion features, and a high-dimensional index structure is established; acquiring a real-time remote sensing picture on the road, and judging whether foreign matters exist on the road or not; if so, marking a foreign matter; if not, not processing; acquiring a historical natural image of a target area, designing a time sequence feature analysis module, a spatial feature analysis module and a spatio-temporal joint prediction module, and outputting a natural environment prediction image of the target area at a future moment; acquiring an actual remote sensing image, and marking an abnormal region in the actual remote sensing image according to the natural environment prediction image; according to the method, the foreign matter recognition efficiency is improved through obtaining and feature extraction of the remote sensing images in the road and natural environment of the target area.

Description

TECHNICAL FIELD

[0001] The application is based on the remote sensing data fusion foreign matter intelligent identification and dynamic monitoring method, and relates to the field of image processing. BACKGROUND

[0002] The existing method for constructing remote sensing data to identify foreign matters has the following shortcomings: Complexity of feature extraction and fusion: Some existing foreign matter identification methods only use single or simple feature extraction methods, which cannot fully utilize the rich information of multi-source remote sensing data; for example, only using color features for foreign matter identification, in the face of complex texture or different shape foreign matters, due to the lack of fusion of key information such as texture and shape, it is difficult to accurately distinguish foreign matters from the background, resulting in low identification accuracy.

[0003] Inaccuracy of identification results: Some existing foreign matter identification methods use fixed weights or simple distance calculation methods, which cannot be dynamically adjusted according to the specific characteristics of the image; for geometrically complex foreign matter images, if the weight of shape features is not increased, the shape information may be ignored due to the dominant role of other features, resulting in misjudgment or omission of foreign matters.

[0004] Insufficient automation: Traditional foreign matter monitoring methods usually require a lot of manual intervention, such as manual feature selection, manual image comparison, etc., which not only increases labor costs, but also introduces subjective errors in manual operation, and due to the speed limit of manual operation, it also affects the efficiency and timeliness of monitoring. SUMMARY

[0005] In view of the shortcomings of the prior art, the application aims to provide a foreign matter intelligent identification and dynamic monitoring method based on remote sensing data fusion, which aims to solve the problem of low efficiency of foreign matter identification.

[0006] In order to achieve the above-mentioned purpose, the application is realized by the following technical scheme: the foreign matter intelligent identification and dynamic monitoring method based on remote sensing data fusion comprises: Obtain remote sensing pictures of various road foreign matters, extract and fuse the texture features, shape features and color features corresponding to each type of road foreign matter to obtain fused features; perform similarity learning on the fused features to establish a high-dimensional index structure; obtain real-time remote sensing pictures on the road, and judge whether there are foreign matters on the road; if so, mark the foreign matters; if not, do not process; Obtain historical natural images of the target area, design a time series feature analysis module, a spatial feature analysis module and a spatio-temporal joint prediction module, and output a natural environment prediction image of the target area at a future time; obtain an actual remote sensing image, and time-align the natural environment prediction image with the actual remote sensing image, and mark the abnormal area in the actual remote sensing image according to the natural environment prediction image; Continuously monitor the foreign matters on the roads in the target area and the abnormal areas in the natural environment.

[0007] Further, the specific steps of extracting the texture features are as follows: All elements of the gray level co-occurrence matrix corresponding to the road foreign matter remote sensing picture are constructed and traversed, and the statistics of the gray level co-occurrence matrix corresponding to each road foreign matter are calculated; The statistics of the gray level co-occurrence matrix corresponding to each road foreign matter are combined into a row vector; All row vectors of each type of road foreign matter are summarized and merged to obtain the original texture matrix of each type of road foreign matter; the style of the original texture matrix is (m （te） ×n （te） ), wherein m （te） represents the number of all row vectors of the road foreign matter, and n （te） represents the type number of all statistics of the road foreign matter; All original texture matrices of each type of road foreign matter are summarized and processed to calculate the covariance matrix; The covariance matrix is subjected to eigenvalue decomposition to obtain the eigenvalue and eigenvector of each type of road foreign matter; The cumulative variance contribution rate of the eigenvector corresponding to each type of road foreign matter is calculated, and the product of the cumulative variance contribution rate of different eigenvectors and the eigenvalue corresponding to the eigenvector is calculated as the feature selection coefficient of each type of road foreign matter different eigenvector; The eigenvector corresponding to the top 30% feature selection coefficient of each type of road foreign matter is taken as the main eigenvector of the road foreign matter; the style of the main eigenvector is a column vector of (n （te） ×1); All main eigenvectors of the road foreign matter are combined into a projection matrix of (n （te） ×k （te） ), and the original texture matrix of each type of road foreign matter is multiplied by the corresponding projection matrix to obtain the texture feature of each type of road foreign matter.

[0008] Further, the specific steps of extracting the shape and color features are as follows: Extraction of shape features: The road foreign matter remote sensing picture is converted from a color image to a gray scale image, and the road foreign matter image and the background in the gray scale image are segmented to obtain a binary image; wherein the road foreign matter is set as white and the background is set as black; Using a lightweight segmentation network, the semantic mask of the white pixels in the binary image is set to 1, and the semantic mask of the black pixels is set to 0, to obtain the mask image corresponding to each type of road foreign matter; The contours of the foreign object area are marked in the binary image of each type of road foreign object, and the geometric moments of the foreign object area contours are calculated; the geometric moments corresponding to each type of road foreign object are converted into Hu moments which are invariant to translation, rotation and scaling, as the shape features of each type of road foreign object; Color features are extracted: The road foreign object image is converted into RGB, HSV and Lab color spaces, and the image in each color space is decomposed by a Gaussian pyramid to obtain a multi-scale image set of each type of road foreign object; According to the mask image corresponding to each type of road foreign object, the local color histogram of the multi-scale image set corresponding to each type of road foreign object in the RGB, HSV and Lab color spaces is calculated; The features of the three color spaces are weighted and fused through a channel attention mechanism to obtain the color features of each type of road foreign object.

[0009] Further, the specific steps of feature fusion are as follows: The texture features, shape features and color features corresponding to each type of road foreign object are fused through a gated attention mechanism, and the specific fusion process is as follows: The three modal features of each type of road foreign object are standardized respectively, and the standardized three modal features are projected through a linear layer respectively; The projected three modal features are L2 normalized; the normalized three modal features are activated using a ReLU function, and then spliced in dimension 1 to form a context vector; The three modal features of the first type of road foreign object are fused through a gated network: The context vector of the first type of road foreign object is input, and the output is 3 logits values; The 3 logits values are converted into a single probability distribution alpha containing 3 probability values through a Softmax function; Each element in alpha is multiplied by the projection features corresponding to the texture features, shape features and color features respectively, and then added to obtain a three-modal mixed feature; The three-modal mixed feature is L2 normalized again to obtain the fusion feature of the first type of road foreign object; The three modal features of other road foreign objects are fused to obtain the fusion feature of each type of road foreign object.

[0010] Further, the specific steps of similarity learning are as follows: Labels are added to all remote sensing images of road foreign objects to mark the type of road foreign object to which each remote sensing image belongs; All remote sensing images of road foreign objects are combined into sample pairs, and a loss function of a twin network is designed: Let the sample pair be (x （i） , x（j） ), wherein i and j represent the labels of the remote sensing pictures; calculating a remote sensing picture x （i） and the embedding distance d （j） of x （ij） ; if the labels i and j are the same, the value of the parameter y （ij） is 1; if different, the value of the parameter y （ij） is 0; Let the loss function be L(x （i） , x （j） ): ; wherein n （mar） represents an interval constant; input all sample pairs into the twin network, adjust the value of n （mar） to minimize the loss function, and construct a feature space; use a clustering algorithm to cluster the fusion features corresponding to all foreign object pictures, classify the fusion features corresponding to each class of foreign object pictures into one class, and generate a multi-layer index tree; According to the distribution of the fusion features of each type of road foreign object in the feature space in the multi-layer index tree, use a high-dimensional index structure to store the fusion features corresponding to each type of road foreign object; Obtain and extract the fusion features of the real-time remote sensing picture on the road, map the fusion features of the real-time remote sensing picture to the feature space, and determine whether the fusion features of the real-time remote sensing picture match the existing fusion features in the high-dimensional index structure; If matched, mark the foreign object and feed back to the traffic management department; If not matched, do not process.

[0011] Further, the specific steps of the time sequence feature analysis module are as follows: Perform multi-scale decomposition and feature coding on the historical natural image to obtain natural features; and fuse external environmental variables as additional features; Standardize and normalize the natural features and additional features to obtain a global feature vector of the historical natural image; Send the global feature vector to the input end of the LSTM / GRU network, and the input end of the LSTM / GRU network receives the global feature vector in time sequence, dynamically adjusts the time dependence through the forget gate, input gate and output gate, and learns the long-term trend of the natural environment change; In the training process of the LSTM / GRU network, introduce the TimeSformer module to calculate the correlation weight corresponding to each parameter of the global feature vector at different times; The ARIMA or SARIMA model is introduced to learn the periodicity or seasonal regularity of natural environment and is combined with the long-term trend of the natural environment change, so that the time change trend of the natural environment of the target area is obtained.

[0012] Further, the specific steps of the spatial feature analysis module are as follows: The spatial feature analysis module is composed of a CNN model and a GNN model. The CNN model extracts local spatial structure: The CNN model learns the spatial features of historical natural images through multi-layer convolution and pooling operations, captures texture and edge information of the spatial features by using shallow convolution, and abstracts high-dimensional spatial semantic features by using deep convolution, so as to divide the historical natural images into environment units. The GNN model models the spatial adjacency relationship: The GNN model regards each environment unit as a graph node, defines the edge weight according to the spatial proximity or geographical similarity, forms a spatial adjacency graph between regions, and realizes the modeling of the influence of environmental change across regions through the information transmission mechanism between adjacent nodes. Each graph node first integrates its own initial features, each graph node sends a message to the adjacent graph node, and the message content integrates the own features and the weight information of the edge; after receiving the message, the adjacent node updates the original features to obtain the spatial change trend of the natural environment of the target area.

[0013] Further, the specific steps of the spatio-temporal joint prediction module are as follows: The ConvLSTM model learns the dependency relationship between the time change trend and the spatial change trend of the natural environment of the target area through the convolution gate mechanism, adjusts the weight distribution in the time and space dimensions through the double self-attention mechanism, and outputs the natural environment prediction image of the target area at the future time. The actual remote sensing image of the natural environment of the target area is obtained in real time, the natural environment prediction image is time-aligned with the actual remote sensing image, the area conflicting with the natural environment prediction image in the actual remote sensing image is marked as an abnormal area, and feedback is given.

[0014] Compared with the prior art, the present application has the following advantages: Efficient feature extraction and fusion: the application adopts feature normalization and attention fusion (GAF) technology to perform Z-score standardization on multi-channel features such as color, texture and shape, eliminate dimensional differences, and dynamically learn the importance of each feature channel and fuse through a gating attention mechanism; this fusion method can adaptively weight the importance of features, so that the model pays more attention to texture features in high-texture complexity images and focuses on shape features in clear structure images, thereby fully utilizing the complementary information of multi-source remote sensing data, extracting more comprehensive and discriminative features, and effectively improving the accuracy of foreign object identification.

[0015] Fast similarity search: the application uses a contrast learning network for deep metric learning to construct a distinguishable feature embedding space, and combines high-dimensional index structures such as FAISS or HNSW to realize approximate nearest neighbor search; through continuous training, the network learns the feature representation of "small semantic distance, similar", and with the help of efficient index structure, it can quickly find a similar picture set in a database containing massive remote sensing images, meeting the real-time requirements of dynamic monitoring and timely grasping the appearance and changes of foreign objects.

[0016] Accurate identification results: the application uses an adaptive similarity reweighting strategy to dynamically adjust the weights according to the complexity of the sample's texture, shape and color, further improving the accuracy of similarity measurement and making the foreign object identification results more reliable; at the same time, the application realizes high automation in the whole process from feature extraction, fusion to similarity search and foreign object identification, reducing the need for manual intervention and reducing the risk of misjudgment and missed judgment. BRIEF DESCRIPTION OF DRAWINGS

[0017] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, with reference to the accompanying drawings: Figure 1 A schematic diagram of the method of the application is shown in Figure 2 A schematic diagram of the road foreign object identification of the application is shown in Figure 3 A schematic diagram of the natural environment foreign object identification of the application is shown in DETAILED DESCRIPTION

[0018] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below with reference to the drawings and specific embodiments.

[0019] Please refer to Figure 1 and Figure 2 The foreign object intelligent identification and dynamic monitoring method based on remote sensing data fusion comprises: Step S1: Obtain remote sensing pictures of various (already appeared) road foreign matters, extract and fuse texture features, shape features and color features corresponding to each type of road foreign matter to obtain fused features; perform similarity learning on the fused features to establish a high-dimensional index structure; obtain real-time remote sensing pictures on the road to determine whether there is a foreign matter on the road; if yes, mark the foreign matter (and feed back to the traffic control department); if not, do not process; It should be noted that the "target area" in the present application refers to the area where the present application (remote sensing data fusion-based foreign matter intelligent recognition and dynamic monitoring method) is used for foreign matter recognition. Obtain remote sensing pictures of various (already appeared) road foreign matters, and extract fused features of the road foreign matters; Preprocess (eliminate sensor errors and atmospheric effects) the remote sensing pictures of the road foreign matters (radiometric calibration, atmospheric correction, geometric correction, etc.), and then extract texture features, shape features and color features of the remote sensing pictures corresponding to each type of road foreign matter; Extract texture features: Construct a gray level co-occurrence matrix (GLCM) of the remote sensing pictures of the road foreign matters; The elements in the gray level co-occurrence matrix (GLCM) represent the frequency or probability of occurrence of a pair of gray level pixels in the remote sensing pictures of the road foreign matters at a specific distance and direction; Traverse all elements of the gray level co-occurrence matrix (GLCM), calculate the mean and standard deviation of all pixel gray levels corresponding to each row direction and each column direction in the gray level co-occurrence matrix, and calculate the statistics (including but not limited to contrast, correlation, energy, entropy, etc.) of the gray level co-occurrence matrix corresponding to each road foreign matter; Combine the statistics of the gray level co-occurrence matrix corresponding to each road foreign matter into a row vector; Summarize and merge all row vectors of each type of road foreign matter to obtain an original texture matrix of each type of road foreign matter; the style of the original texture matrix is (m （te） ×n （te） ), where m （te） represents the number of all row vectors of the type of road foreign matter, and n （te） represents the type number of all statistics of the type of road foreign matter; Summarize all original texture matrices of each type of road foreign matter, and perform Z-score standardization processing (to eliminate the dimensional differences between different statistics) to calculate the covariance matrix of the original texture matrix corresponding to each type of road foreign matter; Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues and eigenvectors of each type of road foreign matter; Calculate the cumulative variance contribution rate of each feature vector corresponding to each type of road foreign matter, and calculate the product of the cumulative variance contribution rate of different feature vectors and the feature value corresponding to the feature vector as the feature selection coefficient of each type of road foreign matter different feature vector; The feature vectors corresponding to the top 30% feature selection coefficients of each type of road foreign matter are selected as the main feature vectors of the road foreign matter of this type in order from large to small (it should be noted that the "30%" in this place of the application is the default selection ratio of the main feature vector, and the user or the relevant technical personnel can adjust the selection ratio of the main feature vector according to the actual needs); The main feature vector is a column vector of (n （te） ×1); The main feature vectors of the road foreign matter are combined into a projection matrix of (n （te） ×k （te） The original texture matrix of each type of road foreign matter is multiplied by the corresponding projection matrix to obtain the texture feature of each type of road foreign matter. Extract the shape feature: Convert the road foreign matter remote sensing picture from a color image to a grayscale image (if the road foreign matter remote sensing picture itself is a grayscale image, skip this step), segment the road foreign matter image and the background ("background" means "non-road foreign matter image" in the grayscale image) in the grayscale image through the OTSU adaptive threshold algorithm, and convert the segmented grayscale image into a binary image, wherein the road foreign matter is set to white and the background is set to black. Using a lightweight segmentation network (such as DeepLabv3-MobileNet or Segment Anything model), set the semantic mask of white pixels in the binary image to 1 and the semantic mask of black pixels to 0 to obtain a mask image corresponding to each type of road foreign matter. Using the cv2.findContours() function, mark the contour of the foreign matter area in the binary image of each type of road foreign matter (i.e. the white pixel area in the binary image of each type of road foreign matter), and then calculate the geometric moments of the contour of the foreign matter area through the moments function. Convert the geometric moments corresponding to each type of road foreign matter into Hu moments that are invariant to translation, rotation, and scaling, as the shape feature of each type of road foreign matter. Extract the color feature: Enhance the image contrast of each type of road foreign matter remote sensing picture using the CLAHE algorithm (contrast limited adaptive histogram equalization) to complete the image preprocessing. Convert the preprocessed image (corresponding to each type of road foreign matter) into RGB, HSV, and Lab color spaces, and perform Gaussian pyramid decomposition on the images in each color space to obtain a multi-scale image set of each type of road foreign matter. According to the mask image corresponding to each type of road foreign matter, the local color histogram of the multi-scale image set corresponding to each type of road foreign matter in RGB, HSV and Lab color spaces is calculated; Through the channel attention (Channel Attention) mechanism, the features of three color spaces (RGB, HSV, Lab, etc.) are weighted and fused, and the specific implementation process is as follows: The local color histogram of the RGB, HSV and Lab corresponding color channel is obtained, and the mean value of the local color histogram corresponding to each color channel is calculated to obtain mean(RGB), mean(HSV) and mean(Lab); A (learnable) weight parameter W （RGB） , W （HSV） and W （Lab） is introduced for each color channel; For each color channel, exp(W （RGB） ×mean(RGB)), exp(W （HSV） ×mean(HSV)) and exp(W （Lab） ×mean(Lab)) are calculated respectively as the (unnormalized) attention score of each color channel; wherein, exp() represents the exponential function; The (unnormalized) attention score of each color channel is normalized by the softmax function to obtain the attention weight of each color channel α （RGB） , α （HSV） and α （Lab） ; According to the attention weights α （RGB） , α （HSV） and α （Lab） , the local color histogram corresponding to each type of road foreign matter in different color channels (i.e. RGB, HSV and Lab color channels) is weighted and fused to obtain the (fusion) color feature of each type of road foreign matter; Feature fusion: The texture feature, shape feature and color feature corresponding to each type of road foreign matter are fused through the gated attention (Gated Attention Fusion, GAF) mechanism, and the specific fusion process is as follows: The three modal features (i.e. texture feature, shape feature and color feature) of each type of road foreign matter are respectively executed Z-score standardization (to eliminate the dimensional difference of different feature sources), and then projected through a linear layer. (Map the three modal features of each type of road foreign matter to the same dimension); The projected three modal features are L2 normalized; the normalized three modal features are activated using the ReLU function, and then concatenated on dimension 1 to form the context vector; The three modal features of the first type of road foreign matter are fused through a gating network (the gating network consists of two fully connected layers, and the ReLU function is used as the activation function in the middle): The context vector of the first type of road foreign matter is input, and the output is 3 logits values (i.e. 3 logits values, corresponding to the attention weights of the three modalities respectively); The 3 logits values are converted into a single probability distribution alpha containing 3 probability values through the Softmax function; (this distribution describes the dynamic weight distribution of the three modal features in the similarity measurement); Each element in alpha is multiplied by the corresponding projection feature of the texture feature, shape feature and color feature respectively, and then added to obtain the three-modal mixed feature; (here, "projection feature" refers to the feature after the linear layer, L2 normalization and ReLU activation) The three-modal mixed feature is L2 normalized again to obtain the fusion feature of the first type of road foreign matter; (The same steps are repeated to obtain the fusion feature of the first type of road foreign matter) The three modal features of other road foreign matters are fused to obtain the fusion feature of each type of road foreign matter; Similarity learning: Labels are added to all remote sensing pictures of road foreign matters to mark the type of road foreign matter to which each remote sensing picture belongs; All remote sensing pictures of road foreign matters are combined into sample pairs (the remote sensing pictures in the sample pair: can be two remote sensing pictures of the same type of road foreign matter, or two remote sensing pictures of different types of road foreign matter); Design the loss function of the Siamese Network: Let the sample pair be (x （i） , x （j） ), where i and j represent the labels of the remote sensing pictures; Calculate the embedding distance d （ij） of the remote sensing pictures x （i） and x （j） ; Determine whether the labels i and j are the same, if they are the same, the value of the parameter y （ij） is 1; if they are different, the value of the parameter y （ij） is 0; Let the loss function be L(x （i） , x （j） ): ; Where n（mar） interval constant (i.e. the hyperparameter preset for the input twin network); input all sample pairs into the twin network (such as Siamese Network), adjust the value of n （mar） through continuous training to minimize the loss function (of the twin network), so that the network learns the embedding space features of "small semantic distance means similarity", and completes the construction of the feature space; After completing the construction of the feature space, use a clustering algorithm (such as K-means or Product Quantization) to cluster the fusion features corresponding to all foreign object images, and classify the fusion features corresponding to each class of foreign object images into one class to generate a multi-layer index tree; According to the distribution of the fusion features of each type of road foreign object in the feature space in the multi-layer index tree, use a high-dimensional index structure (such as FAISS or HNSW) to store the fusion features corresponding to each type of road foreign object; Obtain real-time remote sensing images on the road, (repeat the same steps of extracting the fusion features of the road foreign object remote sensing images) extract the fusion features of the real-time remote sensing images, map the fusion features of the real-time remote sensing images to the feature space, and compare them with the fusion features stored in the high-dimensional index structure to determine whether the fusion features of the real-time remote sensing images match the existing fusion features in the "high-dimensional index structure"; If matched, it means that there is a foreign object on the road, which is marked and fed back to the traffic management department; If not matched, it means that there is no foreign object on the road, which is not processed; Step S2: Obtain historical natural images of the target area, design a time series feature analysis module, a spatial feature analysis module, and a spatio-temporal joint prediction module to output natural environment prediction images of the target area at future time; Obtain actual remote sensing images, and align the natural environment prediction images with the actual remote sensing images in time, and mark the abnormal areas in the actual remote sensing images according to the natural environment prediction images; Please refer to Figure 3 Collect remote sensing images of the natural environment in recent years from the target area as historical natural images; Perform geometric registration and radiation correction (so that the pixel positions of the same geographical area at different times correspond to each other) on the historical natural images, and then enhance the image clarity and texture distinguishability through image enhancement and noise filtering algorithms (such as histogram equalization and adaptive filtering); Time series feature analysis module: In the feature extraction stage, perform multi-scale decomposition and feature coding on the historical natural images to obtain natural features; Natural features include three types: texture features (obtained through gray level co-occurrence matrix or convolution kernel automatic learning); spectral features (reflecting the reflectivity difference of vegetation, soil, and water); geometric features (such as boundaries, corner points, and shape contours); Fusing external environmental variables (such as temperature, precipitation, NDVI, land surface temperature, etc.) as additional features (to improve the model's perception of complex natural evolution rules); Standardizing and normalizing natural features and additional features to obtain global feature vectors of historical natural images; Sending the global feature vectors to the input end of the LSTM / GRU network, which receives the global feature vectors in chronological order, dynamically adjusts the time dependence through the forget gate, input gate, and output gate, and learns the long-term trend of natural environmental changes (i.e., LSTM uses the forget gate, input gate, and output gate to filter and remember the global feature vectors corresponding to different historical natural images, automatically determines the historical information that needs to be retained (such as the cumulative effect of persistent drought), and ignores short-term fluctuations (such as one-time climate anomalies); GRU, as a simplified structure, achieves similar functions as the "LSTM model" through the update gate and reset gate; After the model completes multi-time sequence training, it can abstract patterns that evolve over time from past environmental images, such as decreasing vegetation coverage and seasonal fluctuations in water area, so that it can infer future trends based on the current state during the prediction phase, achieving long-term evolution prediction of natural environmental changes); During the training process of the LSTM / GRU network, the TimeSformer module based on the Transformer structure is introduced; this module uses the self-attention mechanism (Self-Attention) to calculate the correlation weights of the parameters of the global feature vectors at different times in the entire time dimension; Introducing ARIMA or SARIMA models to learn the periodic or seasonal changes of natural environments in the target area, and combining them with the long-term trend of natural environmental changes to obtain the time trend of the natural environment in the target area; Spatial feature analysis module: This module consists of two parts: convolutional neural network (CNN) and graph neural network (GNN); CNN model extracts local spatial structure: The CNN model learns the spatial features of historical natural images in the target area (such as topography, river direction, building density, and vegetation coverage) through multiple convolution and pooling operations, uses shallow convolution to capture texture and edge information of spatial features, and uses deep convolution to abstract high-dimensional semantic features, dividing historical natural images into environmental units (such as "mountain vegetation area", "urban construction area", "water area", etc. Typical environmental units); GNN model modeling spatial adjacency relationship: The GNN model regards each environment unit as a graph node, defines the edge weight based on spatial proximity or geographical similarity, forms a spatial adjacency graph between regions, and propagates features between adjacent nodes through a message passing mechanism to model the influence of environmental changes across regions. The specific process is as follows: Each graph node integrates its initial features (such as vegetation coverage, river area, and biological activity area), and each graph node sends a message to adjacent graph nodes, which combines its own features and edge weight information. After receiving the message, the adjacent node updates its original features to obtain the spatial change trend of the target region's natural environment. Temporal and spatial joint prediction module: The ConvLSTM model introduces convolution operations in the gating unit of the LSTM model, preserving the local structure information of spatial features in the time dimension. The ConvLSTM model (taking a sequence of past images as input) learns the dependency relationship between the time change trend and the spatial change trend of the target region's natural environment through a convolution gating mechanism, and adjusts the weight distribution in the time and spatial dimensions through a double self-attention mechanism to output the predicted image of the target region's natural environment at the future time. Real-time acquisition of the actual remote sensing image of the target region's natural environment, and time alignment of the predicted image of the natural environment with the actual remote sensing image, marking the conflicting area (the area that does not match the predicted image of the natural environment in the actual remote sensing image) as an abnormal area, and feeding back.

[0020] Step S3: Continuously monitor the foreign matter on the target region's road and the abnormal area on the natural environment.

[0021] The above formulas are dimensionless numerical calculations, and the formulas are obtained by software simulation of a large amount of data to obtain the most recent real situation. The preset parameters in the formula are set by the person skilled in the art according to the actual situation, such as the weight coefficient and the proportion coefficient, and the size of the weight coefficient and the proportion coefficient is set to quantify the specific value of each parameter for subsequent comparison. As long as the proportion relationship between the parameter and the quantized value is not affected, the size of the weight coefficient and the proportion coefficient can be set.

[0022] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion, characterized in that, include: Acquire remote sensing images of various types of road debris, extract and fuse the texture features, shape features and color features corresponding to each type of road debris to obtain fused features; Similarity learning is performed on the fused features to establish a high-dimensional index structure; real-time remote sensing images of the road are acquired to determine whether there are foreign objects on the road; if they exist, they are marked; if they do not exist, no action is taken. Historical natural images of the target area are acquired, and a temporal feature analysis module, a spatial feature analysis module, and a spatiotemporal joint prediction module are designed to output a predicted natural environment image of the target area at future times. Acquire actual remote sensing images and time-align the predicted natural environment images with the actual remote sensing images. Mark abnormal areas in the actual remote sensing images based on the predicted natural environment images. Continuously monitor foreign objects on roads and abnormal areas in the natural environment of the target area.

2. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps for extracting texture features are as follows: Construct and iterate through all elements of the gray-level co-occurrence matrix corresponding to the remote sensing image of road debris, and calculate the statistics of the gray-level co-occurrence matrix corresponding to each road debris; The statistics of the gray-level co-occurrence matrix corresponding to each road object are combined into a row vector; Summarize and merge all row vectors of each type of road debris to obtain the original texture matrix of each type of road debris; The original texture matrices of each type of road debris are collected, standardized, and the covariance matrix is ​​calculated. Eigenvalues ​​and eigenvectors of each type of road debris are obtained by performing eigenvalue decomposition on the covariance matrix. Calculate the cumulative variance contribution rate of the feature vector corresponding to each type of road debris, and calculate the product of the cumulative variance contribution rate of different feature vectors and the corresponding feature value of the feature vector, which is used as the feature selection coefficient of different feature vectors for each type of road debris. The feature vectors of the top 30% of the feature selection coefficients for each type of road debris are used as the main feature vectors of that type of road debris. The main feature vectors of all road debris are combined into a projection matrix. The original texture matrix of each type of road debris is multiplied by its corresponding projection matrix to obtain the texture features of each type of road debris.

3. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps for extracting shape and color features are as follows: Extracting shape features: The remote sensing image of road debris was converted from a color image to a grayscale image, and the road debris image and background were segmented in the grayscale image to obtain a binary image. Using a lightweight segmentation network, the semantic mask of white pixels in the binary image is set to 1, and the semantic mask of black pixels is set to 0, to obtain the mask image corresponding to each type of road foreign object. Mark the outline of the foreign object region in the binary image of each type of road foreign object, and then calculate the geometric moments of the foreign object region outline; convert the geometric moments corresponding to each type of road foreign object into Hu moments that are invariant to translation, rotation and scaling, and use them as the shape features of each type of road foreign object. Extract color features: The images of road debris were converted into three color spaces: RGB, HSV, and Lab. Gaussian pyramid decomposition was then performed on the images in each color space to obtain a multi-scale image set for each type of road debris. Based on the mask image corresponding to each type of road foreign object, calculate the local color histogram of the multi-scale image set corresponding to each type of road foreign object in RGB, HSV and Lab color spaces; By using a channel attention mechanism, the features of three color spaces are weighted and fused to obtain the color features of each type of road object.

4. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of feature fusion are as follows: The three modal features of each type of road foreign object are standardized, and the standardized three modal features are projected through a linear layer. The three modal features after projection are L2 normalized; the three normalized modal features are activated and then concatenated in dimension 1 to form a context vector. The three modal features of the first type of road foreign object are fused: the context vector of the first type of road foreign object is input, and the output is 3 logits values; Transform the three logits values ​​into a single probability distribution alpha containing three probability values; Multiply each element in alpha by the projection features corresponding to the texture, shape, and color features respectively, and then add them together to obtain the trimodal mixed feature; The three-modal hybrid features are then L2 normalized to obtain the fused features of the first type of road foreign objects. By fusing the three modal features of other road debris, the fused features of each type of road debris are obtained.

5. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of similarity learning are as follows: Add labels to all remote sensing images of road debris, marking the type of road debris to which each remote sensing label belongs; The remote sensing images of all road debris are combined into sample pairs, and the loss function of the Siamese network is designed as follows: Let the sample pair be (x （i） , x （j） ), where i and j represent the labels of the remote sensing pictures; Calculate remote sensing image x （i） and x （j） Embedding distance d （ij） ; If labels i and j are the same, then parameter y （ij） The value of is 1; if they are different, then the parameter y （ij） The value is 0; Let the loss function be L(x) （i） x （j） ): ； Where, n （mar） Indicates the interval constant; Input all sample pairs into the Siamese network and adjust n. （mar） Find the value of that minimizes the loss function and construct the feature space; Clustering is performed on the fusion features corresponding to all foreign object images, and the fusion features corresponding to each type of foreign object image are grouped into one category to generate a multi-level index tree. Based on the distribution of various road debris fusion features in the feature space in the multi-level index tree, a high-dimensional index structure is used to store the fusion features corresponding to various road debris. The fusion features of real-time remote sensing images on the road are acquired and extracted, the fusion features of real-time remote sensing images are mapped into the feature space, and it is determined whether the fusion features of real-time remote sensing images match the existing fusion features in the high-dimensional index structure. If a match is found, mark the foreign object. If there is no match, then nothing is done.

6. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of the time series feature analysis module are as follows: Historical natural images are decomposed at multiple scales and feature-encoded to obtain natural features; external environmental variables are fused as additional features. The natural features and additional features are standardized and normalized to obtain the global feature vector of the historical natural image. The global feature vector is fed into the input of the LSTM / GRU network. The input of the LSTM / GRU network receives the global feature vector in chronological order. The time dependency is dynamically adjusted through the forget gate, input gate and output gate to learn the long-term trend of natural environment change. During the training of the LSTM / GRU network, the TimeSformer module is introduced to calculate the correlation weights of each parameter of the global feature vector at different times. By introducing ARIMA or SARIMA models to learn the natural changes in the periodic or seasonal patterns of the natural environment, and merging them with the long-term trend of natural environment changes, the temporal change trend of the natural environment in the target area can be obtained.

7. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of the spatial feature analysis module are as follows: The spatial feature analysis module consists of a CNN model and a GNN model; CNN models extract local spatial structure: By using multi-layer convolution and pooling operations, the spatial features of historical natural images are learned. Shallow convolution is used to capture the texture and edge information of spatial features, and deep convolution is used to abstract them into high-dimensional spatial semantic features, dividing historical natural images into environmental units. GNN models model spatial adjacency relationships: Each environmental unit is treated as a graph node, and edge weights are defined based on spatial proximity or geographical similarity to form a spatial adjacency graph between regions. Features are propagated between adjacent nodes through an information transfer mechanism to achieve cross-regional environmental change impact modeling. The specific process is as follows: Each graph node first integrates its own initial features, and then sends a message to its neighboring graph nodes. The message content integrates its own features and edge weight information. After receiving the message, the neighboring nodes combine it with their original features to perform aggregation and update, thereby obtaining the spatial change trend of the natural environment of the target area.

8. The method for intelligent identification and dynamic monitoring of foreign objects based on remote sensing data fusion according to claim 1, characterized in that, The specific steps of the spatiotemporal joint prediction module are as follows: The system learns the dependence between the temporal and spatial changes of the natural environment in the target area through a convolutional gating mechanism, and then adjusts the weight distribution in the temporal and spatial dimensions through a dual self-attention mechanism to output a predicted image of the natural environment in the target area at future times. The system acquires real-time remote sensing images of the natural environment of the target area, aligns the predicted natural environment images with the actual remote sensing images in time, marks areas in the actual remote sensing images that conflict with the predicted natural environment images as anomalous areas, and provides feedback.

Images