Slope monitoring method and device based on multi-modal data

Abstract

The invention relates to a slope monitoring method and device based on multi-modal data, and the method comprises the steps: obtaining monitoring data of a target slope, the monitoring data comprising interference radar data and multi-spectral data at different moments obtained by monitoring the target slope; inputting the monitoring data into the trained preset model to obtain a monitoring result; wherein the monitoring result represents the change condition of the target slope at different moments, the training sample is used for training the preset model to obtain the trained preset model, and the training sample is obtained based on the digital elevation data, the interference radar data and the multispectral data. As the training sample comprises the digital elevation data, the interference radar data and the multispectral data, the model can learn the change information of the side slope in the three-dimensional space in the time dimension, and the interference radar data and the multispectral data in the side slope can be more fully learned. Therefore, the obtained monitoring result can reflect the slope change condition more accurately, and then the slope monitoring efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a slope monitoring method and device based on multimodal data. Background Art

[0002] Slopes are the most common geomorphic geological environment in engineering and economic activities. In recent years, due to the influence of natural environment and human factors, slope instability has occurred frequently, such as landslides, landslides or mud-rock flows, causing serious loss of life and property.

[0003] In order to accurately evaluate and predict the stability of the slope and take effective measures in time, the relevant technology generally uses artificial on-site drilling exploration and placement of sensors to obtain geological information of key test points to monitor the slope, but due to the influence of geological conditions, it is not suitable for automatic monitoring of large-scale slopes. The relevant technology can also monitor the slope through spectral imaging and other methods, using relevant image algorithms, but the surface vegetation, building coverage and other phenomena affect the expression of slope changes in optical images, making it impossible to accurately monitor the slope, which in turn leads to low efficiency of slope monitoring by the relevant technology. Summary of the invention

[0004] The present invention provides a slope monitoring method and device based on multimodal data.

[0005] According to a first aspect of the present disclosure, a slope monitoring method based on multimodal data is provided, the method comprising:

[0006] Acquiring monitoring data of a target slope, wherein the monitoring data includes interferometric radar data and multispectral data obtained from monitoring the target slope at different times;

[0007] The monitoring data is input into a trained preset model to obtain a monitoring result; wherein the monitoring result characterizes the change of the target slope at different times, and the trained preset model is obtained by training the preset model through training samples, and the training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

[0008] According to a second aspect of the present disclosure, a slope monitoring device based on multimodal data is provided, the device comprising:

[0009] A data acquisition module, used to acquire monitoring data of a target slope, wherein the monitoring data includes interferometric radar data and multispectral data obtained from monitoring the target slope at different times;

[0010] A monitoring result acquisition module is used to input the monitoring data into a trained preset model to obtain monitoring results; wherein the monitoring results characterize the changes of the target slope at different times, and the trained preset model is obtained by training the preset model through training samples, and the training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

[0011] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0012] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0013] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above method of the present disclosure when executed by a processor.

[0014] The slope monitoring method and device based on multimodal data provided by the embodiment of the present disclosure can obtain monitoring data including interferometric radar data and multispectral data obtained at different times from monitoring the target slope, and input the monitoring data into the trained preset model to obtain monitoring results. Since the training samples include digital elevation data, interferometric radar data and multispectral data, the model can learn the change information of the slope in the three-dimensional space in the time dimension, and learn the interferometric radar data and multispectral data in the slope more fully, so that the monitoring results obtained can more accurately reflect the change of the slope, thereby improving the monitoring efficiency of the slope. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0016] Figure 1 A schematic diagram of modal slope monitoring provided by an exemplary embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of time alignment of multimodal data provided by an exemplary embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of multimodal data spatial alignment provided by an exemplary embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of wavelet decomposition provided for an exemplary embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of making a multi-modal slope variation data set provided by an exemplary embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of a model training process provided for an exemplary embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of a convolutional neural network structure provided for an exemplary embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of a process flow of automatic slope monitoring provided by an exemplary embodiment of the present disclosure;

[0024] Fig. 9 A flowchart of a slope monitoring method based on multimodal data provided by an exemplary embodiment of the present disclosure;

[0025] Fig.10 A schematic block diagram of functional modules of a slope monitoring device based on multimodal data provided by an exemplary embodiment of the present disclosure;

[0026] Fig.11 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0027] Fig.12 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0036] Slopes are the most common geomorphic geological environment in engineering and economic activities. In recent years, due to the influence of natural environment and human operation, slope instability (especially collapse, landslide and debris flow) has occurred frequently, causing serious loss of life and property. Therefore, accurately evaluating and predicting slope stability has practical engineering significance for regional construction, ecological cities and disaster management.

[0037] In the related technologies, the main basic means of slope monitoring include traditional methods and methods based on image processing. Among them, in the traditional method, sensors can be placed through manual on-site drilling exploration, or the geological information of key test points can be obtained by using equipment such as drones, and the safety factor and possible sliding surface of the slope can be estimated by combining data such as slope key indicators (such as standard cohesion, friction angle, natural bulk density and saturated bulk density) with probability and statistical functions. This method requires a lot of human resources and time, and because it is easily affected by geological conditions, it is not suitable for automatic monitoring of large-scale slopes.

[0038] In the image-based approach, for example, satellite-borne InSAR (Interferometric Synthetic Aperture Radar), ground-based synthetic aperture radar and spectral imaging can be used to process the collected images using subsequent algorithms to generate surface deformation images, which have the characteristics of high accuracy, wide coverage and strong adaptability. However, phenomena such as surface vegetation and building coverage affect the expression of slope changes in optical images, especially the deformation monitoring based on synthetic aperture radar interferometry, resulting in a decrease in monitoring accuracy in large-scale monitoring tasks. Different optical data have unavoidable noise and defects according to their imaging principles. For example, excessive slope displacement usually makes it impossible to perform accurate interferometric measurement of SAR images.

[0039] Therefore, in the relevant technology, slope detection is mainly carried out in the following ways:

[0040] (1) Conventional methods include the Global Navigation Satellite System (GNSS), total station, level measurement, crack meter, etc., which have high monitoring accuracy and relatively mature technology. However, since they are point measurements, they cannot reflect the overall deformation situation, and their coverage is limited and easily affected by the environment. The process from data collection to data interpretation to evaluate the slope status is complicated, and the processing of massive data is difficult, which requires a lot of time and manpower costs.

[0041] (2) Remote sensing technology has the characteristics of long distance, non-contact, and multi-dimensional macroscopic, and has been widely used in landslide and slope monitoring, such as drone laser radar technology (LiDAR) and InSAR technology. However, the monitoring accuracy of InSAR technology is limited by the radar line of sight and the direction of the slope, and the signal strength is affected by the slope direction, vegetation coverage, etc. Vegetation occlusion and excessive slope displacement usually make it impossible to perform accurate interferometric measurement of SAR (Synthetic Aperture Radar) images. In addition, the huge shadow of the mountain can also cause serious interference, resulting in the failure of SAR image correlation and surface displacement monitoring methods.

[0042] (3) LiDAR point cloud can effectively remove vegetation and obtain the real surface information under vegetation cover, capturing high-resolution and high-precision elevation information. However, its scanning range is relatively small, and it is difficult to obtain point cloud data of slopes in a large area in real time.

[0043] (4) Optical images have rich image information such as spectrum and texture, and the image processing theory is mature. However, their accuracy is restricted by factors such as imaging direction, terrain distortion, instrument and atmospheric noise, spatiotemporal decorrelation and co-registration error. Therefore, there is a lack of research on slope displacement monitoring.

[0044] With the 6G (6-Generation, the sixth generation of mobile communication technology) integrated air-space-ground network, by integrating air-space resources such as satellites, drones and high-altitude platforms, a global, ubiquitous, high-speed, intelligent, secure and reliable communication network will be built. It can meet the global communication requirements of low latency and high reliability, making it possible to obtain large-area satellite images in real time and perform high-speed data transmission, processing and analysis.

[0045] Therefore, in view of the technical problems existing in the above-mentioned related technologies, the embodiment of the present disclosure adopts a convolutional network slope automatic monitoring method based on multimodal data. By superimposing SAR images and optical remote sensing images in the channel dimension, CNN (Convolutional Neural Networks) is used to extract the mixed features of multimodal data to fuse different feature information between different data sources, overcome the environmental interference problems such as illumination or occlusion existing in single modal data, and realize all-weather real-time monitoring. Since the LiDAR point cloud acquisition method cannot obtain large-area slope scanning points in real time to generate high-precision DEM (Digital Elevation Model) data for learning, the embodiment of the present disclosure uses an unsupervised learning method to construct the training of a separate branch intervention monitoring model, and uses a feature-level fusion method to introduce slope elevation features for learning in the training stage, implicitly construct the relationship between DEM data and slope change features, and strengthen the information expression of slope changes in the model, which can improve the accuracy of the detection results.

[0046] like Figure 1 As shown, Figure 1 Schematic diagram of modal slope monitoring provided for the embodiment of the present disclosure. The embodiment of the present disclosure fully considers the current natural disaster monitoring needs for the prevention and control of slope disasters such as landslides, collapses and debris flows, and aims at the defects existing in the slope monitoring process and detection results in the related technologies. In order to improve the prevention and control effect and improve the monitoring efficiency, the imaging mechanism of multimodal remote sensing images is integrated, and a slope multimodal data set is produced by image transformation to improve the quality of the data set.

[0047] In the preparation stage, the data can be preprocessed by collecting LiDAR, InSAR and multispectral data of the same slope object in different phases, and the same phase images of different modal data and different phase images of the same modal data can be spatially and temporally aligned to ensure the consistency of the position of the same ground element in different times and different images. Due to the heterogeneity between different modal data, the embodiment decomposes the multispectral image and the InSAR image into the same frequency domain space, and uses different coefficients to fuse the high-frequency elements and the low-frequency elements to form a complementary trend of the data between the modalities. Finally, the fused image is manually annotated with the changed pixels to realize the construction of a multimodal slope monitoring data set.

[0048] During the training stage, in order to combine the advantages of remote sensing data of different modalities, the simultaneous phase SAR image and multispectral remote sensing image information are mapped to the shared feature space for convolution processing in the input layer through joint feature representation. At the same time, an additional DEM learning branch is introduced in an unsupervised learning manner to intervene in the weight information in the main network, thereby supplementing the change information of the slope in the three-dimensional space in the time dimension, so that the model can fully learn the amplitude and phase information, spectral information and elevation information of the slope image.

[0049] During the monitoring phase, the integrated air-space-ground strategic dispatch satellite is used for all-weather earth observation, and the 6G integrated air-space-ground network technology is used for low-latency and high-speed data transmission. By accessing Internet of Things platforms such as OneNet, the system model is used for real-time monitoring and visualization. The monitoring data can support subsequent risk analysis, risk assessment, early warning issuance, and emergency command capabilities, achieving all-weather, high-precision automatic slope detection.

[0050] Specifically, 1) in the multimodal data processing stage:

[0051] When acquiring DEM data, the laser point cloud data collected by the drone-mounted LiDAR system can be processed through POS solution, coordinate conversion, data segmentation and other steps, including all the original surface point clouds collected by the system, which cannot be directly used for DEM production. It is necessary to filter and classify the point clouds to obtain ground points. The embodiment uses software to automatically classify point clouds. The ground point cloud obtained after classification contains high-precision ground point coordinate information, especially elevation information, which can be directly used for DEM production. Exemplarily, the software built-in point cloud conversion tool can be used to produce DEM, the grid size is 1m, and the interpolation method is the neighborhood mean method.

[0052] In the process of multimodal data registration, in order to ensure the consistency of the positioning expression of the same slope feature in different time dimensions, the images of the same data source need to be registered, and then the changed pixels in the two same modal data can be annotated by manual visual interpretation. However, in the task of producing multimodal remote sensing image datasets, it is necessary not only to align the single modal data in time, but also to align the different modal data in space to provide the consistency conditions required for fusion.

[0053] like Figure 2 and Figure 3 As shown, Figure 2 A schematic diagram of time alignment of multimodal data provided by an embodiment of the present disclosure, Figure 3Schematic diagram of multimodal data spatial alignment provided by the embodiment of the present disclosure. Temporal alignment focuses on the changes in ground objects in long time series, and mainly aligns the same ground object pixels in images of different time periods. Spatial alignment refers to the alignment of elements between images of different sensors or different resolutions in a short period of time, that is, facing the same location, matching remote sensing images from different sensors at the pixel level. The process is divided into the following steps:

[0054] (1) Extract key points: Key points are very prominent points that will not disappear due to factors such as lighting, scale or rotation, such as corner points, edge points, bright spots in dark areas and dark spots in bright areas. This step searches for image locations in all scale spaces. Potential interest points that are scale and rotation invariant are identified through Gaussian derivative functions.

[0055] (2) Locate key points and determine feature orientation: At each candidate location, a fine-tuned model is used to determine the position and scale. Key points are selected based on their stability. Then, one or more directions are assigned to each key point location based on the local gradient direction of the image. All subsequent operations on the image data are transformed relative to the orientation, scale, and position of the key points, thereby providing invariance to these transformations.

[0056] (3) Matching feature points: By comparing the feature vectors of each key point, we can find several pairs of matching feature points and establish the corresponding relationship between the scenes.

[0057] In the embodiment, 2) in the stage of preparing a multi-modal slope change data set:

[0058] The slope change dataset is a dataset composed of multiple remote sensing data collected by different sensors at different times for multiple slopes. Due to different imaging mechanisms, multi-source remote sensing data usually has richer expression information about the ground objects, and can obtain more knowledge during comprehensive analysis, which can effectively break through the bottleneck of traditional single sensor data interpretation, thereby achieving a more comprehensive slope detection task. The relevant technology usually only needs to divide and annotate the data to generate training, verification and label samples, while in multimodal tasks, it is also necessary to fuse different modal data to generate label samples for end-to-end training.

[0059] In the embodiment, the registered data has complete spatial consistency, and different modal data need to be fused. The output synthetic image can often maintain the key feature information in the original modal image, thereby achieving more accurate and comprehensive analysis and judgment of the slope target or surrounding scene, and can improve the effectiveness of the sensor system and the efficiency of information use to a certain extent, thereby improving the resolution of the target to be analyzed and improving the slope monitoring effect. In the embodiment, wavelet transform can be used to fuse different modal label data. Wavelet transform has good localization characteristics in both spatial and frequency domains. It can not only show the structure and texture of the image at different resolution levels, but also has the ability of edge detection, and can better retain the detailed features of the slope edge. Therefore, wavelet transform has a better effect in image fusion.

[0060] After two-dimensional wavelet decomposition of a single remote sensing data, four sub-images HH can be obtained: j , WH j , HW j and WW j ,like Figure 4 As shown, Figure 4 A schematic diagram of wavelet decomposition provided for an embodiment of the present disclosure. Figure 4 In the scale 2 j The low frequency, horizontal high frequency, vertical high frequency and diagonal high frequency sub-images on HH j It can also be decomposed into a low-frequency component and three high-frequency components. When the number of wavelet decomposition layers is N, N low-frequency components and 3N high-frequency components can be obtained. The low-frequency subband concentrates the main information of the image, reflecting the basic contour of the image. The noise and details of the image are mainly distributed in the high-frequency area. The disclosed embodiment proposes that in the high-frequency domain of wavelet decomposition, the coefficients with larger average absolute values ​​of the image neighborhood are selected as important coefficients of the fused wavelet to retain the high-frequency detail information; in the low-frequency domain of wavelet decomposition, the low-frequency coefficients of the multi-source images are weighted averaged as the fused wavelet approximation coefficients to reduce the noise of the image, and then they are fused separately and finally inversely transformed to the image matrix. The formulas are shown in (1) and (2) below. Accurate change label images can be generated by manual annotation. Specifically, it can be combined with Figure 5 As shown, Figure 5 A schematic diagram of producing a multi-modal slope variation dataset provided in an embodiment of the present disclosure.

[0061]

[0062] Among them, W(j, k) is the discrete wavelet coefficient, x(t) is the result of inverse wavelet transform, is the wavelet basis function, j is the decomposition series, W j.k is the j-th level, k-th wavelet coefficient.

[0063] In the embodiment, the LiDAR point cloud data can be processed by software to generate DEM elevation data, the algorithm is used to spatially align the multimodal data set, the wavelet transformation method is used to extract the high and low frequency data in the multispectral image and SAR image data, and the data is fused according to the frequency domain characteristics combined with different wavelet coefficients, and finally the fused image is annotated with the changed pixels to generate a multimodal slope change data set with complete data and rich features.

[0064] Deep learning image data enhancement methods increase the diversity of image data by using various transformations or generative models, such as random cropping, rotation or scaling, thereby improving the generalization ability of the model. At the same time, increasing the sample size of the data set can also improve the detection accuracy to a certain extent. Multimodal data enhancement requires image processing based on various factors such as data imaging principles, sensors or environmental influences to improve the robustness of model learning. However, since spectral image imaging is often affected by the weather environment and the sensor itself, Gaussian noise and salt and pepper noise are generated, while InSAR images are affected by the echo of radar scattering and are prone to speckle noise.

[0065] However, in order to ensure the quality of the data set, it is usually necessary to perform radiation correction and geometric correction, image finishing, projection transformation and other processing on the collected images, and the noise generated by various images in the original data is often lost in this process. Therefore, the embodiment of the present disclosure can use random Gaussian noise to process the input multispectral image when the model is input, and use the influence of spot noise on SAR, combined with random rotation and other enhancement methods to process the input image to adapt to the real monitoring scene. In this way, by processing the input image after adding noise processing, the image collected in the real environment can be better simulated, so that the trained model has robustness.

[0066] 3) Constraint-based multimodal neural network model training phase:

[0067] In the process of learning the representation of modalities using CNN, the semantic information contained in different modal data is mainly abstracted into real-valued vectors, and the modalities are converted into vector representations, which is conducive to computer recognition processing and downstream task analysis. First, in the input layer, the mixed input branches of hyperspectral images and SAR images, as well as the DEM data input branch are constructed. Since slope monitoring belongs to a type of image change detection, it is necessary to input images of different phases for change difference comparison. At the same time, considering that multimodal data have consistency in spatial and temporal dimensions for the same monitoring target, the embodiment of the present disclosure proposes to splice the multimodal data in the channel dimension in the model input layer to establish a connection between corresponding elements. Therefore, the input layer requires a total of six branches for data input, such as Figure 6 As shown, Figure 6A schematic diagram of the model training process provided for an embodiment of the present disclosure.

[0068] In the mixed data learning branch: In machine vision tasks, in order to ensure that the acquired features have global semantic features, multiple convolution modules need to be stacked to achieve deep feature extraction. However, model depth often brings negative effects such as overfitting and model degradation. The embodiment uses the cascade-FPN dense residual pyramid structure to extract the input mixed features. Its structure is as follows Figure 7 As shown, Figure 7 Schematic diagram of the convolutional neural network structure provided for the embodiment of the present disclosure. The multimodal features output from the ResNet34 backbone network are fed into the Cascade-FPN module to extract richer deep features. Its main structure includes 4 progressively decreasing FPN modules with the number of layers being (4, 3, 2, 1 layers) respectively. In different FPN structures, weight sharing is used for feature extraction at the same level, which can reduce model parameters, increase model inference speed, and improve the consistency of feature extraction at the same spatial position. The top-down connection transfers the high-level semantics of deep features to the shallow layer, significantly improving the recognition ability of shallow features, thereby alleviating the semantic gap between the encoder and decoder feature maps.

[0069] The 512-dimensional features output by the top-level features are reduced in dimension, and two 3×3 convolution kernels are used to form a residual convolution module to achieve progressive dimensionality reduction. The features with a channel size of 256 are unified in channel dimension with the features output by the next layer of backbone network, and the bilinear interpolation method is used to unify them in the spatial dimension. They are aggregated in the channel dimension to generate a fused feature with a channel size of 512. Two consecutive ensemble residual convolution modules are used to increase the channel size from 512→256→128 dimensions to eliminate the aliasing effect caused by the ensemble operation. The above steps are repeated to process the outputs of the 3rd, 2nd, and 1st layer backbone networks, and finally the feature x1d output by the first-level FPN structure is obtained. Similarly, the top-level features are reduced in dimension from 512 to 256 dimensions using the residual convolution module and the feature size is unified. Next, they are horizontally connected with the intermediate features with a channel size of 256 dimensions output by the third-layer features in the previous level FPN, and the convolution, upsampling and other operations are repeated. The feature maps in different FPN modules are clustered to avoid the possible attenuation of semantic information when the deep features are upsampled layer by layer. At the same time, through the horizontal connection, it is ensured that all features construct high-level semantic feature maps with accurate position representation at the same scale, thereby facilitating the detection of objects and object details of different scales, and also improving the network's detection performance for pseudo changes. The feature of the i-th layer and j-th column can be expressed as the following formula, where Conc represents the superposition operation in the channel domain, Up is bilinear upsampling, and Cov(3×3) represents the convolution operation with a convolution kernel size of 3×3. The formula is shown in (3).

[0070] x i,j =Conc[Up(Cov 3×3 (x i-1，i-1 )), Cov 3×3 (x i，i-1 )] (3)

[0071] Among them, x i，j represents the feature of the jth column of the i-th layer, Up represents the upsampling operation, and Cov represents the convolution operation.

[0072] DEM data learning branch: Since DEM data is a direct mapping of local terrain and landforms in image space, its structure is simple, and the imaging content is single, only containing spatial information such as slope, aspect or water system, but not spectral or texture information. Its image representation is relatively continuous, and each element is strongly correlated with the global elements. The embodiment uses an autoencoder structure to extract elevation change information, captures the representation of elements in the DEM image in its coordinate space, encodes long-distance dependencies between feature elements, and constructs global spatial information constraints to obtain more robust and precise location information and detect spatial relationships between targets. Combined with the idea of ​​decision-level feature fusion, it is mapped to the same spatial dimension as the fused feature to fuse corresponding elements to introduce spatial elevation information.

[0073] Since slope detection requires judging the changed pixels in images of different phases, it can be classified as a binary classification task. Therefore, the present invention uses a binary cross entropy loss function for the proposed monitoring network, as shown in the following formula (4).

[0074]

[0075] Among them, loss is the loss function, N is the number of samples, y i is the true label of the i-th sample, p i is the predicted probability of the i-th sample. C is an integer between 1 and N.

[0076] 4) Deep learning slope monitoring stage of multimodal data fusion:

[0077] During the system operation phase, the sensor can first be used to obtain real-time multispectral images and InSAR images of the ground slope target to be acquired, and the SIFT algorithm can be used to align the two data at each moment in space and size, and the multimodal data acquired at the initial moment is input into the above-mentioned trained module, and the subsequent moment data is continuously input into the above-mentioned trained module. The multi-dimensional ground feature features such as spectrum and amplitude in the mixed multimodal data are used inside the model, and the weight of the elevation feature is introduced into the deep feature for constraint. At the end, the slope change image is continuously output in an end-to-end manner, and the change between the subsequent input image and the previous image is compared to achieve real-time slope monitoring, such as Figure 8 As shown, Figure 8 A schematic diagram of the process of automatic slope monitoring provided in an embodiment of the present disclosure.

[0078] By monitoring the target slope, the embodiment can more realistically restore the expression of slope changes in the feature space. By integrating the algorithm to match and align the modal data before training, the learning cost of the model for positioning features is reduced, and multimodal data enhancement is used to add noise to the original data. This can make the model more in line with actual monitoring requirements and improve the efficiency of slope monitoring.

[0079] In summary, the related existing technologies need to use GNSS-RTK, total station, level measurement or crack meter and other instruments to collect data, and monitoring personnel need to regularly perform monitoring point measurement operations, which have high accuracy but long operation cycles, high work intensity, poor real-time and timeliness, and it is difficult to achieve large-scale real-time monitoring of slopes. The disclosed embodiment adopts a remote sensing non-contact monitoring mode, and uses sensors to obtain large-area surface slope data in real time, which can be combined with machine vision to generate long-term series of slope change images.

[0080] In addition, the related technology uses a single-mode remote sensing data to extract slope deviation change curves, displacement deviation change curves, etc. in images at different times by using algorithms, or uses machine learning methods for analysis. The disclosed embodiment produces a multi-modal slope change data set for deep learning model training. The trained model can output end-to-end monitoring results without multiple algorithm processing. At the same time, the characteristics of slope deformation in each modal image space are fully considered during the training process, and image enhancement methods are used to add noise to the original data to adapt to the actual monitoring scene.

[0081] In addition, related technical solutions require the use of multiple modal data, and use algorithms to extract different modal features from multimodal data to obtain comprehensive results. The disclosed embodiment uses a wavelet algorithm to fuse high-frequency and low-frequency data of multimodal data during the training phase, retaining the rich information in the original data. At the same time, during the detection process, multimodal data is superimposed and input for detection, without the need to learn a single modal feature separately.

[0082] Based on the above embodiments, the present disclosure also provides a slope monitoring method based on multimodal data, such as Fig. 9 As shown, the method may include the following steps:

[0083] In step S910, monitoring data of the target slope is obtained.

[0084] The monitoring data includes interferometric radar data and multispectral data obtained from monitoring the target slope at different times. The interferometric radar data may be specifically an InSAR image, and the multispectral data may be specifically a multispectral image.

[0085] In the embodiment, when the target slope needs to be monitored, the InSAR image and multispectral image of the target slope can be acquired in real time, so that the InSAR image and multispectral image of the target slope with a time series can be obtained. The target slope can be a slope with a certain area, and the embodiment is not limited thereto.

[0086] In step S920, the monitoring data is input into the trained preset model to obtain the monitoring results.

[0087] Among them, the monitoring data characterizes the changes of the target slope at different times. The preset model is trained by training the preset model through training samples. The training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

[0088] In an embodiment, the preset model can be a CNN model. In the model training stage, the DEM data of the slope is added to the training samples, and the interferometric radar data and multispectral data of the slope are used as training samples to train the model. During model training, the weight information in the backbone network of the model can be intervened by introducing the DEM data in the slope, so that the model can learn the change information of the slope in the three-dimensional space in the time dimension, and then the monitoring results obtained by the trained model can more accurately reflect the change of the slope.

[0089] The slope monitoring method based on multimodal data provided by the embodiment of the present disclosure can obtain monitoring data including interferometric radar data and multispectral data obtained at different times from monitoring the target slope, and input the monitoring data into the trained preset model to obtain monitoring results. Since the training samples include digital elevation data, interferometric radar data and multispectral data, the model can learn the change information of the slope in the three-dimensional space in the time dimension, and learn the interferometric radar data and multispectral data in the slope more fully, so that the monitoring results obtained can more accurately reflect the change of the slope, thereby improving the monitoring efficiency of the slope.

[0090] Based on the above embodiment, in another embodiment provided by the present disclosure, the method may further include the following steps:

[0091] In step S930, multimodal data of a plurality of preset slopes at different times are obtained.

[0092] Among them, multimodal data includes digital elevation data, interferometric radar data and multispectral data.

[0093] In an embodiment, the multiple preset slopes may include a target slope, and digital elevation data, interferometric radar data, and multispectral data collected at different times for the multiple preset slopes may be obtained. The digital elevation data, interferometric radar data, and multispectral data of the same preset slope are aligned in time and space to obtain multimodal data of the multiple preset slopes at different times.

[0094] In the embodiment, time alignment mainly focuses on the changes of ground objects in long time series, and aligns the same ground object pixels in images of different time. Spatial alignment refers to the alignment of elements between images of different sensors or different resolutions in a short time, that is, facing the same location, matching remote sensing images from different sensors at the pixel level. For details about the methods of time alignment and space alignment, please refer to the description of the above embodiment, which will not be repeated here.

[0095] In step S940, the interferometric radar data and the multispectral data are fused to obtain fused data, and the fused data and the digital elevation data are used as training samples.

[0096] In an embodiment, the multispectral data can be processed by random Gaussian noise, and the aperture radar data can be processed by spot noise, and the processed multispectral data and the processed aperture radar data can be fused. In this way, by combining the principle of multimodal data imaging and using the noise algorithm to restore the algorithm's neutralization of noise during data processing, the model can adapt to the multi-noise interference in the real monitoring environment, thereby improving the robustness of the monitoring method.

[0097] In an embodiment, when the preset model is trained by training samples to obtain a trained preset model, the preset model can be trained by digital elevation data at different times and fused data at different times. The preset model includes an input layer, a feature extraction layer, and an output layer; the input layer is used to obtain digital elevation data at different times and fused data at different times; the feature extraction layer is used to extract the elevation data features of the digital elevation data and the depth features of the fused data, and superimpose the elevation data features and the depth features of the fused data in the channel dimension to obtain multimodal depth features; the output layer is used to output slope change information between different times based on the multimodal depth features, and obtain a label data set based on logistic regression.

[0098] In the embodiment, by training the DEM constraint model, and taking into account the simple features of DEM data and the continuous image representation, the spatial features in the DEM data are extracted using the autoencoder structure, and the main branch of the model is intervened in combination with the decision-level fusion strategy to introduce the change information of the slope in the vertical space, which can improve the accuracy and authenticity of the monitoring results.

[0099] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S920 may specifically include the following steps:

[0100] In step S921, the interferometric radar data and the multispectral data obtained at the same time are spatially aligned to obtain sequence image data composed at different times.

[0101] In step S922, the sequence image data is input into the trained preset model, and the monitoring result with continuously changing images is output.

[0102] In the embodiment, by monitoring the target slope and obtaining the interferometric radar data and multispectral data at different times, the sensing radar data and multispectral data obtained at the same time are spatially aligned to obtain sequence image data composed of different times. In this way, the sequence image data is input into the preset model, and the image at the next moment in the sequence image can be compared with the image at the previous moment to obtain a monitoring result with a continuously changing image. In this way, the displacement and other changes of the target slope surface can be continuously reflected through the monitoring results, so that effective measures can be taken, such as early warning or other effective measures can be taken to avoid property losses or casualties caused by landslides.

[0103] In the embodiment, satellite resources can be dispatched through the integrated air-space-ground technology to obtain InSAR images and optical images in the same posture in real time for transmission, input the trained model and continuously output the continuous changes of the slope, so as to form all-weather real-time slope monitoring.

[0104] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides a slope monitoring device based on multimodal data, which may be a server, a terminal or a chip applied to a server. Fig.10 A schematic block diagram of the functional modules of a slope monitoring device based on multimodal data provided by an exemplary embodiment of the present disclosure. Fig.10 As shown, the slope monitoring device based on multimodal data includes:

[0105] A data acquisition module 10 is used to acquire monitoring data of a target slope, wherein the monitoring data includes interferometric radar data and multispectral data obtained at different times from monitoring the target slope;

[0106] The monitoring result acquisition module 20 is used to input the monitoring data into the trained preset model to obtain the monitoring results; wherein the monitoring results characterize the changes of the target slope at different times, and the trained preset model is obtained by training the preset model through training samples, and the training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

[0107] Based on the above embodiment, in another embodiment provided by the present disclosure, the apparatus further includes a training sample acquisition module, and the training sample acquisition module is specifically used to:

[0108] Acquire multimodal data of a plurality of preset slopes at different times; the multimodal data includes digital elevation data, interferometric radar data and multispectral data;

[0109] The interferometric radar data and the multispectral data are fused to obtain fused data, and the fused data and the digital elevation data are used as the training samples.

[0110] In another embodiment provided by the present disclosure, the training sample acquisition module is specifically used to:

[0111] Acquire digital elevation data, interferometric radar data and multispectral data collected at different times for the plurality of preset slopes;

[0112] The digital elevation data, interferometric radar data and multispectral data of the same preset slope are time-aligned and spatially aligned to obtain multimodal data of multiple preset slopes at different times.

[0113] In another embodiment provided by the present disclosure, the training sample acquisition module is specifically used to:

[0114] Processing the multispectral data by random Gaussian noise and processing the interferometric radar data by spot noise;

[0115] The processed multispectral data and the processed interferometric radar data are fused.

[0116] In another embodiment provided by the present disclosure, the device further includes:

[0117] A training module is used to train the preset model through the digital elevation data at different times and the fused data at different times; wherein the preset model includes an input layer, a feature extraction layer and an output layer; the input layer is used to obtain the digital elevation data at different times and the fused data at different times; the feature extraction layer is used to extract the elevation data features of the digital elevation data and the depth features of the fused data, and superimpose the elevation data features and the depth features of the fused data in the channel dimension to obtain multimodal depth features; the output layer is used to output the slope change information between different times based on the multimodal depth features, and obtain a label data set based on logistic regression.

[0118] In another embodiment provided by the present disclosure, the monitoring result acquisition module is further configured to:

[0119] The interferometric radar data and multispectral data obtained at the same time are spatially aligned to obtain sequence image data composed of different time periods;

[0120] The sequence image data is input into a trained preset model, and a monitoring result with a continuously changing image is output.

[0121] The slope monitoring device based on multimodal data provided by the embodiment of the present disclosure can obtain monitoring data including interferometric radar data and multispectral data obtained at different times from monitoring the target slope, and input the monitoring data into the trained preset model to obtain monitoring results. Since the training samples include digital elevation data, interferometric radar data and multispectral data, the model can learn the change information of the slope in the three-dimensional space in the time dimension, and learn the interferometric radar data and multispectral data in the slope more fully, so that the monitoring results obtained can more accurately reflect the change of the slope, thereby improving the monitoring efficiency of the slope.

[0122] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0123] Fig.11 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Fig.11 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0124] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0125] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Fig.12 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Fig.12 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0126] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Fig.12As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0128] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0129] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0130] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0131] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0133] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0134] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0139] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A slope monitoring method based on multimodal data, characterized in that: The method comprises: Acquiring monitoring data of a target slope, wherein the monitoring data includes interferometric radar data and multispectral data obtained from monitoring the target slope at different times; The monitoring data is input into a trained preset model to obtain a monitoring result; wherein the monitoring result characterizes the change of the target slope at different times, and the trained preset model is obtained by training the preset model through training samples, and the training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

2. The method according to claim 1, characterized in that The method further comprises: Acquire multimodal data of a plurality of preset slopes at different times; the multimodal data includes digital elevation data, interferometric radar data and multispectral data; The interferometric radar data and the multispectral data are fused to obtain fused data, and the fused data and the digital elevation data are used as the training samples.

3. The method according to claim 2, characterized in that The obtaining of multimodal data of a plurality of preset slopes at different times includes: Acquire digital elevation data, interferometric radar data and multispectral data collected at different times for the plurality of preset slopes; The digital elevation data, interferometric radar data and multispectral data of the same preset slope are time-aligned and space-aligned to obtain multimodal data of multiple preset slopes at different times.

4. The method according to claim 2, characterized in that: The fusing of the interferometric radar data and the multispectral data comprises: Processing the multispectral data by random Gaussian noise and processing the interferometric radar data by spot noise; The processed multispectral data and the processed interferometric radar data are fused.

5. The method according to claim 2, characterized in that: The step of training the preset model by using the training samples to obtain the trained preset model includes: The preset model is trained by the digital elevation data at different times and the fused data at different times; wherein the preset model includes an input layer, a feature extraction layer and an output layer; the input layer is used to obtain the digital elevation data at different times and the fused data at different times; the feature extraction layer is used to extract the elevation data features of the digital elevation data and the depth features of the fused data, and superimpose the elevation data features and the depth features of the fused data in the channel dimension to obtain multimodal depth features; the output layer is used to output the slope change information between different times based on the multimodal depth features, and obtain a label data set based on logistic regression.

6. The method according to claim 1, characterized in that The step of inputting the monitoring data into a trained preset model to obtain monitoring results includes: The interferometric radar data and multispectral data obtained at the same time are spatially aligned to obtain sequence image data composed of different time periods; The sequence image data is input into a trained preset model, and a monitoring result with a continuously changing image is output.

7. A slope monitoring device based on multimodal data, characterized in that: The device comprises: A data acquisition module, used to acquire monitoring data of a target slope, wherein the monitoring data includes interferometric radar data and multispectral data obtained from monitoring the target slope at different times; A monitoring result acquisition module is used to input the monitoring data into a trained preset model to obtain monitoring results; wherein the monitoring results characterize the changes of the target slope at different times, and the trained preset model is obtained by training the preset model through training samples, and the training samples are obtained based on digital elevation data, interferometric radar data and multispectral data.

8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images