Slope rock mass structure plane multi-modal deep learning identification method and system under complex terrain and medium

Abstract

The application discloses a complex terrain under slope rock mass structure surface multi-mode depth learning identification method and system and a medium, relates to the rock-soil engineering monitoring and geological survey technical field; based on the original point cloud data and the hyperspectral image data constructs the fusion feature containing the geometric feature and the spectral feature, the data missing in the original point cloud data is supplemented through the fusion feature, and the rock surface point cloud data is segmented in combination with the fusion feature, the structure surface clustering identification is carried out based on the rock surface point cloud data, and the non-rock surface point cloud data is eliminated; the geometric feature is sensitive to spatial structure, the spectral feature can identify interference such as vegetation, and the fusion feature of the two makes the network infer the missing area with the aid of spectral information when the point cloud is insufficiently illuminated; when the vegetation is covered, the rock mass and the vegetation are distinguished through the spectral feature, and the false closure is inhibited; only the pure rock surface point cloud is clustered, the interference of the non-rock surface point is avoided, and the accuracy and stability of the structure surface occurrence identification are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of geotechnical engineering monitoring and geological exploration technology, specifically to a multimodal deep learning identification method, system, and medium for slope rock mass structure surfaces under complex terrain. Background Technology

[0002] In the evaluation of stability of steep slopes, prevention and control of geological disasters, and geotechnical engineering construction such as hydropower and transportation, accurate acquisition of geometric information such as the attitude, spacing, and extension length of rock mass structural surfaces (e.g., fissures, bedding planes, joints) is a crucial prerequisite for rock mass quality classification and stability calculation. In recent years, with the development of point cloud acquisition technologies such as 3D laser scanning and UAV oblique photography, effective means have been provided for non-contact, high-precision, and high-density acquisition of surface geometric information of slope rock masses.

[0003] However, in practical engineering applications, especially in areas with complex geological environments, existing rock mass structure surface identification technologies based on point cloud data face severe technical bottlenecks, mainly in the following two aspects:

[0004] First, there's the issue of missing (void) point cloud data due to changes in lighting conditions: In field data collection on steep slopes, especially in deeply incised canyons running north-south or with a specific orientation, the apparent motion of the sun causes drastic changes in lighting conditions on the rock mass surfaces throughout the day. When sunlight shines obliquely, the protruding parts of steep terrain cast large, moving shadows on adjacent rock mass surfaces. Due to insufficient light, these shadowed areas lead to failure in feature point matching during photogrammetry or weak lidar echo signals, ultimately forming large, data-free voids in the collected point cloud data. Since these voids are located precisely on critical rock mass surfaces, subsequent point cloud density-based surface identification algorithms cannot fit the correct plane in these areas, resulting in the permanent loss of crucial geological information.

[0005] Secondly, there is the issue of pseudo-closure caused by vegetation cover and noise interference: After the implementation of vegetation concrete greening projects or in areas with abundant natural vegetation, slopes are often covered by vegetation (such as herbs and shrubs). During point cloud acquisition, laser points or image points often preferentially hit the vegetation leaves located in front of the rock surface, generating a large number of noise points representing vegetation. When the vegetation is dense, these noise points can completely obscure the real rock surface points behind them, both visually and spatially, forming a continuous pseudo-surface composed of vegetation points in the point cloud model. Although existing denoising algorithms can remove some outlier noise, they are difficult to effectively separate vegetation points growing close to the rock surface from the real rock surface points. When identifying structural surfaces, the algorithm often incorrectly identifies this pseudo-closed surface composed of vegetation as a rock mass structural surface, thus calculating incorrect attitude information (such as misjudging a steeply dipping structural surface as a gently dipping one), or completely ignoring the covered real structural surfaces, leading to misjudgments of the integrity and stability of the rock mass.

[0006] In summary, existing technologies struggle to reliably and accurately extract true rock mass structural information when faced with the coupled effects of complex lighting variations (leading to voids in structural surfaces) and vegetation cover (causing false closure of structural surfaces) on steep slopes. This hinders the effective application of technologies such as 3D laser scanning in geotechnical engineering investigations in extreme environments. Therefore, a new method is needed to overcome these shortcomings and accurately identify rock mass structural surfaces even under conditions of data loss and strong noise interference. Summary of the Invention

[0007] To address the challenge of existing technologies failing to stably and accurately extract true rock mass structural surface information under the coupled effects of complex lighting variations (leading to voids in structural surfaces) and vegetation cover (causing false closure of structural surfaces) on steep slopes, this invention aims to provide a multimodal deep learning method, system, and medium for identifying rock mass structural surfaces on slopes in complex terrain. Based on original point cloud data and hyperspectral image data, a fusion feature system incorporating both geometric and spectral features is constructed. This fusion feature system fills in data gaps in the original point cloud data to obtain dense point cloud data. Semantic segmentation is then performed on the dense point cloud data to determine the rock surface point cloud data. Structural surface clustering is then performed based on the rock surface point cloud data, eliminating non-rock surface point cloud data. This effectively completes data acquisition and monitoring control. This invention, based on a multimodal deep neural network, simultaneously achieves point cloud void completion and vegetation point removal, solving the problems of lighting and shadow effects and false closure caused by vegetation cover, while also avoiding error accumulation in traditional multi-step processing.

[0008] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0009] This solution provides a multimodal deep learning method for recognizing slope rock mass structural surfaces under complex terrain. The method includes:

[0010] Multimodal data of the target slope area are collected synchronously, and the multimodal data is preprocessed; the multimodal data includes raw point cloud data and hyperspectral image data;

[0011] Preprocessed multimodal data is input into a multimodal deep neural network to construct a rock surface point cloud, and structural surface clustering is performed on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering on the rock surface point cloud data based on the fused features.

[0012] Output the structural surface clustering identification results.

[0013] A further optimized solution is that the preprocessing method includes:

[0014] First, the original point cloud data is subjected to outlier filtering and point cloud normalization. Then, the hyperspectral image data is subjected to radiometric calibration, atmospheric correction, and spectral normalization.

[0015] Then, each pixel of the hyperspectral image data is back-projected into three-dimensional space and matched point by point with the original point cloud data so that each point cloud has corresponding spectral information. For point clouds without corresponding spectral information, spectral values ​​are assigned based on nearest neighbor interpolation or common-view geometric relationships.

[0016] A further optimized scheme is that the feature extraction network includes: a geometric feature extraction branch, a spectral feature extraction branch, and a feature fusion network;

[0017] The geometric feature extraction branch takes the original point cloud data as input and uses the improved PointNet++ network as the backbone network to extract the geometric features of each point cloud.

[0018] The spectral feature extraction branch consists of multiple cascaded dilated convolutional layers, with the porosity of the dilated convolutional layers increasing sequentially, and residual connections introduced into each dilated convolutional layer.

[0019] The feature fusion network includes: a dual-modal channel separation component, a dual-branch spatial attention component, and a feature fusion component;

[0020] The dual-modal channel separation component performs channel dimension separation processing on geometric features and spectral features respectively to obtain important channel features and secondary important fused features;

[0021] The dual-branch spatial attention component enhances spatial attention for important channel features and performs global calibration of channel dimensions for less important channel features.

[0022] The feature fusion component performs weighted fusion of the spatial attention-enhanced features and the globally calibrated features to obtain fused features.

[0023] A further optimization scheme is that the improved PointNet++ network includes an ensemble abstraction layer and a feature propagation layer; the ensemble abstraction layer is composed of multi-level and multi-scale ensemble abstraction modules;

[0024] The collection abstraction module performs the following process:

[0025] T1, select N central sampling points from the original point cloud data, and the central sampling points uniformly cover the rock surface;

[0026] T2, set n search radii; with each central sampling point as the center of the sphere, within a search radius r b The neighborhood points are searched to construct a set of neighborhood points with different search radii; where b = 1, 2, ..., n;

[0027] T3 introduces a normal vector-guided attention weight mechanism on top of depthwise convolution to extract features from the point sets of each domain, obtaining the local aggregated features of each central sampling point under different search radii; under any search radius, the local aggregated features of the central sampling point i are:

[0028] ;

[0029] ;

[0030] Where N(i) represents the set of neighborhood points of the central sampling point i; This represents the relative coordinates between the central sampling point i and its neighboring point j; Let represent the normal vector of the neighboring point j; This represents the original feature vector of the neighboring point j; This represents the contribution weight of neighboring point j to the feature update of the central sampling point i; This represents the consistency coefficient of the normal vectors between the central sampling point i and its neighboring point j. The greater the difference in the normal vectors between the central sampling point i and its neighboring point j, the more consistent the normal vectors become. The closer it is to 1, the less likely it is to be true. The closer it is to 0; MLP() represents a multilayer perceptron; This represents the basic weights calculated based on the spatial distance and relative position between the central sampling point i and its neighboring points j;

[0031] The feature propagation layer splices and fuses local aggregated features under different search radii, and then upsamples them based on a distance-weighted interpolation strategy to obtain the geometric features of each point cloud.

[0032] A further optimized solution is that the method for acquiring the dense point cloud data includes:

[0033] The original point cloud data P is processed by sparse convolutional layers. in Multi-scale semantic features are extracted by downsampling and fusion features. The multi-scale semantic features are then normalized and ReLU activated to obtain sparse point cloud data P. en and feature map F en ;

[0034] Based on the feature map F en Extract global enhancement features F enc Based on the sparse point cloud data P en Extract local neighborhood features F local , fusing the global enhancement feature F enc and local neighborhood features F local Obtain multi-scale extended features F exp ;

[0035] The sparse point cloud data P is processed by a sparse transposed convolutional layer symmetrical to the sparse convolutional layer. en and multi-scale extended features F exp Upsampling is performed, and the features of the sparse convolutional layer are skipped to the features of the corresponding sparse transposed convolutional layer to predict the coordinate offset ΔP of each upsampled point. The coordinate offset ΔP is then superimposed with the coordinates of each upsampled point cloud to obtain the generated point cloud data P. new ;

[0036] The original point cloud data P in With the generated point cloud data P new The data is merged to obtain dense point cloud data.

[0037] A further optimized solution is that the method for determining the rock surface point cloud and non-rock surface point clouds includes:

[0038] Align the dense point cloud data with the fused features, concatenate the coordinates of each point cloud with the fused features along the channel dimension, and linearly map the concatenated features to a low dimension based on a linear layer to obtain the original logical values ​​of each point cloud. Apply a Softmax function to the original logical values ​​to calculate the first probability p of each point cloud belonging to the rock surface. rock And the second probability that it does not belong to the rock surface;

[0039] Preset probability threshold p e In dense point cloud data, if the first probability p of the current point cloud is... rock Greater than the probability threshold p e If the condition is met, the current point cloud is determined to be a rock surface point cloud; otherwise, the current point cloud is determined to be a non-rock surface point cloud.

[0040] A further optimized solution is that the method for identifying structural surface clustering includes:

[0041] The fusion features of each rock surface point cloud are compared with the first probability p. rock Multiplication yields the weighted feature F of the rock surface point cloud. rock ;

[0042] Construct a graph attention convolutional network, and combine it with the weighted features F of the rock surface point cloud. rock Extract the graph convolution features e of the point clouds on each rock surface;

[0043] The slope rock mass structure surface is obtained by clustering the graph convolution feature e based on the differentiable mean drift clustering method.

[0044] A further optimization scheme is that the differentiable mean-shift clustering method includes:

[0045] G31 uses the graph convolution features e of all rock surface point clouds as candidate cluster centers C={e1, e2, ..., e3}. j , ..., e N}; N represents the total number of point clouds on the rock surface;

[0046] G32, for the current cluster candidate center c∈C, calculate the drift vector m(c):

[0047] ;

[0048] Where K() represents the kernel function, which is used to calculate the contribution weight of each point to the current cluster candidate center; h represents the bandwidth of the kernel function;

[0049] G33, update the current cluster candidate center c=c+m(c), repeat steps G32 and G33 until convergence;

[0050] G34 merges cluster candidate centers whose distance is less than the distance threshold τ to obtain a cluster center set;

[0051] G35 assigns each rock surface point cloud to the nearest cluster center to obtain the slope rock mass structure surface identification result.

[0052] This solution also provides a multimodal deep learning recognition system for slope rock mass structure surfaces under complex terrain, used to implement the aforementioned multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain. The system includes:

[0053] The acquisition module is used to simultaneously acquire multimodal data of the target slope area; the multimodal data includes raw point cloud data and hyperspectral image data.

[0054] The preprocessing module is used to preprocess the multimodal data;

[0055] A recognition module is used to input preprocessed multimodal data into a multimodal deep neural network to construct a rock surface point cloud, and to perform structural surface clustering recognition on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data, and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering recognition on the rock surface point cloud data based on the fused features.

[0056] The output module is used to output the structural surface clustering recognition results.

[0057] This solution also provides a computer-readable medium storing a computer program that, when executed by a processor, can implement the multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain as described above.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. This invention provides a multimodal deep learning method, system, and medium for identifying rock mass structural surfaces on slopes in complex terrain. It constructs a fusion feature system containing geometric and spectral features based on original point cloud data and hyperspectral image data. The fusion feature system fills in data gaps in the original point cloud data to obtain dense point cloud data. Semantic segmentation is then performed on the dense point cloud data to determine the rock surface point cloud data. Structural surface clustering is performed based on the rock surface point cloud data, eliminating non-rock surface point cloud data. Geometric features are sensitive to spatial structure, while spectral features can identify interference such as vegetation. The fusion of these two features allows the network to infer missing areas using spectral information when insufficient light causes point cloud voids. In vegetated areas, spectral features can distinguish between rock mass and vegetation, thus suppressing false closures. Clustering based on the fusion feature system and pure rock surface point clouds avoids interference from non-rock surface points, significantly improving the accuracy and stability of structural surface attitude (dip, dip angle) identification.

[0060] 2. This invention provides a multimodal deep learning method, system, and medium for recognizing the structural surfaces of rock masses on slopes under complex terrain. It adaptively fuses the geometric features and hyperspectral features of the point cloud of the target slope area at the bottom layer of the deep network, making full use of spectral information to identify vegetation and using geometric information to characterize the structural surfaces, thereby improving the recognition accuracy of the multimodal deep neural network. The recognition accuracy far exceeds that of single-modal recognition methods.

[0061] 3. This invention provides a multimodal deep learning method, system, and medium for recognizing slope rock mass structural surfaces under complex terrain. The multi-task decoding network achieves joint optimization of point cloud completion, semantic segmentation, and structural surface clustering by sharing a feature extraction layer. Each task promotes the others: the completion task provides complete data for segmentation, and the segmentation task provides clean rock surfaces for clustering. Ultimately, this makes the multimodal deep neural network more adaptable to complex lighting and vegetation, and the recognition results more reliable.

[0062] 4. This invention provides a multimodal deep learning recognition method, system and medium for slope rock mass structure surface under complex terrain. Based on the same multimodal deep neural network, this invention simultaneously realizes point cloud cavity completion and vegetation point removal, solves the problems of light and shadow and false closure caused by cover, and avoids the error accumulation in traditional multi-step processing. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be considered as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort. In the drawings:

[0064] Figure 1 A schematic diagram of the multimodal deep learning method for recognizing the rock mass structure of slopes in complex terrain;

[0065] Figure 2 This is a schematic diagram illustrating the principle of a multimodal deep neural network.

[0066] Figure 3 This is a schematic diagram of the feature extraction network architecture;

[0067] Figure 4 This is a schematic diagram of a multimodal deep learning recognition system for slope rock mass structure surfaces under complex terrain. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments and accompanying drawings. The illustrative embodiments and descriptions of the present invention are only used to explain the present invention and are not intended to limit the present invention.

[0069] Existing technical solutions struggle to stably and accurately extract real rock mass structural information when faced with the coupled effects of complex lighting changes (leading to voids in structural surfaces) and vegetation cover (leading to false closure of structural surfaces) on steep slopes. This severely restricts the effective application of technologies such as three-dimensional laser scanning in geotechnical engineering investigations in extreme environments. The present invention provides the following embodiments to solve the above-mentioned technical problems.

[0070] Example 1

[0071] This embodiment provides a multimodal deep learning method for recognizing slope rock mass structural surfaces under complex terrain, such as... Figure 1 As shown, the method includes:

[0072] Step 1: Synchronously acquire multimodal data of the target slope area and preprocess the multimodal data; the multimodal data includes raw point cloud data and hyperspectral image data;

[0073] In this embodiment, the hyperspectral image data includes reflectance information for each pixel in a continuous spectral band.

[0074] In step one, the preprocessing method includes:

[0075] S11, first perform outlier filtering and point cloud normalization on the raw point cloud data, and then perform radiometric calibration, atmospheric correction and spectral normalization on the hyperspectral image data.

[0076] The purpose of outlier filtering is to remove isolated noise points caused by multiple echoes or atmospheric scattering. Specifically, statistical filtering methods can be used to calculate the average distance between each point and its k neighboring points. If the distance exceeds a certain standard deviation of the global mean, it is identified as an outlier and removed.

[0077] The purpose of point cloud normalization is to eliminate the influence of scale differences between different survey areas on the model input. Specifically, the point cloud coordinates can be centered and translated so that they fall within the interval [0, 1] or [−1, 1]. The point cloud normalization parameters are recorded so that the recognition results can be mapped back to the original coordinate system later.

[0078] Radiometric calibration and atmospheric correction can convert the raw digital quantization (DN) values ​​of hyperspectral image data into surface reflectance, eliminating the influence of illumination conditions and atmospheric attenuation on the spectral curve. Specifically, radiometric calibration can be performed using a standard whiteboard measured before acquisition, and atmospheric correction models (such as the FLAASH model or QUAC model) can be used to eliminate the effects of atmospheric absorption and scattering.

[0079] Spectral normalization can eliminate differences in light intensity and highlight the shape characteristics of the spectral curve. Specifically, spectral normalization methods include L2 normalization or maximum / minimum normalization of the spectral vector of each point cloud.

[0080] S12 combines camera parameters and external environment parameters, then backprojects each pixel of the hyperspectral image data into three-dimensional space, and performs point-by-point matching with the original point cloud data, so that each point cloud has corresponding spectral information. For point clouds without corresponding spectral information (such as radar point clouds that are outside the image coverage area), spectral values ​​are assigned based on nearest neighbor interpolation or common-view geometric relationships.

[0081] Specifically, in the process of identifying the target slope area based on a multimodal deep neural network, the above preprocessing process can be performed after simultaneously collecting the original point cloud data and hyperspectral image data of the target slope area at any time during the day.

[0082] During the training of the target slope area, for the same target slope area, multiple data acquisitions are conducted at different time points (e.g., early morning, morning, noon, and afternoon) using a 3D laser scanner or UAV oblique photography equipment to obtain multiple phases of raw point cloud data covering different shadow distribution states. Simultaneously with at least one raw point cloud data acquisition, a hyperspectral imager mounted on a UAV or ground platform is used to acquire hyperspectral image data of the slope, containing spectral reflectance information for each pixel. Using the coordinate system of the first phase of raw point cloud data as the reference coordinate system, and based on the iterative nearest point (RIB) algorithm or its variants, the point cloud data from other phases are precisely registered to the reference coordinate system. Subsequently, the registered multi-phase raw point cloud data is fused. During the fusion process, shadow void regions (regions with point density below a preset threshold) in each phase of raw point cloud data are identified, and these voids are filled using valid point cloud data from other phases, generating a spatially complete, shadow-free, high-density fused raw point cloud. The aforementioned preprocessing operations are then performed.

[0083] Step two, as Figure 2As shown, preprocessed multimodal data is input into a multimodal deep neural network to construct a rock surface point cloud, and structural surface clustering identification is performed on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering identification on the rock surface point cloud data based on the fused features.

[0084] In step two, the feature extraction network includes: a geometric feature extraction branch, a spectral feature extraction branch, and a feature fusion network;

[0085] The geometric feature extraction branch takes the original point cloud data as input and uses the improved PointNet++ network as the backbone network to extract the geometric features of each point cloud.

[0086] Specifically, in this embodiment, the improved PointNet++ network includes an ensemble abstraction layer and a feature propagation layer; wherein, the ensemble abstraction layer is composed of multi-level and multi-scale ensemble abstraction modules;

[0087] The collection abstraction module performs the following process:

[0088] T1, from the original point cloud data (in this embodiment, the input is the original point cloud matrix P∈R) M×3 Where M represents the number of original point clouds; R represents a real number matrix, N center sampling points are selected from the matrix, and the center sampling points uniformly cover the rock surface; (specifically, the farthest point sampling (FPS) algorithm can be used to select the center sampling points)

[0089] T2, set n search radii; with each central sampling point as the center of the sphere, within a search radius r b The search for neighborhood points constructs a set of neighborhood points with different search radii; where b = 1, 2, ..., n; (specifically, for the characteristics of steep slope rock masses, three scale radii are set, i.e., n = 3, where r1 = 0.1 m, r2 = 0.3 m, r3 = 0.5 m, to capture structural surface features at different scales)

[0090] T3, based on the depthwise convolution operation (PointConv operation), introduces a normal vector-guided attention weight mechanism to extract features from the point sets of each domain, obtaining the local aggregated features of each central sampling point under different search radii; under any search radius, the local aggregated features of the central sampling point i are:

[0091] ;

[0092] ;

[0093] Where N(i) represents the set of neighborhood points of the central sampling point i; This represents the relative coordinates between the central sampling point i and its neighboring point j; Let represent the normal vector of the neighboring point j; This represents the original feature vector of the neighboring point j; This represents the contribution weight of neighboring point j to the feature update of the central sampling point i; This represents the consistency coefficient of the normal vectors between the central sampling point i and its neighboring point j. The greater the difference in the normal vectors between the central sampling point i and its neighboring point j, the more consistent the normal vectors become. The closer it is to 1, the less likely it is to be true. The closer it is to 0; MLP() represents a multilayer perceptron; This represents the basic weights calculated based on the spatial distance and relative position between the central sampling point i and its neighboring points j;

[0094] Finally, the geometric feature F is obtained. g ∈R M×Cg Cg=256 represents the geometric feature dimension.

[0095] This scheme introduces a self-attention mechanism on the basis of the standard PointConv operation, which increases the interactivity and global relevance between data points. When aggregating local information, the set abstraction module can simultaneously consider spatial distance (coordinates) and geometric attribute consistency (normal vector), thereby more accurately identifying points belonging to the same structural surface, while suppressing interference from different structural surfaces (abrupt changes in normal vectors) or noise points.

[0096] The spectral feature extraction branch consists of multiple cascaded dilated convolutional layers with progressively increasing porosity to effectively capture dependencies in the spectral data and avoid loss of spectral detail features. Furthermore, residual connections are introduced into each dilated convolutional layer to alleviate the gradient vanishing problem in deep networks.

[0097] Specifically, the input spectral feature matrix S∈R M×KWhere K represents the number of spectral bands (usually 100-200 consecutive bands), the spectral feature extraction branch consists of three cascaded dilated convolutional layers. The first dilated convolutional layer has a kernel size of 3, a dilation rate of 1, and 64 output channels; the second dilated convolutional layer has a kernel size of 3, a dilation rate of 2, and 128 output channels; the third dilated convolutional layer has a kernel size of 3, a dilation rate of 4, and 256 output channels. After preprocessing, the hyperspectral image data undergoes max pooling (pooling size of 2) after being processed by the three cascaded dilated convolutional layers to gradually reduce the feature dimension. Finally, it passes through two fully connected layers to output spectral features. The two fully connected layers gradually compress the feature dimension to Cs=128, and the output spectral feature map Fs∈R M×Cs , where Cs=128 represents the dimension of the spectral feature.

[0098] To address the heterogeneity between geometric and spectral features, this embodiment employs a dual-branch spatial attention architecture mechanism to achieve deep fusion of the two modalities; for example... Figure 3 As shown, the feature fusion network includes: a dual-modal channel separation component, a dual-branch spatial attention component, and a feature fusion component;

[0099] The dual-modal channel separation component performs channel dimension separation processing on geometric features and spectral features respectively to obtain important channel features and secondary important fused features;

[0100] Channel dimension separation is performed on geometric features and spectral features separately to extract their respective dominant channel information. The specific channel dimension separation processing method includes: first obtaining the channel attention vectors a of geometric features and spectral features. g and a s The preset separation threshold τ = 0.5 is used. Based on the separation threshold τ, if the geometric features or spectral features affect the channel attention a of channel i... gi ∈a g or a si ∈a s If the channel attention of channel i is greater than the separation threshold τ, then channel i is classified as an important channel; if the channel attention of channel i is less than the separation threshold τ, then channel i is classified as a less important channel. Features of different modalities carry different semantic information along the channel dimension. Geometric feature channels mainly encode structural information such as local curvature and normal variation, while spectral feature channels mainly encode material information such as mineral composition and vegetation indices. Directly fusing all channels would cause information contamination and reduce feature discriminative power. This scheme evaluates the importance of channels and separates them into important and less important channels. Important channels handle important geometric and spectral features, while less important channels handle less important geometric and spectral features. This focuses computational resources on important channels while retaining less important channels as supplementary information, avoiding information loss.

[0101] The dual-branch spatial attention component enhances spatial attention for important channel features and performs global calibration of channel dimensions for less important channel features.

[0102] Based on channel separation, important channels carry highly discriminative semantic information and require precise spatial localization; secondary important channels carry background or auxiliary information and can be processed with lightweight methods. This scheme uses a dual-branch spatial attention component to process the two sets of features separately. Spatial attention is enhanced for the geometric and spectral features of important channels, while global calibration of the channel dimensions is performed on secondary important channels. No precise spatial attention calculation is performed to save computational resources. Important channels gain precise spatial localization capabilities, which helps to accurately identify structural boundaries; secondary important channels only undergo global calibration, maintaining feature diversity while controlling computational load.

[0103] The feature fusion component performs a weighted fusion of the spatial attention-enhanced features and the globally calibrated features to obtain the fused feature F. f ∈R M×Cf Where Cf=384 represents the dimension of the fused feature.

[0104] After channel separation and spatial attention enhancement, geometric and spectral features have been sufficiently optimized. An adaptive weighted fusion method can be adopted, which dynamically learns the fusion weights of the two modes according to the characteristics of each spatial location, achieving complementary advantages. Specifically, in the weighted fusion process, higher geometric feature weights can be set in areas with exposed rock surfaces, and higher spectral feature weights can be set in areas with vegetation cover. In transitional areas, the weights of geometric and spectral features are balanced. This dynamic fusion mechanism enables the network to automatically adjust the fusion strategy according to different land cover types, significantly improving the accuracy and robustness of identification.

[0105] The feature propagation layer splices and fuses local aggregated features under different search radii, and then upsamples them based on a distance-weighted interpolation strategy to obtain the geometric features of each point cloud.

[0106] In step two, the method for acquiring the dense point cloud data includes:

[0107] S21, the original point cloud data P is processed by a sparse convolutional layer. in and fusion feature F f Downsampling is performed to extract multi-scale semantic features, which are then normalized and processed with ReLU activation to obtain sparse point cloud data P. en and feature map F en ;

[0108] Specifically, 3-layer sparse convolution can be used for progressive downsampling. The stride of the first layer is 1 (to maintain the resolution), the number of output channels is 128, the stride of the second layer is 2 (for downsampling), the number of output channels is 256, and the stride of the third layer is 2 (for downsampling), the number of output channels is 512. After each layer of convolution, batch normalization and ReLU activation function are connected. After 3-layer encoding, the downsampled sparse point cloud data Pen ∈ R Men×3 and its feature map Fen ∈ R Fen×512 are obtained; where Men represents the number of points in the sparse point cloud data; Fen represents the number of points in the feature map of the sparse point cloud data; Men = Fen < M; Sparse convolution is only calculated on non-empty voxels, which is suitable for dealing with the irregularity of point clouds and extracting high-level semantic features.

[0109] S22. Based on the feature map F en the global enhanced feature F enc is extracted, and based on the sparse point cloud data P en the local neighborhood feature F local is extracted. The global enhanced feature F enc and the local neighborhood feature F local are fused to obtain the multi-scale extended feature F exp ;

[0110] Specifically, global average pooling is performed on the feature map F en to obtain the global feature f global ∈ R 512 . The global feature f global is copied Men times and concatenated with the features of each point to obtain the global enhanced feature F enc ∈ R Men×1024 ; For each point, its K nearest neighbors (K = 16) are searched in the sparse point cloud data P en , and max pooling is performed on the neighborhood features to obtain the local neighborhood feature F local ∈ R Men×512 ; The global enhanced feature and the local neighborhood feature are concatenated to obtain the multi-scale extended feature F exp ∈ R Men×1536 . Finally, the dimension is compressed back to 512 through a fully connected layer (MLP) to obtain the extended feature F exp ∈ R Men×512 .

[0111] S23. The sparse point cloud data P en and the multi-scale extended feature F expUpsampling is performed, and the features of the sparse convolutional layer are skipped to the features of the corresponding sparse transposed convolutional layer to predict the coordinate offset ΔP of each upsampled point. The coordinate offset ΔP is then superimposed with the coordinates of each upsampled point cloud to obtain the generated point cloud data P. new ;

[0112] Specifically, the sparse transposed convolutional layer consists of three upsampling layers, symmetrical to the encoder. The first layer is a transposed convolution with a stride of 2 and 256 output channels, upsampling the point cloud to M. mid1 1 point; the second layer is a transposed convolution with a stride of 2 and 128 output channels, upsampled to M. mid2 The third layer is a transposed convolution with a stride of 1 and 64 output channels, upsampled to M′ points (final density). After upsampling at each layer, skip connections are introduced to concatenate the features of the corresponding layer of the encoder with the current features to preserve detailed information. For example, after upsampling at the first layer, it is concatenated with the output of the second layer of the encoder (256 dimensions), and then fused through an MLP. The last layer outputs the coordinate offset ΔP∈R for each point. M′×3 For the upsampled point, its coordinates are obtained by transposing and convolving the coordinates of the encoded point, and then superimposed with the predicted offset ΔP.

[0113] S24, the original point cloud data P in With the generated point cloud data P new The data is merged to obtain dense point cloud data.

[0114] The final completed dense point cloud data P out =Concat(P in P new To ensure uniform point cloud density, the merged point cloud can be sampled at the farthest point (FPS) up to a fixed number M′.

[0115] In step two, the methods for determining rock surface point clouds and non-rock surface point clouds include:

[0116] F21, align the dense point cloud data with the fused features, concatenate the coordinates of each point cloud with the fused features along the channel dimension, and linearly map the concatenated features to a low dimension based on a linear layer to obtain the original logical values ​​of each point cloud. Apply a Softmax function to the original logical values ​​to calculate the first probability p of each point cloud belonging to the rock surface. rock And the second probability that it does not belong to the rock surface;

[0117] Specifically, the coordinates of each point cloud and the fused features are concatenated along the channel dimension, and then a point-by-point feature transformation is performed through a 3-layer fully connected network MLP. Then, a linear layer maps the 64-dimensional features to 2D to obtain the original logical value of each point cloud. A Softmax function is applied to the 2D original logical value of each point cloud to obtain the first probability p of belonging to the rock surface.rock The second probability is for non-rock surfaces.

[0118] F22, preset probability threshold p e In dense point cloud data, if the first probability p of the current point cloud is... rock Greater than the probability threshold p e If the condition is met, the current point cloud is determined to be a rock surface point cloud; otherwise, the current point cloud is determined to be a non-rock surface point cloud.

[0119] In step two, the method for structural surface clustering identification includes:

[0120] G1 combines the fused features of each rock surface point cloud with the first probability p. rock Multiplication yields the weighted feature F of the rock surface point cloud. rock ;

[0121] F rock =F f ⊙p rock ; ⊙ indicates point-by-point multiplication;

[0122] G2, construct a graph attention convolutional network, and combine it with the weighted features F of the rock surface point cloud. rock Extract the graph convolutional features e of the point clouds of each rock surface; specifically, a KNN graph can be constructed, and a 3-layer graph attention convolution can be used to aggregate and update features to extract the graph convolutional features e of the point clouds of each rock surface.

[0123] G3, based on the differentiable mean drift clustering method, clusters the graph convolution feature e to identify the slope rock mass structure surface.

[0124] In step G3, the differentiable mean-shift clustering method includes:

[0125] G31 uses the graph convolution features e of all rock surface point clouds as candidate cluster centers C={e1, e2, ..., e3}. j , ..., e N}; N represents the total number of point clouds on the rock surface;

[0126] G32, for the current cluster candidate center c∈C, calculate the drift vector m(c):

[0127] ;

[0128] Where K() represents the kernel function, used to calculate the contribution weight of each point to the current cluster candidate center; h represents the bandwidth of the kernel function; in this embodiment, the kernel function K() is a Gaussian kernel function. u represents the normalized distance; points closer to the center contribute more (close to 1); points farther away contribute less (close to 0), making the drift vector mainly affected by local neighbors, which is consistent with the locality of density estimation.

[0129] G33, update the current cluster candidate center c=c+m(c), repeat steps G32 and G33 until convergence;

[0130] G34 merges cluster candidate centers whose distance is less than the distance threshold τ to obtain a cluster center set; in this embodiment, τ is set to 0.3;

[0131] G35 assigns each rock surface point cloud to the nearest cluster center to obtain the slope rock mass structure surface identification result.

[0132] By continuously updating the current clustering candidate centers, each candidate center converges to a density peak point in the feature space. After convergence, closely spaced peak points are merged, and each data point is assigned to the nearest peak point, thus completing the clustering. The differentiable implementation in this embodiment allows this process to be integrated into the end-to-end training of the neural network, and the clustering loss can be backpropagated using this formula to optimize the distribution of embedded features.

[0133] Step 3: Output the structural surface clustering identification results.

[0134] Example 2

[0135] This embodiment provides a multimodal deep learning recognition system for slope rock mass structure surfaces under complex terrain, such as... Figure 4 As shown, the system used to implement the multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain described in Example 1 includes:

[0136] The acquisition module is used to simultaneously acquire multimodal data of the target slope area; the multimodal data includes raw point cloud data and hyperspectral image data.

[0137] The preprocessing module is used to preprocess the multimodal data;

[0138] A recognition module is used to input preprocessed multimodal data into a multimodal deep neural network to construct a rock surface point cloud, and to perform structural surface clustering recognition on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data, and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering recognition on the rock surface point cloud data based on the fused features.

[0139] The output module is used to output the structural surface clustering recognition results.

[0140] Example 3

[0141] This embodiment provides a computer-readable medium storing a computer program. The computer program, when executed by a processor, can implement the multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain as described in Embodiment 1; specifically, it performs the following steps:

[0142] Step 1: Synchronously acquire multimodal data of the target slope area and preprocess the multimodal data; the multimodal data includes raw point cloud data and hyperspectral image data;

[0143] Step two involves inputting the preprocessed multimodal data into a multimodal deep neural network to construct a rock surface point cloud, and then performing structural surface clustering identification on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features from the preprocessed multimodal data and fuses these features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering identification on the rock surface point cloud data based on the fused features.

[0144] Step 3: Output the structural surface clustering identification results.

[0145] Example 4

[0146] This embodiment uses a steep slope in the reservoir area of ​​a hydropower station in Southwest China as an example. The slope has an elevation difference of approximately 300 meters, a slope angle of 65°-80°, and a vegetation coverage of about 40%. Due to the influence of the canyon terrain, the solar radiation and shadow variations are drastic. Using a DJI M300 drone equipped with a LiDAR and a Curt UHD 185 hyperspectral imager, three flight operations were conducted at 8:00 AM, 11:00 AM, and 3:00 PM to collect data. A total of approximately 230 million point cloud data points and 150 hyperspectral images (wavelength range 450-950nm, 125 bands) were acquired.

[0147] Data registration was performed according to step one above to construct a multimodal dataset. A multimodal deep neural network was then constructed according to step two above, in which the geometric branch adopted an improved PointNet++ network, the spectral branch adopted an 8-layer 1D-CNN network, and the feature fusion network adopted a multi-head attention mechanism. The multimodal deep neural network was trained, and the training dataset contained labeled data from 12 similar slopes, totaling approximately 1.5 billion points.

[0148] After training, the slope data of steep slopes were input into a multimodal deep neural network for identification. The results showed:

[0149] Point cloud voids were successfully filled with a filling error of less than 3 cm; vegetation point removal accuracy reached 94.7%, and recall rate was 92.3%; a total of 156 main structural surfaces were identified, and the comparison with manual on-site measurements showed an average dip error of 3.2° and an average tilt error of 2.1°, meeting the accuracy requirements for engineering surveys; the overall processing time was approximately 45 minutes, while traditional manual identification methods require 3-4 people working for 2-3 days.

[0150] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal deep learning method for recognizing slope rock mass structure surfaces in complex terrain, characterized in that, The method includes: Multimodal data of the target slope area are collected synchronously, and the multimodal data is preprocessed; the multimodal data includes raw point cloud data and hyperspectral image data; Preprocessed multimodal data is input into a multimodal deep neural network to construct a rock surface point cloud, and structural surface clustering is performed on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering on the rock surface point cloud data based on the fused features. The feature extraction network includes: a geometric feature extraction branch, a spectral feature extraction branch, and a feature fusion network; The geometric feature extraction branch takes the original point cloud data as input and uses the improved PointNet++ network as the backbone network to extract the geometric features of each point cloud. The spectral feature extraction branch consists of multiple cascaded dilated convolutional layers, with the porosity of the dilated convolutional layers increasing sequentially, and residual connections introduced into each dilated convolutional layer. The feature fusion network includes: a dual-modal channel separation component, a dual-branch spatial attention component, and a feature fusion component; The dual-modal channel separation component performs channel dimension separation processing on geometric features and spectral features respectively to obtain important channel features and secondary important channel features; The dual-branch spatial attention component enhances spatial attention for important channel features and performs global calibration of channel dimensions for less important channel features. The feature fusion component performs weighted fusion of the spatial attention-enhanced features and the globally calibrated features to obtain fused features; Output the structural surface clustering identification results.

2. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 1, characterized in that, The preprocessing method includes: First, the original point cloud data is subjected to outlier filtering and point cloud normalization. Then, the hyperspectral image data is subjected to radiometric calibration, atmospheric correction, and spectral normalization. Then, each pixel of the hyperspectral image data is back-projected into three-dimensional space and matched point by point with the original point cloud data so that each point cloud has corresponding spectral information. For point clouds without corresponding spectral information, spectral values ​​are assigned based on nearest neighbor interpolation or common-view geometric relationships.

3. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 1, characterized in that, The improved PointNet++ network includes a set abstraction layer and a feature propagation layer; the set abstraction layer is composed of multi-level and multi-scale set abstraction modules; The collection abstraction module performs the following process: T1, select N central sampling points from the original point cloud data, and the central sampling points uniformly cover the rock surface; T2, set n search radii; with each central sampling point as the center of the sphere, within a search radius r b The neighborhood points are searched to construct a set of neighborhood points with different search radii; where b = 1, 2, ..., n; T3 introduces a normal vector-guided attention weight mechanism on top of depthwise convolution to extract features from the point sets of each domain, obtaining the local aggregated features of each central sampling point under different search radii; under any search radius, the local aggregated features of the central sampling point i are: ； ； Where N(i) represents the set of neighborhood points of the central sampling point i; This represents the relative coordinates between the central sampling point i and its neighboring point j; Let represent the normal vector of the neighboring point j; This represents the original feature vector of the neighboring point j; This represents the contribution weight of neighboring point j to the feature update of the central sampling point i; This represents the consistency coefficient of the normal vectors between the central sampling point i and its neighboring point j. The greater the difference in the normal vectors between the central sampling point i and its neighboring point j, the more consistent the normal vectors become. The closer it is to 1, the less likely it is to be true. The closer it is to 0; MLP() represents a multilayer perceptron; This represents the basic weights calculated based on the spatial distance and relative position between the central sampling point i and its neighboring points j; The feature propagation layer splices and fuses local aggregated features under different search radii, and then upsamples them based on a distance-weighted interpolation strategy to obtain the geometric features of each point cloud.

4. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 1, characterized in that, The method for acquiring the dense point cloud data includes: The original point cloud data P is processed by sparse convolutional layers. in Multi-scale semantic features are extracted by downsampling and fusion features. These multi-scale semantic features are then normalized and processed with ReLU activation to obtain sparse point cloud data P. en and feature map F en ; Based on the feature map F en Extract global enhancement features F enc Based on the sparse point cloud data P en Extract local neighborhood features F local , fusing the global enhancement feature F enc and local neighborhood features F local Obtain multi-scale extended features F exp ; The sparse point cloud data P is processed by a sparse transposed convolutional layer symmetrical to the sparse convolutional layer. en and multi-scale extended features F exp Upsampling is performed, and the features of the sparse convolutional layer are skipped to the features of the corresponding sparse transposed convolutional layer to predict the coordinate offset ΔP of each upsampled point. The coordinate offset ΔP is then superimposed with the coordinates of each upsampled point cloud to obtain the generated point cloud data P. new ; The original point cloud data P in With the generated point cloud data P new The data is merged to obtain dense point cloud data.

5. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 1, characterized in that, The methods for determining the rock surface point cloud and non-rock surface point cloud include: Align the dense point cloud data with the fused features, concatenate the coordinates of each point cloud with the fused features along the channel dimension, and linearly map the concatenated features to a low dimension based on a linear layer to obtain the original logical values ​​of each point cloud. Apply a Softmax function to the original logical values ​​to calculate the first probability p of each point cloud belonging to the rock surface. rock And the second probability that it does not belong to the rock surface; Preset probability threshold p e In dense point cloud data, if the first probability p of the current point cloud is... rock Greater than the probability threshold p e If the condition is met, the current point cloud is determined to be a rock surface point cloud; otherwise, the current point cloud is determined to be a non-rock surface point cloud.

6. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 5, characterized in that, The method for structural plane clustering identification includes: The fusion features of each rock surface point cloud are compared with the first probability p. rock Multiplication yields the weighted feature F of the rock surface point cloud. rock ; Construct a graph attention convolutional network, and combine it with the weighted features F of the rock surface point cloud. rock Extract the graph convolution features e of the point clouds on each rock surface; The slope rock mass structure surface is obtained by clustering the graph convolution feature e based on the differentiable mean drift clustering method.

7. The multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain according to claim 6, characterized in that, The differentiable mean-shift clustering method includes: G31 uses the graph convolution features e of all rock surface point clouds as candidate cluster centers C={e1, e2, ..., e3}. j , ..., e N }; N represents the total number of point clouds on the rock surface; e j The graph convolution feature representing the point cloud of the j-th rock surface; e N The graph convolution feature representing the point cloud of the Nth rock surface; G32, for the current cluster candidate center c∈C, calculate the drift vector m(c): ； Where K() represents the kernel function, which is used to calculate the contribution weight of each point to the current cluster candidate center; h represents the bandwidth of the kernel function; G33, update the current cluster candidate center c=c+m(c), repeat steps G32 and G33 until convergence; G34 merges cluster candidate centers whose distance is less than the distance threshold τ to obtain a cluster center set; G35 assigns each rock surface point cloud to the nearest cluster center to obtain the slope rock mass structure surface identification result.

8. A multimodal deep learning recognition system for slope rock mass structure surfaces under complex terrain, characterized in that, The system is used to implement the multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain as described in any one of claims 1-7, the system comprising: The acquisition module is used to simultaneously acquire multimodal data of the target slope area; the multimodal data includes raw point cloud data and hyperspectral image data. The preprocessing module is used to preprocess the multimodal data; A recognition module is used to input preprocessed multimodal data into a multimodal deep neural network to construct a rock surface point cloud, and to perform structural surface clustering recognition on the rock surface point cloud. The multimodal deep neural network includes a feature extraction network and a multi-task decoding network. The multi-task decoding network includes a first decoding branch, a second decoding branch, and a third decoding branch. The feature extraction network extracts geometric and spectral features of the point cloud from the preprocessed multimodal data, and fuses the geometric and spectral features to obtain fused features. The first decoding branch is used to complete the original point cloud data based on the fused features to obtain dense point cloud data. The second decoding branch is used to perform semantic segmentation on the dense point cloud data based on the fused features to determine rock surface point cloud data and non-rock surface point cloud data. The third decoding branch is used to perform structural surface clustering recognition on the rock surface point cloud data based on the fused features. The output module is used to output the structural surface clustering recognition results.

9. A computer-readable medium having a computer program stored thereon, characterized in that, The computer program, executed by a processor, can implement the multimodal deep learning recognition method for slope rock mass structure surfaces under complex terrain as described in any one of claims 1-7.

Images