Machine vision dangerous rock mass displacement monitoring method and device fusing image features and AI

Abstract

The invention provides a machine vision dangerous rock mass displacement monitoring method and device fusing image features and AI, and relates to the field of dangerous rock mass displacement monitoring, and the method comprises the steps: carrying out the priori knowledge analysis of a preprocessing image set, and obtaining a priori knowledge feature set; a lightweight CNN-Transform hybrid model is constructed, semantic feature extraction is performed on the preprocessed image set through the lightweight CNN-Transform hybrid model, and a high-level semantic feature set is obtained; performing dynamic weight fusion on the priori knowledge feature set and the high-level semantic feature set through a double-channel attention fusion model to obtain a fused feature set; performing feature matching on the fused feature set through an FLANN matching algorithm to obtain an effective matching feature pair, and performing calculation according to the effective matching feature pair to obtain an initial displacement sequence of the dangerous rock mass; abnormal displacement elimination, time sequence correction and environmental factor compensation are carried out on the initial displacement sequence of the dangerous rock mass through a multi-stage AI model, a final displacement sequence of the dangerous rock mass is obtained, and a dangerous rock mass displacement monitoring result is generated according to the final displacement sequence of the dangerous rock mass.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of dangerous rock mass displacement monitoring, and in particular to a machine vision dangerous rock mass displacement monitoring method and device fusing image features and AI. BACKGROUND

[0002] As a common geological disaster hazard in mountainous areas, the stability monitoring of dangerous rock mass is of great significance to disaster prevention and mitigation. With the development of machine vision technology, non-contact monitoring methods based on images have gradually become one of the mainstream technologies for dangerous rock mass displacement monitoring due to their low cost, wide coverage, and long-term continuous monitoring advantages. Such methods collect sequence images of the dangerous rock mass area, extract features and match analysis, calculate the rock mass displacement, and realize dynamic evaluation of the stability of the dangerous rock mass.

[0003] However, the existing dangerous rock mass displacement monitoring technology based on machine vision cannot accurately calculate the displacement of the dangerous rock mass and does not have high precision in monitoring the displacement of the dangerous rock mass, because it does not comprehensively extract features from the dangerous rock mass image, cannot effectively fuse multi-source features, and does not optimize the displacement results of the dangerous rock mass.

[0004] The existing dangerous rock mass displacement monitoring method has the limitation of single feature extraction, and does not comprehensively extract features from the dangerous rock mass image from the aspects of prior knowledge and semantic association, which leads to insufficient feature robustness and difficulty in stably reflecting the true structure of the rock mass, resulting in inaccurate calculation of the displacement of the dangerous rock mass.

[0005] The existing dangerous rock mass displacement monitoring method often uses traditional methods such as fixed weight splicing or simple element addition to fuse multi-source features, without considering the influence of spatial position and channel on features, resulting in low signal-to-noise ratio of fused features and affecting the calculation accuracy of subsequent dangerous rock mass displacement.

[0006] The calculation results of the displacement of the dangerous rock mass are easily mixed with the influence of camera shaking, vegetation obstruction and environmental factors. The existing dangerous rock mass displacement monitoring method usually only uses median filtering for simple denoising, which cannot effectively filter multiple interferences and directly affects the monitoring results of the displacement of the dangerous rock mass. SUMMARY

[0007] To solve the above problems, the present application provides a machine vision dangerous rock mass displacement monitoring method fusing image features and AI, comprising:

[0008] An original image set of the dangerous rock mass in the monitoring area is collected by a shooting device, and the original image set is preprocessed to obtain a preprocessed image set;

[0009] The pre-processed image set is analyzed according to prior knowledge to obtain a prior knowledge feature set; a light-weight CNN-Transformer hybrid model is constructed, and semantic feature extraction is performed on the pre-processed image set by the light-weight CNN-Transformer hybrid model to obtain a high-level semantic feature set;

[0010] A dual-channel attention fusion model is constructed, and dynamic weight fusion is performed on the prior knowledge feature set and the high-level semantic feature set by the dual-channel attention fusion model to obtain a fusion feature set;

[0011] Feature matching is performed on the fusion feature set by an FLANN matching algorithm to obtain an effective matching feature pair, and an initial displacement sequence of the dangerous rock mass is calculated according to the effective matching feature pair;

[0012] A multi-level AI model is constructed, and abnormal displacement elimination, time sequence correction and environmental factor compensation are performed on the initial displacement sequence of the dangerous rock mass by the multi-level AI model to obtain a final displacement sequence of the dangerous rock mass, and a displacement monitoring result of the dangerous rock mass is generated according to the final displacement sequence of the dangerous rock mass.

[0013] Optionally, the pre-processing of the original image set to obtain a pre-processed image set comprises:

[0014] Each original image in the original image set is obtained, and the original image is subjected to illumination component correction and median filtering to obtain a denoised image;

[0015] The dangerous rock mass region is framed in the denoised image, and the background region other than the dangerous rock mass region is cropped to obtain a pre-processed image, and the pre-processed image set is composed of all the pre-processed images.

[0016] Optionally, the prior knowledge analysis of the pre-processed image set to obtain a prior knowledge feature set comprises:

[0017] The dangerous rock mass region of each pre-processed image in the pre-processed image set is obtained, a stable key point set in the dangerous rock mass region is screened out by a Harris corner point detection algorithm, and an ORB feature set is generated according to the stable key point set by an ORB algorithm;

[0018] The gradient amplitude of each pixel in the dangerous rock mass region is calculated by a Sobel operator, the edge pixels with a gradient amplitude greater than a preset value and located at the contour boundary are retained, and an edge gradient feature set is constructed by the gradient amplitude of the edge pixels;

[0019] A texture analysis window is determined according to the edge gradient feature set, and texture feature extraction is performed on the dangerous rock mass region by the texture analysis window to obtain a texture co-occurrence matrix feature set;

[0020] The prior knowledge feature set is composed of the ORB feature set, the edge gradient feature set and the texture co-occurrence matrix feature set.

[0021] Optionally, the step of extracting semantic features from the preprocessed image set using a lightweight CNN-Transformer hybrid model to obtain a high-level semantic feature set includes:

[0022] The lightweight CNN-Transformer hybrid model includes a CNN network and a Transformer network. The CNN network performs feature mapping on the preprocessed image set to obtain a low-order feature set.

[0023] The low-order feature set is reshaped into a sequence feature vector, and the sequence feature vector is input into a Transformer network. The encoder, attention head and decoder of the Transformer network capture the long-distance dependency between the deformation and semantics of the dangerous rock mass, and obtain a high-level semantic feature set.

[0024] Optionally, the step of dynamically weighting and fusing the prior knowledge feature set and the high-level semantic feature set through a dual-channel attention fusion model to obtain a fused feature set includes:

[0025] The dual-channel attention fusion model includes a spatial attention layer, a channel attention layer, and a shared fully connected layer. The spatial attention layer performs convolution operations on the high-level semantic feature set to obtain local spatial correlation feature vectors. The Sigmoid activation function maps the local spatial correlation feature vectors to spatial attention weight vectors.

[0026] The prior knowledge feature set and the high-level semantic feature set are subjected to global average pooling and channel statistics through the channel attention layer to obtain the first channel vector and the second channel vector. The first channel vector and the second channel vector are then concatenated into a joint channel vector.

[0027] Channel importance is calculated on the joint channel vector by sharing a fully connected layer to obtain a channel importance score vector. The channel importance score vector is then mapped to a channel attention weight vector by the Sigmoid activation function.

[0028] The prior knowledge feature set and the high-level semantic feature set are weighted by spatial attention weight vectors to obtain a first-weighted prior knowledge feature set and a first-weighted high-level semantic feature set; the first-weighted prior knowledge feature set and the first-weighted high-level semantic feature set are weighted by channel attention weight vectors to obtain a second-weighted prior knowledge feature set and a second-weighted high-level semantic feature set.

[0029] Adjustment parameters are set based on the entropy value of the high-level semantic feature set, and the secondary weighted prior knowledge feature set and the secondary weighted high-level semantic feature set are fused into a fused feature set based on the adjustment parameters.

[0030] Optionally, the step of performing feature matching on the fused feature set using the FLANN matching algorithm to obtain effective matching feature pairs, and calculating the initial displacement sequence of the unstable rock mass based on the effective matching feature pairs, includes:

[0031] According to the frame order, the fused features in the fused feature set are divided into reference frame features and matchable frame features. The FLANN matching algorithm is used to match each reference frame feature with each matchable frame feature into multiple feature pairs. The Hamming distance of each feature pair is calculated, and feature pairs with a Hamming distance less than a preset distance threshold are considered as valid matching feature pairs.

[0032] Obtain the camera calibration parameters of the shooting device, calculate the pixel coordinate difference of each valid matching feature pair, convert the pixel coordinate difference into physical displacement according to the camera calibration parameters, and arrange each physical displacement in frame order to form the initial displacement sequence of the unstable rock mass.

[0033] Optionally, the step of performing abnormal displacement removal, temporal correction, and environmental factor compensation on the initial displacement sequence of the unstable rock mass using a multi-level AI model to obtain the final displacement sequence of the unstable rock mass includes:

[0034] The multi-level AI model includes an AI anomaly detection model, an AI time series analysis model, and an AI environmental analysis model. The AI ​​anomaly detection model identifies and removes deformation data from the initial displacement sequence of the unstable rock mass to obtain a denoised displacement sequence.

[0035] The denoised displacement sequence is modeled by time series correlation using an AI time series analysis model to obtain displacement change trend data. The denoised displacement sequence is then time series corrected based on the displacement change trend data to obtain the time series corrected displacement sequence.

[0036] Environmental monitoring data is acquired, and an AI environmental analysis model is used to obtain the correlation matrix between the environmental monitoring data and the displacement of the unstable rock mass. Based on the correlation matrix, environmental factors are compensated for the time-series corrected displacement sequence to obtain the final displacement sequence of the unstable rock mass.

[0037] This invention also provides a machine vision-based unstable rock mass displacement monitoring device that integrates image features and AI, used to implement the aforementioned machine vision-based unstable rock mass displacement monitoring method that integrates image features and AI. The device includes:

[0038] The preprocessed image set acquisition module is used to acquire the original image set of the dangerous rock mass in the monitoring area through the imaging device, and to preprocess the original image set to obtain the preprocessed image set.

[0039] The high-level semantic feature set acquisition module is used to perform prior knowledge analysis on the preprocessed image set to obtain the prior knowledge feature set; a lightweight CNN-Transformer hybrid model is constructed to extract semantic features from the preprocessed image set to obtain the high-level semantic feature set;

[0040] The fusion feature set acquisition module is used to construct a dual-channel attention fusion model. The dual-channel attention fusion model dynamically fuses the prior knowledge feature set and the high-level semantic feature set with weights to obtain the fusion feature set.

[0041] The module for obtaining the initial displacement sequence of the unstable rock mass is used to perform feature matching on the fused feature set using the FLANN matching algorithm to obtain effective matching feature pairs, and to calculate the initial displacement sequence of the unstable rock mass based on the effective matching feature pairs.

[0042] The module for acquiring monitoring results of unstable rock mass displacement is used to construct a multi-level AI model. Through the multi-level AI model, abnormal displacement is removed, time sequence is corrected, and environmental factors are compensated for in the initial displacement sequence of unstable rock mass to obtain the final displacement sequence of unstable rock mass. Based on the final displacement sequence of unstable rock mass, the monitoring results of unstable rock mass displacement are generated.

[0043] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the machine vision method for monitoring the displacement of unstable rock masses by fusing image features and AI.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned machine vision method for monitoring the displacement of unstable rock masses by fusing image features and AI.

[0045] The present invention has the following beneficial effects:

[0046] 1. Prior knowledge feature set is obtained by performing prior knowledge analysis on the preprocessed image set. High-level semantic feature set of the preprocessed image set is extracted by a lightweight CNN-Transformer hybrid model. The prior knowledge feature set and the high-level semantic feature set are fused into a fused feature set by a dual-channel attention fusion model. The fused feature set forms a complementary feature system of structural anchor points and deformation details, which takes into account both the structural stability and deformation sensitivity of the features, and significantly improves the signal-to-noise ratio of the fused features, avoiding the limitations of traditional fixed-weight fusion. Subsequently, the effective feature pairs are efficiently screened by the FLANN matching algorithm. Combined with a multi-level AI model, abnormal displacement removal, temporal correction and environmental factor compensation are completed, which greatly improves the calculation accuracy of the unstable rock mass displacement.

[0047] 2. The prior knowledge feature set and the high-level semantic feature set form a structure-semantic collaborative feature combination. The former provides the latter with accurate structural anchor points, avoiding semantic shift caused by the lack of structural information in the Transformer network; the latter supplements the former with dynamic deformation correlation, avoiding the failure of prior features to identify subtle deformations due to lack of semantic information; thus realizing comprehensive feature extraction of dangerous rock mass images, improving feature robustness, and making the displacement calculation of dangerous rock mass more accurate.

[0048] 3. By convolving high-level semantic features with a spatial attention layer and generating spatial weight vectors, the features of easily deformable areas of unstable rock masses are accurately enhanced, and background interference is suppressed. At the same time, by using a channel attention layer and a shared fully connected layer, the channel statistical vectors of the two types of features are jointly learned to generate channel weight vectors, dynamically increasing the weight of displacement-sensitive channels, completely breaking through the limitations of traditional fixed weights and improving the signal-to-noise ratio of fused features. By spatially weighted alignment of structural anchor points and deformation details, and channel weighted enhancement of prior structural stability and high-level deformation sensitivity through a two-stage weighting, the two types of features are deeply complementary, improving the representational ability of fused features.

[0049] 4. By dividing the reference frame and the frame to be matched according to the frame order, the temporal logic is ensured. The FLANN matching algorithm combined with the Hamming distance threshold is used to improve the matching efficiency of features. Then, based on the camera calibration parameters, the pixel difference is converted into physical displacement to provide a high-quality initial sequence for subsequent processing. Nonlinear anomalies are eliminated by AI anomaly detection model, temporal correlations are captured by AI temporal analysis model to improve trend fitting, false environmental displacements are accurately compensated by AI environmental analysis model, and the calculation accuracy of unstable rock mass displacement is improved by progressive optimization of three-level AI models. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0052] Figure 2 This is a structural diagram of the device according to an embodiment of the present invention. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0054] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0055] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0056] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0057] Reference Figure 1 This invention provides a machine vision-based method for monitoring the displacement of unstable rock masses that integrates image features and AI, comprising:

[0058] The original image set of the dangerous rock mass in the monitoring area is collected by the imaging equipment, and the original image set is preprocessed to obtain the preprocessed image set;

[0059] In some embodiments, the imaging device is a high-definition industrial camera with a resolution of not less than 1920×1080 and a frame rate of not less than 25fps. The original images are uniformly stored in lossless TIFF format, named according to the rule of acquisition timestamp-camera ID-monitored sub-region, and stored in an industrial-grade solid-state drive with a read / write speed of not less than 500MB / s to ensure data retrieval efficiency. The preprocessing operation is based on Python 3.9 or above, using OpenCV 4.8.0 as the core image processing library and NumPy 1.25.2 for numerical calculation. The hardware configuration is Intel Core i7-12700H or above CPU and 32GB or more of memory to support parallel processing of multiple frames of images. If preprocessing needs to be completed in the field, it can be adapted to embedded devices such as NVIDIA Jetson Xavier NX.

[0060] In some embodiments, the preprocessing of the original image set to obtain a preprocessed image set includes:

[0061] The original images in the original image set are obtained, and illumination component correction and median filtering are performed on the original images to obtain the denoised images.

[0062] In the denoised image, the dangerous rock mass area is selected, and the background area other than the dangerous rock mass area is cropped to obtain a preprocessed image. All the preprocessed images are combined to form a preprocessed image set.

[0063] In some embodiments, illumination component correction is performed to eliminate illumination unevenness interference. For a single frame of original image, its RGB format is first converted to grayscale image. Then, the center-surround filtering model of the Retinex algorithm is used to decompose the grayscale image into illumination component, which represents the global brightness distribution, and reflection component, which represents the texture and structural features of the rock mass itself. Subsequently, the gamma correction parameter is dynamically adjusted according to the real-time illumination intensity obtained by the camera's built-in light sensor. When the illumination intensity is greater than 5000 lux, the gamma correction parameter is 0.3-0.5; when the illumination intensity is less than 1000 lux, the gamma correction parameter is 1.5-2.0; and when the illumination intensity is between 1000-5000 lux, the gamma correction parameter is 1.0. Gamma correction is performed on the illumination component according to this parameter. Finally, the corrected illumination component is multiplied by the original reflection component to reconstruct an 8-bit grayscale image after illumination equalization.

[0064] Next, median filtering is performed to remove noise. Using the image after illumination equalization as input, the noise density D is first calculated using the gray-level abrupt pixel statistical method. Then, the filter window size is selected according to the noise density D. When D is less than 5%, a 3×3 sliding window is used. When D is between 5% and 15%, a 5×5 sliding window is used. When D is greater than 15%, a 3×3 Gaussian filter preprocessing with a standard deviation of 1.0 is performed once, followed by a 5×5 sliding window. The sliding window of the selected size is moved pixel by pixel on the image. The median of the gray values ​​of all pixels in the window is taken to replace the gray value of the central pixel of the window. For pixels at the edge of the image, a mirror filling method is used to avoid the loss of edge information. Finally, the denoised image is obtained.

[0065] Subsequently, the unstable rock mass area was selected to determine the region of interest. Using the denoised image as input, a 5×5 Gaussian filter with a standard deviation of 0.8 was first applied for smoothing preprocessing, followed by Canny edge detection using a dual threshold method, with a low threshold T set. l =50, High threshold T h=150, extracting a binary edge image; then, using OpenCV's findContours function to extract all closed contours from the binary edge image, obtaining a contour set C; finally, using the standard contour model C obtained from the early three-dimensional laser scanning of the unstable rock mass... std Based on the baseline, a shape matching algorithm is used to calculate the relationship between each contour in the contour set C and C. std Matching degree S i Select matching degree S i At least 90% of the contour is taken as the outline of the dangerous rock mass. Then, the bounding rectangle of the dangerous rock mass outline is obtained by using the boundingRect function of OpenCV. The coordinates of the upper left and lower right corners of the rectangle are obtained. This rectangular area is the ROI of the dangerous rock mass. Finally, the denoised image is cropped according to the coordinates of the ROI, retaining the dangerous rock mass pixels within the rectangular area and removing the background pixels outside the rectangle. If there are differences in the size of different frames after cropping, bilinear interpolation is used to uniformly scale all cropped images to 512×512 pixels to obtain a single frame preprocessed image.

[0066] Prior knowledge analysis is performed on the preprocessed image set to obtain a prior knowledge feature set; a lightweight CNN-Transformer hybrid model is constructed, and semantic features are extracted from the preprocessed image set through the lightweight CNN-Transformer hybrid model to obtain a high-level semantic feature set;

[0067] In some embodiments, the prior knowledge analysis of the preprocessed image set to obtain a prior knowledge feature set includes:

[0068] Obtain the dangerous rock mass region of each preprocessed image in the preprocessed image set, filter out the set of stable key points within the dangerous rock mass region using the Harris corner detection algorithm, and generate an ORB feature set based on the set of stable key points using the ORB algorithm.

[0069] In some embodiments, a single frame of preprocessed image is taken, converted to grayscale, and then smoothed using a 3×3 Gaussian filter. The Harris corner detection algorithm is used to calculate the gradient and autocorrelation matrix of the grayscale image in the x and y directions. The response value of each pixel is calculated based on the autocorrelation matrix. The formula for calculating the response value R is as follows:

[0070] R = det(M) - k × (trace(M)) 2

[0071] Where M is the autocorrelation matrix, det(M) is the determinant of the matrix, trace(M) is the trace of the matrix, and k is the empirical coefficient;

[0072] Candidate corner points are selected by setting the response value threshold to 0.03, and then 3×3 neighborhood non-maximum suppression is performed on the candidate corner points to retain the points with the largest response values ​​in the neighborhood, thus obtaining a stable set of key points.

[0073] For each stable keypoint, a 16×16 neighborhood window is constructed centered on that point. The principal direction of the keypoint is determined by the gray-level centroid within the window. Based on the principal direction, 256 pairs of random pixel points are selected, and the gray-level values ​​of the point pairs are compared to generate a 512-bit binary descriptor. The coordinates of the stable keypoint are associated with the corresponding descriptor to form the ORB feature set F. orb ={(x p ,y p D p )|p=1,2,...,N}, where (x p ,y p D represents the coordinates of the stable key point. p For descriptors;

[0074] The gradient magnitude of each pixel within the unstable rock mass region is calculated using the Sobel operator. Edge pixels with gradient magnitudes greater than a preset value and located on the contour boundary are retained. An edge gradient feature set is constructed using the gradient magnitudes of the edge pixels.

[0075] In some embodiments, using the grayscale image of the unstable rock mass region and the coordinates of the unstable rock mass outline boundary as input, a 3×3 Sobel-x convolution kernel and a Sobel-y convolution kernel are used to calculate the gradient values ​​of each pixel in the x and y directions within the unstable rock mass region, and to calculate the gradient magnitude of each pixel; a preset gradient magnitude value of 20 is set, and pixels with gradient magnitudes greater than 20 are initially selected. Then, the minimum Euclidean distance between these pixels and the unstable rock mass outline boundary is calculated, and pixels with a distance ≤ 3 pixels are retained as edge pixels; the edge pixels are sorted according to the coordinate order from left to right and from top to bottom within the unstable rock mass region, and the coordinates of the edge pixels are associated with the corresponding gradient magnitudes to form an edge gradient feature set E={(x e ,y e G e )|e=1,2,...,L}, where G e For edge pixels (x) e ,y e The gradient magnitude;

[0076] The texture analysis window is determined based on the edge gradient feature set. Texture features are extracted from the dangerous rock mass area through the texture analysis window to obtain the texture co-occurrence matrix feature set.

[0077] In some embodiments, the edge gradient feature set is converted into a 256-channel feature map. From the generated 256-channel feature map, the number of edge pixels in each channel is counted, and the region with the most edge pixels is determined as the edge gradient dense region. The texture analysis window size is set to 1.5 times the size of the edge gradient dense region. The texture analysis window is slid across the unstable rock mass region with a step size of 1 / 2 of the window side length to obtain multiple texture analysis windows. For the grayscale image in each texture analysis window, the grayscale value is quantized to 16 levels. The distance parameter is set to 1 pixel, and the angle parameters are set to 0° / 45° / 90° / 135°. A 16×16 texture co-occurrence matrix is ​​calculated. From each texture co-occurrence matrix, four texture statistics are calculated: contrast (Con), correlation (Corr), energy (Eng), and entropy (Ent). The coordinates of each texture analysis window are associated with the four texture statistics at the corresponding four angles to form a texture co-occurrence matrix feature set.

[0078] The prior knowledge feature set consists of the ORB feature set, the edge gradient feature set, and the texture co-occurrence matrix feature set.

[0079] In some embodiments, the generated ORB feature set, edge gradient feature set, and texture co-occurrence matrix feature set are stored in a structure associated with feature type, coordinate, and feature value to form a prior knowledge feature set.

[0080] In some embodiments, the step of extracting semantic features from the preprocessed image set using a lightweight CNN-Transformer hybrid model to obtain a high-level semantic feature set includes:

[0081] The lightweight CNN-Transformer hybrid model includes a CNN network and a Transformer network. The CNN network performs feature mapping on the preprocessed image set to obtain a low-order feature set.

[0082] The low-order feature set is reshaped into a sequence feature vector, and the sequence feature vector is input into a Transformer network. The encoder, attention head and decoder of the Transformer network capture the long-distance dependency between the deformation and semantics of the dangerous rock mass, and obtain a high-level semantic feature set.

[0083] In some embodiments, preprocessed image input and format adaptation are first performed. The single-frame preprocessed image has completed background cropping and size unification. Normalization operation is performed on the single-frame image to obtain a normalized image, so that the pixel values ​​are distributed in the range of [-1,1], so as to avoid gradient instability caused by differences in numerical range during model processing.

[0084] The CNN network used is MobileNetV3-Small. Low-order feature mapping is performed through the CNN network, which consists of six feature extraction stages: Stage 1 uses a 3×3 convolutional kernel and a Hard-Swish activation function to compress the 512×512×1 normalized image into a 256×256×16 feature map, initially extracting texture and edge information; Stages 2-5 each contain 2-3 depthwise separable convolutions + SE channel attention modules. The depthwise separable convolutions reduce the number of parameters, and the SE module enhances the feature response of the rock mass's edges and textures. The number of output channels in each stage is 24, 32, 48, and 64 respectively, and the feature map sizes are compressed to 128×128, 64×64, 32×32, and 16×16 respectively; Stage 6 uses a 1×1 convolutional kernel and a Hard-Swish activation function to convert the 16×16×64 feature map into a 16×16×128 low-order feature set. ;

[0085] Then, the low-order feature set is reshaped into a sequence feature vector. For the 16×16×128 feature map of a single sample, it is flattened into a spatial sequence of length 256 along the spatial dimension in row priority. Each spatial position is associated with 128-dimensional channel features and recombined into a sequence feature vector of dimension 256×128. For the low-order feature set with batch_size=8, a sequence feature vector of dimension 8×256×128 is finally generated to adapt to the sequence input format of the Transformer network.

[0086] Then, a Transformer network is used to extract high-level semantic features. The encoder has 6 layers, each containing a multi-head self-attention module and a feedforward neural network module. The multi-head self-attention module has 8 attention heads to capture the deformation dependence of different spatial locations of the unstable rock mass. It also adds positional bias terms to the spatial locations of the rock mass edges and crack areas, increasing the attention weights of these areas by 15%-20%. The feedforward neural network module contains two fully connected layers and a GELU activation function. Residual connections and layer normalization are added to each layer to avoid gradient vanishing and accelerate convergence. The expression for the output feature of each layer of the encoder is as follows:

[0087]

[0088] in, This represents the output feature of the l-th coding layer. Indicates the first The residual results of the attention layer This represents a regularization operator. The weights of the feedforward neural network module, This is the bias for the feedforward neural network module. This represents the activation function. Representation layer normalization;

[0089] The decoder has a single-layer cross-attention module. The encoder output is used as the key vector K and value vector V, and the sequence feature vector is used as the query vector Q. This aligns the semantic association between the input sequence and the encoded features, ultimately outputting a high-level semantic feature set with dimensions 8×16×16×256. ,in The mapping function of the decoder. This is the parameter set for the decoder.

[0090] A dual-channel attention fusion model is constructed, and the prior knowledge feature set and the high-level semantic feature set are dynamically weighted and fused to obtain the fused feature set.

[0091] In some embodiments, the input feature dimension is first adapted. The dimension of the high-level semantic feature set is batch size × 16 × 16 × 256, where the batch size is set to 8, 16 × 16 is the feature map space size, and 256 is the number of channels. The prior knowledge feature set needs to be unified to the same 8 × 16 × 16 × 256 dimension as the high-level semantic feature set through 1 × 1 convolutional layers and bilinear interpolation to ensure dimension matching in subsequent weighted operations.

[0092] In some embodiments, the dynamic weight fusion of the prior knowledge feature set and the high-level semantic feature set through a dual-channel attention fusion model to obtain a fused feature set includes:

[0093] The dual-channel attention fusion model includes a spatial attention layer, a channel attention layer, and a shared fully connected layer. The spatial attention layer performs convolution operations on the high-level semantic feature set to obtain local spatial correlation feature vectors. The Sigmoid activation function maps the local spatial correlation feature vectors to spatial attention weight vectors.

[0094] In some embodiments, a high-level semantic feature set after dimensional unification is used. As input, a convolution operation is performed using a 3×3 convolution kernel to capture the spatial relationships between adjacent pixels, resulting in an 8×16×16×1 local spatial relationship feature vector. ,in This represents a 3×3 convolution;

[0095] The local spatial correlation feature vector is input into the Sigmoid activation function and mapped to the [0,1] interval to generate an 8×16×16×1 spatial attention weight vector. ,in This represents the Sigmoid activation function;

[0096] The prior knowledge feature set and the high-level semantic feature set are subjected to global average pooling and channel statistics through the channel attention layer to obtain the first channel vector and the second channel vector. The first channel vector and the second channel vector are then concatenated into a joint channel vector.

[0097] Channel importance is calculated on the joint channel vector by sharing a fully connected layer to obtain a channel importance score vector. The channel importance score vector is then mapped to a channel attention weight vector by the Sigmoid activation function.

[0098] In some embodiments, global average pooling is performed on the prior knowledge feature set and the high-level semantic feature set respectively to compress the spatial dimension, resulting in a first channel vector and a second channel vector of 8×1×1×256. The two vectors are then concatenated along the channel dimension to obtain a joint channel vector of 8×1×1×512. ,in Represents the first channel vector and the second channel vector;

[0099] The shared fully connected layer first reduces the joint channel vector from 512 dimensions to 64 dimensions, and then increases it back to 512 dimensions to obtain the channel importance score vector. This score vector is then input into the Sigmoid activation function and mapped to the [0,1] interval to generate an 8×1×1×512 channel attention weight vector. ,in Indicates a shared fully connected layer;

[0100] The prior knowledge feature set and the high-level semantic feature set are weighted by spatial attention weight vectors to obtain a first-weighted prior knowledge feature set and a first-weighted high-level semantic feature set; the first-weighted prior knowledge feature set and the first-weighted high-level semantic feature set are weighted by channel attention weight vectors to obtain a second-weighted prior knowledge feature set and a second-weighted high-level semantic feature set.

[0101] Adjustment parameters are set based on the entropy value of the high-level semantic feature set, and the secondary weighted prior knowledge feature set and the secondary weighted high-level semantic feature set are fused into a fused feature set based on the adjustment parameters.

[0102] In some embodiments, the first stage is spatial attention weighting, which multiplies the spatial attention weight vector with the prior knowledge feature set and the high-level semantic feature set pixel by pixel to obtain a first-weighted prior knowledge feature set and a first-weighted high-level semantic feature set; the second stage is channel attention weighting, which extracts the first 256 dimensions of the channel attention weight vector and multiplies it with the first-weighted prior knowledge feature set channel by channel, and extracts the last 256 dimensions and multiplies it with the first-weighted high-level semantic feature set channel by channel to obtain a second-weighted prior knowledge feature set and a second-weighted high-level semantic feature set.

[0103] Entropy of high-level semantic feature set The expression is:

[0104]

[0105]

[0106] in, Indicates the number of channels, height, and width. This represents a double-weighted high-level semantic feature set. Indicates taking the first One channel, spatially located as Features, e i p is the first intermediate parameter. i This is the second intermediate parameter;

[0107] Adjust parameters based on entropy value. When the entropy value is > 0.8, α takes a value of 0.6-0.8; when the entropy value is ≤ 0.8, α takes a value of 0.3-0.5; fused feature set The expression is:

[0108]

[0109] in, This represents a quadratic weighted prior knowledge feature set;

[0110] The fused feature set is matched using the FLANN matching algorithm to obtain effective matching feature pairs. The initial displacement sequence of the unstable rock mass is then calculated based on the effective matching feature pairs.

[0111] In some embodiments, the step of performing feature matching on the fused feature set using the FLANN matching algorithm to obtain effective matching feature pairs, and calculating the initial displacement sequence of the unstable rock mass based on the effective matching feature pairs, includes:

[0112] According to the frame order, the fused features in the fused feature set are divided into reference frame features and matchable frame features. The FLANN matching algorithm is used to match each reference frame feature with each matchable frame feature into multiple feature pairs. The Hamming distance of each feature pair is calculated, and feature pairs with a Hamming distance less than a preset distance threshold are considered as valid matching feature pairs.

[0113] Obtain the camera calibration parameters of the shooting device, calculate the pixel coordinate difference of each valid matching feature pair, convert the pixel coordinate difference into physical displacement according to the camera calibration parameters, and arrange each physical displacement in frame order to form the initial displacement sequence of the unstable rock mass.

[0114] In some embodiments, each batch of samples in the fusion feature set corresponds to the fusion feature of a preprocessed image frame, and is associated with the timestamp of the preprocessed image set acquisition. The fusion feature set is sorted into a frame feature sequence in ascending order of timestamp. The earliest frame feature in the sequence is selected as the reference frame feature, and feature points containing pixel coordinates and 512-bit binary descriptors are extracted from its 16×16 feature map to form the reference frame feature set. All subsequent frame features in the sequence are selected as the frame features to be matched, and the same feature point extraction operation is performed on each frame to obtain the feature set of each frame to be matched.

[0115] To address the characteristics of the 512-bit binary descriptors in the fused features, a FLANN matcher is configured, using a KD-tree index, with a check count of 50, and k-nearest neighbor matching (k=2) is employed. The descriptors of the baseline frame feature set and the descriptors of each frame feature set to be matched are input into the FLANN matcher to generate an initial set of feature pairs. For each initial feature pair's descriptor, the Hamming distance between them is calculated. The expression is:

[0116]

[0117] in, Represents the feature of the i-th reference frame. This represents the feature of the i-th frame to be matched. Let D represent the index of the nearest neighbor of the i-th descriptor, and let D represent the bit length of the descriptor. Representing the features of the reference frame The Bit, Indicates the features of the frame to be matched The Bit, Indicates bitwise XOR;

[0118] Based on the verification of unstable rock mass samples, the Hamming distance threshold is preset to 150. The initial feature pair set is traversed, and feature pairs with a Hamming distance less than 150 are retained to obtain a preliminary effective feature pair set. The random sampling consensus algorithm is applied to the preliminary effective set to remove mismatches. The number of iterations is set to 1000 and the inlier threshold is set to 2 pixels. The homography matrix of the reference frame and the frame to be matched is fitted, and the inlier feature pairs that satisfy the matrix constraints are retained as the effective matching feature pair set.

[0119] The camera calibration parameters of the shooting equipment are acquired and stored in advance, and the focal length and pixel size are called up. For each valid matching feature pair, the horizontal pixel difference and vertical pixel difference of the feature points of the reference frame and the frame to be matched are calculated, and then the sum pixel difference is obtained. According to the formula physical displacement = sum pixel difference × pixel size / focal length, the sum pixel difference is converted into physical displacement in mm. The timestamps of the frames to be matched are arranged in ascending order and associated with the corresponding physical displacements to form the initial displacement sequence of the dangerous rock mass.

[0120] A multi-level AI model is constructed to perform abnormal displacement removal, time-series correction, and environmental factor compensation on the initial displacement sequence of the unstable rock mass, thereby obtaining the final displacement sequence of the unstable rock mass. Based on the final displacement sequence of the unstable rock mass, the displacement monitoring results of the unstable rock mass are generated.

[0121] In some embodiments, the step of performing abnormal displacement removal, temporal correction, and environmental factor compensation on the initial displacement sequence of the unstable rock mass using a multi-level AI model to obtain the final displacement sequence of the unstable rock mass includes:

[0122] The multi-level AI model includes an AI anomaly detection model, an AI time series analysis model, and an AI environmental analysis model. The AI ​​anomaly detection model identifies and removes deformation data from the initial displacement sequence of the unstable rock mass to obtain a denoised displacement sequence.

[0123] In some embodiments, the AI ​​anomaly detection model employs an isolated forest model with 100 trees and 256 subsamples. Training is based on historical normal displacement data, and an anomaly score threshold of 0.6 is set. The normalized initial displacement sequence is divided into subsequences with a time window of 5. After inputting these subsequences into the model, the model assigns an anomaly score >0.6 to subsamples that deviate from the normal deformation pattern, thus identifying anomalous moments. The displacement values ​​at anomalous moments are then interpolated using the mean of the displacements at three preceding and following normal moments, retaining the normal moment data to obtain the denoised displacement sequence. .

[0124] The denoised displacement sequence is modeled by time series correlation using an AI time series analysis model to obtain displacement change trend data. The denoised displacement sequence is then time series corrected based on the displacement change trend data to obtain the time series corrected displacement sequence.

[0125] In some embodiments, an AI time-series analysis model is obtained by training an LSTM network. A two-layer LSTM network is set up, with 64 hidden units in the first layer and 32 hidden units in the second layer. The hidden layers use the ReLU activation function, and the output layer uses the Linear activation function. Training parameters include a batch size of 8, 50 iterations, an Adam optimizer, and an MSE loss function. The denoised displacement sequence is reconstructed with a time step of 10. After being input into the model, the model learns the temporal correlation and outputs the displacement change trend data at each time step. The time-corrected displacement sequence was calculated. The expression is:

[0126]

[0127]

[0128] in, For optimal time delay, Let be the denoised displacement at time t. Let be the displacement after time-corrected time at time t. The trend of displacement change at time t;

[0129] Acquire environmental monitoring data, obtain the correlation matrix between environmental monitoring data and unstable rock mass displacement through AI environmental analysis model, and perform environmental factor compensation on the time-series corrected displacement sequence based on the correlation matrix to obtain the final displacement sequence of unstable rock mass.

[0130] In some embodiments, an AI environment analysis model is obtained by training a multi-input LSTM. A dual-input branch LSTM is set up, which takes a time-corrected displacement sequence and a synchronization environment data sequence as inputs, respectively. After merging the branch features, a hidden layer is set, and the output layer outputs the environment driving component, expressed as:

[0131]

[0132] in, Represents the first element in the environment-displacement correlation matrix. The environmental vector is in lag. Contribution coefficient, Represents the first time at time t An environment vector, The environment-driven component at time t;

[0133] Subtracting the time-corrected displacement from the output environmental driving component yields the final displacement sequence of the unstable rock mass. ,in , Let t be the final displacement of the unstable rock mass at time t.

[0134] In some embodiments, the final displacement sequence is divided into subsets by day / week / month, and the cumulative displacement, daily average displacement, and displacement rate of the unstable rock mass are calculated. A time-series displacement curve is plotted, with the horizontal axis representing timestamps and the vertical axis representing displacement, and the maximum cumulative displacement point and safety threshold line are marked. A report is generated based on the time-series displacement curve. The report includes basic information such as the monitoring area and time range, core data such as cumulative displacement and rate, deformation characteristics and stability conclusions, warning levels and response suggestions, and includes visualization charts for reference by engineers and management.

[0135] Reference Figure 2 This invention provides a machine vision-based unstable rock mass displacement monitoring device 20 that integrates image features and AI, used to implement a machine vision-based unstable rock mass displacement monitoring method that integrates image features and AI. The device includes:

[0136] The preprocessed image set acquisition module 21 is used to acquire the original image set of the dangerous rock mass in the monitoring area through the imaging device, and to preprocess the original image set to obtain the preprocessed image set.

[0137] The high-level semantic feature set acquisition module 22 is used to perform prior knowledge analysis on the preprocessed image set to obtain the prior knowledge feature set; and to construct a lightweight CNN-Transformer hybrid model to extract semantic features from the preprocessed image set to obtain the high-level semantic feature set.

[0138] The feature set acquisition module 23 is used to construct a dual-channel attention fusion model. The dual-channel attention fusion model dynamically fuses the prior knowledge feature set and the high-level semantic feature set to obtain the fused feature set.

[0139] The initial displacement sequence acquisition module 24 of the unstable rock mass is used to perform feature matching on the fused feature set through the FLANN matching algorithm to obtain effective matching feature pairs, and calculate the initial displacement sequence of the unstable rock mass based on the effective matching feature pairs.

[0140] The unstable rock mass displacement monitoring result acquisition module 25 is used to construct a multi-level AI model. Through the multi-level AI model, abnormal displacement is removed, time sequence is corrected and environmental factors are compensated for in the initial displacement sequence of the unstable rock mass to obtain the final displacement sequence of the unstable rock mass. The unstable rock mass displacement monitoring result is generated based on the final displacement sequence of the unstable rock mass.

[0141] This application provides an electronic device, including a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements a machine vision method for monitoring the displacement of unstable rock masses that fuses image features with AI, as described above.

[0142] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0143] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0144] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the machine vision-based method for monitoring rock mass displacement by fusing image features and AI, as described above. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0145] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0146] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A machine vision-based method for monitoring the displacement of unstable rock masses, integrating image features and AI, characterized in that, include: The original image set of the dangerous rock mass in the monitoring area is collected by the imaging equipment, and the original image set is preprocessed to obtain the preprocessed image set; Prior knowledge analysis is performed on the preprocessed image set to obtain a prior knowledge feature set; a lightweight CNN-Transformer hybrid model is constructed, and semantic features are extracted from the preprocessed image set through the lightweight CNN-Transformer hybrid model to obtain a high-level semantic feature set; A dual-channel attention fusion model is constructed, and the prior knowledge feature set and the high-level semantic feature set are dynamically weighted and fused to obtain the fused feature set. The fused feature set is matched using the FLANN matching algorithm to obtain effective matching feature pairs. The initial displacement sequence of the unstable rock mass is then calculated based on the effective matching feature pairs. A multi-level AI model is constructed to perform abnormal displacement removal, time-series correction, and environmental factor compensation on the initial displacement sequence of the unstable rock mass, thereby obtaining the final displacement sequence of the unstable rock mass. Based on the final displacement sequence of the unstable rock mass, the displacement monitoring results of the unstable rock mass are generated.

2. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The preprocessing of the original image set to obtain a preprocessed image set includes: The original images in the original image set are obtained, and illumination component correction and median filtering are performed on the original images to obtain the denoised images. In the denoised image, the dangerous rock mass area is selected, and the background area other than the dangerous rock mass area is cropped to obtain a preprocessed image. All the preprocessed images are combined to form a preprocessed image set.

3. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The prior knowledge analysis of the preprocessed image set to obtain a prior knowledge feature set includes: Obtain the dangerous rock mass region of each preprocessed image in the preprocessed image set, filter out the set of stable key points within the dangerous rock mass region using the Harris corner detection algorithm, and generate an ORB feature set based on the set of stable key points using the ORB algorithm. The gradient magnitude of each pixel within the unstable rock mass region is calculated using the Sobel operator. Edge pixels with gradient magnitudes greater than a preset value and located on the contour boundary are retained. An edge gradient feature set is constructed using the gradient magnitudes of the edge pixels. The texture analysis window is determined based on the edge gradient feature set. Texture features are extracted from the dangerous rock mass area through the texture analysis window to obtain the texture co-occurrence matrix feature set. The prior knowledge feature set consists of the ORB feature set, the edge gradient feature set, and the texture co-occurrence matrix feature set.

4. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The step involves extracting semantic features from the preprocessed image set using a lightweight CNN-Transformer hybrid model to obtain a high-level semantic feature set, including: The lightweight CNN-Transformer hybrid model includes a CNN network and a Transformer network. The CNN network performs feature mapping on the preprocessed image set to obtain a low-order feature set. The low-order feature set is reshaped into a sequence feature vector, and the sequence feature vector is input into a Transformer network. The encoder, attention head and decoder of the Transformer network capture the long-distance dependency between the deformation and semantics of the dangerous rock mass, and obtain a high-level semantic feature set.

5. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The method employs a dual-channel attention fusion model to dynamically weight and fuse the prior knowledge feature set and the high-level semantic feature set to obtain a fused feature set, including: The dual-channel attention fusion model includes a spatial attention layer, a channel attention layer, and a shared fully connected layer. The spatial attention layer performs convolution operations on the high-level semantic feature set to obtain local spatial correlation feature vectors. The Sigmoid activation function maps the local spatial correlation feature vectors to spatial attention weight vectors. The prior knowledge feature set and the high-level semantic feature set are subjected to global average pooling and channel statistics through the channel attention layer to obtain the first channel vector and the second channel vector. The first channel vector and the second channel vector are then concatenated into a joint channel vector. Channel importance is calculated on the joint channel vector by sharing a fully connected layer to obtain a channel importance score vector. The channel importance score vector is then mapped to a channel attention weight vector by the Sigmoid activation function. The prior knowledge feature set and the high-level semantic feature set are weighted by spatial attention weight vectors to obtain a first-weighted prior knowledge feature set and a first-weighted high-level semantic feature set; the first-weighted prior knowledge feature set and the first-weighted high-level semantic feature set are weighted by channel attention weight vectors to obtain a second-weighted prior knowledge feature set and a second-weighted high-level semantic feature set. Adjustment parameters are set based on the entropy value of the high-level semantic feature set, and the secondary weighted prior knowledge feature set and the secondary weighted high-level semantic feature set are fused into a fused feature set based on the adjustment parameters.

6. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The process involves performing feature matching on the fused feature set using the FLANN matching algorithm to obtain valid matching feature pairs, and calculating the initial displacement sequence of the unstable rock mass based on these valid matching feature pairs, including: According to the frame order, the fused features in the fused feature set are divided into reference frame features and matchable frame features. The FLANN matching algorithm is used to match each reference frame feature with each matchable frame feature into multiple feature pairs. The Hamming distance of each feature pair is calculated, and feature pairs with a Hamming distance less than a preset distance threshold are considered as valid matching feature pairs. Obtain the camera calibration parameters of the shooting device, calculate the pixel coordinate difference of each valid matching feature pair, convert the pixel coordinate difference into physical displacement according to the camera calibration parameters, and arrange each physical displacement in frame order to form the initial displacement sequence of the unstable rock mass.

7. The machine vision-based method for monitoring the displacement of unstable rock masses by fusing image features and AI according to claim 1, characterized in that, The process involves using a multi-level AI model to perform abnormal displacement removal, temporal correction, and environmental factor compensation on the initial displacement sequence of the unstable rock mass, resulting in the final displacement sequence of the unstable rock mass. The multi-level AI model includes an AI anomaly detection model, an AI time series analysis model, and an AI environmental analysis model. The AI ​​anomaly detection model identifies and removes deformation data from the initial displacement sequence of the unstable rock mass to obtain a denoised displacement sequence. The denoised displacement sequence is modeled by time series correlation using an AI time series analysis model to obtain displacement change trend data. The denoised displacement sequence is then time series corrected based on the displacement change trend data to obtain the time series corrected displacement sequence. Environmental monitoring data is acquired, and an AI environmental analysis model is used to obtain the correlation matrix between the environmental monitoring data and the displacement of the unstable rock mass. Based on the correlation matrix, environmental factors are compensated for the time-series corrected displacement sequence to obtain the final displacement sequence of the unstable rock mass.

8. A machine vision-based unstable rock mass displacement monitoring device integrating image features and AI, used to implement the machine vision-based unstable rock mass displacement monitoring method integrating image features and AI as described in any one of claims 1 to 7, characterized in that, The device includes: The preprocessed image set acquisition module is used to acquire the original image set of the dangerous rock mass in the monitoring area through the imaging device, and to preprocess the original image set to obtain the preprocessed image set. The high-level semantic feature set acquisition module is used to perform prior knowledge analysis on the preprocessed image set to obtain the prior knowledge feature set; a lightweight CNN-Transformer hybrid model is constructed to extract semantic features from the preprocessed image set to obtain the high-level semantic feature set; The fusion feature set acquisition module is used to construct a dual-channel attention fusion model. The dual-channel attention fusion model dynamically fuses the prior knowledge feature set and the high-level semantic feature set with weights to obtain the fusion feature set. The module for obtaining the initial displacement sequence of the unstable rock mass is used to perform feature matching on the fused feature set using the FLANN matching algorithm to obtain effective matching feature pairs, and to calculate the initial displacement sequence of the unstable rock mass based on the effective matching feature pairs. The module for acquiring monitoring results of unstable rock mass displacement is used to construct a multi-level AI model. Through the multi-level AI model, abnormal displacement is removed, time sequence is corrected, and environmental factors are compensated for in the initial displacement sequence of unstable rock mass to obtain the final displacement sequence of unstable rock mass. Based on the final displacement sequence of unstable rock mass, the monitoring results of unstable rock mass displacement are generated.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the machine vision method for monitoring the displacement of dangerous rock masses that integrates image features and AI as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the machine vision method for monitoring the displacement of unstable rock masses that integrates image features and AI as described in any one of claims 1 to 7.

Images